[chenghao-mou]

Adds simulator flows to agent simulation scenarios.

A scenario today describes one persona. For an outbound call, the simulator needs to play the far end: a voicemail greeting and beep, an IVR menu that routes on DTMF, a screener, a hold, and the person who picks up after it. A flow is a set of named nodes; the simulator walks them as the call goes.

• ScenarioGroup.nodes: named SimulatorNodes, shared by every scenario in the group. Nodes can also name the built-ins livekit/hangup, livekit/voicemail, and livekit/unavailable.
• Scenario.start: the node a call starts at. A scenario sets start or instructions, not both.

livekit/voicemail is the only voicemail box, and nodes cannot configure one. Each run picks the box: whether it beeps, how long it waits for silence, and whether a send-or-re-record menu follows the message. Repeated runs then show how an agent handles different boxes.

A node with routes, such as a menu, plays at most three times; the next visit ends the call, as with a real IVR. Nodes have no retry setting. A node timeout is at most 20 seconds, and scenarios have no duration setting: 30 seconds of silence ends a call, as in a persona run.

In scenarios.yaml:

nodes:
  # a menu: single-key routes, checked in order
  menu:
    say: "Thanks for calling Acme Dental. For appointments, press 1. To pay a bill, press 2."
    speaker: MACHINE
    routes:
      - { match: "1", to: hold }
      - { match: "2", to: card }
      - { match: "[0-9*#]", to: invalid }  # any other key
    timeout: { after: 8s, to: menu }       # no key: play again, at most 3 times in all
  invalid:
    say: "Sorry, that is not a valid option."
    speaker: MACHINE
    then: menu                             # when say ends
  # a multi-key entry: keys collect until #
  card:
    say: "Enter your 16-digit card number, then press pound."
    speaker: MACHINE
    routes:
      - { match: '\d{16}', to: paid }
      - { match: '[0-9*]+', to: card }     # anything else: ask again
    timeout: { after: 15s, to: livekit/hangup }
  paid:
    say: "Thank you, your payment is complete. Goodbye."
    speaker: MACHINE
    then: livekit/hangup
  # silence: only the clock moves on
  hold:
    say: "Please hold while I connect you."
    speaker: MACHINE
    timeout: { after: 4s, to: screener }
  # converses in a machine voice; `then` once the agent replies, `timeout` if it never does
  screener:
    say: "This is Sam's assistant. May I ask who is calling?"
    instructions: You screen calls for Sam. Once the caller says who they are and why, thank them.
    speaker: MACHINE
    then: sam
    timeout: { after: 10s, to: livekit/hangup }
  # a person, in the scenario's voice; hangs up after 15s without the agent speaking
  sam:
    say: "Hi, this is Sam."
    instructions: You are Sam Lee. Confirm tomorrow's 10am cleaning, then say goodbye.
    timeout: { after: 15s, to: livekit/hangup }
  # a person's recorded greeting, then the built-in voicemail box
  greeting:
    say: "Hi, you've reached Sam. Leave a message after the beep."
    then: livekit/voicemail
scenarios:
  - label: menu-to-sam
    start: menu
    agent_expectations: Presses 1, stays silent on hold, tells the screener who is calling, and confirms the appointment with Sam.
  - label: pay-bill
    start: menu
    userdata: { card_number: "4242424242424242" }
    agent_expectations: Presses 2, enters the card number followed by #, and ends the call after the payment.
  - label: voicemail
    start: greeting
    agent_expectations: Waits for the greeting, then leaves one short message.
  - label: number-not-in-service
    start: livekit/unavailable
    agent_expectations: Does not leave a message and ends the call.

All fields are new, so the change is wire-compatible. The server compiles and validates each scenario's flow at CreateSimulationRun and keeps the nodes out of SimulationDispatch, so the agent under test never reads them.

Companion PRs: livekit/livekit-cli#1031 reads nodes and start from scenarios.yaml; the agent-service compile, validator, and runtime are in a private PR.

Addresses AGT-3725

[changeset-bot]

🦋 Changeset detected

Latest commit: 7b7ed8c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
github.com/livekit/protocol	Patch
@livekit/protocol	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[theomonnom]

I think a lot of those options may not be that relevant to users?

They seem like a lot of "options" or "randomness" that we should handle automatically.

Then, the failure rate across multiple runs (let's say 5) would determine how robust your agent is at handling voicemails. (because we just pick "randomly")

[chenghao-mou]

This is mainly for the machine node, where you want some degree of rigidity. But we can fix it, or at least, most of them, for the user too.