Skip to content

feat(simulation): list a scenario group's runs with their scenario outcomes - #1870

Open
u9g wants to merge 7 commits into
mainfrom
jason/sim-list-scenario-group
Open

u9g wants to merge 7 commits into
mainfrom
jason/sim-list-scenario-group

Conversation

@u9g

@u9g u9g commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What

Two fields on SimulationRun.List.Request, and one on SimulationRun:

  • optional string scenario_group_id = 8 narrows the list to one scenario group (its SCNG_ id).
  • bool include_scenario_outcomes = 9 fills each listed run's new repeated ScenarioOutcome scenario_outcomes = 24: one entry per scenario with samples, passed, failed, the first failure's failure_reason, and the job_id to open. jobs stays empty on List; Get still returns whole jobs.

Why

The Simulations dashboard is getting a page per scenario group that shows every scenario's verdict across the group's recent runs. Today that would take one Get per run, since List has no group filter and returns no jobs. With these fields it is one List call. A page holds 20 runs of up to 250 jobs; whole jobs (per-turn metrics, scenario text) are several KB each and would pass NATS' 8 MB message cap. Tallies are tens of bytes per scenario, plus a failure reason only where a scenario failed.

Runs created before the agent service stored the group id carry none, so a scenario_group_id filter never matches them; the field comment says so.

Follow-ups

  • livekit/agents-private#579: store the group id on create, honour both fields in ListSimulationRuns.
  • livekit/web#6617: the scenario-group cards and page that call it.

SimulationRun.List.Request gains scenario_group_id, to narrow the list to one
scenario group, and include_jobs, to return each run's jobs, which the list
otherwise omits. Together they let a scenario-group page read every scenario's
verdict across its recent runs in one call.
@changeset-bot

changeset-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: d5a0023

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
github.com/livekit/protocol Patch
@livekit/protocol Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions Bot and others added 3 commits October 16, 2026 23:06
A list page holds 20 runs of up to 250 jobs, and whole jobs (per-turn metrics,
instructions, expectations) can reach several KB each: past NATS' message cap
for a large scenario group. The page that sets include_jobs reads only each
job's outcome, so that is all the list returns; Get still returns whole jobs.
include_jobs is replaced by include_scenario_outcomes, which fills
SimulationRun.scenario_outcomes: one entry per scenario with its sample,
passed and failed counts, the first failure's reason and the job to open.
A scenario group page reads 20 runs of up to 250 jobs; tallies keep that to
tens of bytes per scenario where jobs, even trimmed to their outcome, carried
a row per sample.
@u9g u9g changed the title feat(simulation): list a scenario group's runs with their jobs feat(simulation): list a scenario group's runs with their scenario outcomes Oct 16, 2026
optional string scenario_group_id = 8;
// Populates each run's `scenario_outcomes`. `jobs` stays empty either
// way; `Get` returns them.
bool include_scenario_outcomes = 9;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
bool include_scenario_outcomes = 9;

No need for it, check the review on agents-private

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants