[sarah-chitty-1]

Stacked on sarahchitty/fd-5364-always-on-step (the always-on step, at 974a6aba). This PR adds the user sim: one
deploy_user_sim_agent step, the built-in user sim it renders, their tests and docs, and a few small edits to
always_on.py so it reads the user sim from that step. The gateway is unchanged.

One step for the user sim (deploy_user_sim_agent, from Edgar's review)

The user_sim settings step and the deploy_agent that played the user are replaced by one step, as in Edgar's
"DeployUserSimAgentTaskStep" design (9c861380).

• It deploys the agent that plays the user and sets it up there: it renders the system prompt with the built-in
  user sim from persona and opening_prompt, deploys the agent with it, and sets the model (this step's step_params, then the
  run's user_model, then the step's model). If the agent's card refuses the system prompt or the model, the step
  fails (a new opt-in system_prompt_required on DeployAgentTaskStep; plain deploy_agent is unchanged).
• The tested agent never plays the user: _resolve_agent_ref ignores a run-wide a2a_agent_id; the step's own
  step_params still apply, and its a2a_agent_id is task-scoped. The step sets plays_user = True, which feat: bundle, training, task run and agent picks understand always_on and user sims (FD-5371) #565's role
  checks read (a step without it doesn't play the user).
• It records the user in metadata["user_sims"][agent_name] with the model's source and the prompt's sha256.
  always_on reads that record and sends the user's opening_prompt as the first message; it takes no prompt of its
  own (files in the first message: FD-5494). It no longer sets the persona mid-run.
• The user sim's words are unchanged: the prompt and every read are byte-for-byte the tested ones (golden tests).
  A temporary multi_day setting keeps the tested done line; always_on refuses to start if it disagrees with the run.
• Fields dropped: email, plugin, user_version, versions, output_format. max_replies and the user's
  time_zone (in persona; else the clock's) are kept.
• prompt_agent works with it too (FD-5409): a chat task whose user was deployed by this step reads it the way
  always_on does, one read per agent reply. reply is the next prompt; done with a message reaches the agent and
  ends the chat after its answer; done without one, no_reply or max_replies ends it. A failed user sim turn sets
  error_type. Each read is kept short in metadata and stored in full. Before the first turn it refuses
  multi_day: true, its own user_model / user_output_format, a schedule or reply times, and a prompt that isn't the
  user's opening_prompt (written twice for now: FD-5495). Other prompt_agent tasks are unchanged.
• Where it lives: task_steps/agent/ (deploy_user_sim_agent.py, user_sim_versions.py, user_sim_common.py,
  user_sim.md). deploy_agent.py moves there in its own PR after feat: bundle, training, task run and agent picks understand always_on and user sims (FD-5371) #565–feat(user_sim): the built-in user sim for always-on tasks #567.

The sections below describe the built-in user sim itself; where they say "the user_sim step", read
deploy_user_sim_agent.

What and why

What it adds

• user_sim_versions.py is now the built-in user sim, one version, ours. It replaces the test rig's adapter, which
  had "today" and ablations. A task that wants another version names a plug-in file.

  • The system prompt (setup), written once per run:

    • fixed lines on how a real person reads and answers: not the assistant; never do its work; can't open apps;
      what other people and apps send you is for you; say you don't know, never guess; what "done" means; used;
    • the task writer's description, as tagged sections;
    • the opening request, from person["opening"], right after the first "who you are" line;
    • the user's rules, written in plain English from permissions, then keep_private;
    • the answer format.

  • Each read (answer), rendered as one labelled message:

    • the time, and when the user last wrote;
    • YOUR NEW EMAILS, TEXTS AND NOTIFICATIONS in full, then YOUR EARLIER EMAILS, TEXTS AND NOTIFICATIONS, first lines
      only, each with its sender from channel. The earlier list keeps the last 7 days' items, at most the 20 most recent, and ends with "(older messages not shown)"
      when it drops any, so a month-long run's reads stay short;
    • YOU SENT YOUR AI ASSISTANT: only the user's own lines;
    • everything new from the assistant at once.

    Earlier items and "last messaged" come from history. Keys of read it doesn't know are ignored. The read
    after the final check-in, of the agent's wrap-up, is rendered the same way.

  • The answer: {outcome: reply | no_reply | done, message, used?, score?}.

    • A reply with no text is no_reply.
    • A malformed answer gets one retry in the same session: "Your last answer wasn't in the required format. Answer
      again, only as JSON: {…}". The answer records retried, the first answer and turns: 2, which the step logs
      as user_sim_turns. If the retry is malformed too, it's no_reply, with both answers' schema_errors
      recorded. An error from the user sim agent gets no retry.
    • used is recorded as a pulse check only. It never decides a grade and never reaches the agent.

• user_sim.py: the step checks its settings when the task loads, so a mistake fails before a paid run.

  • For the built-in user sim:
    • the description sections;
    • permissions and keep_private;
    • the version.
  • For any plug-in: schedule, response_times and max_replies, under the setting names the loop already reads.
  • There's no apps setting.

• user_sim.md: "Writing a user sim". It covers the description sections and the rules that mattered most, each with
  its reason:

  • a specific "don't mention X unless asked" line per tempting fact;
  • "say you don't know, never guess";
  • "done is only a short sign-off";
  • decisions with one answer;
  • nothing about the future.

  It also covers marking triggers to reach the user's phone, with the sender in channel, and general triggers that
  fill in real details with ${args.*}.

Why

The user sim today sees only the assistant's raw text: no time, none of the emails or texts that reach the user, none of its own lines. In multi-day tasks it
makes mistakes that change the assistant's grade.

The final replay:

• 45 real moments from two full runs of a parents' visit week, each replayed 10 times;
• the same assistant messages and fixed earlier history for every version;
• right answers written by someone who hadn't seen the design;
• two judges, agreeing on 94% of items.

What it showed:

• Fewer grade-changing mistakes. This design made 5.6–6.9 per 100 moments that matter. The current user sim made
  9.4–10.6.
• The instructions drive most of the gain. A 2×2 swapped how the prompt is written with how each read is
  delivered. Our prompt beat the current one under either delivery: pooled, 5.9–7.2 vs 8.8–9.7 grade-changing mistakes
  per 100. It was ahead in 88–95% of resamples.
• Our delivery adds a smaller gain.
  • On grade-changing mistakes, it's within noise: ahead in 74–81% of resamples.
  • "Didn't do what the moment expected" fell from 40–53 to 13–28 per 450.
• Events shown as they happen let the user catch mistakes only the confirmation email shows. She caught 50 of 50,
  with 0 false alarms. A list of events in her instructions can't do this.

How a plug-in file loads. A plugin file artifact is loaded and run in the runner's own process. With no
file_artifact_version, it loads the latest version. This is the same pattern as post_turn_hooks.

How long a run's history gets

Each read adds to the user sim's one session, so a long run's context grows with every read. Estimate for an 80-read
run, about a month at two to three reads a day:

• Visible text only: the system prompt, every read and every answer, with the tested runs' assistant messages and
  two phone items a read.
  • With the 20-item, 7-day cap: about 61k–74k tokens.
  • Without it: about 157k–190k. The EARLIER section alone grew past 14k characters a read.
• Measured in a live run: the user sim in our 21-read parents' visit run averaged about 31.6k prompt tokens a read
  (663k in all). That's about 2.0k–2.9k tokens of context added per read, several times its visible text. Our 24-read
  runs ended near 48k. We haven't measured where the rest comes from; the likely sources are her thinking kept in the
  session and the CLI's own additions to each turn.
• At that measured rate, 80 reads come to about 160k–230k tokens. On a 200k-token window, Claude Code compacts
  automatically at about 187k (from the 2.1.224 build's code), and compaction is untested for the user sim. 40 reads
  (about 80k–115k) fit.
• Before an 80-read task, measure context per read on its first live run. If needed, give the user sim a
  1M-token model, or less thinking.

Not in this PR

• The always-on wiring, which is the always-on step's.
• Removing the real date the user sim's harness adds: the user sim's agent image does it (FD-5380, below).
• Read-only app tools for the user sim.
• The satisfaction-score grader. score is parsed and recorded if the user sim gives one, nothing more.

The real date (FD-5380)

Claude Code adds the computer's date to the user sim's first message ("Today's date is …"), and "The date has
changed …" when that date changes. No setting turns either off. In a run set in another week, the user sim can take the
real date as today; in the head-to-head, 3 thinking traces did.

This PR doesn't fix it; the user sim's agent image does. psi-e2e-claude-code-mcp-only-nodate is the image ML's tasks
already use for simulated people (Claude Code 2.1.177), with a filter in front of the model endpoint that removes both
texts, including the "date has changed" note inside the answer tool's result. A test fails if a new Claude Code version
gets the date past the filter. A task uses it by naming it as the user sim's a2a_agent_id. The user sim's
instructions still say any other date is the computer's clock, which covers runs without it.

How it was tested

• Golden tests, byte for byte, against what the final replay sent, from read, history and person shaped as
  the always-on step builds them:

  • the system prompt (tst/data/user_sim/hannah_system_prompt.txt), with the opening request coming from person;
  • all 45 reads of the two replayed runs (tst/data/user_sim/hannah_reads.json; 21 and 24 reads). Each is rendered
    from the earlier reads and answers in history, with senders in channel.

• Unit tests (tst/unit/task_step/test_user_sim_versions.py, test_user_sim.py):

  • the parser: reply, no_reply, done, a reply with no text, used optional, score kept, fenced or embedded JSON,
    malformed answers;
  • the retry: malformed then valid uses the valid answer; malformed twice is no_reply with both errors; the retry is
    recorded; an empty reply or a valid answer gets none;
  • the earlier list dropping items older than 7 days, with its closing line;
  • a simulated 80-read run with two phone items a read. EARLIER never shows more than 20 items, and once the cap
    holds, the read's length stays within 100 characters. The tested runs had at most 5 earlier items, none older than
    4.3 days, so the cap never applies to the golden reads;
  • YOU SENT filtering: a ping in you_sent is dropped and doesn't count as her last message;
  • "last messaged" counting a reply but not a no_reply;
  • phone items in full, then as earlier items, each shown once at its time;
  • an earlier phone item or own line with no time keeps the time of the read it came in;
  • a time zone like "Pacific", which is a whole region, is named as a bad setting;
  • the done wording;
  • the rules and empty sections;
  • not asking when the assistant sent nothing;
  • the read after the final check-in, rendered like any other, and its answer parsed;
  • the load checks, for the built-in user sim and for any plug-in.

• The always-on loop with this plug-in, through the always-on step's own test harness:

  • one run where the user sim agent got exactly the rendered read, and her reply reached the agent;
  • one run where a malformed answer was retried once, in the same session, and the valid answer reached the agent;
  • one run where, after the final check-in, she read the agent's wrap-up like any other read. Her "done" was
    recorded, never delivered, and the run still ended at its end time.

• The always-on step's loop-side tests of what it hands the user sim (test_always_on_user_sim_inputs.py, 16
  tests) pass, as do all its other always-on tests.

• The full unit suite (pytest tst/unit packages/agentenv-protocol/tests -n 6):

  	passed	failed	skipped
  base (12e03f5d, before the last rebase)	14,131	6	67
  this branch on 12e03f5d	14,244	6	67
  this branch on 3bfdf7e8	14,325	6	67
  this branch on 3d27a238	14,398	3	67

  • The failures on 3d27a238 are three of the five that every run had (the two test_mcp_cli_builder failures
    passed this time).
  • Five failures are in every run, on the base and on the branch. They're all outside the user sim and all come from
    this laptop: test_mcp_cli_builder (2), test_docker_image_github_clone (2, timeouts) and a Windows disk-recipe
    test.
  • The sixth differs by run:
    • the branch's disk-recipe launcher test fails under load and passes when run alone;
    • the base's version_lookup_benchmark_test needs a git checkout, and the base ran from an export.

Worth a look

• setup gets no read, so it words the done line from person["done_ends_run"], which the always-on step sets before
  setup. When it's missing, it uses the multi-day wording.
• The one-sitting done line now ends "after "done", the conversation is over." The tested line ended "a "done" message
  never reaches the assistant", which stopped being true once done messages reach the agent. The multi-day line,
  which the final replay tested, is unchanged.
• The wording names what reaches the user as "emails, texts and notifications", not "your phone" (b5bf5ef1).
  It tested level with the old wording on Sonnet 4.6: 6.9–7.5 vs 5.6–6.9 grade-changing mistakes per 100 on the final
  test's 20 moments that matter (inside the noise), and identical on the 12 planted mistakes (50/50, 20/20, 0 false
  alarms). The golden prompt matches the tested prompt byte for byte.
• The built-in user sim refuses a description written as one string. A task with its own plug-in can still use one.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new blocking issue was established.

Summary

Adds deploy_user_sim_agent to deploy and configure the built-in user, then shares its reads between always_on and prompt_agent.

• One deploy step sets up the user agent for an always-on run.
• The built-in user sim gives each read a persona prompt and recent conversation context.
• A deployed user agent can answer each reply in a prompt-agent chat.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Persona and opening] --> B[deploy_user_sim_agent]
  B --> C[Configure user prompt and model]
  C --> D[Save user record]
  D --> E[always_on]
  D --> F[prompt_agent]
  E --> G[Render user read]
  F --> G
  G --> H[Ask user agent]
  H --> I{Answer valid?}
  I -->|No| J[Retry once]
  I -->|Yes| K[Record answer]
  J --> K
  K --> L[Deliver message or stop]

Loading

_{Reviews (28) · Last reviewed commit: "Merge sarahchitty/fd-5364-always-on-step..." · Reviewed by Greptile}