[sarah-chitty-1]

Builds on #544 (its commit 11c8a8e is included here). Merge #544 first and this PR drops it. How the user sim behaves comes in its own PR, stacked on this one.

What

Two new step types.

• always_on runs the agent under test as an always-on assistant:
  • It wakes the agent when something happens: an alert from a trigger, the user's reply, a scripted user line, or a check the agent scheduled for itself.
  • Between wakes it moves the task clock straight to the next event, never past one.
  • It ends at end_time, or when the user says done in a one-sitting task.
• user_sim says who the user is: description, time zone, awake and busy hours, how long they take to reply, and the agent that plays them.
  • The loop asks the user sim only when the agent has messaged them, at the time they would read it.

The agent reaches the user only through two new gateway tools: message_user, and stay_quiet for when there's nothing worth saying. Its normal reply goes to a log the user never reads.

Today, multi-day tasks are built with prompt_agent plus a per-task hook script, and each task does it differently. Our stress test of ML's three tasks found:

• events landing hours late;
• 22% of contact replies cut off;
• graders seeing only the last wake;
• a spending check that passed without running.

FD-5332 has the evidence.

Change

• The loop (task_steps/always_on.py):
  • wakes from trigger alerts, user replies, scripted lines and scheduled checks; anything due in the same minute arrives in one wake;
  • a clock that runs only during the agent's turns;
  • waits until contacts have actually replied before the next wake, up to each contact's own time limit (set on register_env_triggers with executor_timeout_seconds).
• User sim (task_steps/user_sim.py): the settings, and a plug-in interface (configure / setup / answer) for how the user behaves. A built-in default is included. The user sim's behaviour gets its own PR (FD-5380).
• Gateway:
  • message_user, stay_quiet and the user chat (env/gateway/user_chat.py);
  • the trigger engine reports each fire's progress, and can run contact replies side by side (opt-in);
  • a new clock call, clock/v1 set_rate.
• Endings:
  • a final check-in at the end time, with a last-turn notice;
  • the user reads the wrap-up;
  • what was still pending is recorded;
  • an end marker with the reason; "finished" and "stuck" are told apart.
• Grading:
  • the graded response is the user's whole chat with times, never the agent's private text;
  • failed turns set error_type;
  • each app change is tagged with the wake that made it;
  • each reply is linked to the request it answered;
  • reply-time records in the shape ML's existing asked_before_spending check reads, so it now grades instead of passing;
  • a rubrics_verifier subclass, always_on_agent_verifier, grades an always_on prompt from the user's chat, the run context and a wake-by-wake summary, never every wake's trajectory. It refuses screenshot-citing output formats and grade_up_to_turn when the task loads, and a plain rubrics_verifier on an always_on prompt is refused with a pointer to it. is_rubrics_verifier() recognises the subclass wherever code checks for a rubrics verifier.
• Limits:
  • a real-time budget so a run ends cleanly inside the hub's 8-hour job;
  • the run record keeps a short summary, with the full timeline in storage;
  • the process log carries no message text.

Behaviour changes for existing users

• #544: the executor read timeout goes from 30 s to the executor's own limit (its own PR).
• Trigger state gains fields (running, waits, executor_calls, settings) and the card gains a feature, fires_in_state. Additive.
• A contact reply that completes just after its time limit is now kept instead of failing the fire.
• Side-by-side contact replies are off unless a step turns them on.
• When the clock moves, a contact reply waiting on a delay goes out at exactly its time, including on a clock frozen between turns. Replies that also alert the agent run side by side. Tasks with a clock at rate 0 and prompt_agent tasks behave as before.

Everything else is new code paths, used only by always_on and user_sim.

Tests

• Unit tests: over 100 new across the loop, endings, grading, budget, clock, settling, user sim inputs and gateway tools. They use a real in-process gateway with a scripted agent and user sim.
• Full unit suite: per folder, the same as clean main except 2 known failures (test_mcp_cli_builder, a Python without click).
• End to end on a laptop, Opus 5.5 on the neutral harness, with the user sim and contacts played by agents:

Task	Cost	Wakes	Ended	Checks
Surprise 30th	$9.18	26	end time	32/32
Tree Removal	$14.13	80	wake cap	31/31
Holiday Rush (browser)	$15.60	3	user done	29/29
Holiday season (a month)	$18.16	47	end time	31/31

The checks cover:

• the graded chat;
• no private text reaching graders or the user;
• every contact reply landing before the next wake;
• the final check-in;
• the record size;
• thinking settings.

ML's own graders ran too; their scores reflect the agent, not the step.

After the review fixes, everything was merged again with the plumbing and user sim PRs: 13,024 unit tests pass, with 1 known failure (test_disk_recipe). A Tree Removal run at clock rate 1 then passed 37 of 37 checks ($1.75), confirming each fix:

• the process log has no message text;
• the clock is frozen between turns;
• undelivered replies are marked;
• the gateway's settings are put back at the end.

A full-week run of a parents' visit task (Opus 5.5, $20.92) then ran to its end time: every clock move settled, and contact replies arrived at their set times.

Worth a look

• The size of the step: always_on.py is the bulk. Its module docstring describes the loop in order; start there.
• The clock freeze uses the new set_rate rather than set_time. set_time re-arms the clock, which would push every recurring time trigger back on each turn.
• Ending a run with no end_time: it ends after 24 clock jumps in a row where nothing that could wake the agent is ahead (IDLE_JUMPS_MAX). A run with an end_time always goes on to it.
• The default time budget is 6.62 h at 600 s turns: budget plus one worst-case wake fits 8 h with 10 minutes spare.
• The user sim plug-in loads from a file artifact, in the runner's process, like post_turn_hooks. With no file_artifact_version it loads the latest.

Linear: FD-5364, FD-5365, FD-5366, FD-5367, FD-5368, FD-5369, FD-5370, FD-5371, FD-4408.

🤖 Generated with Claude Code

Confidence Score: 3/5

Fix the two user-timing bugs before merging.

Findings

1.  User answers while asleep ▶
2.  Reply delays are ignored ▶

Fix with agent prompt

### Issue 1
src/agent_env/task_step/task_steps/always_on.py:1013-1015
When an agent sends a message before bedtime but its turn ends after bedtime, `_next()` calls `read(now)` immediately. It never checks whether the user is still awake or free, and it keeps the earlier `read_mode`. The user can therefore answer overnight or during a busy block, with the wrong reply delay. This changes when the assistant can act and can change its grade.

Recheck `_available(now)` before an overdue read, update its mode, and leave the read pending if the user is unavailable.

### Issue 2
src/agent_env/task_step/task_steps/always_on.py:932-936
Setting `response_times` without `schedule` silently removes every reply delay. `_available()` returns mode `"now"` when no schedule is set, and `_latency()` returns zero for that mode even when `response_times.free` is configured. For example, `response_times={"free": 30}` produces an immediate reply instead of waiting 30 task minutes. The assistant can then act earlier than the task settings allow.

Use the configured fallback delay when no schedule is supplied, or reject this combination rather than silently ignoring it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds always_on tasks, simulated-user settings, gateway chat tools, contact settling, and chat-based grading.

• The always-on step wakes the assistant for events and advances simulated time between turns.
• The assistant reaches the simulated user through a separate message tool and chat queue.
• Trigger state reports work in progress, and contact replies can run side by side when enabled.
• A user-sim step sets the user's profile and connects the always-on run to a simulated user agent.
• Always-on runs are graded from the user's chat and a wake-by-wake run summary.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Agent wake] --> B[Run clock during turn]
  B --> C[Freeze clock and settle contacts]
  C --> D{Next event}
  D -->|User read| E[Ask simulated user]
  E --> F[Hold reply until delivery time]
  F --> A
  D -->|Alert or scheduled check| A
  D -->|Future event| G[Advance clock]
  G --> D
  D -->|End time or budget| H[Final wake and last read]
  H --> I[Store chat and timeline]

Loading

_{Reviews (19) · Last reviewed commit: "Merge origin/main into sarahchitty/fd-53..." · Reviewed by Greptile}

[earakely-scale]

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from a sibling open PR rather than from this diff: #492 ("the gateway's control routes take the deploy's control token") puts the harness control routes behind the deploy's token by matching scope["path"] against a hardcoded set in the ASGI wrapper — CONTROL_ROUTES in src/agent_env/env/gateway/constants.py, which enumerates /tools/{enable,disable}, /triggers/{register,remove,clear}, /clock/{set-time,advance,clear} and /notifications/{config,take,clear}, and says everything else "stays open for now".

This PR adds five more routes on that same listener that its own card text calls the harness control path: /triggers/config (gateway.py:301), /clock/rate (:343), /user_chat/config (:432), /user_chat/take (:461), /user_chat/clear (:472). None of them appear in that frozenset, so whichever of the two branches merges second leaves them callable without the token by anything that can reach the gateway — which, per #492's own framing, includes the agent on the MCP url. Concretely: PUT /clock/rate {"virtual_seconds_per_real_second": 0} freezes the clock the always-on step is pacing (0 freezes, per the card description added here), and POST /user_chat/take drains the queue the harness reads. /user_chat/state is a read, so it lines up with /clock/state and /triggers/state, which #492 leaves open deliberately.

Probably just a one-line addition to CONTROL_ROUTES on whichever branch rebases last — flagging it because neither diff shows the other half.

---

Generated by Claude Code

[earakely-scale]

contact_timeout_seconds doesn't reach contacts registered by register_env_triggers, so in the usual setup it has no effect.

The gateway applies executor_timeout_seconds only to executors registered without a timeout_seconds of their own (_executor_timeout in triggers.py, and the unset set built at registration). register_env_triggers always sends one: "timeout_seconds": self.executor_timeout_seconds, which defaults to 120 (register_env_triggers.py:67 and :243).

So with register_env_triggers then always_on, contacts are still cut off at 120 s, not this step's 300 s. To reproduce: register an executor with timeout_seconds: 120, then call /triggers/config with {"executor_timeout_seconds": 300}. The executor call still uses 120.

The simplest fix may be for register_env_triggers to send timeout_seconds only when the task sets executor_timeout_seconds (default None), so the gateway's default, or this step's, applies.

[sarah-chitty-1]

Good catch, it was dead config. Removed contact_timeout_seconds in aebf480. The wait for contacts now reads each contact's own time limit from the gateway (what register_env_triggers sets with executor_timeout_seconds), and the run record shows the longest one.

[earakely-scale]

A follow-up for always_on grading. Rather than rubrics_verifier spotting always_on runs itself (_for_always_on, the copy and the always_on import), we'd like a rubrics_verifier subclass, AlwaysOnAgentVerifier (type always_on_agent_verifier):

• It grades an always_on prompt from the user's chat, the run context and the wake-by-wake summary, as _for_always_on does today.
• It refuses screenshot-citing output formats and grade_up_to_turn when the task loads, not after the run.
• The per-run wake summary goes into the judge-prompt builder rather than onto the shared step, so no copy is needed.
• An is_rubrics_verifier helper (a registry lookup plus issubclass, for step objects and stored dicts) lets code that checks the type string recognise subclasses.

Not blocking this PR. It's tracked internally as a follow-up, covering the framework step, the helper and hub support.

[sarah-chitty-1]

Done in 7dda611: always-on runs are now graded by always_on_agent_verifier, a rubrics_verifier subclass. It grades from the user's chat, the run context and a wake-by-wake summary. When the task loads, it refuses screenshot-based formats, grade_up_to_turn and use_trajectory: true. A plain rubrics_verifier on an always-on prompt is refused at load. The hub PR (scaleapi/scaleapi#173960) treats the new type as a rubrics verifier.

[earakely-scale]

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from main rather than from this diff: the clock card block this PR edits moved on main after this branch's merge base, and set_rate already exists there at a different path.

• This PR adds it as a new capability — the route @self._mcp.custom_route("/clock/rate", methods=["PUT"]) (src/agent_env/env/gateway/gateway.py:344) and the card entry "set_rate": {"method": "PUT", "endpoint": "/clock/rate", …} (src/agent_env/env/gateway/constants.py:229). At the merge base (0d171e0, Nov 9; the branch is 90 behind) the clock card carried no set_rate at all, so the diff reads as an addition rather than a rename.
• 6344340 ("computers follow the clock through a watch, and clock writes wait for them", feat(gateway): computers follow the clock through a watch, and clock writes wait for them (FD-5471) #614, Nov 11) added one on main at /clock/set-rate: declared at constants.py:243, served by gateway.py:328, over a Clock.set_rate (clock.py:200) that keeps t0 and the generation so time triggers keep their marks — the semantics this PR's new description names.

Two parts of this hunk may be worth dropping in the rebase rather than merging:

1. Trigger re-evaluation. This route calls self._trigger_engine.clock_changed() itself. On main every clock write goes through _clock_write → _change_clock (gateway.py:613, :633), which re-evaluates the time triggers on any change that keeps the generation — a rate change does — and answers held watches.
2. The rest of the block. main's clock methods now carry wait_followers_s (_WAIT_FOLLOWERS) on set_time, set_rate, advance and clear, read at gateway.py:623 to wait for declared followers and answer 504 when they're behind, plus a watch method. Those sit as context lines in this hunk, so taking this side of constants.py would drop them from the card. The advertised path is the half worth watching: tst/unit/env/gateway/clock_watch_test.py:79 pins the card's set_rate to ("PUT", "/clock/set-rate"), so a card/route mismatch there fails loudly, but only in that one assertion.

Worth knowing alongside it: open #631 adds a set_env_clock_rate task step that reaches set_rate through the card (deployed.invoke(EXT_CLOCK_URI, "set_rate", …)), so whichever path the card ends up advertising is the one that step calls.

Not a review of this diff — just the half that is only visible from main. Nothing to do here if the rebase plan already covers it.

---

Generated by Claude Code

[greptile-apps]

User answers while asleep

When an agent sends a message before bedtime but its turn ends after bedtime, _next() calls read(now) immediately. It never checks whether the user is still awake or free, and it keeps the earlier read_mode. The user can therefore answer overnight or during a busy block, with the wrong reply delay. This changes when the assistant can act and can change its grade.

Recheck _available(now) before an overdue read, update its mode, and leave the read pending if the user is unavailable.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/always_on.py
Line: 1013-1015

Comment:
**User answers while asleep**

When an agent sends a message before bedtime but its turn ends after bedtime, `_next()` calls `read(now)` immediately. It never checks whether the user is still awake or free, and it keeps the earlier `read_mode`. The user can therefore answer overnight or during a busy block, with the wrong reply delay. This changes when the assistant can act and can change its grade.

Recheck `_available(now)` before an overdue read, update its mode, and leave the read pending if the user is unavailable.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Reply delays are ignored

Setting response_times without schedule silently removes every reply delay. _available() returns mode "now" when no schedule is set, and _latency() returns zero for that mode even when response_times.free is configured. For example, response_times={"free": 30} produces an immediate reply instead of waiting 30 task minutes. The assistant can then act earlier than the task settings allow.

Use the configured fallback delay when no schedule is supplied, or reject this combination rather than silently ignoring it.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/always_on.py
Line: 932-936

Comment:
**Reply delays are ignored**

Setting `response_times` without `schedule` silently removes every reply delay. `_available()` returns mode `"now"` when no schedule is set, and `_latency()` returns zero for that mode even when `response_times.free` is configured. For example, `response_times={"free": 30}` produces an immediate reply instead of waiting 30 task minutes. The assistant can then act earlier than the task settings allow.

Use the configured fallback delay when no schedule is supplied, or reject this combination rather than silently ignoring it.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.