roadmap: bounded asynchronous RL with a checked event core #8

Motivation

Asynchronous and distributed execution changes when samples arrive, which policy version generated them and which process reports them. Invar must say which of these changes leave training unchanged and which change the algorithm, and check that a run followed its declared rules. Today the loop collects a full cohort, updates and publishes in lockstep, and rejects samples from any policy other than the current one.

Proposal

Drive the loop from a pure core, step :: State -> Event -> Either ProtocolError (State, [Command]), over checked events: completed requests, reported probabilities, finished steps, commits, failures and connections. Support bounded staleness with a decoupled objective: behavior probabilities recorded by the engine, proximal probabilities scored by the learner with the weights frozen at the start of each update, current probabilities per optimizer step, and a fixed reference. Each request's policy version, its update and the order of use are functions of logical identities, never of timing. Updates may run several declared optimizer steps; the core computes advantages, objective terms and cotangents, and the learner runs the backward pass and the optimizer. Publication has one commit point, and recovery redoes an unpublished update as a whole. One transport runs workers over SSH. MLX adopts the new protocol synchronously; asynchronous acceptance runs on CUDA. Distributed learning is outside this plan.

Acceptance

• Reordering independent events, duplicate deliveries and restarts lead to the same consumed samples, update plans and publication decisions, shown by property tests on the core.
• Every cotangent is bound to its update, step, forward observation and pre-step state; a late or stale reply cannot be applied elsewhere.
• Rollout and learning overlap on CUDA with staleness 1, and a use contract compares the asynchronous checkpoint with a synchronous one under a budget, evaluation protocol and checkpoint rule fixed in advance.
• A two-host run survives killed workers, a restarted learner and dropped messages without consuming a sample twice, re-running a published update or accepting a stale publication.
• Documentation states what each guarantee covers: the core fixes the logical assignment and use of samples; identical weights additionally require evidence of the workers' numerical invariance.
• Quality, throughput and total resource cost of the new execution are reported for the recorded configurations.