Skip to content

Event feed connector: the run loop and tier-2 driver (2/3) - #705

Merged
jeremy merged 51 commits into
mainfrom
event-feed-go-connector
Sep 16, 2026
Merged

jeremy merged 51 commits into
mainfrom
event-feed-go-connector

Conversation

@jeremy

@jeremy jeremy commented Aug 12, 2026

Copy link
Copy Markdown
Member

This PR has been split and force-pushed. It now carries the state machine
only, and is stacked on #777 (foundations). #778 (conformance corrections)
stacks on this. The pre-split head is preserved at tag pre-split/705-head;
the exact commit this PR pointed at before the force-push is
pre-split/705-remote-head (d379f2e11). All 63 review threads are intact,
but line anchors on foundation files now resolve against #777.

Why: eight bot rounds here did not converge (12→3→5→2→2→1→3 threads, with late
findings in files no earlier round had touched), and a review pass then found a
P1 credential defect all eight missed because it composes two files across a
package boundary. That defect is fixed in #777.

The Go reference implementation of the SPEC.md §23 Event Feed connector — BC3's
account-wide event feed over Action Cable push plus polling catch-up — and the
tier-2 conformance driver. All 22 fixtures pass (23 with #778's addition).

Layer 2 only: the connector reaches the wire through TicketMinter /
PollSource seams and one sanctioned cable dial (AGENTS.md Hard Rule 2). The
Layer-1 adapters are deferred to G1b, so the package is experimental and a
consumer supplies the seams today.

What is here

connector.go (New, the options, Events, Close), loop.go (states,
transitions, timers, live buffer), catchup.go (the poll walk, its page
boundary, the drain), recovery.go (the 400/409/410 matrix), and the tier-2
driver with its fixture model, harness and self-tests.

The foundations — seams, wire types, transports, filters, checkpoint identity,
the file store, dedupe, backoff, clock, cable codec, feedtest/ — are #777.

Rebase note

The 29 commits were re-cut, not rebased. Rebasing onto #777 was attempted
and abandoned: 8 of them touch only foundation files and would replay empty, and
21 of the remaining 22 mix both halves, so every one conflicts against the four
fixes #777 applies. Resolving 22 interleaved conflicts by hand is a worse
guarantee than re-cutting, which is byte-exact by construction — the foundation
paths come from #777's tip, these 17 from the pre-split head, verified with an
empty git diff against both.

The five fixes on top

Blocker 2 — a poll page carrying no position is malformed

The walk took page.Position on trust, and an empty one silently skipped
history in two different ways.

Position-resume: acceptPosition("") sets l.position = "", and
entryCursor selects on l.position != "" — so it does not preserve the old
cursor, it falls through to a bare present entry. The feed resumes at the
server's head with everything between skipped, reporting nothing.

Present-class is worse. held uses "" as its sentinel for "the final entry
was not present-class", so an empty position is not saved as empty — it
collapses into the sentinel, the held != "" guard skips acceptPosition and
saveCheckpoint outright, and caught_up announces anyway. The position is
discarded, with the drain's deliveries already handed to the consumer.

Refused before delivery, counter resets, and every mutation. That placement is
what the mutation check exercises: moving the guard after the delivery loop
fails three of four subtests on delivered ids = [101], want [].

Blocker 6 — narrow the promise, and make Wait the quiescence point

Close's doc claimed "no seam call and no delivery can begin after Close has
returned". That is not true and cannot be made true: the run goroutine checks
its context at each dispatch point and then acts, so a Close landing in between
cannot stop the call from starting — only from starting on a live context.
Closing that window means holding a lock across arbitrary host code, trading a
benign race for a deadlock reachable from any callback.

The one effect that outlives the process is the checkpoint save, and this PR
deliberately does not gate it. An accepted page's position saves even when
Close lands first — its events were already delivered, and dropping the write
would silently re-deliver them — and the save runs under a context detached
from the run's cancellation (context.WithoutCancel: values kept, cancellation
dropped), because a store that honors its ctx is compliant and would
otherwise lose the position Close raced. A save decided just before Close can
land just after it; that is intended, not residual.

Ordering a second connector over the same store is therefore the consumer's
to do, and Wait is the tool: it blocks until the run goroutine has exited, so
no save can be in flight — by construction, not by a narrow window. Await the
iterator's termination, or Wait, before opening a second connector over the
same checkpoint store. (An earlier revision ordered saves through a
durableGate claimed inside Close and tracked the unclosable [claim, write]
residual in #784; the gate is gone, Wait replaces it, and #784 is closed with
it.)

Also: a cancelled checkpoint load is no longer Terminal(checkpoint_load).

Blocker 7 — observers see origins only

#777's redactor applied to Observer.Gap and both CatchUpStarted sites. An
accepted 410 latches the server's resume URL as reconnect state, so the
reconnect announces its walk carrying it — redacting Gap alone would have left
the identical URL leaving through a different callback one reconnect later.

Observer.Disconnected is redacted, and this paragraph used to say the
opposite. The original reasoning — an error is opaque text, and stripping a
credential out of arbitrary text means modelling the credential, which §23's
"opaque bearer" contract forbids — argued for leaving it alone. A later round
found a ticket reaching the callback through the raw seam read error, which no
seam obligation can repair from the connector's side, so both arguments now go
through closed vocabularies: observableDisconnectReason maps every
unrecognized peer reason to "other", and observableSocketError reduces every
cause to one the connector owns, degrading anything unrecognized to
errSocketFailed.

"Reduces" is load-bearing and was the last hole: matching a sentinel is not the
same as being one, so a seam returning fmt.Errorf("read %s: %w", cableURL, context.Canceled) matched a recognized arm while its text carried the ticket.
Every arm now returns the connector's own value rather than its argument;
errors.Is still matches, and the wrapper does not survive.

The seam obligation remains on Dial, ReadFrame and WriteFrame — it is what
keeps a preserved typed error safe — but the connector no longer depends on it
being honored.

#763 — staleness arms at socket open

Observer.Connected fired between the socket opening and the window that
measures silence on it. Whatever a host's callback spent was time the window
never counted. Observed from inside the callback, which is the only place the
ordering is visible.

#760 — an occupied deferral slot no longer blinds the drain's fatal scan

drainScan returned the moment it found the slot occupied, on the reasoning
that everything queued "arrived behind this one". That is a claim about arrival
order, where §23's carve-out is a claim about which verdict governs. With
any non-fatal outcome parked ahead of it the scan looked at nothing — and the
budget it runs under is pumpDepth+1, sized in drain's own comment to reach
every frame the pump had already read. An occupied slot spent none of it.

The planned fix was a bounded deferral queue carved out of pumpDepth; that
turned out to be unnecessary.
Only one deferral is ever dispatched —
pumpExited, dispatchDisconnect and the invalid-frame teardown all end the
cycle — so a queue would be a buffer sized for a delivery that cannot happen.
The scan keeps the first outcome and discards the rest as it passes them, which
removes the capacity question entirely: no share of pumpDepth, no channel
resize, no change to the published memory bound, no fixture sweep.

The new subtest needs both halves of the two around it: deferring during the
entry poll occupies the slot, and queuing the fatal frame mid-drain puts it
where only the scan can find it. Queued earlier, the ownership cut consumes it
first — which is how the first draft passed against un-fixed code.

On #758/#759: I could not reproduce the missing-wake-source hang. Every path
leaves a wake. The overshoot is real (close to two staleness windows) and
provably cannot exceed two, so the bound is documented in place rather than
patched, and carried to bc3 as an open §23 contract question.

Verification

Pristine worktree, one pass, clean tree before and after:

  • go build / go vet / -race -count=1 / -count=5 — pass
  • make go-lint 0 issues; gosec on the CI-pinned v2.23.0 (hash-verified) 0 issues
  • full make checkexit 0
  • 22/22 fixtures
  • TestWalkFailureBetweenPages both subtests — the invariant that killed the
    reviewers' one-liner survives
  • staleness soak 8 × 500 under -race: 0 failures, 0 data races, re-earned
    because this PR rewrites the files that wait

Kill-matrix correction

Row 15's claim was inherited from the family README and is wrong; #778
corrects it. Tier 2 cannot prove zero egress to a foreign redirect target,
because the driver is the seam and manufactures the verdict. That is a
Layer-1 property, tracked for G1b.

Reconciled with the shipped feed contract (e28b337)

Stacked on the reconciled #777. What changed in the run loop against the shipped contract:

  • The 410 resume URL is a position-resume entry. The server documents it as re-entering at the epoch with the canonical filter set preserved (the inbox's at since=0) — positioned in served history, not at the present. Pages save on acceptance and the buffered live events drain after them; SPEC §23's present-class definition, row 17 and "Accept on FeedGap" say so, and fixtures 16/23/25/27 spell the resume as since=<epoch_after_id>.
  • Observer.FilterConflict(positionDigest, filtersDigest) fires on a 409 before PositionRejected(filter_changed). Diagnostics only — the re-entry is unchanged; a filters_digest differing from the SDK's own Filters.Digest() for the same set is the drift signal.
  • The tier-2 driver follows the foundations (config and identifier params for the new dimensions; poll rows forward performed_by_id/details; push events pinned at eleven keys). Tests spell continuations as the position-carrying URLs the server returns; SPEC notes the header echoes and since's signed 64-bit range.

Copilot AI balanced review requested due to automatic review settings August 12, 2026 02:40
@github-actions github-actions Bot added the go label Aug 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds the experimental Go Event Feed reference connector, deterministic conformance infrastructure, WebSocket transport, and checkpoint persistence.

Changes:

  • Implements the push/poll state machine, recovery, deduplication, and checkpointing.
  • Adds real and fake transports plus tier-2/tier-3 conformance tests.
  • Documents the experimental API and architecture exception.

Tip

If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

Reviewed changes

Copilot reviewed 54 out of 55 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
AGENTS.md Registers Event Feed infrastructure.
CONTRIBUTING.md Documents conformance verification.
go/README.md Adds Event Feed usage guidance.
go/go.mod Adds WebSocket dependency.
go/go.sum Locks WebSocket dependency.
eventfeed/backoff.go Implements retry timing.
eventfeed/backoff_test.go Tests retry timing.
eventfeed/cable.go Implements cable framing.
eventfeed/cable_test.go Tests cable framing.
eventfeed/catchup.go Implements catch-up and streaming.
eventfeed/catchup_test.go Tests catch-up behavior.
eventfeed/checkpoint.go Defines checkpoint contracts.
eventfeed/clock.go Implements production timers.
eventfeed/clock_test.go Tests production timers.
eventfeed/connector.go Defines the public connector.
eventfeed/connector_test.go Tests connector construction.
eventfeed/continuation.go Validates continuation URLs.
eventfeed/dedupe.go Implements delivered-ID deduplication.
eventfeed/dedupe_test.go Tests deduplication.
eventfeed/digest.go Implements filter digests.
eventfeed/digest_test.go Tests shared digest vectors.
eventfeed/doc.go Documents the package contract.
eventfeed/errors.go Defines terminal errors.
eventfeed/errors_test.go Tests error taxonomy.
eventfeed/event.go Defines feed events.
eventfeed/event_test.go Tests event decoding.
eventfeed/export_test.go Exposes test-only hooks.
eventfeed/filestore.go Implements file checkpoints.
eventfeed/filestore_test.go Tests file checkpoints.
eventfeed/filters.go Defines filter validation.
eventfeed/filters_test.go Tests filter validation.
eventfeed/loop.go Implements connector lifecycle.
eventfeed/loop_test.go Tests lifecycle behavior.
eventfeed/reconnect_test.go Tests reconnect and staleness.
eventfeed/recovery.go Implements poll recovery.
eventfeed/recovery_test.go Tests recovery paths.
eventfeed/scenario_conformance_test.go Replays conformance fixtures.
eventfeed/scenario_fixture_test.go Decodes fixture contracts.
eventfeed/scenario_harness_test.go Provides scenario harnessing.
eventfeed/scenario_selftest_test.go Tests driver strictness.
eventfeed/seams.go Defines connector seams.
eventfeed/transport.go Enforces cable URL policy.
eventfeed/transport_contract_test.go Defines transport contract tests.
eventfeed/transport_test.go Tests cable URL policy.
eventfeed/websocket_transport.go Implements WebSocket transport.
eventfeed/websocket_transport_test.go Tests real WebSocket behavior.
eventfeed/feedtest/clock.go Adds deterministic virtual time.
eventfeed/feedtest/clock_test.go Tests virtual time.
eventfeed/feedtest/minter.go Adds scripted ticket minting.
eventfeed/feedtest/minter_test.go Tests scripted minting.
eventfeed/feedtest/polls.go Adds scripted polling.
eventfeed/feedtest/polls_test.go Tests scripted polling.
eventfeed/feedtest/store.go Adds scripted checkpoints.
eventfeed/feedtest/transport.go Adds scripted cable transport.
eventfeed/feedtest/transport_test.go Tests scripted transport.
Suppressed comments (1)

go/pkg/basecamp/eventfeed/filestore.go:230

  • The rename is atomic but not durable without syncing the staged file and parent directory. After a system crash, the first checkpoint file can disappear; the next Load then reports Missing and starts at the present, which can skip history rather than merely replay from an older position. Since Save and the package advertise durable checkpointing, sync the file before rename and the directory after rename, or stop claiming crash durability and avoid treating disappearance as a safe present entry.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/recovery.go
Comment thread go/pkg/basecamp/eventfeed/loop.go
Comment thread go/pkg/basecamp/eventfeed/feedtest/clock.go
Comment thread go/pkg/basecamp/eventfeed/filestore.go Outdated
Comment thread go/pkg/basecamp/eventfeed/connector.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ceb8398f4c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go
Comment thread go/pkg/basecamp/eventfeed/cable.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/loop.go
Comment thread go/pkg/basecamp/eventfeed/connector.go
Comment thread go/pkg/basecamp/eventfeed/connector.go Outdated
Comment thread go/pkg/basecamp/eventfeed/cable.go
Comment thread go/pkg/basecamp/eventfeed/recovery.go
Comment thread go/pkg/basecamp/eventfeed/checkpoint.go
Copilot AI review requested due to automatic review settings August 12, 2026 05:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.

Suppressed comments (4)

go/pkg/basecamp/eventfeed/websocket_transport.go:73

  • The offered subprotocol is not verified after the handshake. coder/websocket v1.8.15 accepts a 101 response with an empty Sec-WebSocket-Protocol, so this can return a connection even though actioncable-v1-json was never negotiated, contrary to the CableTransport contract. Check conn.Subprotocol() and reject/close a missing selection; add a server case that intentionally selects none.
    go/pkg/basecamp/eventfeed/catchup.go:302
  • This parks an already-observed overflow until the poll returns. SPEC §23 requires semantic signals at the first consumer-context opportunity after the condition arises (with “before the next save” only as the outer bound), so a stalled poll can postpone the handler forever even though this goroutine has received the dropping frame. Dispatch the overflow immediately here; an Accept disposition can continue awaiting the poll, while Terminate should cancel the attempt/poll.
			} else if over {
				// The buffer, not the socket: the call is unaffected and is
				// awaited to completion, and the drop's disposition runs
				// before this page's position moves anything durable.
				l.deferred = &deferredFrame{item: item, overflow: true}
				if l.hooks.frameDeferred != nil {
					l.hooks.frameDeferred(true)
				}
				r := <-done
				return r.page, false, r.err

go/pkg/basecamp/eventfeed/websocket_transport.go:73

  • checkCableURL explicitly accepts case-insensitive schemes (the new unit test includes WSS://), but this passes that original spelling to coder/websocket. In v1.8.15 its handshake switch recognizes only lowercase ws/wss, so a URL accepted by policy fails as a transient dial and is retried indefinitely. Normalize only the scheme before dialing, while leaving the ticket-bearing remainder unchanged.

This issue also appears on line 70 of the same file.
go/pkg/basecamp/eventfeed/websocket_transport.go:183

  • This synchronous graceful close can block teardown for several seconds: coder/websocket v1.8.15 waits up to 5 seconds to write the close frame and another 5 seconds for the peer response. Because dispose calls this before cancelling the attempt, caller cancellation, Connector.Close, terminal outcomes, and reconnects can all stall on a live peer that ignores the handshake. Use a bounded teardown strategy that preserves the required close frame without letting the library's full handshake timeout delay the universal Closed edge.

Copilot AI review requested due to automatic review settings August 12, 2026 05:29

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated 3 comments.

Suppressed comments (1)

go/pkg/basecamp/eventfeed/catchup.go:504

  • A live event admitted by drainScan is stranded when the buffer was empty at the start of this iteration: batch stays empty, so this returns even though the scan just repopulated l.buffer. Streaming never drains that buffer, delaying the event until a later repair walk and allowing caught_up (and a held save) to happen first. Continue the loop when the scan added events.
		batch := l.buffer.take()
		if out, done := l.drainScan(at, &budget); done {
			return out, true
		}
		if len(batch) == 0 {
			return cycleOutcome{}, false

Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/filestore.go
Comment thread go/pkg/basecamp/eventfeed/catchup.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6eb58e6010

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/cable.go Outdated
Comment thread go/pkg/basecamp/eventfeed/websocket_transport.go Outdated
Comment thread go/pkg/basecamp/eventfeed/loop.go Outdated
Comment thread go/pkg/basecamp/eventfeed/transport.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
@jeremy

jeremy commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

Round-2 note on the suppressed comments, since three of them are load-bearing and one changed the design.

catchup.go:302 (overflow parked until the poll returns) — taken, and it reframed the visible finding above it. Rather than add a third dispatch site I deleted the overflow deferral outright; admitDuringPoll now dispatches at the drop's instant. Details in the thread. Fixture 22's own description already called drop-time dispatch normative and its post-poll rendezvous "scheduling slack only", so the implementation now matches what the merged fixture says it should do.

catchup.go:504 (live event stranded by drainScan) — real, and a defect introduced by my own round-1 change: drain returned on len(batch) == 0 even when the scan had just repopulated the buffer, so the event waited for a later repair walk while caught_up and the held save went first. Now gated on l.buffer.empty() too; red proof showed [event 41 save pos-1] where [event 41 event 42 save pos-1] was required.

websocket_transport.go:73 ×2 and :183 — all three taken and in progress: the negotiated subprotocol is never verified after the handshake, a WSS://-spelled URL passes policy but the library's dial switch is lowercase-only (so it fails transient and retries forever), and the graceful close can block teardown on a peer that ignores the handshake. That last one has to keep the close frame that nine fixtures pin — an earlier round caught the opposite bug — so it is getting a bound rather than a reordering.

Thanks for putting the sharp ones in the suppressed block; they've been the most useful part of both rounds.

Copilot AI review requested due to automatic review settings August 12, 2026 06:11
@jeremy

jeremy commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

Closing out the remaining suppressed comments from round 2 (02fec5f). Two fixed, one declined with evidence.

Unverified subprotocol — real, fixed. Confirmed in the library rather than assumed: coder/websocket@v1.8.15/dial.go:270-283, verifySubprotocol returns nil when Sec-WebSocket-Protocol is absent, and the server side (accept.go, selectSubprotocol) sets no header at all when it selects nothing — so a 101 that negotiated nothing yielded a live connection. A mismatched selection the library does reject, leaving the empty case as the only reachable one, exactly as you said. Now conn.Subprotocol() must match actioncable-v1-json after the handshake; on mismatch the socket is torn down and the dial fails DialPolicy. Policy rather than transient is a deliberate call: a fresh mint returns a URL pointing at the same server, which will select the same nothing, so retrying forever against a server that cannot speak the protocol is the wrong shape — the redirect refusal already lands there for the same structural reason. Test includes the server-selects-none case you asked for.

Unbounded graceful close — real, fixed. close.go:99-128,157-228: Close writes the close frame under a hardcoded 5s context, then waits another 5s for the peer, then waitGoroutines. The phases can't be bounded separately (closeHandshake is unexported, no exported close-frame writer), and CloseNow is not an escape hatch once Close is in flight — casClosing has already flipped, so it just waits too. Bound is 1s, run off-caller: only the write is contractual (§23 needs the peer to see the frame, which is why dispose closes before cancelling — this is a bound, not the reordering an earlier round rejected), and that write is a control frame to an open socket bounded by the kernel send buffer, not by the peer. All 12 fixtures carrying expectClientClose still pass. Red proof: Close blocked past 3s against a peer that never answers; green returns at exactly the 1s budget, so it passes via the timeout path rather than a lucky response.

Case-insensitive scheme — not a defect, declined. net/url.Parse lowercases the scheme before anything downstream sees it ($(go env GOROOT)/src/net/url/url.go:454), and coder/websocket's dial switch runs on u.Scheme from its own url.Parse, so a WSS:// spelling arrives there already wss. Verified empirically, not just by reading: the new test dials a WS://-spelled loopback URL and passed against un-fixed code, with the ticket-bearing remainder byte-identical. No normalization added; the test stays as a regression pin binding checkCableURL's deliberate case-insensitivity to what actually dials.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.

Suppressed comments (3)

go/pkg/basecamp/eventfeed/catchup.go:158

  • A socket outcome deferred while this poll was in flight is skipped when the poll itself fails. For terminal poll branches, recoverPoll calls disposeAttempt, which clears l.deferred; for retryable failures, the deferred frame can remain undispatched through arbitrarily many retries. In particular, an invalid_event_stream_command observed during CatchingUp can be replaced by poll_failed/authorization_failed or delayed indefinitely, despite SPEC §23 requiring protocol-fatal to terminate from every socket-open state. Dispatch the already-observed socket outcome here before applying poll recovery; a failed poll has no successful page boundary left to finish.
		if p.err != nil {
			step, out, done := l.recoverPoll(at, cursor, p.err)

go/pkg/basecamp/eventfeed/transport.go:40

  • Checking u.Host does not ensure that the URL has a hostname. For example, wss://:443/cable has Host == ":443" but an empty Hostname(), so it passes the policy check and is classified as a transient dial failure instead of terminal invalid_cable_url, causing repeated re-mints/dials for a structurally unusable URL.
    go/pkg/basecamp/eventfeed/cable.go:250
  • The tier-2 push-event schema requires all nine keys, including presence-bearing visible_to_clients, and SPEC §23 treats a correlated message missing a required event key as an invalid frame. Omitting this field from the presence checks accepts both an absent value and JSON null, exposing a nil value on a push event instead of taking the socket-failure recovery edge.
		{"creator_id", p.CreatorID != nil},
		{"recording_id", p.RecordingID != nil},

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 02fec5ff1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread go/pkg/basecamp/eventfeed/connector.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/cable.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Copilot AI review requested due to automatic review settings August 12, 2026 06:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated no new comments.

Suppressed comments (1)

go/pkg/basecamp/eventfeed/catchup.go:160

  • A deferred socket outcome is skipped when the in-flight poll returns an error. recoverPoll may retry, increment authorization failures, or terminate, and disposal then clears l.deferred; this can even swallow an already-observed invalid_event_stream_command instead of producing protocol_fatal. Since no page succeeded, dispatch the deferred outcome before classifying the poll error (the finish-page ordering only applies to successful pages).
		if p.err != nil {
			step, out, done := l.recoverPoll(at, cursor, p.err)
			if done {
				return out, "", true

@jeremy

jeremy commented Aug 12, 2026

Copy link
Copy Markdown
Member Author

Holding two findings open deliberately: the in-flight-poll mechanism has drawn four rounds

Two round-4 findings — the suspendable bound in awaitSupersededPoll (thread-adjacent) and the deferred socket outcome swallowed on the poll-error path (suppressed, catchup.go:160) — are not being fixed in this round. Both are correct. I am not writing the next patch on that mechanism until its shape is settled, because the pattern is now the finding.

The ledger on one mechanism, in order:

  1. Round 1 (Codex P1): a stalled PollSource holds the consumer's goroutine when the socket dies → added awaitSupersededPoll, bounding the wait by the staleness window.
  2. Round 2 (Copilot + Codex, P1): a deferred overflow dispatches after the save, and is dropped entirely on failed-poll paths → removed the overflow deferral; admitDuringPoll dispatches at the drop's instant.
  3. Round 3 (Codex): the fatal-frame scan's budget was sized by the live-buffer capacity, so a fatal could hide behind one ping → rebounded on the pump queue's own depth.
  4. Round 4 (Codex, this round): the staleness bound from step 1 is suspendable by the very pump backpressure that creates the problem, so a misbehaving peer can keep a compliant stalled poll alive indefinitely. Plus (suppressed): the socket deferral is cleared by disposal when the poll returns an error, which can swallow an already-observed invalid_event_stream_command instead of producing protocol_fatal.

Each fix was principled and each was verified, but four rounds of edge-findings on one structure is evidence about the structure. What they all orbit: the state machine both awaits a poll seam call and services the frame queue during it, parking one out-of-band frame in a single l.deferred slot, with a bound borrowed from a timer whose evaluation the queue itself can suspend.

The concurrency is not optional — fixtures 01 and 19 require a live frame admitted after confirm but before the entry page is served, and transition 21 requires the in-flight page to be finished before the dying socket is observed. So this is not a mechanism that can simply be deleted, which is precisely why it wants a decision rather than a fifth patch. The candidate shapes:

  • (a) Patch in place — give the superseded-poll wait its own deadline from the injected clock (unsuspendable), and dispatch a deferred socket outcome before recoverPoll classifies a poll error, since the finish-page ordering only applies to pages that succeeded. Small, local, and both findings close.
  • (b) One event loop — stop blocking on the poll at all: make the poll result just another case in the same select that reads frames, so there is no deferral slot, no superseded wait, and no borrowed bound. Larger, restructures the core of catchup.go, and dissolves this whole class.

I lean (a) plus a note, because the deferral that remains is the one SPEC explicitly sanctions (transition 21's deferred consumption) and (b) risks the delivery-ordering guarantees that nine fixtures and the save-ordering invariant pin. But it is a judgment call about this PR's core, so it goes to a human rather than to me.

The other three round-4 findings (synchronous cancellation on Close, visible_to_clients required on push frames, and the drain batch escaping the live-buffer ceiling) are independent of this and are being fixed now.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a9103dd83a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread go/pkg/basecamp/eventfeed/websocket_transport.go Outdated
Comment thread go/pkg/basecamp/eventfeed/transport.go
Comment thread go/pkg/basecamp/eventfeed/loop.go Outdated
Comment thread go/pkg/basecamp/eventfeed/loop.go Outdated
Comment thread go/pkg/basecamp/eventfeed/catchup.go
Comment thread go/pkg/basecamp/eventfeed/loop.go
Copilot AI review requested due to automatic review settings August 12, 2026 07:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 54 out of 55 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

go/pkg/basecamp/eventfeed/loop.go:152

  • ctx is the attempt context, which is a child of runCtx. Connector.Close (or caller cancellation) therefore cancels the pump immediately, before the state machine can reach dispose and call conn.Close. The default WebSocket read may abort the socket on that cancellation, so normal shutdown can still produce the abrupt disconnect that the close-before-cancel ordering is meant to prevent. Give the pump a cancellation scope that is canceled only by disposal after the socket close; HTTP seam calls can continue using the attempt context.
    go/README.md:615
  • This says the connector performs no wire I/O, but the package directly performs the sanctioned WebSocket cable dial. That contradicts both the implementation and the architecture rule. Limit the claim to HTTP requests and explicitly name the cable dial exception.
**Experimental: the Layer-1 seam adapters have not landed yet.** The connector performs
no wire I/O of its own — every HTTP exchange reaches the wire through a seam backed by
a generated operation — and the adapters that build those seams over the generated
`CreateStreamTicket` and `PollEvents` operations are still to come. Until they do, a
consumer must supply the `TicketMinter` and `PollSource` implementations itself, and the
exported surface may still change as they land.

go/pkg/basecamp/eventfeed/loop.go:638

  • Cancellation is not re-checked after Load. A custom store that honors the supplied context can return context.Canceled when the caller cancels or calls Close, and this path then yields checkpoint_load even though cancellation is documented to end iteration cleanly. Match the mint/poll paths by giving runCtx cancellation precedence over the load error.

Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93a3d5a457

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread go/pkg/basecamp/eventfeed/catchup.go Outdated
Comment thread go/pkg/basecamp/eventfeed/cable.go Outdated
Comment thread go/pkg/basecamp/eventfeed/loop.go Outdated
Comment thread go/pkg/basecamp/eventfeed/loop.go
jeremy and others added 29 commits September 16, 2026 01:21
Completes 6ea9d12 (event-feed-foundations), which typed the transport's
read-limit abort: coder/websocket rejects an over-limit message during the
read, and ReadFrame now returns the exported ErrFrameOversize sentinel flat.
This branch's run loop never learned that vocabulary. observableSocketError
preserves package-owned sentinels, *CloseError, and *invalidFrameError, and
degrades every stranger to errSocketFailed — so the one invalid-frame shape
the SPEC assigns to the transport arrived typed and left generic, and
Observer.Disconnected lost the indication §23 requires it to carry.

The contract is the invalid-frame class (SPEC.md §23, the three-shapes
bullet — SPEC.md:2810-2821 at this revision): "a frame exceeding
EVENT_FEED_MAX_FRAME_BYTES" is one of "three shapes, one disposition", each
dispatched as a socket failure "with Observer.disconnected carrying an
invalid-frame indication; never an untyped decoder error escaping", and
"the size check binds inside the transport". The disposition was already
right — a pump read error takes the current state's socket-failure edge to
Backoff regardless of type — so this is observability, not a verdict change.

Classification design: the sentinel passes through, rather than widening
invalidFrameError with a third shape. The sanitizer's own taxonomy decides
it. invalidFrameError names violations the CONNECTOR judged in frames it
read; the oversize frame is never materialized, so there is nothing for the
connector to classify — the seam's sentinel IS the classification. A third
shape would also widen the security invariant's pinned rendering vocabulary
("names its shape (frame_parse / event_decode) and nothing else") for no
gain. The new arm follows the sentinel rule already stated there: reduced to
the bare package value, never the argument, because a seam's wrapper text is
where a cable URL rides. ErrFrameOversize is exported so seams can return
it, but a var can only be referenced — unlike the exported struct types the
sanitizer evicted, it cannot be rebuilt around peer text.

feedtest.Conn now wraps the sentinel in its read-limit violation, aligning
the fake with the seam contract its interface documents — an untyped fake
would keep every test green against a classification the real transport
defeats. The red proof stages an actually oversized frame through the pump
(one byte past the dial's own recorded cap) and failed against the pre-fix
sanitizer with "Disconnected err = event feed socket failed, want the
ErrFrameOversize indication"; green after, asserting both the errors.Is
match and the bare-sentinel rendering, with the Backoff timer arming to pin
the disposition unchanged.
…as the bug

Five of this round's eight findings taken; the notable decline is recorded
at the end.

The continuation validator had the defect validateConfig was cured of two
commits ago, on its other consumer: checkContinuation canonicalized the
server-supplied URL first, and CanonicalOrigin's ToLower rewrites every
invalid byte to U+FFFD — so a continuation host carrying a raw 0xff
collapsed to the same canonical form as a configured origin that
legitimately contains the replacement character (valid UTF-8, so
construction accepts it), and §8 same-origin validation equated two
distinct byte strings on the authenticated path. The raw bytes are refused
before the lossy step; red proof: the collapsed URL was accepted against a
U+FFFD base.

Two waits treated a staleness firing as a verdict where §23 says it is a
wake. In awaitSupersededPoll, the stale=false arm returned superseded on an
authoritative expiry — but a read error does not reset staleness, so the
PRE-EXISTING window, mostly spent when the failure was deferred, abandoned
the in-flight poll almost immediately: a failure landing 7s into a 7.5s
window granted the poll 0.5s of the grace phase §23 measures from the
deferral ("wakes may be early or late, and the deadline is what decides").
The arm now latches through graceWake exactly as the stale caller does, and
the deadline alone ends the wait — red proof: teardown on the old firing,
green: a wake, then teardown only when virtual time reaches
deferral + EVENT_FEED_STALE_AFTER. In waitPollRetry, a select with both
poll-retry and staleness ready could take the retry and start another Poll
on a socket whose expired window was already evidence — and a blocking call
then defers the stale verdict and buys a fresh grace phase, overrunning
§23's published bound ("detection window + grace phase... nothing waits
longer than their sum"). The retry arm now re-checks staleness before
polling; the red proof parks the run goroutine in an overflow handler while
both timers fire, and 17/40 rounds issued the second poll.

Transition 6 was announced late: newLiveConn opens the socket, arms
staleness and starts the pump — the transition's own definition — but the
state stayed Connecting until awaitConfirmation, so Observer.Connected ran
with AwaitingWelcome's timer set {handshake-deadline, staleness} against an
announced state whose set is {handshake-deadline} alone. The announcement
now precedes the callback; red proof: Connected observed "connecting".

Wait could report quiescence for a run that had started: the single-shot
claim was an atomic CompareAndSwap taken BEFORE the mutex section that
publishes runDone, and the claim is observable in that window — a second
consumption yields the usage terminal — while Wait still read nil and
returned. The claim now lives in the same critical section as the
publication, so Wait runs wholly before it (no run to await) or wholly
after runDone exists. No red proof discriminates this one honestly: the
window was two adjacent statements with no seam between them, and a hook
would exist for nothing else — the same account the terminal-claim fix gave
in an earlier round. The existing single-shot and Wait suites hold the
contract on both sides of the change.

AGENTS.md's architecture row still called this package foundations-only
with the run loop pending; this branch is where that stops being true, so
the row now names foundations, the run loop, and the tier-2 driver, with
the Layer-1 adapters and the other SDKs still pending.

Declined: terminating on a post-confirmation reject_subscription. The state
machine draws transition 12 solely from AwaitingConfirmation; §23's "always
terminal — first attempt or reconnect" pins which ATTEMPT a rejection lands
on, not a state-independent verdict. Action Cable rejects only from the
subscribe callback and the connector subscribes exactly once per socket, so
a post-confirmation rejection is an unsolicited frame — and honoring it
would hand one such frame the connector's most severe verdict, ZERO
reconnects, which is precisely what the pre-subscribe gate was added to
prevent in an earlier round. The default arm's liveness-only comment says
this on purpose.
All five of this round's findings taken — four expired-verdict races in the
run loop's family, and a conformance-driver fidelity hole.

The dial result could outrun the handshake deadline: both ready, the select
random, and accepting the dial installed the pump, armed staleness, and
announced Connected for an attempt whose window had already closed —
transition 7 bypassed by a coin flip. The success path now drains the
deadline before transition 6, as a drain rather than a Stop-probe because
an unfired deadline must keep running to welcome. The red proof fires the
deadline from inside Dial, sequencing it strictly before every dial result,
so any Connected at all is a violation: 1/100 rounds pre-fix, 0 after.

The subscribe write's bounded wait selected only {written, close, phase
deadline}, on the recorded premise that the phase deadline is always the
tighter bound. It is not, twice: staleness (7.5s) undercuts even the
default 10s handshake deadline on an immediate first welcome, and a
duplicate welcome resends under the confirmation deadline, which is
configurable to anything — a dead socket sat blocked in the write
arbitrarily past the expiry rows 9/15 say tears it down. The wait now
carries the staleness case like every other socket-open wait; red proof:
a stalled write plus a fired window produced no StaleConnection within the
watchdog.

Streaming could deliver past a latched expiry: a frame received AFTER the
window fired does not reset it — staleHolder.arm latches, which is §23's
authoritative-firing rule working — but the fired timer and that frame were
then both ready, and the frame arm delivered first. The arm now re-checks
the verdict before delivering, with evaluate as the arbiter: a frame the
pump received first moved the generation and delivers as ever, a latched
expiry takes transition 25 with the frame discarded to the reconnect walk.
Red proof, rounds-driven off a collector parked mid-delivery: 17/40 rounds
delivered event 102 past the latched expiry.

A throttled Retry-After of zero was read as absence. SPEC §6 fixes the
adapter mapping — a retryable outcome whose last response carried a parsed
Retry-After maps to throttled(retry_after) whatever its status, one without
maps to transient — so PollThrottled always carries a server-directed wait,
zero included, waited exactly and cap-exempt. pollRetryDelay is now
kind-keyed; a parsed zero stays zero instead of drawing up to 60s of local
jitter (the red run drew 315ms), and a negative value clamps rather than
arming a negative timer.

The tier-2 driver's arrival-strict lookahead judged expectBuffered against
GLOBAL occupancy history, so occupancy reached and left in a long-departed
era satisfied a pending expectBuffered and let the scan read through to a
following expectCheckpoint — an out-of-order save accepted by the very
instrument that exists to reject it. History is now scoped to an era
boundary captured whenever the step pointer moves (enterStep and the
await advance), and expectBuffered's own rendezvous scans from the same
boundary, which also closes the reached-and-left-before-runStep flake its
run-time capture had. Red proof: the scan judged an arriving save against
step 3 (expectCheckpoint) on stale history. Every committed fixture still
passes under the scoped scan.
…o ask it

Four follow-ups taken, one of them a correction to this branch's own
previous round — stated plainly: the round-4 throttled-zero change overshot,
and Copilot caught it.

The two §6 texts compose; they do not conflict. The seam mapping is
presence-keyed — throttled iff the last response carried a PARSED
Retry-After — but "parsed" is defined by §6's parsing algorithm, and that
algorithm cannot yield zero: step 1 requires an integer > 0, step 2 returns
max(0, date − now) only when > 0, and the rounding rationale says why —
"zero is read as 'no usable value' and drops the request onto the local
backoff curve". So a conformant adapter's throttled ALWAYS carries a
positive value, the original value gate honored every one of them exactly,
and keying the delay on the kind bought nothing while turning a
nonconforming adapter's throttled-with-zero into a zero-delay poll-retry —
a tight loop against a server that is throttling the caller. The red run
shows it literally: "poll-retry armed for 0s". pollRetryDelay is reverted
to the value gate with the reconciliation written above it; the round-4
test asserting the wrong contract is replaced by one asserting the jitter
fallthrough. What WAS wrong is the seam doc, which still described both
throttled kinds by status ("429/503") — the exact status-keyed mapping the
§6 sentence exists to forbid — and both docs now state the presence-keyed
definition with the always-positive consequence.

The tier-2 driver had the same status-keying in the flesh, on both lanes: a
schema-valid 500/502/504 carrying Retry-After became transient and its
delay never reached classifyMintFailure's floor, while a bare 429 became a
throttled zero. Both outcome mappers now classify retryable statuses by
presence of the parsed value. And the driver's Retry-After parse was
integer-only — a valid RFC 7231 HTTP-date failed the scenario outright
(red: strconv.Atoi rejecting the date), zero and negatives became
durations. retryAfterFrom now implements §6 against the harness's virtual
clock — integer > 0; else HTTP-date with the sub-second remainder rounded
UP; else undefined — returning presence separately from the value.

The round-3 grace-phase fix also overshot, in the other direction: an
authoritative firing late in the phase re-armed graceWake for a FULL
window, so a firing 1ms short of the deadline pushed the only remaining
wake almost a whole window past it — the wait ended nearly two windows
after the deferral instead of one. graceWake now takes its wake distance
and every caller passes the remainder to its deadline (the full window at
the deferral itself, where they coincide). The new red proof pins the upper
bound the round-3 test could not see: advance to 1ms short, fire, advance
the last 1ms, and demand the teardown there — red sat parked until the
watchdog; green ends at the deadline exactly.
… lost to an announcement

All seven of this round's findings taken — four in the loop's established
families, three in the README, each verified against the code first.

A superseded poll is now cancelled AT the grace deadline, in time and not
merely in control flow. Disposal closes the socket gracefully before it
cancels — deliberately, so the peer sees a close frame — but a peer that
never acknowledges holds that close for the transport's full grace budget,
and the abandoned seam call (carrying the caller's bearer) stayed live
through it, stalling reconnection past §23's detection-plus-grace bound.
The deadline branch cancels the attempt context itself; the disposal that
follows finds its own cancel a no-op. The red proof captures the pump's
read context and parks the teardown's close: pre-fix the context was still
live at the park.

AwaitingConfirmation's frame arm gains the latched-expiry drain the
Streaming arm got last round — the same shape for a worse consequence: a
frame received after the window fired could win the select and either admit
a correlated message to the loop-wide buffer, which survives the teardown
and resurfaces after reconnect, or hand a confirmation a live catch-up on a
socket whose verdict was already in. Rounds red: 11/40 announced Confirmed
past a latched expiry.

Two Close-outranks-the-announcement gaps, both on the far side of observer
callbacks that are documented Close sites. outcomeFailed announced Backoff
and asked the product clock for a reconnect timer with runCtx already
cancelled by a Close inside Disconnected or StaleConnection; finishDrain
continued past CaughtUp into dispatchDeferred and stream, which re-arms the
repair cadence through the same product seam and announces Streaming. Both
now take the Closed edge first — a host clock that blocks in NewTimer would
otherwise keep the iteration and Wait from ever reaching Closed. Red
proofs: the state ledger carried "backoff" (and "streaming") after Close
returned; green goes straight to closed.

Three README corrections, each checked against the code. The terminal-vs-
continuable section implied every continuable outcome waits a timer;
rejected positions do not — reenterWalk re-polls a replacement cursor
immediately on the same socket, and the text now says which failure
continues how. The unwrap claim promised a generated error behind every
poll_failed; the connector's own verdicts carry none (a 200 whose page has
no position), so the claim is now conditional. And the consumer-surface
section stated "Close stops the feed" without the lifecycle this branch
actually ships: Close only cancels and can return with a save still in
flight, Wait is the quiescence point, and a replacement connector opened
straight after Close races the prior run's last save — the README now says
so where the store guidance lives.
…dy race

All six of this round's findings taken.

The three dispatch frame arms — AwaitingConfirmation, Streaming, and the
poll-retry wait — now let the Closed edge win the both-ready race Close
itself creates: cancellation makes the conforming ReadFrame return its
cancellation error, the pump hands it off, and the frame case dispatching
it reported the consumer's own Close to Observer.Disconnected as a socket
failure, after Close had returned. Rounds red: 17/40 fired
Disconnected(context canceled) post-Close.

A panic in host code — an observer, the signal handler, a checkpoint
store, the consumer's range body, all of which run on the run goroutine —
unwound with only the run context cancelled: no deferred path closed the
CableConn, and a compliant conn is only required to unblock reads on
CLOSE, so under an outer recovery the socket stayed open with its pump
parked and a later Connector.Close could not reach it (cancelRun cleared
on unwind). runCycle now re-disposes the live attempt inside a recover and
re-panics; disposal is idempotent, so paths that already disposed are
unharmed, and phase timers held as locals are deliberately not chased —
unfired, they fire into nothing. Red: the socket stayed open and the
staleness timer stayed armed through a recovered panic.

The repair-poll arm gains the family's expired-verdict drain: entering the
walk made an already-rendered stale verdict look in-flight — pollPage
deferred it into a grace phase, and the poll's page could be accepted
first — where transition 25 fires directly from every socket-open state.
Rounds red: repair polls issued on an expired socket.

Both unauthorized paths forwarded a seam-supplied RetryAfter into the
reconnect floor. The SPEC's loop table names `unauthorized` "a kind
carrying no retry_after" and row 4 says the rest: "the backoff draw alone
governs". Both arms drop the field; red on each showed the 5-minute floor
verbatim.

The tier-2 driver's redirect leg now runs the SHIPPED per-hop predicate.
Fixture 30's description promised a sentinel listener the harness never
binds, and the driver classified the scripted 302 by an ad-hoc origin
comparison — synthesizing the refusal rather than exercising the decision,
and turning an unreducible hostile Location into a SCENARIO error instead
of a refusal. redirectRefusalFrom now classifies through checkContinuation
(exported to the driver as a test shim), so the decision tier 2 pins is
the one that ships. What tier 2 cannot see — an adapter's HTTP client
auto-following inside one seam call — is below the poll seam by
construction, and pretending otherwise was the finding: the fixture's
description now claims exactly what the tier verifies (the refusal
decision, the continuation terminal, the closed socket, no further seam
calls) and names Layer-1 adapter conformance as the owner of the zero-
egress property, where the pending adapter PR's sentinel can actually
observe a request. The description edit is the only fixture change, one
line, and the fixture gate stays green.
Copilot's exact-head review carried one suppressed finding, and it is a real
residual of the round-5 grace fix: the latched, deadline-remainder wake was
armed only when the DEFERRED OUTCOME was a staleness expiry. A deferred
disconnect or invalid frame left the phase's wakes riding the ordinary
staleness window — which the pump still re-arms on every inbound frame — so
a peer whose last frame landed just short of the deadline pushed the only
remaining wake to frameTime + staleAfter: the rearm's own wake ran the
deadline check a hair early, then nothing fired until almost a full window
past the deadline, and the abandoned poll outlived the one-window phase §23
measures from the deferral. The SPEC's immunity is unconditional — "a frame
arriving inside the phase re-arms staleness in the ordinary way and MUST
NOT move the deadline" — and a deadline nothing wakes is moved in effect.
The existing TestGracePhaseIsImmuneToFrameResets could not see it: its
staleDeferralHarness drives a silent socket, so the deferral is always the
stale kind, the one path that already had the latch.

pollPage now arms graceWake at the deferral for EVERY deferred kind. The
latch is what buys both published immunities — pump resets are refused, a
latched window cannot be suspended — and the deferred outcome is untouched:
the walk still dispatches the PARKED item, never a substituted staleness
verdict (the new test pins that too — the teardown reports the disconnect's
redacted "other", not staleness's empty reason). With the wake armed at the
deferral, awaitSupersededPoll's firing arm collapses to one case: every
firing is a wake re-armed for the remainder, the two-regime split and its
evaluate call are gone, and the `stale` parameter with them.

Red proof, the frame-deferral sibling of the existing immunity test: a
deferred remote disconnect, the peer's last ping 1ms short of the deadline,
then silence — pre-fix the wait sat parked past the deadline until the
watchdog; green tears down at the deadline exactly, dispatching the parked
disconnect, with Backoff arming after.
… and two honest declines

Five findings; three taken, two declined with the mechanism named — and one
of the takes ships with a no-red account rather than a proof, stated
plainly.

The subscribe write's result arm gains the family's Closed-edge check:
cancellation surfaces through a conforming WriteFrame as its own error, so
`written` and runCtx.Done() can be ready together and the result arm
dispatched the consumer's own Close to Observer.Disconnected. No red proof
discriminates this one, and two attempts taught why. The written channel's
readiness is causally downstream of runCtx's, so a machine PARKED in the
select always takes the Closed arm first — the race lives only in the
spawn-to-select entry window, which no hook can hold open. A staged version
that raced a genuine socket failure against Close fired on nearly every
round on BOTH sides of the fix, because a real failure dispatched before
the cancel lands is legitimately Disconnected and no assertion can tell
the orderings apart. The check is kept because the family closed every
sibling arm and the entry window is real; the terminal-claim fix set the
precedent for saying so instead of shipping a test that proves nothing.

Declined: the pump-exit (!ok) racing runCtx.Done. The interleaving is
structurally unreachable: handOff's FIRST send is non-blocking, so with
queue room the error item always lands and close-without-item requires a
full queue — whose drained items hit the ready-item Closed check first, one
round's fix ago. Forty staged rounds produced zero occurrences; the test
was dropped rather than kept as decoration.

Declined: redacting the generated error out of Terminal(poll_failed). The
SPEC pins the opposite twice — the out-of-inventory edge list and the
terminal-reason table both say poll_failed is "passed through with the
generated error attached" — and the URL it may render is the VALIDATED
same-origin continuation: the consumer's own configured origin and a feed
cursor, carrying no credential (the bearer rides the header; tickets never
touch the poll lane). The redirect arm redacts a different trust class —
its URL is the hostile Location that FAILED validation. The observer
surface is already closed-vocabulary either way.

The live buffer is now a head-indexed ring. The finding's magnitude was
wrong — reslice-and-append amortizes to O(1), not an O(capacity) copy per
event — but its direction was right: each growth step copied the full
backing and transiently retained a SECOND buffer's worth of payload, the
exact class add's zeroing exists to prevent. The store still grows lazily
while filling, wraps only at capacity, zeroes vacated slots, and exposes
logical order through a snapshot the hand-off boundary now uses. Red: a
full window of sustained overflow made 10004 allocations against the
10000 the dropped-ids slices account for; green makes exactly 10000, and
the eviction/zeroing/order pins all hold.

The driver's Retry-After parse now matches the SDK's parseRetryAfter
exactly: RFC 9110's 1*DIGIT checked before ParseInt (which honours a sign
Atoi also accepted — a schema-valid "+5" classified as throttled), int64 so
the verdict cannot vary with the platform's int width, digits beyond int64
malformed, and a representable value clamped to the portable
MaxInt32-seconds ceiling instead of overflowing the duration multiply into
garbage. Red: "+5" → throttled(5s); green: the sign, width, overflow and
saturation rows all land where client.go's parser puts them.
The Race Detection job on the conformance-driver branch failed
TestProtocolFatalBehindADeferredRecoverableOutranksIt/page_boundary with
mint_failed where protocol_fatal must win — while the same loop code passed
this branch's own CI, and the driver branch's eventfeed diff is
comment-only. The defect is the TEST's rendezvous, latent since the wedge
was added, and a loaded two-core runner is what finally scheduled it.

The choreography serves the wedge ping without awaiting it, drains the
hand-off signal channel, serves the fatal, and waits for ONE signal. When
the ping's hand-off lands after the drain, that one signal is the PING's:
the fatal is still between the pump's read and its hand-off — where §23's
normative boundary says no scan need find it — when the poll returns. The
entry cut spends its capacity-1 budget on the ping, the probe scans a
queue the fatal has not reached, the deferred recoverable's teardown wins,
and the reconnect lands on the terminally scripted second mint:
mint_failed, exactly the CI signature. The loop is correct; the test
counted one signal for two frames.

The rendezvous now owes one awaited signal per frame served after the
drained baseline (the deferral rendezvous proves every earlier signal had
landed), so the fatal is handed off — the state the guarantee is stated
over — before the poll is allowed to return. The CI failure is the red
capture; the interleaving cannot be forced locally without a hook into the
pump's hand-off, and 20 through 100 stress runs under GOMAXPROCS=2 never
hit it here. Post-fix: 100 race runs green under the same constraints.
…cts that lost their turn

Five findings from the exact-head review; four taken, one declined.

The panic disposal now stops the attempt's active phase timer. The
round's own fix handed disposal a nil deadline — its commit even said
phase timers were "deliberately not chased" because an unfired timer
fires into nothing — and that reasoning holds for the system clock only:
the clock is a product seam, and a handshake deadline left registered in
a host's custom clock outlives the run indefinitely, contaminating every
later user. The attempt retains its currently armed phase timer
(handshake and confirmation deadlines, repair cadence, poll-retry), the
recover stops it, and ordinary teardowns double-stop harmlessly. Red: a
panic from Observer.Connected left map[handshake-deadline:1]
outstanding.

The in-flight poll's done arm now parks an authoritative expiry before
returning the result. Both can be ready, and accepting a FAILED verdict
let recoverPoll's disposal discard the socket's fired window — an
unauthorized verdict incrementing the shared counter, a 410 raising
FeedGap, where transition 21 should have torn the dead socket down. The
walk's existing precedence (probe, then deferral, then recoverPoll)
dispatches the parked expiry ahead of any failed poll, and a succeeded
page keeps finish-the-page ordering exactly as if the expiry had been
observed mid-flight. Rounds red: 20/40 dispatched the unauthorized
verdict over the fired window, with StaleConnection never reported.

Declined: arbitrating the superseded wait's done arm against the grace
deadline. The deadline bounds WAITING — "nothing waits longer than their
sum" — and a result that is already ready costs no wait to accept: the
page's deliveries and save are §23's own finish-the-page preference, and
the deferred outcome still dispatches at the very next boundary.
Discarding a served page because its acceptance raced the final wake
would trade data for nothing the bound asks for.

Both semantic dispatch sites now let Close outrank the handler.
Observer.BufferOverflow and Observer.Gap are supported Close sites, and
the handler that followed them is host code that may block — run after
Close returned, it kept the iteration and Wait from terminating, and its
disposition would govern a feed the consumer already ended. Red, both
sites: the handler ran after Close returned.

The unauthorized-threshold and filter-invalid terminals rebuild their
causes without the generated error. §23's terminal table mandates the
attachment for poll_failed and mint_failed ONLY — the previously declined
poll_failed finding stands on exactly that mandate — while these two
reasons' contracts are the counter message and the server's verbatim
filter message, and a generated error routinely renders the full request
URL: on this lane the server-controlled continuation, path and query
included. Red, verbatim: both terminals rendered
"?page_token=SECRET-CURSOR" through Error and the unwrap chain; green
keeps the reason, the counter text, and the verbatim filter message, on
a connector-authored cause.
Foundations' round-12 P1 made every rejected-continuation rendering
zero-interpolation closed vocabulary; these two end-to-end pins still
required the rejected origin in the terminal message and went red on the
merge. They now assert the absence of every server-authored component,
matching the checkContinuation pin that owns the exact phrase.
…spelling

The routed half of foundations' round: PollError.LocationOrigin is now
data-not-rendering by contract — a hostile redirect can reflect the
caller's bearer into a host label, the CloseError.Reason precedent — and
PollError.Error already omits it, but this branch's refused-redirect
terminal still concatenated the origin into TerminalError.Msg, undoing
the field's demotion at the one surface consumers read first. The
terminal now renders the fixed violation-class phrase the continuation
rejections converged on ("the poll refused a cross-origin redirect");
the origin survives as a field on the retained sanitized cause for a
caller that reads it, which no rendering includes.

Red proof, bearer-canary form: a LocationOrigin of
https://bearer-canary-abc123.invalid appeared verbatim in the terminal's
Error() pre-fix and appears nowhere in the rendering or the unwrap chain
after. Two tests pinned the old origin-bearing message —
TestRedirectRefusalExposesOnlyTheLocationOrigin (now
TestRedirectRefusalRendersNoServerValue, inverted into the canary proof)
and TestPollFailureClassification's redirect row, which now expects the
fixed phrase and points at the negative pin.
The live buffer's ring wrapped an insert whenever the tail's index fell
below the head, and grew the backing whenever it did not. Those are the
right two moves in the wrong order. A drain's shifts move the head off zero
long before the store reaches capacity, so a pre-capacity store whose tail
has reached len(events) satisfied the wrap test first: the insert circled
to index 0, and the next insert — now past the head — appended, changing
the modulus every existing index is taken against. With capacity 3, add 1
and 2, shift 1, add 3, add 4 read back as 2, 4, 3. fatalScan interleaves
admissions with the drain's shifts, so this was the drain's own delivery
order.

The insert now asks whether the store has reached capacity, which is the
invariant the layout actually rests on: below it the occupied range is
contiguous and ends at or before len(events), so growth is the only move
that keeps every index valid, and a wrap can exist only at capacity, where
the modulus never changes again. The regression test drives the three
partial-wrap shapes and reads the buffer back in logical order.
…ived

Three receiving arms were missed when the others learned that a ready item
does not outrank a Close: the in-flight-poll servicing, the bounded
admission pass, and the drain's protocol-fatal scan. Each dequeued past a
cancelled run context, and each did post-Close work the Closed edge exists
to rule out. The admission pass dispatched the pump's own cancellation
error through the live-frame path, so Observer.Disconnected reported the
consumer's Close as a socket failure after Close returned. The other two
admitted frames into a buffer the next run never reads, and when that
admission dropped, called Observer.BufferOverflow after Close returned — a
registered handler that closed the connector and returned Accept saw the
pass it returned into keep admitting.

The check is added at the three receives, and once more at the one place
every admission passes: admitLive refuses an admission under a cancelled
run context before it touches the buffer. That is what covers the
accepting-handler case wherever the pass that dispatched it receives next,
rather than only at that pass's own dispatch point.

Three tests pin the three arms, each red against the previous revision:
the handler queues a third frame before closing and asserts no overflow
observer call follows; the pass is parked by a frame-handled callback while
Close lands and the cancellation queues behind the parked frame; and the
drain's loop body closes with a protocol-fatal disconnect and two events
already queued behind the event in hand — the fatal queued first, because
it bypasses admitLive's guard and so pins the scan's own check.
Four places acted on a socket whose verdict was already rendered, and each
made that verdict look like something else.

The walk's page boundary probed the socket before dispatching the outcome
parked during the page's own poll. A deferral latches a grace wake on the
staleness timer, so a page whose delivery ran past the grace deadline
reached the boundary with that wake fired; the probe read it as an expiry,
disposed the attempt — which clears the slot — and reported staleness where
the parked verdict was the server's own invalid_event_stream_command. The
boundary now dispatches a parked verdict first, and only when one is
parked: the fatal probe admits what it dequeues, so with nothing parked the
ordinary order — the expired window before any admission — stands, or an
overflow the probe manufactured would end the cycle ahead of a staleness
verdict already in.

The walk issued its first poll without probing staleness, so a window that
expired during Observer.CatchUpStarted, or during a re-entry's callbacks,
was deferred as if it had fired mid-call and granted a grace phase, with
the page possibly accepted first. The repair cadence and the poll-retry
arms already probe before entering the walk; the walk's own entries do now.

The subscribe write's wait has no frame arm, because a write in flight
cannot dispatch frames without re-entering the handshake it belongs to. A
conforming transport is only required to unblock a write on Close or
cancellation, so a verdict the pump queued behind a blocked write died with
the socket when the wait ended for its own reason, and the connector
reconnected into a rejection §23 says is terminal with zero reconnects: a
raw invalid_event_stream_command, or a correlated reject_subscription
queued behind a duplicate welcome's blocked retransmit. The same loss
turned out to exist wherever the connector tears the attempt down for a
LOCAL reason with the server's last word already queued behind it — a
confirmation deadline against a queued rejection, an expired window
against a queued fatal, a garbled frame ahead of a queued fatal, a lapse
ahead of a queued pre-welcome `unauthorized` that the shared counter
needed — and which arm won the select, or which frame arrived first,
decided it. So the rule is stated once: every recoverable teardown
(failSocket, lapse, staleTeardown) scans the queue for the server's own
verdict before disposing — any disconnect frame, dispatched by reason, or
a correlated rejection — and a frame arm holding one dispatches it ahead
of its expiry probe.

The non-blocking probe those arms each spelled out is one helper now,
expiredStaleness, and the StaleConnection-then-Disconnected pair every
staleness edge reports is observeStale; the five inline copies and seven
pairs are gone. Eight tests pin the defects, each red against the previous
revision. The written-arm probe carries no test of its own: the fake
transport's write and the fired window cannot be made ready together from
outside the select, and a test that passes for either behavior is worse
than none.
…aking

Two shapes of host-seam misbehavior reached the consumer's goroutine as a
crash or a leak where the documented outcome exists.

A seam returning a nil *MintError, *PollError, *DialError or *CloseError
through a non-nil error interface — the ordinary Go footgun — satisfied
errors.As with a nil target, and the classification that followed
dereferenced it. The four sites now treat a nil target as the unclassified
case: the mint and poll lanes take their unclassified terminal with a named
cause rather than the nil value (rendering it is the same dereference), a
dial is a transient failure into backoff rather than a policy verdict read
off a nil Kind, and the observable-error reduction degrades to the generic
socket failure.

A product Clock that panics arming the staleness timer does so inside
newLiveConn, after the dial succeeded and before the attempt owns the
connection. The recovery disposed a live attempt through at.lc and stopped
the phase timer otherwise, so the dialed socket stayed open with its pump
never started — a permanent leak in a process an outer recovery keeps
alive. The connection is declared ahead of the recovery and closed on
unwind when no liveConn owns it yet.

Tests: each typed-nil site, and a clock that panics on its first staleness
timer, all red against the previous revision.
The third unauthorized mint's authorization_failed terminal retained the
generated error as its cause, where the poll lane's threshold terminal
rebuilds a bare classified cause: §23's terminal table attaches the
generated error to mint_failed and poll_failed only, and this reason's
contract is its counter message. The mint lane now rebuilds the same way.

The redirect-refused terminal rendered "a cross-origin redirect", but the
kind covers every Location that fails the per-hop rule — downgraded,
non-HTTP, unparseable — so the phrase misdiagnosed the other three. It
names the rule the Location failed now, which is the violation class the
continuation terminals already render.

The CableTransport.Dial contract described a policy error's Reason and
cause as reaching the consumer through the iteration's terminal, and hung
the never-render-the-URL obligation on that. The loop reads the Kind alone
and forwards nothing a transport wrote, for every kind; the doc says so,
and states what the obligation still protects: the seam's return is opaque
text to everything outside this package, and a host that wraps or logs it
directly has no reduction in front of it.
Wait's exclusion named consumer callbacks alone, and §23 said the same;
the run also awaits every seam it calls synchronously — minter, poll
source, transport and connection, store, clock — so a wait() from inside
one of those blocks the very call the run is blocked on. Both now say
"anything the run is waiting on" and list the seams.

doc.go called the package unusable against the live API while the next
sentence, and the README, describe hosts supplying TicketMinter and
PollSource over the generated operations. Nothing ships those two seams
yet; a host-supplied adapter is the supported path, and the paragraph says
so instead of telling that host its path is impossible.

Fixture 30's kill claim and the kill matrix's row 15 are deliberately not
touched here: #778, stacked on this branch, already rewrites both
(6135ca7) and scopes the zero-egress obligation to the Layer-1 adapter
(a3189b5), and repeating that work here would only conflict with it.
…ader sees twice

The driver's mint mapping populated RetryAfter before classifying, so an
unauthorized or unrecoverable outcome carried a value §23 defines for the
throttled kind alone — and its comment said row 4 floors the reconnect on
it, which is the opposite of what row 4 and classifyMint say. The value
rides on the throttled branch now, and the comment names the tier-3 pin
that proves the connector ignores a nonconforming one.

The driver's HTTP-date branch could report a negative Retry-After: Sub
saturates at time.Duration's ~292-year maximum, and rounding that up to
whole seconds and multiplying back wrapped. It clamps to the digit
branch's MaxInt32-seconds ceiling, and a year-9999 date is pinned.

The loader promised that nothing in a fixture is silently ignored, and
encoding/json broke that promise for one shape it cannot see: a member
named twice keeps the last value and DisallowUnknownFields never meets the
first, so a duplicated expectation was discarded while the fixture stayed
green — and the schema gate's own parser collapses duplicates the same
way. Fixtures are now walked as a token stream before decoding and any
object naming a member twice, at any depth and under escape-equivalent
spellings, is refused; the four levels are probed.
Streaming arms its repair cadence before the state is announced, and says
why: the announcement is made with the exact set §23 publishes for the
state already outstanding, so an observer reacting to it sees that set and
not the empty one a statement earlier. Connecting and Backoff were the two
states that announced first and armed after — Connecting's handshake
deadline sat behind the cable-URL check, Backoff's timer behind the
announcement — and a tier-2 scenario that awaits a state and then asserts
its timers could read {} where the table says {handshake-deadline} or
{backoff}. Both arm first now; a cable-URL refusal stops the deadline on
its way to the terminal, whose set is {}.
…not the process

Dial, WriteFrame and Poll run on worker goroutines so the state machine
can keep selecting while they are in flight, and Go's recovery is
goroutine-local: a host seam panicking on one of them ended the process,
bypassing the recovery that — for the same panic on the consumer's
goroutine — disposes the attempt and lets the consumer's own recover see
it. Each worker now captures the panic and carries it back over its result
channel, and the receiving arm re-raises it on the state machine's
goroutine, where runCycle's deferred recovery runs as for any host-code
panic: the socket is disposed where one exists, the phase timer is stopped
where none does, and the consumer recovers the seam's own value. A poll
abandoned at the grace deadline has no receiver left for its panic — its
attempt is already torn down, and there is no consumer context to
propagate into — which the result type says. One test per seam, each
ending the test binary against the previous revision.
…ts join

The two arms that cancel a pending dial — the handshake deadline and Close
— join the worker and inspect only the connection it may have returned. A
dial that answers cancellation by panicking had that panic captured by the
worker (the previous commit) and then dropped at the join, so the
connector reconnected, or closed, over a host panic the consumer never
saw. Both arms re-raise it once any returned connection is closed, with
the handshake deadline stopped first on the Close arm as on its ordinary
path. One test per arm, each red against the previous revision.
The holder latches an expiry on arm's Stop-false verdict — a frame that
arrived after the window closed — and the shipped SystemClock fires
through time.AfterFunc, whose callback deregisters and then sends, so Stop
reports false a moment before the timer's channel is ready. The
non-blocking probe every arm runs before acting on the socket consulted
the channel alone, so inside that gap a frame was delivered, or a poll
issued, from a socket whose verdict the holder had already decided. The
probe reads the latch first. The test wraps the clock so a staleness
firing is held back from its channel after Stop has reported it, serves a
frame, and asserts nothing is delivered; red against the previous revision.
The shipped feed's 410 `resume` URL re-enters at the epoch with the
canonical filter set preserved — positioned in served history, so the
servable history above the fence is not skipped — not at the present as
the pre-merge branch documented. The recovery matrix accordingly treats an
accepted gap's resume as a position-resume entry: pages save on acceptance
and the buffered live events drain after them, since everything between
the fence and the present is poll-served and nothing behind such an entry
depends on the live buffer. The inbox lane's resume (since=0, the earliest
retained item) is the same class for the same reason.

A 409's two digests reach the host: Observer.FilterConflict fires with
the body's position_digest and filters_digest before
PositionRejected(filter_changed). The re-entry is unchanged — the digests
are diagnostics, and the one that matters is a filters_digest differing
from the SDK's own Filters.Digest() for the same set.

The tier-2 driver follows the foundations: config and identifier params
carry performers, exclude_performers and actor_types; poll rows forward
performed_by_id and details; push events are pinned at the shipped
eleven-key shape. Fixtures spell the resume URL as the epoch re-entry
(since=<epoch_after_id>), tests spell continuations as the position-
carrying URLs the server actually returns, and SPEC §23 names the class
change, the observer callback, the header echoes and since's signed
64-bit range.
…and settle two §23 sentences

The tier-2 driver now proves more of what it claims. A feedGap signal
expectation can pin the resume URL the handler received — whole, as the
record contract says — and fixtures 16, 23, 25 and 27 do; a scripted
409's digests reach the connector on the poll error; every expected
client close must carry status 1000; a terminal element must be exactly
(Event{}, error); an observation record no expect step consumed —
a second Gap, a stray PositionRejected, an unmatched signal — fails the
scenario at `finally` as an unmatched outbound action does; and the
query allowlist admits the three feed dimensions the schema already
permits.

Two §23 sentences are settled against the code rather than left in
disagreement with it. Subscribe is sent on the welcome that opens each
connection, which is the only welcome Action Cable sends; a later frame
of that type is liveness only. And the two URL-bearing observer
callbacks carry poll and resume URLs reduced to their origin — they are
server-chosen text on a logging surface — while the FeedGap signal hands
the handler the resume URL whole, since its disposition is a decision
about which URL to follow.
…blishes quiescence last, and tier 2 pins the observer's reduced gap URL

A consumer that closes the connector from Observer.FilterConflict no
longer receives PositionRejected after Close returned: the re-entry
checks the run context first, as every callback that precedes a further
act does. Events' deferred cleanup unwinds in the order the guarantee
needs — cancel the run context, clear the registered cancel, then close
the done channel — so Wait never returns while the run context is live.

The tier-2 driver records Observer.gap's URL argument, and an expectGap
step can pin it: the resume URL reduced to its origin, or the fixed
cross-origin placeholder — never the whole URL, which only the FeedGap
signal carries. The four 410 fixtures pin both.
…ns the 409 re-entry

The rejection ledger is one ordered record for both Observer.positionRejected
and Observer.filterConflict, so a script that places expectFilterConflict —
the 409 body's two digests — before expectPositionRejected(filter_changed)
pins the payload and the ordering. Fixture 34 serves a 409 with both
digests, watches the conflict callback precede the rejection, and follows
the walk into its present-class re-entry at since=now: the 409 pin the
family README had deferred.
@jeremy jeremy mentioned this pull request Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conformance Conformance test suite go

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants