A frozen opt-in cursor lets stale stars reverse a completed opt-out

[neo-opus-grace]

Context

DevIndex's opt-in cursor (optin-sync.json) has not advanced since 2026-03-29T09:58:10Z — commit fee7327841, 143 days before filing. optout-sync.json has never advanced at all; it still reads {"lastCheck": null}.

That is not a stuck file. It is the visible symptom of the working set's read side and write side naming different artifacts, and the consequence is that a completed opt-out can be silently reversed and the person re-published.

Found while verifying the DevIndex extraction (#17375). Observation and inference are separated below: the mechanism is verified from code and live probes; the population of affected people is explicitly not measured, because enumerating it means handling exactly the personal data this ticket is about.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-19T17:56Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages: no overlapping [lane-claim].

The Problem

The DevIndex pipeline hydrates its working set from config.publishedWorkingSet.baseUrl, which resolves to neomjs.com/node_modules/neo.mjs/apps/devindex/resources/data/ — the copy served from neomjs/pages. Since neomjs/neo#17375, the pipeline publishes to a content-plane prefix named by the DEVINDEX_PUBLISH_BUCKET repository variable. Those are different artifacts. Each run adopts the pages copy, publishes its own working set where nothing serves it, and the next run starts from the pages copy again.

Neo's own publish step never restored the cursors either: .github/workflows/data-sync-pipeline.yml:283-287 stages users.jsonl and the portal data, and nothing else. So the cursors have been frozen in the committed tree since March and reset on every run since.

Why a frozen cursor is not merely wasteful. OptIn.collectStargazerOptIns() pages the opt-in repository's stargazers newest-first and stops at lastCheck:

if (lastCheck && starredAt <= lastCheck) { stopFetching = true; break }

With lastCheck pinned to March, every star placed since then is re-collected on every run — and stargazers are one of the two signals that reverse a blocklist entry (OptIn.mjs, "Remove from blocklist if they are there (ONLY stargazers and self-issues)"). So an opt-in star that predates a later opt-out is replayed indefinitely, removing that person from the blocklist again on every run.

And the un-block reaches the published index. Stage order is opt-in → opt-out → spider → updater. Manager.mjs runs Cleanup.run() as pre-run hygiene for spider and update only — optin and optout run none. By the time Cleanup performs blocklist enforcement it is reading the blocklist.json that OptIn has already rewritten, so the person is no longer blocklisted, is not pruned, and is enriched into users.jsonl and published.

The opt-out is not repaired on a later run either: OptOut re-adds only people it can still see. Somebody who opted out via the issue template had that issue closed after processing, and somebody who opted out by starring is invited by our own guides to remove the star afterwards. Neither is visible to the next run.

The Architectural Reality

surface	file	what it does
read side	apps/devindex/services/config.mjs (publishedWorkingSet.baseUrl)	resolves to neo's pages copy
write side	buildScripts/publishWorkingSet.mjs	writes to the prefix in DEVINDEX_PUBLISH_BUCKET
cursor stop	apps/devindex/services/OptIn.mjs (collectStargazerOptIns)	stops paging at lastCheck
blocklist reversal	apps/devindex/services/OptIn.mjs	removeFromBlocklist for stargazers + self-issues
enforcement	apps/devindex/services/Cleanup.mjs	hard-deletes blocklisted users — but runs before OptIn's write, not after
stage order	.github/workflows/data-sync-pipeline.yml (devindex)	opt-in, opt-out, spider, updater

Paths are in neomjs/devindex; the workflow reference under "The Problem" is neomjs/neo.

Measured, not inferred:

• the working-set manifest returns HTTP 404 at the read URL — pages carries no manifest, so digest verification is inert
• neomjs.com/dist/devindex/… returns HTTP 404; that host proxies to pages and the content-plane prefix is not routed
• the bucket's own public endpoint returns HTTP 403 — private, mounted into Cloud Run
• devindex run 32280343186 logged No published manifest — adopting the fetched set unverified in all three hydrating stages, then published successfully. Both halves are true at once, which is the whole defect.

The Fix

Point the read side at the artifact the pipeline publishes, so the working set round-trips through one place. config.publishedWorkingSet.baseUrl becomes the served URL of the published prefix; nothing else in Storage.hydrateWorkingSet() changes, and manifest verification becomes live the moment it can fetch a manifest.

Reaching that URL is a hosting decision, not a code choice, and it is the reason this is filed rather than fixed:

1. Route the content plane — serve the published prefix through the existing middleware, which already mounts the bucket. Needs a middleware-v2 deploy. Recommended: it keeps the artifact private-by-default and same-origin.

2. Public prefix — expose the prefix directly and point baseUrl at it. No deploy, but it adds a second public surface for the same data and moves egress off the current path.

3. Publish the working set to neomjs/devindex's own GitHub Pages, via actions/deploy-pages. The artifact path deploys straight from the workflow — no commit, no branch, so none of the history cost DevIndex collection leaves neo, and stops committing what it generates neo#17375 exists to remove. Measured against the published limits: a site may be no larger than 1 GB, bandwidth is a soft 100 GB/month, and builds a soft 10 per hour. The working set is ~26.45 MiB and the schedule is 12 runs/day, so all three have wide margin. Enabling Pages on that repository is a settings change, not a deploy.

   The honest cost, and it is why this is listed third rather than first: the dated archive/ copy must stay in object storage. A Pages deployment replaces the whole site each run, so carrying a 30-day archive would mean re-uploading it every two hours and would reach the 1 GB cap inside two months. That splits one lifecycle across two systems — working set on Pages, archive in the bucket — and a set that is "published" in two places is exactly the shape the manifest exists to make unambiguous. Option 1 keeps one destination.

   No new exposure either way: this data is already served publicly from neomjs.com. What differs between the options is the number of surfaces to reason about, not whether the data is public.

All three need the operator — a middleware-v2 deploy, a bucket ACL change, or a repository settings change. None is a code choice, and the code change is the same one line in every case.

Either unblocks the cursor: once the pipeline reads what it wrote, lastCheck advances and the replay stops.

Contract Ledger Matrix

Target Surface	Source of Authority	Proposed Behavior	Fallback	Docs	Evidence
config.publishedWorkingSet.baseUrl	apps/devindex/services/config.mjs (devindex)	resolves to the published working set	unchanged — rejectWorkingSet keeps the local copy on any fetch/digest failure	learn/data-factory/Storage.md	two consecutive runs where the second reports a lastCheck newer than the first
working-set-manifest.json	Storage.recordWorkingSetManifest()	fetched and verified rather than absent	absent manifest still adopts unverified, but the warning now names the real condition	learn/data-factory/Storage.md	a run log with no "adopting the fetched set unverified" line

Decision Record impact

none — verified rather than assumed. ADR 0004 governs the on-disk shape of resources/content/ (its "bucket" is an archive chunk bucket, not object storage); ADR 0014 classifies Agent OS scheduler lanes and the Brain's deployment topology. Neither governs the DevIndex published working set or the website content plane. No ADR mentions the content plane except ADR 0004 in the unrelated sense above.

Acceptance Criteria

• config.publishedWorkingSet.baseUrl resolves to the artifact publishWorkingSet.mjs writes — demonstrated by fetching the manifest from that URL and getting a document rather than a 404
• A pipeline run logs no adopting the fetched set unverified line, i.e. the manifest was fetched and every digest matched
• optin-sync.json advances across two consecutive scheduled runs: the second run's published lastCheck is strictly newer than the first's
• A spec covers the replay directly: given a blocklisted login whose opt-in star predates the cursor, OptIn does not remove it from the blocklist. The spec must fail when the cursor is pinned to a stale value — a test that cannot fail on the frozen-cursor case does not cover this
• learn/OptOut.md's statement that removal is permanent is true under the shipped behaviour, or the page is corrected
• Post-merge only: confirm on a real scheduled run, not a dispatch — the freeze is only observable across runs

Out of Scope

• The middleware-v2 deploy itself, and any change to what other routes serve
• Hosting for the DevIndex app (same decision, different artifact — it needs its own resolution)
• Reclaiming generated-artifact history — that is Reclaim the generated-artifact history in neo and pages neo#17376
• Re-designing opt-in/opt-out as a mechanism. The issue flow and the star flow are both fine; the cursor is what is broken
• Quantifying who was affected. It requires joining the blocklist against a stargazer list, which is the personal data this ticket exists to protect

Avoided Traps

• Adding a timestamp comparison so an old star cannot reverse a newer opt-out. Treats the symptom, and blocklist.json is a flat array of logins with no timestamps, so it would need a schema change to fix something that disappears when the cursor advances.
• Committing the cursors back to git. That is the publication state machine DevIndex collection leaves neo, and stops committing what it generates neo#17375 exists to delete, reintroduced for two small files — and it would restore the per-run commit that made neo's .git 4.1 GB.
• Failing closed when the manifest is absent. Every run would fail until the loop closes, which converts a silent defect into an outage without fixing it. The warning was corrected instead (devindex Storage.mjs).
• Treating the frozen cursor as a neo bug. It reads that way — the stale value lives in neo's tree — but neo never persisted it either. The freeze belongs to the publication path, not to a repository.

Related

• DevIndex collection leaves neo, and stops committing what it generates neo#17375 — DevIndex collection leaves neo (the extraction that surfaced this)
• Reclaim the generated-artifact history in neo and pages neo#17376 — Reclaim the generated-artifact history
• Register neomjs/devindex as a pull-mode tenant on the local Agent OS neo#17377 — Register neomjs/devindex as a pull-mode tenant
• feat(ci): DevIndex collection leaves the neo pipeline (#17375) neo#17391 — merge-order dependency: dev still refreshes the pages copy the pipeline reads. Merging it before this ticket closes freezes the index rather than merely leaving it stale.

Handoff Retrieval Hints

• query_raw_memories: "devindex published working set read side write side different artifacts"
• query_raw_memories: "opt-in cursor frozen stale star reverses opt-out"
• devindex commit range: 62fb114..HEAD (the extraction's read-side work, including the corrected unverified-adoption warning)
• devindex run 32280343186 — a full collection that succeeded and logged unverified adoption in the same run

Origin Session ID: 44746e37-a5f9-44c4-8c9d-f664247f0e38

[neo-opus-grace]

Risk note from neomjs/neo#17421's D1 investigation: this path has no test coverage in the repo that owns it

While checking test parity for the apps/devindex removal I measured the coverage of the hydration path this ticket is about.

Storage#fetchAndAdoptWorkingSet() (Storage.mjs:394 in neomjs/devindex) is the code that adopts the published working set — the mechanism by which a completed opt-out gets reversed. Counting publishedWorkingSet / baseUrl / fetch( across all nine devindex unit specs: zero. StorageWriteDirectory covers the write side only.

neomjs/neo still holds StoragePublishedIndex.spec.mjs (8 tests), but it pins the superseded single-file publishedIndex contract, not the three-member publishedWorkingSet one that actually runs. So the tests that exist do not cover the code that runs, and the code that runs has none.

Two things follow:

• whoever takes this should expect to write the harness as part of the fix, not inherit one;
• indexProvenance — the digest-provenance file neo's spec asserts against — does not exist in devindex. If provenance verification was dropped rather than relocated during the move, that is a second defect on this same path and is arguably the more serious one, since it is what makes "adopt only what is provably ours" enforceable.

I have not verified which of those it is. Naming it rather than assuming either.

Origin Session ID: 3e4f33e0-fb23-4a61-a2a0-7f396950f3d6

[neo-opus-grace]

Resolved: digest verification was relocated, not dropped

I flagged on this ticket that indexProvenance's absence from neomjs/devindex might mean provenance verification was lost during the move, and said explicitly that I had not verified which. Now checked.

It was relocated. config.publishedWorkingSet carries a fetched manifest, and Storage#fetchAndAdoptWorkingSet compares a digest per member against it. working-set-provenance.json currently ships with digests: null deliberately — while neomjs/neo is still the publisher, that repository cannot hold a digest for bytes it did not write, so a null record takes the documented absence branch and adopts unverified rather than rejecting every hydration.

So: not a second defect, and not silent — the absence branch emits a loud UNVERIFIED warning naming exactly this ticket's mechanism:

the read side and the write side name different artifacts, so nothing this run produces is read by the next one, and the sync cursors reset every time

That is this ticket, stated by the code itself.

And it now has coverage. StorageWorkingSetHydration.spec.mjs (neomjs/devindex PR neomjs/neo#4) pins the absence branch, the digest-mismatch rejection, and — most relevant here — that a failure anywhere in the set adopts nothing, mutation-verified. blocklist and optoutSync are both named individually as working-set members, because that is the file whose loss turns an honoured opt-out into a re-index.

The fix for this ticket is still what its body says: point baseUrl at the set this pipeline publishes. What changes is that the surrounding contract is no longer untested, so whoever takes it inherits a harness instead of writing one.

Origin Session ID: 3e4f33e0-fb23-4a61-a2a0-7f396950f3d6