Skip to content

fix: sync only changed rows on a learnings-fingerprint miss (GIT-98) - #38

Merged
2 commits merged into
mainfrom
bugfix/GIT-98-per-row-delta-sync
Sep 22, 2026
Merged

2 commits merged into
mainfrom
bugfix/GIT-98-per-row-delta-sync

Conversation

@nTEG-dev

Copy link
Copy Markdown
Member

What

A learnings-fingerprint miss no longer means a full download (GIT-98, per-row delta).

  • On a miss, when there's a cache from this store, gitmem downloads the manifest: id,updated_at only, no vectors, using the bulk load's filters, order and limit.
  • A row whose id and updated_at match is reused from disk. Changed and new ids are fetched in full, 100 per request. Rows no longer in the manifest are dropped. The result is in manifest order, so it's exactly what a full download returns.
  • Fallbacks, unchanged from before: a failed manifest or delta fetch, an empty manifest, or no cache from this store all take the full download. cache-flush always takes the full download.
  • The bulk load, fingerprint, manifest and delta fetch now share one filter set (INDEXED_LEARNING_FILTERS), so they can't drift apart.

Why

On a Pro store the fingerprint misses on nearly every session_start. refresh_scar_behavioral_scores() rewrites each scar with 3 or more recent usage rows and bumps its updated_at. So each miss re-downloaded the whole index, every row with its 1536-dim embedding, to pick up a handful of changed scores.

How to test

Standing rule: the venue runs the v1.8.0 setup.sql first. The venue vujndhcumgroztfkkdus was provisioned from v1.8.0:schema/setup.sql. Production was not touched. No schema change.

VENUE_ENV=<env outside repo> node tests/e2e/blank-supabase.mjs egress --label git98 --out <dir> --rows 250 --usage 5

250 learnings; --usage 5 gives 5 scars ≥3 usage rows, so each session_start rewrites 5 rows (confirmed on both rounds). Bytes received from the venue:

Start origin/main 1.10.0 This branch
1 cold 921,700 B 921,750 B
2 warm 100 B 100 B
4: first start after a session_start 921,780 B (full re-download) 42,823 B: 245 reused from disk, 5 fetched, 0 dropped
5: in-process reload during the 2nd session_start 924,465 B 45,176 B: 245 reused, 5 fetched

−95% (921,780 → 42,823 B) for the start after a session_start. Of the 42.8 KB, the manifest is 24,248 B (about 97 B a row) and the 5 full rows are 18,475 B. The manifest is now the floor, and it scales with the row count.

New egress gate: the driver now fails if any start after the cold one logs a full index download. main exits 1 (two ~922 KB re-downloads); this branch exits 0. When the delta lands varies from run to run, because the in-process reload timing moves, but no start after the cold one pays full price. flow on this branch exits 0: close persisted, relevance readable, thread counts match.

Unit tests: 1282/1282 pass (+8). The pure planning covers reuse, fetch, drop, rows with no updated_at, and ids that vanish before the fetch. Through the real loadScarsFromSupabase: 5 changed rows fetch exactly those 5 ids and the result equals a full download, order included; inserts and deactivations; manifest failure falls back to the full download; cache-flush bypasses the delta; a cold cache does no manifest.

Existing tests

None changed. vector-disk-cache.test.ts "misses when the row count / newest updated_at changed" still holds: readVectorCache still misses, and the delta is a separate path layered on top.

Assumption (same as the fingerprint's)

updated_at moves on every edit (setup.sql's trigger). An in-place edit that leaves updated_at unchanged would be reused as-is. The fingerprint already couldn't see such an edit; cache-flush is the manual way out.

Checklist

  • npm run test:unit passes (1282)
  • npm run test:e2e: not run; the venue driver is the e2e check here
  • New tests added
  • No schema change; works on a v1.8.0 store as-is
  • Docs: none describe the cache

Breaking changes

None.

🤖 Generated with Claude Code

Claude and others added 2 commits September 21, 2026 20:11
…isses (GIT-98)

On a Pro store the fingerprint misses on nearly every session_start:
refresh_scar_behavioral_scores() rewrites each scar with >=3 recent usage rows
and bumps its updated_at. Each miss re-downloaded the whole index, every row
with its 1536-dim embedding.

- A miss with a cache from this store downloads the manifest (id, updated_at;
  no vectors) with the bulk load's filters, order and limit.
- Rows whose id and updated_at match are reused from disk; changed and new
  ids are fetched in full, 100 ids per request; rows gone from the manifest are
  dropped. The result is in manifest order, what a full download returns.
- Any failure, an empty manifest, or no cache from this store: the full
  download, as before. cache-flush still always takes the full download.
- The bulk load, fingerprint, manifest and delta fetch share one filter set.

Venue (v1.8.0 setup.sql, 250 learnings, --usage 5): the start after a
session_start drops from 921,780 B to 42,823 B — 245 reused, 5 fetched.

+8 unit tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rt (GIT-98)

Exits 1 on 1.10.0 (the start after each session_start re-downloads ~922 KB);
exits 0 with per-row delta sync.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@nTEG-dev nTEG-dev closed this pull request by merging all changes into main in f93c3a3 Sep 22, 2026
@nTEG-dev
nTEG-dev deleted the bugfix/GIT-98-per-row-delta-sync branch September 23, 2026 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant