Skip to content

fix(reflection): batch source deprecation, revert it on failure, cap sources per merge - #445

Open
L4XB wants to merge 1 commit into
EverMind-AI:mainfrom
L4XB:fix/reflection-deprecation-batch-and-cap
Open

fix(reflection): batch source deprecation, revert it on failure, cap sources per merge#445
L4XB wants to merge 1 commit into
EverMind-AI:mainfrom
L4XB:fix/reflection-deprecation-batch-and-cap

Conversation

@L4XB

@L4XB L4XB commented Sep 9, 2026

Copy link
Copy Markdown

Summary

Addresses the first two defects in #443 (Reflection V1 on large clusters):

Why the run failed. _deprecate_lance_episodes / _deprecate_lance_facts issued one LanceDB update per source episode and per fact parent, all started concurrently with asyncio.gather. Every update takes the table's write lock, so on a 228-member cluster the calls queued behind one another until the 15 s write-lock deadline (lancedb_write_lock_deadline_exceeded storm), the run was reported failed, and the updates that had already gone through stayed applied: sources half-deprecated toward a merge that was never committed to the cluster.

Changes

  1. Batched, sequential deprecation. Sources are deprecated in batches of 100 ids with one entry_id IN (...) (episodes) / parent_id IN (...) (facts) update each, issued sequentially. A merge now takes a handful of lock acquisitions instead of hundreds.
  2. Compensation on failure. The LanceDB rows are written first and the markdown frontmatter last, so the durable record only lists deprecations that were actually applied. If any write fails, the deprecated_by values written by this run (only rows pointing at this merge's entry id) are cleared before the error propagates. A failed run now leaves the sources, the markdown record and the cluster as they were; the merged episode itself is still detected by the existing orphan check on the next run, as before.
  3. Per-merge source cap. _MAX_SOURCES_PER_MERGE = 50 (module constant next to _MAX_CLUSTERS_PER_RUN). A larger cluster merges its existing merged episode(s) plus the oldest sources up to the cap; deferred members stay in the cluster, are neither merged nor deprecated in this run, and are folded in by later runs in update mode. The cluster's count after a merge now reflects the members that remain (previously hard-coded to 1). Wiring the cap into [reflection] config can follow if you prefer it configurable.

The third point in the issue (prompt / output-language configuration) is a product decision and is left out.

Area

  • Architecture method

Verification

uv run pytest tests/unit/test_memory          711 passed
make lint                                     ruff, import-linter, datetime discipline, openapi drift: OK

New tests in tests/unit/test_memory/test_reflection/test_orchestrator.py:

  • test_deprecate_lance_episodes_batches_updates: 250 ids → 3 updates with IN predicates covering every id.
  • test_deprecate_failure_reverts_applied_writes: the second batch raises VectorStoreBusyError → the markdown record is never patched, both applied batches are reverted (deprecated_by = NULL only where it equals this merge's entry id), and the error propagates so the run is still reported failed.
  • test_run_caps_sources_per_merge_and_keeps_the_rest: with the cap at 2 and a 3-member cluster, only the two oldest sources are reflected, deprecated and removed; the deferred member plus the merged episode remain (count=2).

Checklist

  • I kept the change scoped to the relevant area.
  • I am opening this from a separate branch, not pushing directly to main.
  • I updated docs, examples, or setup notes when behavior changed.
  • I added or updated tests when the change affects behavior.
  • I did not commit secrets, .env files, dependency folders, or generated output.
  • Active relative links in Markdown files resolve.

Notes for Reviewers

The reorder (LanceDB first, markdown last) is what makes the compensation complete without having to un-patch frontmatter. If you would rather keep markdown first, the revert still works for the LanceDB side, but the frontmatter would then re-apply the deprecation on the next cascade reconcile.

Refs #443

…sources per merge

Reflection deprecated the sources of a merge with one LanceDB update per
episode and per fact parent, all started concurrently. Every update
takes the table write lock, so on a large cluster (228 sources in the
report) the calls queued behind each other until they hit the 15 s
write-lock deadline; the run was then reported failed while the updates
that had already gone through stayed applied, leaving sources
half-deprecated toward a merge that was never committed to the cluster.

- Deprecate in batches of 100 ids with one `entry_id IN (...)` /
  `parent_id IN (...)` update each, issued sequentially, so a merge takes
  a handful of lock acquisitions instead of hundreds.
- Write the LanceDB rows first and the markdown frontmatter last, and on
  any failure clear the `deprecated_by` values this run wrote (only rows
  pointing at this merge's entry) before propagating the error. A failed
  run now leaves the sources, the markdown record and the cluster as
  they were.
- Cap one merge at 50 source episodes. A larger cluster merges its
  existing merged episode plus the oldest sources; the deferred members
  stay in the cluster (its count now reflects them) and are folded in by
  later runs in update mode.

Refs EverMind-AI#443 (the prompt/language configuration is left for a separate change)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant