Skip to content

Add incremental Workspace storage and configurable history retention - #33

Open
haolpku wants to merge 4 commits into
mainfrom
codex/incremental-workspace
Open

haolpku wants to merge 4 commits into
mainfrom
codex/incremental-workspace

Conversation

@haolpku

@haolpku haolpku commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Workspace publication previously copied the complete raw/KB/Graph/DB state for each revision. Add opt-in shared incremental storage with fixed native version bindings, plus configurable history retention without losing pinned reads or coordinated rollback.

  • Share hashed source objects, vectors and parsed artifacts; reuse unchanged LanceDB rows, source-scoped indexed SQLite graph assertions, and Dolt table trees. Publication still uses one manifest and revision/epoch CAS.
  • Add latest/recent/all retention, protected versions, durable reader leases, operation close, preview/apply GC and native compaction. New workspaces default to latest; existing histories remain all. Interrupted builds and private Dolt child processes recover without exposing partial output.
  • Migrate the current legacy snapshot explicitly from frozen evidence/stored embeddings with no model calls; preserve old revision IDs until retention permits cleanup. Add CLI, HTTP/UI and MCP controls; update both packaged Skills and the Codex plugin.

Validation: 500 local tests passed, 40 skipped; strict optional-versioning suite 79 passed with no skips; real Dolt/LanceDB integration, all-surface migration, process-death/ABA/cancellation cases, cross-connector startup retry and concurrent legacy schema upgrades; installed-wheel smoke and SQLite demo passed. Browser verified saving/reloading retention and no JavaScript errors. All seven GitHub CI jobs passed on f9cc10f, including real MySQL/Dolt integration without skips: https://github.com/OpenDCAI/DataMind/actions/runs/39586609483. CI includes the new incremental and real-provider tests in required jobs.

Limits: single-host coordinator; incremental KB/DB require local LanceDB/Dolt and explicit setup/migration. Changed tables are replaced as a unit (no primary-key row upserts yet); source inventory/reference metadata still scale with total workspace size. Private Dolt SQL sessions currently serialize and start a local server per operation. See docs/WORKSPACE_STORAGE.md for setup, retention, migration and physical reclamation behavior.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant