feat(documents): a sync classifies the prose it reads - #127
Merged
AndrejK666 merged 1 commit intoSep 11, 2026
Merged
Conversation
Classification was a button. A repository could be synced, its files sitting in the graph, and nothing knew what any of them were until somebody opened the Documents tab and pressed Scan — which was never what was asked for: the type of a document should be worked out at ingest. The obstacle was that neither gear can answer alone. `studio-artifact-ingest` walks the repository and ends up holding every file's path and text; it knows nothing about document types. `studio-documents` owns the type catalogue and deliberately does not read the graph, so that classification stays a pure function of (path, content, catalogue). So the seam is a trait this gear declares and that one calls: `documents::port::DocumentClassifier`, one method, published on the ClientHub in the documents gear's `init` and resolved in the ingest gear's REST phase — the same ordering the graph-storage client uses, for the same reason. An absent client is a normal state, not a failure: the documents gear stands down without a database, and the ingest gear then logs that files stay unclassified and syncs them anyway. Checked both ways on a running stack — with the gear configured, "studio-documents wired"; with its `database:` section removed, "studio-documents unavailable — files stay unclassified", and the backend still serving. Classification never fails the sync either. The repository is ingested whatever happens, and the pass is idempotent, so a failure costs a re-run of the cheap half rather than the clone. Only prose is collected, filtered by the catalogue's own `is_prose_path` rather than a second copy of that rule — a repository of five thousand files hands over the hundred and fifty that could be documents, not its whole source tree. And it is collected from a checkout only: a tree-API sync has no text to offer. `file_instance_id` is split out of `file_node_cloned` so the classification pass names the node the walk is about to store rather than recomputing the key beside it. Two tests hold that: the clone's node and the tree's node both agree with it. If they ever stopped, every binding would point at a node that does not exist — silently, because a binding keeps working and simply never matches anything. The portal's Scan repository stays, for repositories synced before this existed and for a checkout the IDE cloned after the fact. Signed-off-by: Andrej Kuchma <Andrej.Kuchma@constructor.tech>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
Classification was a button. A repository could be synced, its files sitting in
the graph, and nothing knew what any of them were until somebody opened the
Documents tab and pressed Scan.
Which was never what was asked for: the type of a document should be worked out
at ingest.
Why it was a button
Neither gear can answer alone.
studio-artifact-ingestwalks the repository and ends up holding every file'spath and text. It knows nothing about document types.
studio-documentsowns the type catalogue and deliberately does not readthe graph, so that classification stays a pure function of
(path, content, catalogue).So the seam is a trait this gear declares and that one calls:
Published on the ClientHub in the documents gear's
init, resolved in theingest gear's REST phase — the same ordering the graph-storage client uses, for
the same reason.
Absent is a normal state
The documents gear stands down without a database. A consumer that finds no
client should simply not classify, not fail a repository sync. Checked both ways
on a running stack:
Classification never fails the sync either. The repository is ingested whatever
happens, and the pass is idempotent, so a failure costs a re-run of the cheap
half rather than the clone.
Only the prose, and only from a checkout
Filtered by the catalogue's own
is_prose_path, re-exported from the portrather than copied — filtering by a second, drifting copy of that rule is how
the two ends start disagreeing about what a document is. A repository of five
thousand files hands over the hundred and fifty that could be documents, not its
whole source tree. A tree-API sync has no text to offer and hands over nothing.
One identity, named once
file_instance_idis split out offile_node_clonedso the classification passnames the node the walk is about to store rather than recomputing the key beside
it. Two tests hold it: the clone's node and the tree's node both agree with it.
If those ever stopped agreeing, every binding would point at a node that does
not exist — silently, because a binding keeps working and simply never
matches anything in the graph.
Still there
The portal's Scan repository stays, for repositories synced before this
existed and for a checkout the IDE cloned after the fact.
491 backend tests (the Postgres suites against a real server), clippy and fmt
clean.