Skip to content

feat(documents): a sync classifies the prose it reads - #127

Merged
AndrejK666 merged 1 commit into
constructorfabric:mainfrom
AndrejK666:AndrejK666/classify-on-ingest
Sep 11, 2026
Merged

feat(documents): a sync classifies the prose it reads#127
AndrejK666 merged 1 commit into
constructorfabric:mainfrom
AndrejK666:AndrejK666/classify-on-ingest

Conversation

@AndrejK666

Copy link
Copy Markdown
Contributor

The gap

Classification was a button. A repository could be synced, its files sitting in
the graph, and nothing knew what any of them were until somebody opened the
Documents tab and pressed Scan.

Which was never what was asked for: the type of a document should be worked out
at ingest.

Why it was a button

Neither gear can answer alone.

  • studio-artifact-ingest walks the repository and ends up holding every file's
    path and text. It knows nothing about document types.
  • studio-documents owns the type catalogue and deliberately does not read
    the graph
    , so that classification stays a pure function of
    (path, content, catalogue).

So the seam is a trait this gear declares and that one calls:

// documents::port
#[async_trait]
pub trait DocumentClassifier: Send + Sync + 'static {
    async fn classify_ingested(
        &self, ctx: &SecurityContext,
        workspace_id: Uuid, project_id: Option<Uuid>,
        files: Vec<IngestedDocument>,
    ) -> anyhow::Result<ClassifiedCounts>;
}

Published on the ClientHub in the documents gear's init, resolved in the
ingest gear's REST phase — the same ordering the graph-storage client uses, for
the same reason.

Absent is a normal state

The documents gear stands down without a database. A consumer that finds no
client should simply not classify, not fail a repository sync. Checked both ways
on a running stack:

with the gear configured
  INFO  studio-artifact-ingest: studio-documents wired — a sync classifies the prose it reads

with its `database:` section removed
  WARN  studio-documents: no database configured — gear inert
  WARN  studio-artifact-ingest: studio-documents unavailable — files stay unclassified
  … and /account-management/v1/me still answers 200

Classification never fails the sync either. The repository is ingested whatever
happens, and the pass is idempotent, so a failure costs a re-run of the cheap
half rather than the clone.

Only the prose, and only from a checkout

Filtered by the catalogue's own is_prose_path, re-exported from the port
rather than copied — filtering by a second, drifting copy of that rule is how
the two ends start disagreeing about what a document is. A repository of five
thousand files hands over the hundred and fifty that could be documents, not its
whole source tree. A tree-API sync has no text to offer and hands over nothing.

One identity, named once

file_instance_id is split out of file_node_cloned so the classification pass
names the node the walk is about to store rather than recomputing the key beside
it. Two tests hold it: the clone's node and the tree's node both agree with it.

If those ever stopped agreeing, every binding would point at a node that does
not exist — silently, because a binding keeps working and simply never
matches anything in the graph.

Still there

The portal's Scan repository stays, for repositories synced before this
existed and for a checkout the IDE cloned after the fact.

491 backend tests (the Postgres suites against a real server), clippy and fmt
clean.

Classification was a button. A repository could be synced, its files sitting in
the graph, and nothing knew what any of them were until somebody opened the
Documents tab and pressed Scan — which was never what was asked for: the type of
a document should be worked out at ingest.

The obstacle was that neither gear can answer alone. `studio-artifact-ingest`
walks the repository and ends up holding every file's path and text; it knows
nothing about document types. `studio-documents` owns the type catalogue and
deliberately does not read the graph, so that classification stays a pure
function of (path, content, catalogue).

So the seam is a trait this gear declares and that one calls:
`documents::port::DocumentClassifier`, one method, published on the ClientHub in
the documents gear's `init` and resolved in the ingest gear's REST phase — the
same ordering the graph-storage client uses, for the same reason.

An absent client is a normal state, not a failure: the documents gear stands
down without a database, and the ingest gear then logs that files stay
unclassified and syncs them anyway. Checked both ways on a running stack — with
the gear configured, "studio-documents wired"; with its `database:` section
removed, "studio-documents unavailable — files stay unclassified", and the
backend still serving.

Classification never fails the sync either. The repository is ingested whatever
happens, and the pass is idempotent, so a failure costs a re-run of the cheap
half rather than the clone.

Only prose is collected, filtered by the catalogue's own `is_prose_path` rather
than a second copy of that rule — a repository of five thousand files hands over
the hundred and fifty that could be documents, not its whole source tree. And it
is collected from a checkout only: a tree-API sync has no text to offer.

`file_instance_id` is split out of `file_node_cloned` so the classification pass
names the node the walk is about to store rather than recomputing the key beside
it. Two tests hold that: the clone's node and the tree's node both agree with it.
If they ever stopped, every binding would point at a node that does not exist —
silently, because a binding keeps working and simply never matches anything.

The portal's Scan repository stays, for repositories synced before this existed
and for a checkout the IDE cloned after the fact.

Signed-off-by: Andrej Kuchma <Andrej.Kuchma@constructor.tech>
@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: beb9bc9b-f3f4-4403-a1c2-1d0e2dee92af


Comment @coderabbitai help to get the list of available commands.

@AndrejK666
AndrejK666 merged commit ffbbd97 into constructorfabric:main Sep 11, 2026
3 checks passed
@AndrejK666
AndrejK666 deleted the AndrejK666/classify-on-ingest branch September 11, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant