Parsed result contract #45

Activity

1. suguanYang

   added

   enhancementNew feature or requestNew feature or request

   on May 12, 2026
2. suguanYang

   changed the title [-]Parsed result ZIP contract: sync mechanism for API → SDKs → dependent services[/-] [+]Parsed result contract[/+] on May 12, 2026
3. nuemaan commented on Jun 12, 2026

   

   nuemaan

   on Jun 12, 2026

   More actions

   Hi @EricNGOntos, I'd like to take this on if it's still open.

   Reading the current state: packages/shared-python/shared/services/storage/zip_package_writer.py writes chunks.json, doc_nav.json, and manifest.json directly from dict payloads, and apps/worker/app/services/document_ingestion/success_finalization.py:finalize_parse_success is the natural pre-completion hook but doesn't validate output shape today. The existing contract test (apps/worker/tests/contract/test_parse_task_contract.py) covers end-to-end behaviour but only asserts presence of fields like result_s3_key and chunk counts, not the inner shape of the three JSON artifacts.

   Concrete shape I'd build:

   1. Pydantic V2 models (in packages/shared-python/shared/contracts/parse_result/) for Manifest, ChunkRecord, and DocNav capturing the current shape produced by the worker. Each model carries a schema_version field starting at 1 so future shape changes are versioned, not silent.
   2. A single validate_parse_result(manifest, chunks, doc_nav) helper called from finalize_parse_success before the job is marked done. Validation failure fails the job with a structured error that names the violated field and the schema version, so SDK maintainers immediately see a contract break instead of a malformed ZIP reaching consumers.
   3. SDK-discoverable schema: dump the Pydantic models to JSON Schema under packages/shared-python/shared/contracts/parse_result/schemas/ so the Node + Python SDKs can vendor or reference them, and add a make sync-contracts target (or equivalent) that regenerates the schema files from the models so they cannot drift.
   4. Unit tests for each model covering the happy path plus one violation per required field. Extend the existing parse-task contract test with one assertion that the validator is called and rejects a deliberately-malformed payload.

   Scope questions before I start:

   1. Schema location: would you prefer packages/shared-python/shared/contracts/ as proposed, or a separate top-level packages/contracts/ so the Node SDK can pin to it without depending on the Python shared package?
   2. Contract violation on the worker side: fail the job hard (status=failed), or finalize with the partial result and surface the violation through a non-fatal warning channel? My read of the issue ("detect the break before it reaches consumers") is hard-fail, but I want to confirm before locking in.
   3. Versioning policy: when a field is added in a backwards-compatible way, do we bump schema_version minor, or only on breaking changes? A short note in the repo will make this explicit for future contributors.

   If you'd rather scope this differently or split it across PRs, I can adjust. Otherwise, please confirm and I'll open a draft.
4. Frankie-Xu commented on Sep 14, 2026

   

   Frankie-Xu

   on Sep 14, 2026

   More actions

   @suguanYang @EricNGOntos @nuemaan

   Draft RFC for this issue (no implementation): #420

   It answers the three scope questions from 2026-06-12 against current main (9489fa2c):

   1. Schema location: Pydantic models in packages/shared-python/shared/contracts/parse_result/ as source of truth; generated JSON Schema in language-neutral packages/contracts/parse_result/ for Node/Python SDK vendoring; make sync-contracts + CI drift check.
   2. Violation: hard-fail the job (status=failed), do not upload the ZIP. Validate in ZipResultService.generate_zip_package after payloads exist and before ZipPackageWriter.write.
   3. Versioning: integer schema_version starting at 1, independent of existing manifest.version / doc_nav.version strings. Additive optional fields do not bump; breaking changes bump. One ZIP, one version.

   Also asks whether doc_nav.json is required on every successful job (today it can be omitted after a warning).

   Please confirm or redirect on those points before any models/validator PR.