Skip to content

DOCSORT.md

Latest commit

 

History

History
180 lines (157 loc) · 15.8 KB

DOCSORT.md

File metadata and controls

180 lines (157 loc) · 15.8 KB
Raw

╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ Docsort Pipeline Integration Plan

Context

The REFACTOR.md describes "docsort" -- a fully offline, privacy-first, German+English document sorting daemon. The existing OcrTextSnap (SnapText) codebase is a well-architected Rust app with OCR, SQLite persistence, state machine, event bus, and configuration hot-reload. This plan integrates docsort into the existing codebase, reusing patterns and dependencies wherever possible.

Goal: Add a src/docsort/ module and src/bin/docsort.rs binary that takes scanner-dropped documents (PDF, TIFF, images) from an inbox folder through OCR processing, LLM metadata extraction, and deterministic file naming -- all crash-safe and offline.

Phased approach: Phase 1 (this implementation) covers the full pipeline from inbox watcher through OCR, LLM extraction, filename generation, and file output. Phase 2 (future) adds embeddings/FAISS vector similarity.

Architecture Decision: Separate Binary

The existing main.rs is a synchronous macOS GUI event loop (tray icon, hotkeys, 60fps polling). Docsort is an async tokio daemon. These are fundamentally different runtimes, so docsort gets its own [[bin]] entry in Cargo.toml. Both binaries share the library crate (src/lib.rs) and all existing modules.

Files to Modify (minimal changes) ┌────────────────────────┬───────────────────────────────────────────────────────────────────────────────────┐ │ File │ Change │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤ │ Cargo.toml │ Add [[bin]] for docsort, add 6 new deps (clap, sha2, hex, reqwest, uuid, walkdir) │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤ │ src/lib.rs │ Add pub mod docsort; and 3 new SnapTextError variants │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤ │ src/domain/mod.rs │ Add pub mod document; and re-exports │ ├────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤ │ src/persistence/mod.rs │ Add pub mod docsort_migrations; and pub mod docsort_repository; │ └────────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘ Files to Create

  1. Domain Layer
  • src/domain/document.rs -- DocumentStage enum, ErrorCode enum, Document struct, DocumentMetadata (LLM-extracted fields)
    • Pattern: follows src/domain/snapshot.rs (value objects, no dependencies)
  1. Docsort Module (src/docsort/)
  • mod.rs -- Module root with re-exports
  • config.rs -- DocsortConfig struct (base_dir, ocr_languages, ollama_url, llm_model, concurrency settings)
    • Pattern: follows src/config/config.rs (serde + TOML + defaults + load/save)
  • state.rs -- Pure transition(DocumentStage, PipelineEvent) -> Result function
    • Pattern: follows src/app/state_machine.rs exactly (exhaustive match, tests)
  • watcher.rs -- InboxWatcher using notify crate with stability checks (size+mtime unchanged x3)
    • Reuses: notify dependency, pattern from src/config/watcher.rs
  • ocr.rs -- OcrmypdfEngine calling ocrmypdf -l deu+eng --sidecar via tokio::process::Command
  • llm.rs -- OllamaClient for metadata extraction via HTTP /api/generate (JSON mode)
    • Uses: reqwest (new dep), serde_json (existing dep)
  • normalize.rs -- Detects file types, converts TIFF/images to PDF
    • Reuses: image crate (existing dep)
  • filename.rs -- Deterministic filename builder from metadata, sanitization, collision handling (__v2, __v3)
  • quality.rs -- QualityReport struct + assess_quality() + clean_text() (whitespace collapse, page separators)
  • recovery.rs -- Queries interrupted documents on startup, re-enqueues or fails with RECOVERY_MISSING_ARTIFACT
  • pipeline.rs -- Main orchestrator: watcher -> normalize -> OCR -> clean -> LLM extract -> filename -> move
    • Bounded concurrency via tokio::sync::Semaphore
    • SHA256 dedup before processing
    • Atomic file moves (rename within same filesystem)
    • Privacy: never logs OCR text or LLM content
  • cli.rs -- Clap subcommands: daemon, status, retry, reocr, reextract, gc
  1. Persistence Layer
  • src/persistence/docsort_migrations.rs -- Migration 002: WAL mode, documents table (all columns from REFACTOR.md), embeddings table, settings table, indices
    • Pattern: follows src/persistence/migrations.rs exactly
  • src/persistence/docsort_repository.rs -- DocumentRepository trait + SqliteDocumentRepository impl
    • Pattern: follows src/persistence/sqlite_repository.rs (r2d2 pooling, params!, error mapping)
    • Methods: insert, update_stage, get_by_id, get_by_sha256, list_by_stage, count_by_stage, update_ocr_fields, update_llm_fields, mark_failed
  1. Binary Entry Point
  • src/bin/docsort.rs -- #[tokio::main], clap CLI, structured JSON logging, startup checks (ocrmypdf, ollama, folders), runs pipeline or CLI command

Existing Code Reused ┌───────────────────────────────────┬────────────────────────────────────┐ │ Component │ Reuse │ ├───────────────────────────────────┼────────────────────────────────────┤ │ rusqlite + r2d2 + r2d2_sqlite │ Same connection pooling pattern │ ├───────────────────────────────────┼────────────────────────────────────┤ │ persistence/migrations.rs pattern │ Versioned schema migrations │ ├───────────────────────────────────┼────────────────────────────────────┤ │ notify crate │ Inbox folder watching │ ├───────────────────────────────────┼────────────────────────────────────┤ │ config/config.rs pattern │ TOML config with serde defaults │ ├───────────────────────────────────┼────────────────────────────────────┤ │ app/state_machine.rs pattern │ Pure-function state transitions │ ├───────────────────────────────────┼────────────────────────────────────┤ │ SnapTextError enum │ Extended with new variants │ ├───────────────────────────────────┼────────────────────────────────────┤ │ tracing │ Structured logging │ ├───────────────────────────────────┼────────────────────────────────────┤ │ tokio │ Async runtime │ ├───────────────────────────────────┼────────────────────────────────────┤ │ serde + serde_json │ LLM JSON parsing │ ├───────────────────────────────────┼────────────────────────────────────┤ │ chrono │ Timestamps │ ├───────────────────────────────────┼────────────────────────────────────┤ │ dirs │ Config/data paths │ ├───────────────────────────────────┼────────────────────────────────────┤ │ image │ File type detection, TIFF handling │ └───────────────────────────────────┴────────────────────────────────────┘ New Dependencies ┌─────────┬─────────────────────┬───────────────────────────────────────┐ │ Dep │ Version │ Purpose │ ├─────────┼─────────────────────┼───────────────────────────────────────┤ │ clap │ 4 (derive) │ CLI subcommands │ ├─────────┼─────────────────────┼───────────────────────────────────────┤ │ sha2 │ 0.10 │ SHA256 for duplicate detection │ ├─────────┼─────────────────────┼───────────────────────────────────────┤ │ hex │ 0.4 │ Hex encoding of hashes │ ├─────────┼─────────────────────┼───────────────────────────────────────┤ │ reqwest │ 0.12 (json, rustls) │ Ollama HTTP client │ ├─────────┼─────────────────────┼───────────────────────────────────────┤ │ uuid │ 1 (v4) │ Document ID generation for file moves │ ├─────────┼─────────────────────┼───────────────────────────────────────┤ │ walkdir │ 2 │ Directory traversal for GC │ └─────────┴─────────────────────┴───────────────────────────────────────┘ Pipeline Flow (Phase 1)

inbox/ file detected -> stability check (size+mtime unchanged x3, 2s intervals) -> atomic rename to processing/{uuid}.{ext} -> SHA256 -> check duplicate -> if dup, move to output/_duplicates/ -> INSERT documents (stage=queued) -> normalize to PDF (if TIFF/image, convert; else copy) -> stage=staged -> ocrmypdf -l deu+eng --sidecar -> ocr.pdf + ocr.txt -> stage=ocr_done -> clean_text() -> quality gate (alphabetic ratio, length) -> Ollama /api/generate (JSON mode) -> extract metadata -> stage=extract_done -> build deterministic filename -> sanitize + collision resolve -> atomic move ocr.pdf to output/{final_filename} -> stage=processed -> on failure: bundle artifacts to failed/{id}/, stage=failed

Folder Structure (created on startup)

docsort/ inbox/ # scanner drops files here processing/ # atomic move target (same fs) tmp/ # per-doc artifacts: {doc_id}/... output/ # final OCR PDFs with deterministic names _duplicates/ # by sha256 failed/ # failed/{doc_id}/ with diagnostic bundle db/ docsort.sqlite # WAL mode logs/ # structured JSON tracing

Implementation Order

  1. Cargo.toml -- add [[bin]] + deps
  2. src/domain/document.rs + src/domain/mod.rs -- domain types
  3. src/lib.rs -- add pub mod docsort;, error variants
  4. src/docsort/mod.rs -- module skeleton
  5. src/docsort/config.rs -- configuration
  6. src/docsort/state.rs -- state machine with tests
  7. src/persistence/docsort_migrations.rs -- DB schema
  8. src/persistence/docsort_repository.rs + src/persistence/mod.rs -- repository
  9. src/docsort/quality.rs -- text quality assessment
  10. src/docsort/normalize.rs -- file type normalization
  11. src/docsort/ocr.rs -- ocrmypdf adapter
  12. src/docsort/llm.rs -- Ollama LLM client
  13. src/docsort/filename.rs -- deterministic naming
  14. src/docsort/watcher.rs -- inbox watcher
  15. src/docsort/recovery.rs -- crash recovery
  16. src/docsort/pipeline.rs -- pipeline orchestrator
  17. src/docsort/cli.rs -- CLI subcommands
  18. src/bin/docsort.rs -- entry point

Verification

  1. cargo build -- both binaries compile
  2. cargo test -- existing tests still pass + new state machine and migration tests pass
  3. Create test inbox with a sample PDF: docsort daemon should detect, OCR, and move to output/
  4. docsort status should show pipeline counts
  5. Kill and restart daemon mid-processing -- recovery should resume