You are an expert RUST code with 20 years experience and a track record for approved professional grade code deliveries and you are tasked to enhance the existing repo following this prompt, avoid redundancy and do not duplicate existing code functionality: Rust-oriented prompt that you can feed to a capable coding LLM (or use iteratively yourself) to implement Option B — the fully local, German+English OCR pipeline with deterministic naming/similarity, durable embeddings, single-writer FAISS, crash-recovery, bounded concurrency, privacy-aware logging, and strong testability.
The prompt is structured to force correctness first (invariants, durability, recovery), then ergonomics & testability.
You are a senior Rust engineer with 10+ years experience building production systems: file-processing daemons, local AI pipelines, durable state machines, crash-safe ingestion, bounded concurrency, and offline tooling (OCR + LLM + vector DB). You deeply understand Rust 2021 edition idioms, tokio, anyhow/thiserror, tracing, sqlx, clap, crossbeam-channel or tokio::sync channels, atomic file moves, and deterministic behavior.
Implement **docsort** — a fully offline, privacy-first, German+English document sorting daemon according to the detailed spec below.
Strict invariants — never violate these:
• 100% local/offline: ocrmypdf (Tesseract), Ollama localhost:11434, no outbound network except loopback.
• Durability & rebuildability: SQLite (WAL) is source of truth. FAISS index is derived → must be rebuildable via reindex command.
• Exactly one writer mutates index.faiss (use tokio::sync::Mutex or dedicated single-consumer task + channel).
• Crash-safe state machine: every document is always in exactly one stage; restart resumes deterministically or fails loudly with missing artifacts.
• Bounded parallelism + backpressure: never unbounded queues or temp file explosion (use bounded crossbeam or tokio mpsc channels).
• Privacy: never log full OCR text, LLM prompts/responses, or document content — only ids, stages, timings, error codes, final filenames.
• Deterministic similarity: cluster_key = normalized(sender) + "::" + normalized(doc_type) + "::" + template_sig (sha1 of first ~40 normalized lines)
Fixed external dependencies & choices:
• OCR → ocrmypdf binary (call via tokio::process::Command), always -l deu+eng --sidecar
• LLM → Ollama HTTP localhost:11434 /api/generate (stream:false, JSON mode) + /api/embed
• Embeddings → L2-normalized vectors → IndexFlatIP (cosine via IP)
• Metadata → SQLite WAL + NORMAL sync
• Folder watch → notify (preferred) or inotify / recommended watcher crate in 2026
• PDF handling → pdf crate or lopdf or poppler-rs or pdfium-render if needed for extraction/validation
• TIFF→PDF → if needed, call external tool or find pure-Rust way (image + printpdf or similar)
• CLI → clap 4.x with subcommands (status, retry, reocr, reextract, reembed, reindex, gc)
Folder structure (create on startup if missing):
docsort/
├── inbox/ # scanner drops files here
├── processing/ # atomic move target (same fs!)
├── tmp/ # per-doc artifacts: {doc_id}/...
├── output/ # final OCR PDFs with deterministic names
│ └── _duplicates/ # by sha256
├── failed/ # failed/{doc_id}/ → full diagnostic bundle
├── db/
│ ├── docsort.sqlite # WAL
│ └── index.faiss # atomic write + temp→rename
└── logs/ # structured (tracing JSON or similar)
On-disk per-doc artifacts (tmp/{doc_id}/...):
• staged.<ext> # original after stability check
• normalized.pdf # PDF/A-ish or converted from images/TIFF
• ocr.pdf # ocrmypdf result
• ocr.txt # --sidecar
• clean.txt # post-processed text (whitespace collapse, page sep)
• llm_raw.txt # verbatim Ollama /api/generate response
• error_report.txt # concise failure reason
SQLite schema (use sqlx, migrations with sqlx migrate or refinery):
PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
TABLE documents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ingest_path TEXT NOT NULL,
sha256 TEXT UNIQUE NOT NULL,
original_ext TEXT,
staged_path TEXT,
normalized_pdf TEXT,
ocr_pdf TEXT,
ocr_sidecar TEXT,
ocr_clean TEXT,
llm_json TEXT, -- verbatim JSON
llm_raw_path TEXT,
doc_type TEXT,
sender TEXT,
doc_date TEXT, -- ISO or empty
key_id TEXT,
title TEXT,
filename_suggestion TEXT,
template_sig TEXT,
template_sig_version TEXT, -- bump on sig algo change
cluster_key TEXT,
final_filename TEXT,
final_path TEXT,
stage TEXT NOT NULL, -- queued | staged | ocr_done | extract_done | embed_done | moved | processed | failed | duplicate
error_code TEXT,
error_detail_path TEXT,
prompt_version TEXT,
extract_schema_version TEXT,
llm_model TEXT,
embed_model TEXT,
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
);
TABLE embeddings (
doc_id INTEGER UNIQUE REFERENCES documents(id) ON DELETE CASCADE,
model TEXT NOT NULL,
dim INTEGER NOT NULL,
vector BLOB NOT NULL, -- f32 bytes, serialized
faiss_id INTEGER UNIQUE,
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
);
TABLE settings (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
Indices:
CREATE INDEX documents_sha_idx ON documents(sha256);
CREATE INDEX documents_stage_idx ON documents(stage);
CREATE INDEX documents_cluster_idx ON documents(cluster_key);
CREATE INDEX documents_sender_date ON documents(sender, doc_date);
Startup checks — hard exit if any fail:
• ocrmypdf --version works & tesseract deu+eng data files exist
• Ollama reachable → /api/tags returns 200
• embedding model (from settings or default) returns consistent dimension → store in settings.embed_dim
• all folders exist or can be created
Core pipeline stages (each atomic + transactional):
1. Watcher → detect inbox file → stability wait (size+mtime unchanged × 3 checks, timeout → fail)
2. Atomic rename → processing/{uuid}.{ext}
3. SHA256 → check duplicate → if exists → move to _duplicates & mark duplicate
4. Insert documents (stage=staged, paths, sha, timestamps)
5. Normalize to PDF (if image(s)/TIFF → convert, else copy)
6. OCR → ocrmypdf -l deu+eng --sidecar ... → update paths, stage=ocr_done
7. Clean text → collapse ws, page breaks → quality gate (alphabetic ratio, length) → fail OCR_LOW_QUALITY if bad
8. LLM extract → strict JSON mode, retry once, minimal fallback prompt → parse → validate schema → compute template_sig → cluster_key → store fields + versions
9. Build deterministic filename → sanitize + collision __v2 …
10. Embed → /api/embed → L2 norm → store BLOB → single-writer adds to FAISS → atomic persist index.faiss
11. Atomic move ocr.pdf → output/{final_filename}
12. Update final_path, stage=processed
Failure → create failed/{id}/ bundle (copy relevant artifacts) → update stage=failed + error_code + detail_path
Recovery on startup:
• SELECT * FROM documents WHERE stage IN ('staged','ocr_done','extract_done','embed_done','moved')
• For each: check required files exist → enqueue to correct worker queue → else fail RECOVERY_MISSING_ARTIFACT
Workers (bounded):
• watcher task
• ocr pool (tokio::task::spawn_blocking or ProcessPool-like via rayon/tokio-executor)
• llm pool (async HTTP, bounded semaphore)
• embed/index single task (consumes from mpsc, owns Mutex<Index> or similar)
CLI subcommands (docsortctl):
• status → counts by stage + recent errors
• retry --id <id> | --all
• reocr --id <id> | --all
• reextract --id <id> | --all (reuse ocr_clean.txt)
• reembed --id <id> | --all
• reindex → truncate embeddings.faiss_id, rebuild FAISS from vectors BLOB
• gc → clean tmp/ older than N days (configurable)
Deliverables — provide in this order:
1. Complete Cargo.toml (2021 edition, dependencies: tokio, sqlx(sqlite), anyhow, thiserror, tracing, tracing-subscriber, clap, sha2, hex, notify, ollama-rs or ollama-sdk or reqwest-based, serde_json, tempfile, walkdir, etc. — justify unusual choices)
2. src/main.rs — clap CLI + daemon entry (or separate bin for daemon vs ctl)
3. src/lib.rs or modules: db.rs, state.rs, pipeline.rs, ocr.rs, llm.rs, embed.rs, filename.rs, normalize.rs, recovery.rs, cli.rs
4. QualityReport struct + assess function
5. Enums: Stage, ErrorCode, ExtractionMethod
6. Use tracing spans per stage/doc_id
7. Transaction discipline with sqlx::Transaction
8. Atomic file ops (rename, fs::copy for bundles)
9. Comment invariants, non-obvious parts (stability wait, cluster_key, single-writer, recovery)
Focus first on correctness, crash-safety, durability, deterministic behavior — then performance & clean code.
Start writing the skeleton: folder setup, SQLite schema & migrations, startup checks, watcher loop stub.