Skip to content

feat(documents): pluggable document conversion with converter chain - #182

Merged
chaliy merged 1 commit into
mainfrom
claude/project-thread-gtqhrp
Oct 12, 2026
Merged

feat(documents): pluggable document conversion with converter chain#182
chaliy merged 1 commit into
mainfrom
claude/project-thread-gtqhrp

Conversation

@chaliy

@chaliy chaliy commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Requested by Михайло · project thread

What changed

PDF and Office conversion is now configurable and pluggable instead of two fixed processors.

  • One DocumentProcessor runs an ordered chain of DocumentConverter backends, set per format. When a backend fails, doesn't support the input, or returns nothing usable (for example a scanned PDF), the next one runs.
  • Built-in backends sit behind Cargo features:
    • docs-pdf (default on): pdf-inspector.
    • docs-office (default on): the local Office converter from feat(content): convert Office documents to Markdown locally #181.
    • docs-cloudflare (opt-in): Cloudflare Workers AI toMarkdown.
    • docs-http (opt-in): a generic multipart HttpConverter for self-hosted services such as Docling or MarkItDown.
  • Remote backends upload documents to a third party. They run only with an explicit allow_remote(true), and never under ToolBuilder::hardened(). Uploads go through a pluggable ConverterClient, so a host can route them through its own egress. Tokens are redacted from debug output and errors.
  • Documents have their own download cap, max_document_bytes (25 MB default), separate from max_body_size. Formats are confirmed by magic bytes before any converter runs.
  • The same config is available from every surface:
    • Library: ToolBuilder::document_conversion, plus ToolBuilder::content_processor to add custom processors ahead of the built-ins.
    • CLI and MCP: --convert, --converter, --allow-remote-conversion, --max-document-bytes. The Cloudflare converter reads CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN.
    • Python: keyword arguments of the same names.
  • For extension, the public traits are DocumentConverter, FormatDetector, ConverterClient and ContentProcessor.
  • When a document format is recognised but no converter handles it, the error now names the format.

Why

Design: Fetchkit: PDF and Office document conversion design. Conversion should be configurable, feature-flagged and pluggable. It should allow third-party services such as Cloudflare, and leave room for markkit or other backends later.

Before / After

Before: Office conversion was a fixed OfficeProcessor (from #181), with no way to turn it off, choose a backend or add a remote one.

After, default build (unchanged output for supported formats):

$ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx
quality_score: 1.00
extraction_method: "office_docx"
---
# Demonstration of DOCX support in calibre

Restricting formats:

$ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx --convert pdf -o json
"error": "Binary content is not supported. ... DOCX conversion is not enabled in this fetchkit configuration; enable a document converter to convert it."

Remote converter without its feature:

$ fetchkit fetch …/demo.docx --converter cloudflare --allow-remote-conversion
Error: converter `cloudflare` requires the docs-cloudflare feature

The Cloudflare path is covered end to end with wiremock: the tool downloads the DOCX, uploads it to a mock toMarkdown, and returns cloudflare_tomarkdown. It has not been run against a real Cloudflare account yet.

Risk

  • Medium.
  • Unreleased API change: PdfProcessor and OfficeProcessor are replaced by DocumentProcessor plus PdfInspectorConverter and OfficeConverter.
  • pdf-inspector, calamine, zip and quick-xml are now optional dependencies. Default features keep them on.
  • Default behaviour changes slightly in two ways:
    • Documents may now be up to 25 MB, where the old cap was 10 MB.
    • application/msword and application/epub+zip are now treated as binary.
  • New threats are documented as TM-LEAK-006 (document disclosure to a remote converter) and TM-LEAK-007 (converter credentials).

Checklist

  • Unit tests are passed (cargo test --workspace; clippy with default, no-default, docs-all and single-backend feature sets)
  • Smoke tests are passed (real DOCX and PDF through the release CLI)
  • Documentation is updated (README, CHANGELOG, docs/hardening.md)
  • Knowledge is up to date and not in conflict (fetchers, tool contract, threat model, log; check_okf.py passes)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi

Generated by Claude Code

@claude
feat(documents): pluggable document conversion with converter chain
236177b 
Replace the fixed PDF and Office processors with one DocumentProcessor that
runs an ordered, per-format chain of DocumentConverter backends configured by
DocumentConversion (formats, chain order, 25 MB document cap, remote opt-in).

- Built-in converters behind features: pdf-inspector (docs-pdf, default),
  office (docs-office, default), Cloudflare toMarkdown (docs-cloudflare) and a
  generic multipart HttpConverter (docs-http).
- Remote converters run only with allow_remote(true), never under hardened();
  uploads go through a pluggable ConverterClient; tokens are redacted.
- Fall through to the next converter on error, unsupported input, or an
  empty/OCR-required result; formats are confirmed by magic bytes.
- Config reachable from ToolBuilder, FetchOptions, CLI/MCP flags and Python;
  ToolBuilder::content_processor adds custom processors ahead of built-ins.
- Rejection message names the recognised format when conversion is disabled.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi
@chaliy chaliy self-assigned this Oct 12, 2026
@chaliy chaliy mentioned this pull request Oct 12, 2026
Merged
@chaliy
chaliy merged commit c20b730 into main Oct 12, 2026
12 checks passed
@chaliy
chaliy deleted the claude/project-thread-gtqhrp branch October 12, 2026 18:30
chaliy added a commit to everruns/markkit that referenced this pull request Oct 12, 2026
@chaliy @claude
feat(markkit): bytes-in document API with DOCX and spreadsheet conver…
f82b7b2 
…sion (#9)

Before: markkit had no way to convert document bytes to Markdown. DOCX
went through docx-rs with no limits, and the only office path pulled in
the HTTP client stack. `cargo deny` also failed on newly published
advisories.

After:
- `Markkit::extract_document(bytes, DocumentHint)` and
`extract_document_sync` convert DOCX, XLSX/XLSM, XLS and ODS to
Markdown. They return an `Extraction` with the title, the detected
`Format`, warnings and stats.
- The `docx` and `spreadsheet` features build without reqwest or hyper,
so fetchkit can embed markkit as its Office backend.
- `cargo deny check` and `cargo audit` pass.

How:
- **Format detection.** `Format::detect`, `sniff` and `from_file_name`
use the content type first, then the file extension, then magic bytes.
Detection works with no format feature enabled.
- **DOCX.** Read directly with `zip` and `quick-xml`; docx-rs is
removed. Every part is decompressed with caps and parsed into a
depth-limited XML tree.
- **Spreadsheets.** Converted with calamine 0.36.
  - XLSX is streamed.
- XLS and ODS are pre-scanned against `max_spreadsheet_cells`, so a tiny
file cannot force a huge dense-grid allocation.
- Hidden sheets are skipped by default, and formulas are not evaluated.
- **New `MarkkitBuilder` limits.** `max_archive_entries`,
`max_archive_entry_size`, `max_archive_total_size`, `max_sheet_rows`,
`max_sheet_cols`, `max_spreadsheet_cells` and `include_hidden_sheets`.
- **Unsupported formats.** PDF, PPTX, legacy `.doc` and XLSB are
detected and return an explicit `UnsupportedFormat` error.
- **Advisories.**
- Removed `pdf-extract`, which was never called. This clears lopdf
RUSTSEC-2026-0187 and the unmaintained `ttf-parser`. The `pdf` feature
is now a dependency-free placeholder.
- Upgraded `pyo3` and `pyo3-async-runtimes` to 0.29
(RUSTSEC-2026-0176/0177).
- Upgraded `tabled` in the bench crate to 0.22, which drops the
unmaintained `proc-macro-error2`.
- **Security tests.** They cover:
  - zip bombs
  - entry and total size caps
  - grid blow-ups
  - billion laughs
  - XXE
  - deep nesting
  - hidden sheets
  - fuzz-style proptests

  Fixtures are built in memory.
- **CI.** A new step tests `--no-default-features --features
docx,spreadsheet` and fails if an HTTP client appears in the dependency
tree.
- **Docs.** Specs, the threat model (T6, S8–S11), the README and the
CHANGELOG are updated.

Tests:
- `cargo test --workspace` passes.
- `cargo clippy -D warnings` passes for both the default and the
all-features builds.
- The doc build and the release build pass.
- `cargo deny check` and `cargo audit` pass.

Part of the fetchkit document conversion work (everruns/fetchkit#182).
markkit will replace fetchkit's in-crate Office converter.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@chaliy chaliy mentioned this pull request Oct 12, 2026
Merged
4 tasks done
chaliy added a commit that referenced this pull request Oct 12, 2026
@chaliy @claude
feat(documents): markkit as default Office converter, deprecate office (
0d2f262 
#183)

## What changed
markkit (0.1.0, crates.io) becomes the default Office backend. The
in-crate `office` converter from #181 is deprecated and off by default.

- **New `docs-markkit` feature, on by default.** It adds
`MarkkitConverter` (name `markkit`, local) for DOCX, XLSX/XLSM, XLS and
ODS.
- Extraction methods are `markkit_docx`, `markkit_xlsx`, `markkit_xls`
and `markkit_ods`.
- Limits are tuned with
`MarkkitConverter::with_markkit(Markkit::builder()…)`. `Markkit` and
`MarkkitBuilder` are re-exported from `fetchkit::documents`.
- **Feature defaults:** `docs-pdf` + `docs-office` becomes `docs-pdf` +
`docs-markkit`. `docs-all` now includes markkit and excludes office.
- **`OfficeConverter` is `#[deprecated]`.** It still works behind
`docs-office`, and when that feature is enabled it runs after markkit as
a fallback.
- **Settings, CLI and Python:** `--converter markkit` /
`converters=["markkit"]` are accepted. When a feature is missing, the
error says so.
- markkit is built without its HTTP client, and the only reqwest in the
tree is fetchkit's own.

## Why
The repo owner decided that #181's in-crate Office parsing is not the
implementation to keep. Office parsing moves to markkit, which
fetchkit's design already planned as a pluggable backend. markkit has
zip-bomb, archive and spreadsheet-grid limits, and it skips hidden
sheets by default.

## Before / After
Before (default build): `fetchkit fetch …/demo.docx` gave
`extraction_method: "office_docx"`.

After:
```
$ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx
quality_score: 1.00
extraction_method: "markkit_docx"
---
# Demonstration of DOCX support in calibre
…
| ITEM | NEEDED |
| Books | 1 |
```
Local files: `.xlsx` gives `markkit_xlsx` (with a `hidden_sheet_skipped`
warning), `.xls` gives `markkit_xls`, `.ods` gives `markkit_ods`, and a
large sheet carries a `truncated_sheet` warning. Legacy `.doc` is still
rejected with "DOC conversion is not enabled".

## Risk
- Medium. Unreleased API change on top of #182.
- PPTX and XLSB are no longer converted by default, because markkit
doesn't support them yet. Enabling `docs-office` restores them.
- Spreadsheet titles are no longer extracted.
- markkit's default sheet caps are looser: 10,000 rows × 256 columns,
where `office` used 1,000 × 50. Output is still capped by
`max_body_size`.
- New transitive crates: `cfb`, `uuid`, `regex`, `anyhow`, `mime`,
`unicode-segmentation`.

### Checklist
- [x] Unit tests are passed: `cargo test --workspace`, plus feature sets
with no defaults, `docs-markkit` only, and `docs-all,docs-office`.
clippy `-D warnings` passes across the same sets.
- [x] Smoke tests are passed: the release CLI on a real DOCX, and local
XLSX/XLS/ODS files.
- [x] Documentation is updated: README, CHANGELOG (Added and
Deprecated), docs/hardening.md.
- [x] Knowledge is up to date and not in conflict: fetchers, tool
contract, TM-DOS-008, and the log. `check_okf.py` passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi

---
_Generated by [Claude
Code](https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@chaliy chaliy

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@chaliy @claude