feat(documents): pluggable document conversion with converter chain

CHANGELOG.md

@@ -17,12 +17,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Preserve richer HTML semantics in Markdown, including collision-safe code
   fences, footnotes, math, callouts, highlights, strikethrough, image sources,
   figure captions, task checkboxes, and valid tables with formatted cells.
-- Add a pluggable post-download `ContentProcessor` registry and built-in
-  `PdfProcessor` that extracts Markdown from text-based PDFs with local
-  `pdf-inspector` processing and explicit OCR guidance for unsupported pages.
-- Add a built-in `OfficeProcessor` that converts Word (DOCX), PowerPoint (PPTX)
-  and spreadsheet (XLSX, XLSM, XLS, ODS) documents to Markdown locally, with
-  zip-bomb limits checked before any part is inflated.
+- Add a pluggable post-download `ContentProcessor` registry.
+- Add pluggable document conversion: `DocumentProcessor` runs an ordered,
+  per-format chain of `DocumentConverter` backends configured with
+  `DocumentConversion` (formats, chain order, 25 MB document cap, remote opt-in)
+  through `ToolBuilder`, the CLI/MCP flags `--convert`, `--converter`,
+  `--allow-remote-conversion`, `--max-document-bytes`, and Python keyword arguments.
+- Built-in converters behind Cargo features: `pdf-inspector` for text-based PDFs
+  (`docs-pdf`, default) and `office` for Word (DOCX), PowerPoint (PPTX) and
+  spreadsheets (XLSX, XLSM, XLSB, XLS, ODS) with zip-bomb limits (`docs-office`,
+  default), plus opt-in remote `CloudflareConverter` (Workers AI `toMarkdown`,
+  `docs-cloudflare`) and generic `HttpConverter` (`docs-http`). Remote converters
+  run only when explicitly allowed and never under `ToolBuilder::hardened()`.
+- `ToolBuilder::content_processor` adds custom processors ahead of the built-ins,
+  and `FetchOptions` carries `document_conversion` and `content_processors`.
 - Discover agent-facing resources from HTTP and HTML declarations plus bounded
   conventional probes, including `llms.txt`, `auth.md`, OAuth metadata, MCP
   server cards, A2A agent cards, and Agent Skills indexes. Markdown responses

README.md

@@ -46,16 +46,59 @@ requests.
 
 - `HtmlProcessor` - metadata and focused-content extraction followed by native
   Markdown or text conversion; accepts a custom `HtmlToMarkdownConverter`
-- `PdfProcessor` - text-based PDF classification and Markdown extraction via
-  [`pdf-inspector`](https://github.com/firecrawl/pdf-inspector); scanned or
-  image-only pages are reported as requiring OCR
-- `OfficeProcessor` - Word (DOCX), PowerPoint (PPTX) and spreadsheet (XLSX, XLS,
-  ODS) documents to Markdown, parsed locally with zip-bomb limits
+- `DocumentProcessor` - PDF and Office documents to Markdown through a
+  configurable chain of document converters (see [Document conversion](#document-conversion))
 
-Custom processors implement `ContentProcessor` and can be registered in a
-`ContentProcessorRegistry`. Pass that registry to
-`FetcherRegistry::with_content_processors` to retain the built-in fetchers while
-customizing post-download processing.
+Custom processors implement `ContentProcessor` and are added with
+`ToolBuilder::content_processor` (checked before the built-ins) or registered in a
+`ContentProcessorRegistry` passed to `FetcherRegistry::with_content_processors`.
+
+## Document Conversion
+
+Documents are always downloaded by fetchkit itself, under the same URL policy,
+DNS pinning, transport and timeouts as any other fetch. `DocumentProcessor` then
+detects the format (Content-Type, then file extension for generic binary types,
+confirmed by magic bytes) and runs an ordered chain of `DocumentConverter`
+backends. The next converter runs when one fails, does not support the input, or
+returns nothing usable (for example a scanned PDF that needs OCR).
+
+| Converter | Cargo feature | Default | Where it runs | Formats |
+| --- | --- | --- | --- | --- |
+| `PdfInspectorConverter` (`pdf-inspector`) | `docs-pdf` | on | local | PDF (text layer; reports OCR-only pages) |
+| `OfficeConverter` (`office`) | `docs-office` | on | local | DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS |
+| `CloudflareConverter` (`cloudflare`) | `docs-cloudflare` | off | remote | PDF, DOCX, XLSX, XLSB, XLS, ODT, ODS |
+| `HttpConverter` (any name) | `docs-http` | off | remote (configurable) | any format your service accepts |
+
+`docs-all` enables every backend. Remote converters upload document bytes to a
+third party, so they only run when remote conversion is explicitly allowed, and
+never under `ToolBuilder::hardened()`.
+
+```rust
+use fetchkit::documents::CloudflareConverter;
+use fetchkit::{DocumentConversion, DocumentFormat, ToolBuilder};
+
+let tool = ToolBuilder::new()
+    .document_conversion(
+        DocumentConversion::default()                 // pdf-inspector + office
+            .converter(CloudflareConverter::new(account_id, api_token))
+            .allow_remote(true)                       // opt in to uploads
+            .chain(DocumentFormat::Pdf, ["pdf-inspector", "cloudflare"])
+            .formats([DocumentFormat::Pdf, DocumentFormat::Docx, DocumentFormat::Xlsx])
+            .max_document_bytes(25 * 1024 * 1024),    // separate from max_body_size
+    )
+    .build();
+```
+
+Extension points: implement `DocumentConverter` for a new backend,
+`FormatDetector` to change format detection, `ConverterClient` to route remote
+uploads through your own egress, or `ContentProcessor` to bypass the chain.
+
+CLI and MCP server flags: `--convert pdf,docx`, `--converter office,cloudflare`,
+`--allow-remote-conversion`, `--max-document-bytes`. The Cloudflare converter
+reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` (build with
+`--features docs-cloudflare`). Python: `FetchkitTool(document_formats=...,
+converters=..., allow_remote_conversion=..., max_document_bytes=...,
+cloudflare_account_id=..., cloudflare_api_token=...)`.
 
 ## Installation
 
@@ -336,14 +379,16 @@ Partial content is returned on body timeout or body-size limit with `truncated:
 
 ### PDF And Binary Content
 
-When Markdown is requested, text-based PDFs are downloaded within the configured
-body limit and converted to Markdown. PDF parsing runs locally; no OCR service or
+When Markdown is requested, text-based PDFs are downloaded within the document
+size limit (25 MB default) and converted to Markdown. PDF parsing runs locally; no OCR service or
 additional network request is used. Scanned/image-only PDFs report `use_ocr` as the
 suggested next action.
 
 Office documents are converted the same way, locally: Word headings, lists and
 tables, one section per PowerPoint slide, and one Markdown table per spreadsheet
-sheet (DOCX, PPTX, XLSX, XLS, ODS). Legacy DOC and PPT files are not supported.
+sheet (DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS). Legacy DOC and PPT files are not
+converted by the built-in local converters; route them to a remote or custom
+converter if needed.
 
 Other binary content returns metadata only:
 - Images, audio, video, fonts

crates/fetchkit-cli/Cargo.toml

@@ -20,6 +20,9 @@ doc = false
 default = []
 bot-auth = ["fetchkit/bot-auth"]
 render-rakers = ["fetchkit/render-rakers"]
+docs-cloudflare = ["fetchkit/docs-cloudflare"]
+docs-http = ["fetchkit/docs-http"]
+docs-all = ["fetchkit/docs-all"]
 
 [dependencies]
 fetchkit = { path = "../fetchkit", version = "0.5.0" }

crates/fetchkit-cli/src/main.rs

@@ -13,8 +13,8 @@
 
 mod mcp;
 
-use clap::{Parser, Subcommand, ValueEnum};
-use fetchkit::{CrawlPage, FetchRequest, Tool, TOOL_LLMTXT};
+use clap::{Args, Parser, Subcommand, ValueEnum};
+use fetchkit::{CrawlPage, DocumentConversionSettings, FetchRequest, Tool, TOOL_LLMTXT};
 use std::io::{self, Write};
 
 /// Output format for fetch subcommand
@@ -27,6 +27,41 @@ enum OutputFormat {
     Json,
 }
 
+/// Document (PDF, Office) conversion options shared by `fetch` and `mcp`.
+#[derive(Args, Debug, Default, Clone)]
+struct DocumentArgs {
+    /// Document formats to convert, comma-separated (e.g. pdf,docx,xlsx)
+    #[arg(long = "convert", value_delimiter = ',')]
+    formats: Option<Vec<String>>,
+
+    /// Converter order, comma-separated (pdf-inspector, office, cloudflare). Cloudflare
+    /// reads CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN.
+    #[arg(long = "converter", value_delimiter = ',')]
+    converters: Option<Vec<String>>,
+
+    /// Allow converters that upload documents to a third-party service
+    #[arg(long)]
+    allow_remote_conversion: bool,
+
+    /// Maximum document download size in bytes (default 25 MB)
+    #[arg(long)]
+    max_document_bytes: Option<usize>,
+}
+
+impl DocumentArgs {
+    fn settings(&self, respect_proxy_env: bool) -> DocumentConversionSettings {
+        DocumentConversionSettings {
+            formats: self.formats.clone(),
+            converters: self.converters.clone(),
+            allow_remote: self.allow_remote_conversion,
+            max_document_bytes: self.max_document_bytes,
+            cloudflare_account_id: std::env::var("CLOUDFLARE_ACCOUNT_ID").ok(),
+            cloudflare_api_token: std::env::var("CLOUDFLARE_API_TOKEN").ok(),
+            respect_proxy_env,
+        }
+    }
+}
+
 /// Fetchkit - AI-friendly web content fetching tool
 #[derive(Parser, Debug)]
 #[command(name = "fetchkit")]
@@ -59,6 +94,9 @@ enum Commands {
         /// Agent FQDN for Signature-Agent header (requires --bot-auth-key)
         #[arg(long)]
         bot_auth_agent: Option<String>,
+
+        #[command(flatten)]
+        documents: DocumentArgs,
     },
     /// Fetch URL and output as markdown with metadata frontmatter
     Fetch {
@@ -104,6 +142,9 @@ enum Commands {
         /// Render HTML with the rakers backend before conversion
         #[arg(long)]
         render_rakers: bool,
+
+        #[command(flatten)]
+        documents: DocumentArgs,
     },
 }
 
@@ -123,6 +164,7 @@ async fn main() {
             allow_env_proxy,
             bot_auth_key,
             bot_auth_agent,
+            documents,
         }) => {
             mcp::run_server(build_tool(
                 None,
@@ -131,6 +173,7 @@ async fn main() {
                 bot_auth_key,
                 bot_auth_agent,
                 false,
+                &documents,
             ))
             .await;
         }
@@ -146,8 +189,10 @@ async fn main() {
             crawl,
             max_pages,
             render_rakers,
+            documents,
         }) => {
             let options = FetchCommandOptions {
+                documents,
                 output,
                 user_agent,
                 hardened,
@@ -177,9 +222,19 @@ fn build_tool(
     bot_auth_key: Option<String>,
     bot_auth_agent: Option<String>,
     render_rakers: bool,
+    documents: &DocumentArgs,
 ) -> Tool {
     let mut builder = Tool::builder().enable_markdown(true);
 
+    let settings = documents.settings(allow_env_proxy);
+    if !settings.is_default() {
+        let conversion = settings.build().unwrap_or_else(|e| {
+            eprintln!("Error: {e}");
+            std::process::exit(1);
+        });
+        builder = builder.document_conversion(conversion);
+    }
+
     if hardened {
         builder = builder.hardened();
     }
@@ -229,6 +284,7 @@ fn build_tool(
 }
 
 struct FetchCommandOptions {
+    documents: DocumentArgs,
     output: OutputFormat,
     user_agent: Option<String>,
     hardened: bool,
@@ -260,6 +316,7 @@ async fn run_fetch(url: &str, options: FetchCommandOptions) {
         options.bot_auth_key,
         options.bot_auth_agent,
         options.render_rakers,
+        &options.documents,
     );
 
     // Execute request

crates/fetchkit-python/Cargo.toml

@@ -12,6 +12,11 @@ publish = false
 name = "fetchkit_py"
 crate-type = ["cdylib"]
 
+[features]
+default = []
+docs-cloudflare = ["fetchkit/docs-cloudflare"]
+docs-all = ["fetchkit/docs-all"]
+
 [dependencies]
 fetchkit = { path = "../fetchkit" }
 pyo3 = { version = "0.29", features = ["extension-module"] }

crates/fetchkit-python/src/lib.rs

@@ -12,7 +12,10 @@
 //! print(response.content)
 //! ```
 
-use fetchkit::{FetchError, FetchRequest, FetchResponse, HttpMethod, Tool, ToolBuilder};
+use fetchkit::{
+    DocumentConversionSettings, FetchError, FetchRequest, FetchResponse, HttpMethod, Tool,
+    ToolBuilder,
+};
 use pyo3::exceptions::{PyDeprecationWarning, PyValueError};
 use pyo3::prelude::*;
 
@@ -214,7 +217,13 @@ impl PyFetchkitTool {
         allowed_ports=None,
         blocked_hosts=None,
         same_host_redirects_only=None,
-        hardened=false
+        hardened=false,
+        document_formats=None,
+        converters=None,
+        allow_remote_conversion=false,
+        max_document_bytes=None,
+        cloudflare_account_id=None,
+        cloudflare_api_token=None
     ))]
     fn new(
         enable_markdown: bool,
@@ -229,6 +238,12 @@ impl PyFetchkitTool {
         blocked_hosts: Option<Vec<String>>,
         same_host_redirects_only: Option<bool>,
         hardened: bool,
+        document_formats: Option<Vec<String>>,
+        converters: Option<Vec<String>>,
+        allow_remote_conversion: bool,
+        max_document_bytes: Option<usize>,
+        cloudflare_account_id: Option<String>,
+        cloudflare_api_token: Option<String>,
     ) -> PyResult<Self> {
         let mut builder = ToolBuilder::new()
             .enable_markdown(enable_markdown)
@@ -279,6 +294,20 @@ impl PyFetchkitTool {
             }
         }
 
+        let documents = DocumentConversionSettings {
+            formats: document_formats,
+            converters,
+            allow_remote: allow_remote_conversion,
+            max_document_bytes,
+            cloudflare_account_id,
+            cloudflare_api_token,
+            respect_proxy_env,
+        };
+        if !documents.is_default() {
+            builder =
+                builder.document_conversion(documents.build().map_err(PyValueError::new_err)?);
+        }
+
         let runtime = tokio::runtime::Runtime::new()
             .map_err(|e| PyValueError::new_err(format!("Failed to create runtime: {}", e)))?;
 
@@ -406,6 +435,12 @@ fn deprecated_fetch_kit_tool(
         blocked_hosts,
         same_host_redirects_only,
         hardened,
+        None,
+        None,
+        false,
+        None,
+        None,
+        None,
     )
 }
 
@@ -423,7 +458,8 @@ fn fetch(
     max_pages: Option<usize>,
 ) -> PyResult<PyFetchResponse> {
     let tool = PyFetchkitTool::new(
-        true, true, None, None, None, None, true, false, None, None, None, false,
+        true, true, None, None, None, None, true, false, None, None, None, false, None, None,
+        false, None, None, None,
     )?;
     tool.fetch(
         url,

crates/fetchkit/Cargo.toml

@@ -11,7 +11,13 @@ categories.workspace = true
 readme = "../../README.md"
 
 [features]
-default = []
+default = ["docs-pdf", "docs-office"]
+# Document conversion backends (see knowledge/foundations/fetchers.md)
+docs-pdf = ["dep:pdf-inspector"]
+docs-office = ["dep:calamine", "dep:zip", "dep:quick-xml"]
+docs-cloudflare = []
+docs-http = []
+docs-all = ["docs-pdf", "docs-office", "docs-cloudflare", "docs-http"]
 bot-auth = ["dep:ed25519-dalek", "dep:base64", "dep:sha2", "dep:rand"]
 live-tests = []
 render-rakers = ["dep:rakers"]
@@ -30,10 +36,10 @@ futures = { workspace = true }
 bytes = { workspace = true }
 async-trait = { workspace = true }
 tower = { workspace = true }
-pdf-inspector = { workspace = true }
-calamine = { workspace = true }
-zip = { workspace = true }
-quick-xml = { workspace = true }
+pdf-inspector = { workspace = true, optional = true }
+calamine = { workspace = true, optional = true }
+zip = { workspace = true, optional = true }
+quick-xml = { workspace = true, optional = true }
 libc = { workspace = true }
 rakers = { workspace = true, optional = true }