Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Built-in converters behind Cargo features: `pdf-inspector` for text-based PDFs
(`docs-pdf`, default) and `office` for Word (DOCX), PowerPoint (PPTX) and
spreadsheets (XLSX, XLSM, XLSB, XLS, ODS) with zip-bomb limits (`docs-office`,
default), plus opt-in remote `CloudflareConverter` (Workers AI `toMarkdown`,
now deprecated and off by default; see Deprecated), plus opt-in remote `CloudflareConverter` (Workers AI `toMarkdown`,
`docs-cloudflare`) and generic `HttpConverter` (`docs-http`). Remote converters
run only when explicitly allowed and never under `ToolBuilder::hardened()`.
- Add `MarkkitConverter` (`markkit`, `docs-markkit` feature, default): local
DOCX, XLSX/XLSM, XLS and ODS conversion through the
[markkit](https://crates.io/crates/markkit) crate, with configurable limits via
`MarkkitConverter::with_markkit` (`fetchkit::documents::Markkit`). It is the
default Office backend; `extraction_method` is `markkit_docx`, `markkit_xlsx`,
`markkit_xls` or `markkit_ods`.
- `ToolBuilder::content_processor` adds custom processors ahead of the built-ins,
and `FetchOptions` carries `document_conversion` and `content_processors`.
- Discover agent-facing resources from HTTP and HTML declarations plus bounded
conventional probes, including `llms.txt`, `auth.md`, OAuth metadata, MCP
server cards, A2A agent cards, and Agent Skills indexes. Markdown responses
include a compact navigation appendix.

### Deprecated

- `OfficeConverter` (`office`, `docs-office` feature) is deprecated in favour of
`MarkkitConverter` and no longer a default feature or part of `docs-all`. It
still works when `docs-office` is enabled explicitly and remains the only
built-in PPTX and XLSB converter for this release; PPTX and XLSB responses are
rejected as unconverted binaries under default features.

### Changed

- Markdown requests now ask for `text/markdown` first in the `Accept` header, so
Expand Down
46 changes: 46 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ tower = { version = "0.5", features = ["util"] }

# Binary document extraction
pdf-inspector = "0.1.7"
markkit = { version = "0.1", default-features = false, features = ["docx", "spreadsheet"] }
calamine = { version = "0.36", default-features = false }
zip = { version = "8", default-features = false, features = ["deflate"] }
quick-xml = "0.41"
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,19 @@ returns nothing usable (for example a scanned PDF that needs OCR).
| Converter | Cargo feature | Default | Where it runs | Formats |
| --- | --- | --- | --- | --- |
| `PdfInspectorConverter` (`pdf-inspector`) | `docs-pdf` | on | local | PDF (text layer; reports OCR-only pages) |
| `OfficeConverter` (`office`) | `docs-office` | on | local | DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS |
| `MarkkitConverter` (`markkit`) | `docs-markkit` | on | local | DOCX, XLSX, XLSM, XLS, ODS (via [markkit](https://crates.io/crates/markkit)) |
| `OfficeConverter` (`office`), deprecated | `docs-office` | off | local | DOCX, PPTX, XLSX, XLSM, XLSB, XLS, ODS |
| `CloudflareConverter` (`cloudflare`) | `docs-cloudflare` | off | remote | PDF, DOCX, XLSX, XLSB, XLS, ODT, ODS |
| `HttpConverter` (any name) | `docs-http` | off | remote (configurable) | any format your service accepts |

`docs-all` enables every backend. Remote converters upload document bytes to a
`markkit` is the default Office backend. The in-crate `office` converter is
deprecated and kept for one release behind the non-default `docs-office` feature;
it is still the only built-in PPTX and XLSB converter. With both features enabled,
`markkit` runs first and `office` is the fallback. `docs-all` enables every
backend except the deprecated `docs-office`. Tune markkit limits (rows and
columns per sheet, hidden sheets, archive budgets) by registering
`MarkkitConverter::with_markkit(Markkit::builder()...build())`, which replaces the
default `markkit` converter in place. Remote converters upload document bytes to a
third party, so they only run when remote conversion is explicitly allowed, and
never under `ToolBuilder::hardened()`.

Expand All @@ -79,7 +87,7 @@ use fetchkit::{DocumentConversion, DocumentFormat, ToolBuilder};

let tool = ToolBuilder::new()
.document_conversion(
DocumentConversion::default() // pdf-inspector + office
DocumentConversion::default() // pdf-inspector + markkit
.converter(CloudflareConverter::new(account_id, api_token))
.allow_remote(true) // opt in to uploads
.chain(DocumentFormat::Pdf, ["pdf-inspector", "cloudflare"])
Expand All @@ -93,7 +101,7 @@ Extension points: implement `DocumentConverter` for a new backend,
`FormatDetector` to change format detection, `ConverterClient` to route remote
uploads through your own egress, or `ContentProcessor` to bypass the chain.

CLI and MCP server flags: `--convert pdf,docx`, `--converter office,cloudflare`,
CLI and MCP server flags: `--convert pdf,docx`, `--converter markkit,cloudflare`,
`--allow-remote-conversion`, `--max-document-bytes`. The Cloudflare converter
reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` (build with
`--features docs-cloudflare`). Python: `FetchkitTool(document_formats=...,
Expand Down
2 changes: 2 additions & 0 deletions crates/fetchkit-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ doc = false
default = []
bot-auth = ["fetchkit/bot-auth"]
render-rakers = ["fetchkit/render-rakers"]
docs-markkit = ["fetchkit/docs-markkit"]
docs-office = ["fetchkit/docs-office"]
docs-cloudflare = ["fetchkit/docs-cloudflare"]
docs-http = ["fetchkit/docs-http"]
docs-all = ["fetchkit/docs-all"]
Expand Down
2 changes: 1 addition & 1 deletion crates/fetchkit-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ struct DocumentArgs {
#[arg(long = "convert", value_delimiter = ',')]
formats: Option<Vec<String>>,

/// Converter order, comma-separated (pdf-inspector, office, cloudflare). Cloudflare
/// Converter order, comma-separated (pdf-inspector, markkit, office, cloudflare). Cloudflare
/// reads CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN.
#[arg(long = "converter", value_delimiter = ',')]
converters: Option<Vec<String>>,
Expand Down
2 changes: 2 additions & 0 deletions crates/fetchkit-python/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ crate-type = ["cdylib"]

[features]
default = []
docs-markkit = ["fetchkit/docs-markkit"]
docs-office = ["fetchkit/docs-office"]
docs-cloudflare = ["fetchkit/docs-cloudflare"]
docs-all = ["fetchkit/docs-all"]

Expand Down
11 changes: 9 additions & 2 deletions crates/fetchkit/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,17 @@ categories.workspace = true
readme = "../../README.md"

[features]
default = ["docs-pdf", "docs-office"]
default = ["docs-pdf", "docs-markkit"]
# Document conversion backends (see knowledge/foundations/fetchers.md)
docs-pdf = ["dep:pdf-inspector"]
# Office documents (DOCX, XLSX/XLSM, XLS, ODS) via markkit.
docs-markkit = ["dep:markkit"]
# Deprecated in-crate Office converter (DOCX, PPTX, spreadsheets); use docs-markkit.
docs-office = ["dep:calamine", "dep:zip", "dep:quick-xml"]
docs-cloudflare = []
docs-http = []
docs-all = ["docs-pdf", "docs-office", "docs-cloudflare", "docs-http"]
# Every supported backend; excludes the deprecated docs-office.
docs-all = ["docs-pdf", "docs-markkit", "docs-cloudflare", "docs-http"]
bot-auth = ["dep:ed25519-dalek", "dep:base64", "dep:sha2", "dep:rand"]
live-tests = []
render-rakers = ["dep:rakers"]
Expand All @@ -37,6 +41,7 @@ bytes = { workspace = true }
async-trait = { workspace = true }
tower = { workspace = true }
pdf-inspector = { workspace = true, optional = true }
markkit = { workspace = true, optional = true }
calamine = { workspace = true, optional = true }
zip = { workspace = true, optional = true }
quick-xml = { workspace = true, optional = true }
Expand All @@ -59,3 +64,5 @@ base64 = { workspace = true }
sha2 = { workspace = true }
rand = { workspace = true }
mira-eval = "0.3"
# Builds in-memory DOCX/XLSX fixtures for document tests.
zip = { workspace = true, features = ["deflate"] }
Loading
Loading