diff --git a/.agents/skills/process-issues/SKILL.md b/.agents/skills/process-issues/SKILL.md new file mode 100644 index 0000000..b3d0af7 --- /dev/null +++ b/.agents/skills/process-issues/SKILL.md @@ -0,0 +1,52 @@ +Process open GitHub issues and resolve them via merged pull requests. + +## Qualifying Issues + +Only process issues created by authorized users or containing approval +comments from those users. + +## Core Principle + +One issue = one PR. No bundling. No skipping. + +## Process Per Issue + +### 1. Understand & Scope +- Read the issue thoroughly +- Classify: bug, feature, docs, refactor +- Identify affected crates and files +- Create branch: `fix/issue-{N}-{short-slug}` + +### 2. Test First +- For bugs: create a failing test that reproduces the issue +- For features: scaffold validation tests +- Place tests alongside existing test modules + +### 3. Implement +- Make minimal, focused changes +- Cover positive and negative test cases +- Consider security implications for parsing changes + +### 4. Ship +- Use the `/ship` skill for: + - Spec updates + - Quality gates + - PR creation + - CI verification + - Merge + +### 5. Return to Main +```bash +git checkout main && git pull origin main +``` +Then proceed to next issue. + +## Post-Resolution + +After all issues are resolved, identify any `#[ignore]` tests +that now pass and un-ignore them in a single PR. + +## Arguments + +Optional: specific issue numbers or labels. +Default: process all qualifying open issues. diff --git a/.agents/skills/ship/SKILL.md b/.agents/skills/ship/SKILL.md new file mode 100644 index 0000000..621c7d4 --- /dev/null +++ b/.agents/skills/ship/SKILL.md @@ -0,0 +1,51 @@ +Run the full ship flow: verify quality, ensure test coverage, update +artifacts, smoke test, then push, create PR, and merge when CI is green. + +## Phases + +### 1. Pre-flight +- Verify on a feature branch (not main) +- Check for uncommitted changes +- Fetch latest main and rebase if needed + +### 2. Test Coverage +- Identify changed code paths +- Ensure tests exist for: + - Positive cases (happy path) + - Negative cases (errors, edge cases) + - Security scenarios (malicious input) +- Add missing tests + +### 3. Artifact Updates +- Update specs if behavior changed +- Update CHANGELOG.md for user-facing changes +- Update documentation for API changes + +### 4. Simplification +- Remove dead code introduced by changes +- Reduce duplication +- Simplify where possible without changing behavior + +### 5. Security Review +- Check for injection risks in new parsing code +- Verify resource limits are enforced +- Review input validation + +### 6. Smoke Test +- Run `just test` for full test suite +- Run `just lint` for code quality +- Build CLI and verify basic extraction works + +### 7. Quality Gates +- `cargo fmt --all -- --check` +- `cargo clippy --workspace -- -D warnings` +- `cargo deny check` + +### 8. Push, PR, Merge +- Push branch to remote +- Create or update PR with summary +- Wait for CI to pass +- Merge when green + +**IMPORTANT**: Phases 2-5 (tests, artifacts, simplification, security review, +smoke testing) are the quality core — do NOT skip them. diff --git a/.claude/commands/maintain.md b/.claude/commands/maintain.md new file mode 100644 index 0000000..40cf3ac --- /dev/null +++ b/.claude/commands/maintain.md @@ -0,0 +1,49 @@ +Run pre-release maintenance checks on the markkit codebase. + +Each section below is an outcome to achieve, not a script to follow. +Use whatever tools and approaches make sense to verify and fix. + +Fix issues as you find them — don't just report. Use conventional +commit messages for any fixes. + +Sections (run all unless $ARGUMENTS specifies a subset): + +## 1. Dependencies +- Run `cargo update` and verify tests pass +- Check for outdated dependencies +- Review Dependabot PRs if any + +## 2. Security +- Run `cargo audit` for vulnerabilities +- Run `cargo deny check` for license compliance +- Review any new advisories + +## 3. Tests +- Run full test suite: `just test` +- Check coverage isn't regressing +- Identify any `#[ignore]` tests that should be re-enabled + +## 4. Documentation +- Verify README is current +- Check rustdoc builds without warnings +- Ensure specs match implementation + +## 5. Examples +- Verify examples compile and run +- Update examples for any API changes + +## 6. Specs +- Review specs/ for accuracy +- Update implementation status + +## 7. Simplification +- Look for dead code, unused dependencies +- Identify duplication opportunities +- Reduce complexity where possible + +## 8. CI Health +- Check recent GitHub Actions runs +- Verify nightly builds are passing +- Review any flaky tests + +Deliver a summary of findings and actions taken. diff --git a/.claude/hooks/fix-git-identity.sh b/.claude/hooks/fix-git-identity.sh new file mode 100755 index 0000000..8b0fbc8 --- /dev/null +++ b/.claude/hooks/fix-git-identity.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# SessionStart hook: override agent-like git identity with real user + +set -euo pipefail + +git rev-parse --git-dir >/dev/null 2>&1 || exit 0 + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" + +source "$REPO_ROOT/scripts/lib/common.sh" + +configure_commit_git_identity_if_needed 2>/dev/null || true diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..2918fd0 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,20 @@ +{ + "gitAttribution": false, + "attribution": { + "commit": "", + "pr": "" + }, + "hooks": { + "SessionStart": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": ".claude/hooks/fix-git-identity.sh" + } + ] + } + ] + } +} diff --git a/.claude/skills b/.claude/skills new file mode 120000 index 0000000..bf44df0 --- /dev/null +++ b/.claude/skills @@ -0,0 +1 @@ +../../.agents/skills \ No newline at end of file diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..83df277 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,25 @@ +version: 2 +updates: + # Rust dependencies + - package-ecosystem: "cargo" + directory: "/" + schedule: + interval: "weekly" + commit-message: + prefix: "chore(deps)" + groups: + rust-dependencies: + patterns: + - "*" + + # GitHub Actions + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + commit-message: + prefix: "chore(ci)" + groups: + github-actions: + patterns: + - "*" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..fccd3ce --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,73 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +env: + CARGO_TERM_COLOR: always + RUSTFLAGS: -Dwarnings + +jobs: + lint: + name: Lint + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + with: + components: rustfmt, clippy + - uses: Swatinem/rust-cache@v2 + - name: Check formatting + run: cargo fmt --all -- --check + - name: Clippy + run: cargo clippy --workspace --all-targets -- -D warnings + - name: Build docs + run: cargo doc --workspace --no-deps + env: + RUSTDOCFLAGS: -Dwarnings + + audit: + name: Audit + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: Install cargo-audit + run: cargo install cargo-audit + - name: Install cargo-deny + run: cargo install cargo-deny + - name: Security audit + run: cargo audit + - name: License and dependency check + run: cargo deny check + + test: + name: Test + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + - name: Run tests + run: cargo test --workspace + - name: Run tests (all features) + run: cargo test --workspace --all-features + + check: + name: Check + needs: [lint, audit, test] + runs-on: ubuntu-latest + if: always() + steps: + - name: All jobs passed + run: | + if [[ "${{ needs.lint.result }}" != "success" ]] || \ + [[ "${{ needs.audit.result }}" != "success" ]] || \ + [[ "${{ needs.test.result }}" != "success" ]]; then + echo "One or more jobs failed" + exit 1 + fi diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml new file mode 100644 index 0000000..cd3d84b --- /dev/null +++ b/.github/workflows/coverage.yml @@ -0,0 +1,25 @@ +name: Coverage + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + coverage: + name: Code Coverage + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + - name: Install cargo-tarpaulin + run: cargo install cargo-tarpaulin + - name: Generate coverage + run: cargo tarpaulin --workspace --out xml + - name: Upload coverage + uses: codecov/codecov-action@v4 + with: + file: cobertura.xml + fail_ci_if_error: false diff --git a/.github/workflows/js.yml b/.github/workflows/js.yml new file mode 100644 index 0000000..2b28d1b --- /dev/null +++ b/.github/workflows/js.yml @@ -0,0 +1,28 @@ +name: JavaScript + +on: + push: + branches: [main] + paths: + - "crates/markkit-js/**" + - "crates/markkit/**" + pull_request: + branches: [main] + paths: + - "crates/markkit-js/**" + - "crates/markkit/**" + +jobs: + build: + name: Build Node.js bindings + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: actions/setup-node@v4 + with: + node-version: "20" + - name: Install dependencies + run: cd crates/markkit-js && npm install + - name: Build + run: cd crates/markkit-js && npx napi build --release diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml new file mode 100644 index 0000000..35161b5 --- /dev/null +++ b/.github/workflows/nightly.yml @@ -0,0 +1,21 @@ +name: Nightly + +on: + schedule: + - cron: "0 4 * * *" + workflow_dispatch: + +jobs: + nightly: + name: Nightly checks + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@nightly + with: + components: rustfmt, clippy + - uses: Swatinem/rust-cache@v2 + - name: Build on nightly + run: cargo build --workspace + - name: Test on nightly + run: cargo test --workspace diff --git a/.github/workflows/publish-js.yml b/.github/workflows/publish-js.yml new file mode 100644 index 0000000..74b27be --- /dev/null +++ b/.github/workflows/publish-js.yml @@ -0,0 +1,30 @@ +name: Publish JavaScript + +on: + workflow_dispatch: + inputs: + dry_run: + description: "Dry run" + required: false + default: "true" + +jobs: + publish: + name: Publish npm package + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: actions/setup-node@v4 + with: + node-version: "20" + registry-url: "https://registry.npmjs.org" + - name: Install dependencies + run: cd crates/markkit-js && npm install + - name: Build + run: cd crates/markkit-js && npx napi build --release + - name: Publish + if: inputs.dry_run != 'true' + run: cd crates/markkit-js && npm publish + env: + NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} diff --git a/.github/workflows/publish-python.yml b/.github/workflows/publish-python.yml new file mode 100644 index 0000000..4bd1890 --- /dev/null +++ b/.github/workflows/publish-python.yml @@ -0,0 +1,30 @@ +name: Publish Python + +on: + workflow_dispatch: + inputs: + dry_run: + description: "Dry run" + required: false + default: "true" + +jobs: + publish: + name: Publish Python package + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install maturin + run: pip install maturin twine + - name: Build wheels + run: cd crates/markkit-python && maturin build --release + - name: Publish + if: inputs.dry_run != 'true' + run: twine upload target/wheels/*.whl + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }} diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml new file mode 100644 index 0000000..586f8f7 --- /dev/null +++ b/.github/workflows/publish.yml @@ -0,0 +1,25 @@ +name: Publish + +on: + workflow_dispatch: + inputs: + dry_run: + description: "Dry run (no actual publish)" + required: false + default: "true" + +jobs: + publish: + name: Publish to crates.io + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - name: Publish markkit + run: cargo publish -p markkit ${{ inputs.dry_run == 'true' && '--dry-run' || '' }} + env: + CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} + - name: Publish markkit-cli + run: cargo publish -p markkit-cli ${{ inputs.dry_run == 'true' && '--dry-run' || '' }} + env: + CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }} diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml new file mode 100644 index 0000000..fd725f4 --- /dev/null +++ b/.github/workflows/python.yml @@ -0,0 +1,32 @@ +name: Python + +on: + push: + branches: [main] + paths: + - "crates/markkit-python/**" + - "crates/markkit/**" + pull_request: + branches: [main] + paths: + - "crates/markkit-python/**" + - "crates/markkit/**" + +jobs: + build: + name: Build Python bindings + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install maturin + run: pip install maturin + - name: Build + run: cd crates/markkit-python && maturin build + - name: Test import + run: | + pip install target/wheels/*.whl + python -c "import markkit_python; print('Import OK')" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml new file mode 100644 index 0000000..a2ecd8f --- /dev/null +++ b/.github/workflows/release.yml @@ -0,0 +1,21 @@ +name: Release + +on: + push: + tags: + - "v*" + +jobs: + release: + name: Create GitHub Release + runs-on: ubuntu-latest + permissions: + contents: write + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - name: Create release + uses: softprops/action-gh-release@v2 + with: + generate_release_notes: true diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3e88272 --- /dev/null +++ b/.gitignore @@ -0,0 +1,39 @@ +# Rust +/target/ +Cargo.lock + +# Environment +.env +.env.* +.envrc + +# Local files +local* + +# IDE +.vscode/ +*.swp + +# OS +.DS_Store + +# Coverage reports +coverage/ +.venv/ + +# Claude Code worktrees +.claude/worktrees/ + +# Python build artifacts +__pycache__/ +*.pyc +*.pyo +*.so +*.egg-info/ +dist/ + +# Node +node_modules/ + +# WASM build artifacts +*.wasm diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a5451f9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,33 @@ +# Markkit – Coding Agent Guidance + +Minimal tokens. Telegraph style. Drop filler. + +## Quick Reference + +- Build: `just build` +- Test: `just test` +- Lint: `just lint` +- Pre-PR: `just pre-pr` +- Specs: `specs/` directory + +## Rules + +- Fetch `origin/main` and rebase before editing +- Small PRs, incremental changes +- Conventional Commits: `type(scope): description` +- Real human git identity only +- No Claude session links in commits/PRs +- Write failing test first, then fix +- Security tests for parsing changes +- Run `just pre-pr` before submitting + +## Crate Map + +| Crate | Purpose | +|-------|---------| +| `markkit` | Core extraction library | +| `markkit-cli` | CLI tool | +| `markkit-bench` | Benchmarks | +| `markkit-eval` | LLM evaluation | +| `markkit-python` | Python bindings | +| `markkit-js` | Node.js bindings | diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..89f2dc3 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,18 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](https://keepachangelog.com/), +and this project adheres to [Semantic Versioning](https://semver.org/). + +## [Unreleased] + +### Added + +- Initial project structure +- Core markdown extraction library (`markkit` crate) +- CLI tool (`markkit-cli` crate) +- Python bindings (`markkit-python` crate) +- Node.js bindings (`markkit-js` crate) +- Benchmark harness (`markkit-bench` crate) +- LLM evaluation harness (`markkit-eval` crate) diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..e1c766d --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,44 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity and +orientation. + +## Our Standards + +Examples of behavior that contributes to a positive environment: + +- Using welcoming and inclusive language +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members + +Examples of unacceptable behavior: + +- The use of sexualized language or imagery and unwelcome sexual attention or + advances +- Trolling, insulting/derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information without explicit permission +- Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the project maintainers at **conduct@everruns.com**. + +All complaints will be reviewed and investigated promptly and fairly. The +project team is obligated to maintain confidentiality with regard to the +reporter of an incident. + +## Attribution + +This Code of Conduct is adapted from the +[Contributor Covenant](https://www.contributor-covenant.org), version 2.1. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..b5e2271 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,62 @@ +# Contributing to Markkit + +Thank you for your interest in contributing to Markkit! + +## Getting Started + +```bash +# Clone the repository +git clone https://github.com/everruns/markkit.git +cd markkit + +# Install just (task runner) +cargo install just + +# Build +just build + +# Run tests +just test +``` + +## Development Workflow + +1. Fork the repository +2. Create a feature branch: `git checkout -b feat/your-feature` +3. Make your changes +4. Run `just pre-pr` to validate +5. Submit a pull request + +## Code Quality + +- Format with `cargo fmt` +- Lint with `cargo clippy -- -D warnings` +- Run license checks with `cargo deny check` +- All tests must pass + +## Commit Convention + +We use [Conventional Commits](https://www.conventionalcommits.org/): + +- `feat(scope): description` – new feature +- `fix(scope): description` – bug fix +- `docs(scope): description` – documentation +- `test(scope): description` – test changes +- `refactor(scope): description` – code refactoring +- `chore(scope): description` – maintenance + +## Testing + +- Write failing tests before fixing bugs +- Cover positive and negative scenarios +- Security tests for input parsing changes +- Run `just bench` for performance-sensitive changes + +## Specs + +The `specs/` directory contains architecture and design documentation. New +features should either comply with existing specs or propose changes. + +## Questions? + +Open an issue on the [GitHub repository](https://github.com/everruns/markkit/issues). diff --git a/Cargo.toml b/Cargo.toml new file mode 100644 index 0000000..ae719c1 --- /dev/null +++ b/Cargo.toml @@ -0,0 +1,90 @@ +# Markkit – Extract markdown from any format +# Workspace root + +[workspace] +resolver = "2" +members = [ + "crates/markkit", + "crates/markkit-cli", + "crates/markkit-bench", + "crates/markkit-eval", + "crates/markkit-python", + "crates/markkit-js", +] + +[workspace.package] +version = "0.1.0" +edition = "2024" +license = "MIT" +authors = ["Everruns "] +repository = "https://github.com/everruns/markkit" +keywords = ["markdown", "extraction", "conversion", "html", "pdf"] +categories = ["text-processing", "command-line-utilities", "parser-implementations"] + +[workspace.dependencies] +# Async +tokio = { version = "1", default-features = false } +async-trait = "0.1" +futures = "0.3" + +# Error handling +thiserror = "2" +anyhow = "1" + +# Serialization +serde = { version = "1", features = ["derive"] } +serde_json = "1" + +# CLI +clap = { version = "4", features = ["derive"] } + +# HTTP +reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"] } + +# HTML parsing +scraper = "0.26" +html5ever = "0.39" +lol_html = "2" + +# Text processing +regex = "1" +unicode-segmentation = "1" + +# PDF +pdf-extract = "0.10" + +# Document formats +calamine = "0.34" # Excel/spreadsheet +docx-rs = "0.4" # DOCX +zip = "2" # Archive extraction + +# Image / OCR (optional) +image = "0.25" + +# Encoding +encoding_rs = "0.8" +base64 = "0.22" +mime = "0.3" + +# Logging +tracing = "0.1" +tracing-subscriber = { version = "0.3", features = ["env-filter"] } + +# Python bindings +pyo3 = { version = "0.28" } +pyo3-async-runtimes = { version = "0.28", features = ["tokio-runtime"] } + +# Node.js bindings +napi = { version = "3", features = ["async", "serde-json"] } +napi-derive = "3" + +# Testing +tempfile = "3" +criterion = { version = "0.5", features = ["html_reports"] } +proptest = "1" + +[profile.release] +lto = "thin" +codegen-units = 1 +panic = "abort" +strip = true diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..40cd13e --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024-2026 Everruns + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..b0d3111 --- /dev/null +++ b/NOTICE @@ -0,0 +1,37 @@ +Markkit +Copyright 2024-2026 Everruns + +Licensed under the MIT License. + +=== Design Inspirations === + +Markkit is an independent implementation. The following projects +inspired aspects of its design: + +* Readability (Apache 2.0) – Mozilla's approach to content extraction + informed our HTML-to-markdown pipeline design. + +* Pandoc (GPL-2.0) – The universal document converter's format + coverage and conversion philosophy inspired our multi-format + architecture. + +* turndown (MIT) – The JavaScript HTML-to-markdown converter's + rule-based approach influenced our conversion engine design. + +Markkit's parser, converter, and extraction pipeline are original +Rust implementations. This project is not affiliated with, endorsed +by, or derived from any of the above projects. + +=== Key Dependencies === + +Markkit relies on several permissively licensed crates: + +* scraper (MIT) – HTML parsing and CSS selector matching +* html5ever (MIT/Apache-2.0) – HTML5 parsing +* lol_html (BSD-3-Clause) – Low-latency HTML rewriting +* pdf-extract (MIT) – PDF text extraction +* reqwest (MIT/Apache-2.0) – HTTP client +* tokio (MIT) – Async runtime +* serde / serde_json (MIT/Apache-2.0) – Serialization + +Full dependency license details are available via `cargo deny check licenses`. diff --git a/README.md b/README.md index 3ca4983..8333a65 100644 --- a/README.md +++ b/README.md @@ -1 +1,127 @@ -# markkit \ No newline at end of file +# Markkit + +Extract clean markdown from any format. + +Markkit is a Rust library and CLI tool for converting HTML, PDF, DOCX, spreadsheets, and other document formats into well-structured markdown. It provides native bindings for Python and Node.js. + +## Features + +- **HTML** – Convert web pages and HTML documents to markdown +- **PDF** – Extract text from PDF documents +- **DOCX** – Convert Word documents +- **Spreadsheets** – Convert Excel/ODS files to markdown tables +- **URL fetching** – Fetch and extract from URLs directly +- **Configurable** – CSS selectors for include/exclude, resource limits +- **Safe** – Input sanitization, resource limits, no code execution + +## Quick Start + +### Rust Library + +```rust +use markkit::Markkit; + +let markkit = Markkit::builder().build(); +let md = markkit.extract_html("

Hello

World

").await?; +``` + +### CLI + +```bash +# From a URL +markkit extract https://example.com + +# From a file +markkit extract --file document.html + +# With CSS selector filtering +markkit extract https://example.com --include "article" --exclude "nav,footer" + +# JSON output with metadata +markkit extract https://example.com --json +``` + +### Python + +```python +import markkit_python + +markdown = await markkit_python.extract_html("

Hello

") +markdown = await markkit_python.extract_url("https://example.com") +``` + +### Node.js + +```javascript +const { extractHtml, extractUrl } = require('@everruns/markkit'); + +const markdown = await extractHtml('

Hello

'); +const markdown = await extractUrl('https://example.com'); +``` + +## Installation + +### Rust + +```toml +[dependencies] +markkit = "0.1" +``` + +### CLI + +```bash +cargo install markkit-cli +``` + +### Python + +```bash +pip install markkit +``` + +### Node.js + +```bash +npm install @everruns/markkit +``` + +## Configuration + +```rust +let markkit = Markkit::builder() + .max_input_size(10 * 1024 * 1024) // 10MB limit + .max_processing_secs(30) + .preserve_links(true) + .preserve_images(false) + .include_selector("article") + .exclude_selector("nav") + .exclude_selector(".sidebar") + .build(); +``` + +## Feature Flags + +| Feature | Default | Description | +|---------|---------|-------------| +| `html` | Yes | HTML parsing and conversion | +| `http_client` | Yes | URL fetching via reqwest | +| `pdf` | No | PDF text extraction | +| `docx` | No | DOCX conversion | +| `spreadsheet` | No | Excel/ODS conversion | +| `all_formats` | No | Enable all format support | +| `logging` | No | Structured tracing output | + +## Development + +```bash +just build # Build workspace +just test # Run all tests +just lint # Run clippy +just pre-pr # Full pre-PR validation +just bench # Run benchmarks +``` + +## License + +MIT License. See [LICENSE](LICENSE) for details. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..bac4de5 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,45 @@ +# Security Policy + +## Reporting a Vulnerability + +**Please do not report security vulnerabilities through public GitHub issues.** + +Instead, please report them via email to **security@everruns.com**. + +Please include: + +- Description of the vulnerability +- Steps to reproduce +- Potential impact +- Suggested fix (if any) + +## Response Timeline + +- **Acknowledgment**: Within 48 hours +- **Initial assessment**: Within 1 week +- **Critical fixes**: Target 30 days + +## Supported Versions + +| Version | Supported | +|---------|-----------| +| 0.1.x | Yes | + +## Security Architecture + +Markkit processes untrusted input (HTML, PDF, DOCX, etc.) and converts it to markdown. Key security considerations: + +- **Input sanitization**: All parsed content is sanitized before output +- **Resource limits**: Configurable limits on input size and processing time +- **No code execution**: Extracted content is treated as data, never executed +- **Network isolation**: URL fetching is opt-in with configurable allowlists +- **Memory budgets**: Configurable memory limits to prevent DoS via large documents + +## Scope + +The following components are in scope for security reports: + +- `markkit` core library crate +- `markkit-cli` command-line tool +- Official Python and Node.js bindings +- Published documentation diff --git a/crates/markkit-bench/Cargo.toml b/crates/markkit-bench/Cargo.toml new file mode 100644 index 0000000..b6c978e --- /dev/null +++ b/crates/markkit-bench/Cargo.toml @@ -0,0 +1,31 @@ +# Benchmark tool for measuring markkit extraction performance + +[package] +name = "markkit-bench" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +description = "Benchmark tool for markkit markdown extraction" + +[[bin]] +name = "markkit-bench" +path = "src/main.rs" + +[dependencies] +markkit = { path = "../markkit", features = ["all_formats", "http_client"] } +tokio = { workspace = true, features = ["rt-multi-thread"] } +serde.workspace = true +serde_json.workspace = true +clap.workspace = true +anyhow.workspace = true + +# Timing +instant = "0.1" + +# Statistics +statrs = "0.18" + +# Terminal output +colored = "3" +tabled = "0.20" diff --git a/crates/markkit-bench/src/main.rs b/crates/markkit-bench/src/main.rs new file mode 100644 index 0000000..91f4396 --- /dev/null +++ b/crates/markkit-bench/src/main.rs @@ -0,0 +1,21 @@ +use anyhow::Result; +use clap::Parser; + +#[derive(Parser)] +#[command(name = "markkit-bench", about = "Benchmark markkit extraction")] +struct Cli { + /// Number of iterations per benchmark + #[arg(long, default_value = "100")] + iterations: usize, + + /// Category to benchmark (html, pdf, docx, all) + #[arg(long, default_value = "all")] + category: String, +} + +fn main() -> Result<()> { + let _cli = Cli::parse(); + // TODO: Implement benchmark harness + println!("markkit-bench: benchmark harness not yet implemented"); + Ok(()) +} diff --git a/crates/markkit-cli/Cargo.toml b/crates/markkit-cli/Cargo.toml new file mode 100644 index 0000000..89cf8bd --- /dev/null +++ b/crates/markkit-cli/Cargo.toml @@ -0,0 +1,37 @@ +# Markkit CLI – Command line interface for markkit +# Extract markdown from files and URLs + +[package] +name = "markkit-cli" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true +keywords.workspace = true +categories.workspace = true +description = "Command line interface for Markkit markdown extraction" +readme = "../../README.md" + +[[bin]] +name = "markkit" +path = "src/main.rs" +doc = false + +[features] +default = ["all_formats"] +all_formats = ["markkit/all_formats"] +pdf = ["markkit/pdf"] +docx = ["markkit/docx"] +spreadsheet = ["markkit/spreadsheet"] + +[dependencies] +markkit = { path = "../markkit", version = "0.1.0", features = ["http_client"] } +tokio = { workspace = true, features = ["macros", "net", "rt", "rt-multi-thread", "time", "fs"] } +clap.workspace = true +anyhow.workspace = true +serde.workspace = true +serde_json.workspace = true + +[dev-dependencies] +tempfile = "3" diff --git a/crates/markkit-cli/src/main.rs b/crates/markkit-cli/src/main.rs new file mode 100644 index 0000000..c8b52e2 --- /dev/null +++ b/crates/markkit-cli/src/main.rs @@ -0,0 +1,112 @@ +use anyhow::Result; +use clap::{Parser, Subcommand}; +use markkit::Markkit; + +#[derive(Parser)] +#[command( + name = "markkit", + about = "Extract clean markdown from any format", + version +)] +struct Cli { + #[command(subcommand)] + command: Commands, +} + +#[derive(Subcommand)] +enum Commands { + /// Extract markdown from a URL or file + Extract { + /// URL or file path to extract from + source: String, + + /// Treat source as a file path instead of URL + #[arg(long)] + file: bool, + + /// CSS selector to include (can be repeated) + #[arg(long = "include", short = 'i')] + include_selectors: Vec, + + /// CSS selector to exclude (can be repeated) + #[arg(long = "exclude", short = 'e')] + exclude_selectors: Vec, + + /// Maximum input size in bytes + #[arg(long, default_value = "52428800")] + max_size: usize, + + /// Output as JSON with metadata + #[arg(long)] + json: bool, + }, +} + +#[tokio::main] +async fn main() -> Result<()> { + let cli = Cli::parse(); + + match cli.command { + Commands::Extract { + source, + file, + include_selectors, + exclude_selectors, + max_size, + json: output_json, + } => { + let mut builder = Markkit::builder().max_input_size(max_size); + + for sel in &include_selectors { + builder = builder.include_selector(sel); + } + for sel in &exclude_selectors { + builder = builder.exclude_selector(sel); + } + + let markkit = builder.build(); + + let markdown = if file { + let bytes = tokio::fs::read(&source).await?; + let ext = std::path::Path::new(&source) + .extension() + .and_then(|e| e.to_str()) + .unwrap_or(""); + let content_type = match ext { + "html" | "htm" => Some("text/html"), + "pdf" => Some("application/pdf"), + "docx" => Some( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ), + _ => None, + }; + markkit.extract_bytes(&bytes, content_type).await? + } else { + markkit.extract_url(&source).await? + }; + + if output_json { + let output = serde_json::json!({ + "source": source, + "markdown": markdown, + }); + println!("{}", serde_json::to_string_pretty(&output)?); + } else { + print!("{markdown}"); + } + } + } + + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use clap::CommandFactory; + + #[test] + fn verify_cli() { + Cli::command().debug_assert(); + } +} diff --git a/crates/markkit-eval/Cargo.toml b/crates/markkit-eval/Cargo.toml new file mode 100644 index 0000000..d904ace --- /dev/null +++ b/crates/markkit-eval/Cargo.toml @@ -0,0 +1,25 @@ +# LLM evaluation harness for markkit extraction quality +# See specs/012-eval.md for design + +[package] +name = "markkit-eval" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +description = "LLM evaluation harness for markkit extraction quality" + +[[bin]] +name = "markkit-eval" +path = "src/main.rs" + +[dependencies] +markkit = { path = "../markkit", features = ["all_formats", "http_client"] } +tokio = { workspace = true, features = ["rt-multi-thread", "macros"] } +serde.workspace = true +serde_json.workspace = true +clap.workspace = true +anyhow.workspace = true +reqwest.workspace = true +regex.workspace = true +async-trait.workspace = true diff --git a/crates/markkit-eval/src/main.rs b/crates/markkit-eval/src/main.rs new file mode 100644 index 0000000..e76d643 --- /dev/null +++ b/crates/markkit-eval/src/main.rs @@ -0,0 +1,22 @@ +use anyhow::Result; +use clap::Parser; + +#[derive(Parser)] +#[command(name = "markkit-eval", about = "Evaluate markkit extraction quality")] +struct Cli { + /// Provider (anthropic, openai) + #[arg(long, default_value = "anthropic")] + provider: String, + + /// Evaluation dataset path + #[arg(long)] + dataset: Option, +} + +#[tokio::main] +async fn main() -> Result<()> { + let _cli = Cli::parse(); + // TODO: Implement evaluation harness + println!("markkit-eval: evaluation harness not yet implemented"); + Ok(()) +} diff --git a/crates/markkit-js/Cargo.toml b/crates/markkit-js/Cargo.toml new file mode 100644 index 0000000..fd26309 --- /dev/null +++ b/crates/markkit-js/Cargo.toml @@ -0,0 +1,25 @@ +# Markkit Node.js bindings via NAPI-RS +# Expose markkit as a native Node.js module + +[package] +name = "markkit-js" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +description = "Node.js bindings for Markkit markdown extraction via NAPI-RS" + +[lib] +crate-type = ["cdylib"] +doc = false + +[dependencies] +markkit = { path = "../markkit", features = ["all_formats", "http_client"] } +napi.workspace = true +napi-derive.workspace = true +serde.workspace = true +serde_json.workspace = true +tokio = { workspace = true, features = ["rt-multi-thread", "macros"] } + +[build-dependencies] +napi-build = "2" diff --git a/crates/markkit-js/build.rs b/crates/markkit-js/build.rs new file mode 100644 index 0000000..9fc2367 --- /dev/null +++ b/crates/markkit-js/build.rs @@ -0,0 +1,5 @@ +extern crate napi_build; + +fn main() { + napi_build::setup(); +} diff --git a/crates/markkit-js/package.json b/crates/markkit-js/package.json new file mode 100644 index 0000000..1d1df88 --- /dev/null +++ b/crates/markkit-js/package.json @@ -0,0 +1,25 @@ +{ + "name": "@everruns/markkit", + "version": "0.1.0", + "description": "Extract clean markdown from any format – Node.js bindings", + "main": "index.js", + "types": "index.d.ts", + "napi": { + "name": "markkit", + "triples": { + "defaults": true + } + }, + "license": "MIT", + "repository": { + "type": "git", + "url": "https://github.com/everruns/markkit" + }, + "files": [ + "index.js", + "index.d.ts" + ], + "devDependencies": { + "@napi-rs/cli": "^3" + } +} diff --git a/crates/markkit-js/src/lib.rs b/crates/markkit-js/src/lib.rs new file mode 100644 index 0000000..0f049ce --- /dev/null +++ b/crates/markkit-js/src/lib.rs @@ -0,0 +1,31 @@ +use napi_derive::napi; + +/// Extract markdown from an HTML string. +#[napi] +pub async fn extract_html(html: String) -> napi::Result { + let m = markkit::Markkit::builder().build(); + m.extract_html(&html) + .await + .map_err(|e| napi::Error::from_reason(e.to_string())) +} + +/// Extract markdown from a URL. +#[napi] +pub async fn extract_url(url: String) -> napi::Result { + let m = markkit::Markkit::builder().build(); + m.extract_url(&url) + .await + .map_err(|e| napi::Error::from_reason(e.to_string())) +} + +/// Extract markdown from a buffer with optional content type. +#[napi] +pub async fn extract_bytes( + data: napi::bindgen_prelude::Buffer, + content_type: Option, +) -> napi::Result { + let m = markkit::Markkit::builder().build(); + m.extract_bytes(&data, content_type.as_deref()) + .await + .map_err(|e| napi::Error::from_reason(e.to_string())) +} diff --git a/crates/markkit-python/Cargo.toml b/crates/markkit-python/Cargo.toml new file mode 100644 index 0000000..c0d6f49 --- /dev/null +++ b/crates/markkit-python/Cargo.toml @@ -0,0 +1,22 @@ +# Markkit Python bindings via PyO3 +# Expose markkit as a Python module + +[package] +name = "markkit-python" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +description = "Python bindings for Markkit markdown extraction via PyO3" + +[lib] +name = "markkit_python" +crate-type = ["cdylib"] +doc = false + +[dependencies] +markkit = { path = "../markkit", features = ["all_formats", "http_client"] } +pyo3.workspace = true +pyo3-async-runtimes.workspace = true +tokio = { workspace = true, features = ["rt-multi-thread", "macros"] } +serde_json.workspace = true diff --git a/crates/markkit-python/src/lib.rs b/crates/markkit-python/src/lib.rs new file mode 100644 index 0000000..6320465 --- /dev/null +++ b/crates/markkit-python/src/lib.rs @@ -0,0 +1,46 @@ +use pyo3::prelude::*; + +/// Extract markdown from an HTML string. +#[pyfunction] +fn extract_html(py: Python<'_>, html: String) -> PyResult> { + pyo3_async_runtimes::tokio::future_into_py(py, async move { + let m = markkit::Markkit::builder().build(); + m.extract_html(&html) + .await + .map_err(|e| PyErr::new::(e.to_string())) + }) +} + +/// Extract markdown from a URL. +#[pyfunction] +fn extract_url(py: Python<'_>, url: String) -> PyResult> { + pyo3_async_runtimes::tokio::future_into_py(py, async move { + let m = markkit::Markkit::builder().build(); + m.extract_url(&url) + .await + .map_err(|e| PyErr::new::(e.to_string())) + }) +} + +/// Extract markdown from bytes with optional content type. +#[pyfunction] +fn extract_bytes( + py: Python<'_>, + data: Vec, + content_type: Option, +) -> PyResult> { + pyo3_async_runtimes::tokio::future_into_py(py, async move { + let m = markkit::Markkit::builder().build(); + m.extract_bytes(&data, content_type.as_deref()) + .await + .map_err(|e| PyErr::new::(e.to_string())) + }) +} + +#[pymodule] +fn markkit_python(m: &Bound<'_, PyModule>) -> PyResult<()> { + m.add_function(wrap_pyfunction!(extract_html, m)?)?; + m.add_function(wrap_pyfunction!(extract_url, m)?)?; + m.add_function(wrap_pyfunction!(extract_bytes, m)?)?; + Ok(()) +} diff --git a/crates/markkit/Cargo.toml b/crates/markkit/Cargo.toml new file mode 100644 index 0000000..09afd79 --- /dev/null +++ b/crates/markkit/Cargo.toml @@ -0,0 +1,79 @@ +# Markkit – Core markdown extraction library +# Extract clean markdown from HTML, PDF, DOCX, and more + +[package] +name = "markkit" +version.workspace = true +edition.workspace = true +license.workspace = true +authors.workspace = true +repository.workspace = true +keywords.workspace = true +categories.workspace = true +description = "Extract clean markdown from any format – HTML, PDF, DOCX, and more" +readme = "../../README.md" + +[features] +default = ["html", "http_client"] +html = ["dep:scraper", "dep:html5ever", "dep:lol_html"] +pdf = ["dep:pdf-extract"] +docx = ["dep:docx-rs", "dep:zip"] +spreadsheet = ["dep:calamine"] +http_client = ["dep:reqwest"] +logging = ["dep:tracing", "dep:tracing-subscriber"] +all_formats = ["html", "pdf", "docx", "spreadsheet"] + +[dependencies] +# Async +tokio = { workspace = true, features = ["sync", "io-util"] } +async-trait.workspace = true +futures.workspace = true + +# Error handling +thiserror.workspace = true +anyhow.workspace = true + +# Serialization +serde.workspace = true +serde_json.workspace = true + +# Text processing +regex.workspace = true +unicode-segmentation.workspace = true + +# Encoding +encoding_rs.workspace = true +base64.workspace = true +mime.workspace = true + +# HTML (optional) +scraper = { workspace = true, optional = true } +html5ever = { workspace = true, optional = true } +lol_html = { workspace = true, optional = true } + +# PDF (optional) +pdf-extract = { workspace = true, optional = true } + +# DOCX (optional) +docx-rs = { workspace = true, optional = true } +zip = { workspace = true, optional = true } + +# Spreadsheet (optional) +calamine = { workspace = true, optional = true } + +# HTTP client (optional) +reqwest = { workspace = true, optional = true } + +# Logging (optional) +tracing = { workspace = true, optional = true } +tracing-subscriber = { workspace = true, optional = true } + +[dev-dependencies] +tokio = { workspace = true, features = ["macros", "rt-multi-thread"] } +tempfile.workspace = true +proptest.workspace = true +criterion.workspace = true + +[[bench]] +name = "extraction" +harness = false diff --git a/crates/markkit/benches/extraction.rs b/crates/markkit/benches/extraction.rs new file mode 100644 index 0000000..a147f0b --- /dev/null +++ b/crates/markkit/benches/extraction.rs @@ -0,0 +1,18 @@ +use criterion::{Criterion, criterion_group, criterion_main}; + +fn bench_format_detection(c: &mut Criterion) { + c.bench_function("format_from_content_type", |b| { + b.iter(|| { + markkit::format::Format::from_content_type("text/html; charset=utf-8"); + }); + }); + + c.bench_function("format_from_extension", |b| { + b.iter(|| { + markkit::format::Format::from_extension("html"); + }); + }); +} + +criterion_group!(benches, bench_format_detection); +criterion_main!(benches); diff --git a/crates/markkit/src/config.rs b/crates/markkit/src/config.rs new file mode 100644 index 0000000..a98386a --- /dev/null +++ b/crates/markkit/src/config.rs @@ -0,0 +1,77 @@ +use crate::extract::Markkit; + +/// Builder for configuring a Markkit instance. +#[derive(Debug, Clone)] +pub struct MarkkitBuilder { + /// Maximum input size in bytes (default: 50MB). + pub max_input_size: usize, + /// Maximum processing time in seconds (default: 60). + pub max_processing_secs: u64, + /// Whether to preserve link URLs in output. + pub preserve_links: bool, + /// Whether to preserve image references in output. + pub preserve_images: bool, + /// Whether to extract content from code blocks. + pub extract_code_blocks: bool, + /// CSS selectors for content to include (empty = include all). + pub include_selectors: Vec, + /// CSS selectors for content to exclude. + pub exclude_selectors: Vec, +} + +impl Default for MarkkitBuilder { + fn default() -> Self { + Self { + max_input_size: 50 * 1024 * 1024, + max_processing_secs: 60, + preserve_links: true, + preserve_images: true, + extract_code_blocks: true, + include_selectors: Vec::new(), + exclude_selectors: vec![ + "nav".into(), + "footer".into(), + "header".into(), + "script".into(), + "style".into(), + "noscript".into(), + ], + } + } +} + +impl MarkkitBuilder { + pub fn max_input_size(mut self, size: usize) -> Self { + self.max_input_size = size; + self + } + + pub fn max_processing_secs(mut self, secs: u64) -> Self { + self.max_processing_secs = secs; + self + } + + pub fn preserve_links(mut self, yes: bool) -> Self { + self.preserve_links = yes; + self + } + + pub fn preserve_images(mut self, yes: bool) -> Self { + self.preserve_images = yes; + self + } + + pub fn include_selector(mut self, selector: impl Into) -> Self { + self.include_selectors.push(selector.into()); + self + } + + pub fn exclude_selector(mut self, selector: impl Into) -> Self { + self.exclude_selectors.push(selector.into()); + self + } + + pub fn build(self) -> Markkit { + Markkit::new(self) + } +} diff --git a/crates/markkit/src/error.rs b/crates/markkit/src/error.rs new file mode 100644 index 0000000..c291664 --- /dev/null +++ b/crates/markkit/src/error.rs @@ -0,0 +1,30 @@ +/// Errors that can occur during markdown extraction. +#[derive(Debug, thiserror::Error)] +pub enum Error { + #[error("HTML parsing error: {0}")] + HtmlParse(String), + + #[error("PDF extraction error: {0}")] + PdfExtract(String), + + #[error("DOCX extraction error: {0}")] + DocxExtract(String), + + #[error("Spreadsheet extraction error: {0}")] + SpreadsheetExtract(String), + + #[error("HTTP error: {0}")] + Http(String), + + #[error("Encoding error: {0}")] + Encoding(String), + + #[error("Unsupported format: {0}")] + UnsupportedFormat(String), + + #[error("Resource limit exceeded: {0}")] + ResourceLimit(String), + + #[error("I/O error: {0}")] + Io(#[from] std::io::Error), +} diff --git a/crates/markkit/src/extract.rs b/crates/markkit/src/extract.rs new file mode 100644 index 0000000..c8be920 --- /dev/null +++ b/crates/markkit/src/extract.rs @@ -0,0 +1,119 @@ +use crate::config::MarkkitBuilder; +use crate::error::Error; + +/// Core extraction engine. +/// +/// Use [`MarkkitBuilder`] to create a configured instance. +pub struct Markkit { + config: MarkkitBuilder, +} + +impl Markkit { + /// Create a new Markkit instance with the given configuration. + pub fn new(config: MarkkitBuilder) -> Self { + Self { config } + } + + /// Create a builder for configuring extraction. + pub fn builder() -> MarkkitBuilder { + MarkkitBuilder::default() + } + + /// Extract markdown from an HTML string. + pub async fn extract_html(&self, html: &str) -> crate::Result { + if html.len() > self.config.max_input_size { + return Err(Error::ResourceLimit(format!( + "Input size {} exceeds limit {}", + html.len(), + self.config.max_input_size + ))); + } + + // TODO: Implement HTML-to-markdown conversion + Ok(String::new()) + } + + /// Extract markdown from raw bytes, auto-detecting the format. + pub async fn extract_bytes( + &self, + bytes: &[u8], + content_type: Option<&str>, + ) -> crate::Result { + if bytes.len() > self.config.max_input_size { + return Err(Error::ResourceLimit(format!( + "Input size {} exceeds limit {}", + bytes.len(), + self.config.max_input_size + ))); + } + + let mime = content_type.unwrap_or("application/octet-stream"); + + match mime { + m if m.contains("html") => { + let text = String::from_utf8_lossy(bytes); + self.extract_html(&text).await + } + m if m.contains("pdf") => self.extract_pdf(bytes).await, + _ => Err(Error::UnsupportedFormat(mime.to_string())), + } + } + + /// Extract markdown from a URL by fetching and converting. + #[cfg(feature = "http_client")] + pub async fn extract_url(&self, url: &str) -> crate::Result { + let response = reqwest::get(url) + .await + .map_err(|e| Error::Http(e.to_string()))?; + + let content_type = response + .headers() + .get("content-type") + .and_then(|v| v.to_str().ok()) + .map(|s| s.to_string()); + + let bytes = response + .bytes() + .await + .map_err(|e| Error::Http(e.to_string()))?; + + self.extract_bytes(&bytes, content_type.as_deref()).await + } + + /// Extract markdown from PDF bytes. + async fn extract_pdf(&self, _bytes: &[u8]) -> crate::Result { + #[cfg(feature = "pdf")] + { + // TODO: Implement PDF extraction + Ok(String::new()) + } + #[cfg(not(feature = "pdf"))] + Err(Error::UnsupportedFormat( + "PDF support requires the 'pdf' feature".to_string(), + )) + } + + /// Access the current configuration. + pub fn config(&self) -> &MarkkitBuilder { + &self.config + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn test_builder_defaults() { + let m = Markkit::builder().build(); + assert_eq!(m.config().max_input_size, 50 * 1024 * 1024); + assert!(m.config().preserve_links); + } + + #[tokio::test] + async fn test_resource_limit() { + let m = Markkit::builder().max_input_size(10).build(); + let result = m.extract_html("hello world!").await; + assert!(matches!(result, Err(Error::ResourceLimit(_)))); + } +} diff --git a/crates/markkit/src/format.rs b/crates/markkit/src/format.rs new file mode 100644 index 0000000..ead1d3a --- /dev/null +++ b/crates/markkit/src/format.rs @@ -0,0 +1,74 @@ +//! Format detection and content type handling. + +/// Detected document format. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Format { + Html, + Pdf, + Docx, + Xlsx, + Csv, + PlainText, + Markdown, + Unknown, +} + +impl Format { + /// Detect format from a MIME content type string. + pub fn from_content_type(content_type: &str) -> Self { + let ct = content_type.to_lowercase(); + if ct.contains("text/html") || ct.contains("application/xhtml") { + Format::Html + } else if ct.contains("application/pdf") { + Format::Pdf + } else if ct.contains("wordprocessingml") || ct.contains("msword") { + Format::Docx + } else if ct.contains("spreadsheetml") || ct.contains("ms-excel") { + Format::Xlsx + } else if ct.contains("text/csv") { + Format::Csv + } else if ct.contains("text/plain") { + Format::PlainText + } else if ct.contains("text/markdown") { + Format::Markdown + } else { + Format::Unknown + } + } + + /// Detect format from a file extension. + pub fn from_extension(ext: &str) -> Self { + match ext.to_lowercase().as_str() { + "html" | "htm" | "xhtml" => Format::Html, + "pdf" => Format::Pdf, + "docx" | "doc" => Format::Docx, + "xlsx" | "xls" | "ods" => Format::Xlsx, + "csv" | "tsv" => Format::Csv, + "txt" => Format::PlainText, + "md" | "markdown" => Format::Markdown, + _ => Format::Unknown, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_content_type_detection() { + assert_eq!( + Format::from_content_type("text/html; charset=utf-8"), + Format::Html + ); + assert_eq!(Format::from_content_type("application/pdf"), Format::Pdf); + assert_eq!(Format::from_content_type("text/plain"), Format::PlainText); + } + + #[test] + fn test_extension_detection() { + assert_eq!(Format::from_extension("html"), Format::Html); + assert_eq!(Format::from_extension("PDF"), Format::Pdf); + assert_eq!(Format::from_extension("docx"), Format::Docx); + } +} diff --git a/crates/markkit/src/lib.rs b/crates/markkit/src/lib.rs new file mode 100644 index 0000000..36e58a4 --- /dev/null +++ b/crates/markkit/src/lib.rs @@ -0,0 +1,44 @@ +//! # Markkit +//! +//! Extract clean markdown from any format. +//! +//! Markkit is a Rust library for converting HTML, PDF, DOCX, spreadsheets, +//! and other document formats into clean, well-structured markdown. +//! +//! ## Quick Start +//! +//! ```rust,no_run +//! use markkit::Markkit; +//! +//! # async fn example() -> anyhow::Result<()> { +//! let markkit = Markkit::builder().build(); +//! +//! // Extract markdown from HTML +//! let md = markkit.extract_html("

Hello

World

").await?; +//! assert!(md.contains("# Hello")); +//! +//! // Extract from a URL +//! let md = markkit.extract_url("https://example.com").await?; +//! # Ok(()) +//! # } +//! ``` +//! +//! ## Features +//! +//! - **HTML**: Convert HTML to markdown with configurable rules +//! - **PDF**: Extract text from PDF documents (requires `pdf` feature) +//! - **DOCX**: Convert Word documents (requires `docx` feature) +//! - **Spreadsheet**: Convert Excel/ODS files (requires `spreadsheet` feature) +//! - **URL fetching**: Fetch and extract from URLs (requires `http_client` feature) + +pub mod config; +pub mod error; +pub mod extract; +pub mod format; + +pub use config::MarkkitBuilder; +pub use error::Error; +pub use extract::Markkit; + +/// Result type alias for markkit operations. +pub type Result = std::result::Result; diff --git a/deny.toml b/deny.toml new file mode 100644 index 0000000..eed71ab --- /dev/null +++ b/deny.toml @@ -0,0 +1,32 @@ +[advisories] +ignore = [ + { id = "RUSTSEC-2024-0436", reason = "paste is a transitive dep of statrs/nalgebra, no security impact" }, + { id = "RUSTSEC-2024-0384", reason = "instant is a transitive dep of markkit-bench, no security impact" }, +] + +[licenses] +allow = [ + "MIT", + "Apache-2.0", + "BSD-2-Clause", + "BSD-3-Clause", + "ISC", + "Unicode-3.0", + "Unicode-DFS-2016", + "Zlib", + "OpenSSL", + "BSL-1.0", + "MPL-2.0", + "CDLA-Permissive-2.0", + "Apache-2.0 WITH LLVM-exception", +] + +[bans] +multiple-versions = "warn" +wildcards = "allow" + +[sources] +unknown-registry = "warn" +unknown-git = "warn" +allow-registry = ["https://github.com/rust-lang/crates.io-index"] +allow-git = [] diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..b207219 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,56 @@ +# Getting Started with Markkit + +## Installation + +Add markkit to your `Cargo.toml`: + +```toml +[dependencies] +markkit = "0.1" +``` + +For additional format support: + +```toml +[dependencies] +markkit = { version = "0.1", features = ["all_formats"] } +``` + +## Basic Usage + +```rust +use markkit::Markkit; + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + let markkit = Markkit::builder().build(); + + // Extract from HTML + let html = "

Title

Some content here.

"; + let markdown = markkit.extract_html(html).await?; + println!("{markdown}"); + + // Extract from a URL + let markdown = markkit.extract_url("https://example.com").await?; + println!("{markdown}"); + + Ok(()) +} +``` + +## Configuration + +```rust +let markkit = Markkit::builder() + .max_input_size(10 * 1024 * 1024) // 10MB + .preserve_links(true) + .include_selector("article") + .exclude_selector("nav") + .build(); +``` + +## Next Steps + +- See `specs/002-html-extraction.md` for HTML conversion details +- See `specs/003-format-support.md` for supported formats +- See `specs/005-security.md` for security considerations diff --git a/justfile b/justfile new file mode 100644 index 0000000..739986b --- /dev/null +++ b/justfile @@ -0,0 +1,111 @@ +# Markkit – Extract markdown from any format +# Run `just` to see available recipes + +set dotenv-load := false + +# Default recipe: list all available commands +default: + @just --list + +# ─── Build & Check ────────────────────────────────────────── + +# Build the workspace +build: + cargo build --workspace + +# Build in release mode +build-release: + cargo build --workspace --release + +# Run all checks (format, lint, test) +check: fmt-check lint test + +# Format code +fmt: + cargo fmt --all + +# Check formatting without modifying +fmt-check: + cargo fmt --all -- --check + +# Lint with clippy +lint: + cargo clippy --workspace --all-targets -- -D warnings + +# ─── Test ──────────────────────────────────────────────────── + +# Run all tests +test: + cargo test --workspace + +# Run tests with all features +test-all: + cargo test --workspace --all-features + +# Run a specific test +test-one name: + cargo test --workspace -- {{name}} + +# ─── Run ───────────────────────────────────────────────────── + +# Run the CLI with arguments +run *args: + cargo run -p markkit-cli -- {{args}} + +# Extract markdown from a URL +extract url: + cargo run -p markkit-cli -- extract {{url}} + +# Extract markdown from a file +extract-file path: + cargo run -p markkit-cli -- extract --file {{path}} + +# ─── Benchmarks ────────────────────────────────────────────── + +# Run benchmarks +bench *args: + cargo run -p markkit-bench --release -- {{args}} + +# ─── Eval ──────────────────────────────────────────────────── + +# Run LLM evaluation harness +eval *args: + cargo run -p markkit-eval --release -- {{args}} + +# ─── Security & Audit ─────────────────────────────────────── + +# Run cargo-deny checks +deny: + cargo deny check + +# Run cargo-audit +audit: + cargo audit + +# Run cargo-vet +vet: + cargo vet + +# ─── Release ──────────────────────────────────────────────── + +# Pre-PR checklist +pre-pr: fmt lint test deny + @echo "All pre-PR checks passed" + +# Check current version +version: + @grep '^version' Cargo.toml | head -1 | cut -d'"' -f2 + +# Dry-run publish +publish-dry-run: + cargo publish -p markkit --dry-run + +# ─── Docs ─────────────────────────────────────────────────── + +# Build documentation +docs: + cargo doc --workspace --no-deps --open + +# Build documentation (no open) +docs-build: + cargo doc --workspace --no-deps diff --git a/rust-toolchain.toml b/rust-toolchain.toml new file mode 100644 index 0000000..73cb934 --- /dev/null +++ b/rust-toolchain.toml @@ -0,0 +1,3 @@ +[toolchain] +channel = "stable" +components = ["rustfmt", "clippy"] diff --git a/scripts/init-cloud-env.sh b/scripts/init-cloud-env.sh new file mode 100755 index 0000000..f1802d8 --- /dev/null +++ b/scripts/init-cloud-env.sh @@ -0,0 +1,90 @@ +#!/usr/bin/env bash +# Initialize cloud agent environment +# Installs: just, gh, doppler + +set -euo pipefail + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +info() { echo -e "${GREEN}[info]${NC} $*"; } +warn() { echo -e "${YELLOW}[warn]${NC} $*"; } +error() { echo -e "${RED}[error]${NC} $*" >&2; } + +ARCH=$(uname -m) +case "$ARCH" in + x86_64) ARCH_LABEL="amd64" ;; + aarch64) ARCH_LABEL="arm64" ;; + *) error "Unsupported architecture: $ARCH"; exit 1 ;; +esac + +INSTALL_DIR="/usr/local/bin" + +# ─── Install just ──────────────────────────────────────────── + +install_just() { + info "Installing just..." + curl --retry 3 --connect-timeout 10 -sSfL https://just.systems/install.sh | bash -s -- --to "$INSTALL_DIR" + info "just installed: $(just --version)" +} + +# ─── Install gh ────────────────────────────────────────────── + +GH_VERSION="2.63.2" + +install_gh() { + info "Installing gh ${GH_VERSION}..." + local tmp + tmp=$(mktemp -d) + curl --retry 3 --connect-timeout 10 -sSfL \ + "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${ARCH_LABEL}.tar.gz" \ + -o "$tmp/gh.tar.gz" + tar -xzf "$tmp/gh.tar.gz" -C "$tmp" + cp "$tmp"/gh_*/bin/gh "$INSTALL_DIR/gh" + chmod +x "$INSTALL_DIR/gh" + rm -rf "$tmp" + info "gh installed: $(gh --version | head -1)" +} + +# ─── Install doppler ──────────────────────────────────────── + +install_doppler() { + info "Installing doppler..." + curl --retry 3 --connect-timeout 10 -sSfL https://cli.doppler.com/install.sh | sh + info "doppler installed: $(doppler --version)" +} + +# ─── Run installations in parallel ────────────────────────── + +install_just & +install_gh & +install_doppler & +wait + +# ─── Configure authentication ─────────────────────────────── + +# Doppler +if [[ -n "${DOPPLER_TOKEN:-}" ]]; then + info "Configuring Doppler..." + echo "$DOPPLER_TOKEN" | doppler configure set token --scope / 2>/dev/null || true +fi + +# GitHub CLI +GH_TOKEN="${GH_TOKEN:-}" +if [[ -z "$GH_TOKEN" ]] && command -v doppler &>/dev/null; then + GH_TOKEN=$(doppler secrets get GITHUB_TOKEN --plain 2>/dev/null || true) +fi + +if [[ -n "$GH_TOKEN" ]]; then + info "Configuring gh authentication..." + echo "$GH_TOKEN" | gh auth login --with-token 2>/dev/null || true +fi + +# Git remote configuration +if [[ -n "${GITHUB_REPOSITORY:-}" ]]; then + gh repo set-default "$GITHUB_REPOSITORY" 2>/dev/null || true +fi + +info "Cloud environment initialized." diff --git a/scripts/lib/common.sh b/scripts/lib/common.sh new file mode 100755 index 0000000..3eb90cb --- /dev/null +++ b/scripts/lib/common.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Shared utility functions for markkit scripts + +set -euo pipefail + +# Check if a git identity value looks like an AI agent +git_identity_looks_agent_like() { + local value="$1" + local lower + lower=$(echo "$value" | tr '[:upper:]' '[:lower:]') + case "$lower" in + *claude*|*copilot*|*gpt*|*bot*|*agent*|*ai*|*assistant*) + return 0 + ;; + esac + return 1 +} + +# Resolve a human git identity for commits +resolve_commit_git_identity() { + local name email source + + # Try git config first + name=$(git config user.name 2>/dev/null || true) + email=$(git config user.email 2>/dev/null || true) + + if [[ -n "$name" && -n "$email" ]]; then + if ! git_identity_looks_agent_like "$name" && ! git_identity_looks_agent_like "$email"; then + RESOLVED_NAME="$name" + RESOLVED_EMAIL="$email" + RESOLVED_SOURCE="git-config" + return 0 + fi + fi + + # Fall back to environment variables + name="${GIT_USER_NAME:-}" + email="${GIT_USER_EMAIL:-}" + + if [[ -n "$name" && -n "$email" ]]; then + RESOLVED_NAME="$name" + RESOLVED_EMAIL="$email" + RESOLVED_SOURCE="env" + return 0 + fi + + echo "ERROR: No human git identity found." >&2 + echo "Set GIT_USER_NAME and GIT_USER_EMAIL to a real user before committing." >&2 + return 1 +} + +# Configure git identity if current one looks agent-like +configure_commit_git_identity_if_needed() { + if resolve_commit_git_identity; then + if [[ "$RESOLVED_SOURCE" == "env" ]]; then + git config user.name "$RESOLVED_NAME" + git config user.email "$RESOLVED_EMAIL" + fi + fi +} diff --git a/specs/architecture.md b/specs/architecture.md new file mode 100644 index 0000000..bbf3178 --- /dev/null +++ b/specs/architecture.md @@ -0,0 +1,45 @@ +# Architecture + +## Overview + +Markkit is a Rust library for extracting clean markdown from any document format. +It follows a pipeline architecture: **detect → parse → normalize → convert → output**. + +## Design Principles + +1. **Format-agnostic API**: One entry point regardless of input format +2. **Streaming where possible**: Process large documents without loading entirely into memory +3. **Configurable output**: Users control what gets included/excluded +4. **Safe by default**: All input is treated as untrusted; no code execution +5. **Feature-gated formats**: Only compile what you need + +## Pipeline + +``` +Input (bytes/URL/file) + → Format Detection (MIME, magic bytes, extension) + → Parser (HTML, PDF, DOCX, etc.) + → Normalized AST (intermediate representation) + → Markdown Converter (AST → markdown string) + → Output (String / streaming) +``` + +## Crate Structure + +| Crate | Purpose | +|-------|---------| +| `markkit` | Core library – parsing, conversion, API | +| `markkit-cli` | Command-line tool | +| `markkit-bench` | Performance benchmarks | +| `markkit-eval` | LLM-based quality evaluation | +| `markkit-python` | Python bindings (PyO3) | +| `markkit-js` | Node.js bindings (NAPI-RS) | + +## Configuration + +Via `MarkkitBuilder`: +- Input size limits +- Processing time limits +- CSS selectors for include/exclude +- Link and image preservation +- Code block extraction diff --git a/specs/documentation.md b/specs/documentation.md new file mode 100644 index 0000000..5814dd0 --- /dev/null +++ b/specs/documentation.md @@ -0,0 +1,26 @@ +# Documentation + +## Documentation Structure + +### Public Documentation (`docs/`) +- Getting started guide +- Format support reference +- Configuration options +- API examples + +### Rustdoc +- All public types and functions documented +- Code examples in doc comments +- Feature flag documentation + +### Specs (`specs/`) +- Architecture decisions +- Format-specific implementation details +- Security model +- Testing strategy + +## Documentation Requirements + +- All public API items must have rustdoc +- Breaking changes documented in CHANGELOG.md +- New features include usage examples diff --git a/specs/eval.md b/specs/eval.md new file mode 100644 index 0000000..f6a4019 --- /dev/null +++ b/specs/eval.md @@ -0,0 +1,25 @@ +# Evaluation Harness + +## Overview + +`markkit-eval` measures extraction quality using LLM-as-judge +to compare markkit output against reference extractions. + +## Methodology + +1. Curate a dataset of (URL, reference markdown) pairs +2. Extract markdown from each URL using markkit +3. Score extraction quality via LLM comparison +4. Track metrics over time + +## Metrics + +- **Completeness**: Is all important content preserved? +- **Accuracy**: Is the content faithful to the original? +- **Formatting**: Is the markdown well-structured? +- **Noise**: Is non-content (nav, ads, etc.) excluded? + +## Providers + +- Anthropic (Claude) +- OpenAI (GPT) diff --git a/specs/format-support.md b/specs/format-support.md new file mode 100644 index 0000000..d29e3a7 --- /dev/null +++ b/specs/format-support.md @@ -0,0 +1,33 @@ +# Format Support + +## Supported Formats + +### Tier 1 (Core) +- **HTML/XHTML**: Full support via html5ever + scraper +- **Plain text**: Pass-through with minimal formatting + +### Tier 2 (Feature-gated) +- **PDF**: Text extraction via pdf-extract +- **DOCX**: Word document conversion via docx-rs +- **XLSX/ODS**: Spreadsheet to markdown tables via calamine + +### Tier 3 (Planned) +- **EPUB**: E-book extraction +- **RTF**: Rich text format +- **CSV/TSV**: Tabular data to markdown tables +- **Markdown**: Normalize/clean existing markdown + +## Format Detection + +Priority order: +1. Explicit content type parameter +2. MIME type from HTTP headers +3. File extension +4. Magic bytes detection + +## Resource Limits + +All formats enforce: +- Maximum input size (default: 50MB) +- Maximum processing time (default: 60s) +- Maximum output size (prevent markdown explosion) diff --git a/specs/html-extraction.md b/specs/html-extraction.md new file mode 100644 index 0000000..a04c231 --- /dev/null +++ b/specs/html-extraction.md @@ -0,0 +1,45 @@ +# HTML Extraction + +## Overview + +HTML-to-markdown is the primary extraction path. It must handle real-world +HTML from the web: malformed markup, complex layouts, embedded scripts, etc. + +## Approach + +1. **Parse** with html5ever for spec-compliant DOM construction +2. **Clean** by removing non-content elements (nav, footer, script, style, ads) +3. **Extract** content using CSS selectors or readability heuristics +4. **Convert** DOM nodes to markdown elements +5. **Normalize** whitespace, collapse empty lines, clean up formatting + +## Element Mapping + +| HTML | Markdown | +|------|----------| +| `

`–`

` | `#`–`######` | +| `

` | Paragraph (double newline) | +| `` | `[text](url)` | +| `` | `![alt](src)` | +| `