Repository navigation
feat: HTML-to-Markdown pipeline via canonical IR - #4
Merged
Merged
Conversation
Adds the core Parse → Normalize → Render pipeline: - ir.rs: Canonical intermediate representation (Document, Node, ListItem) with serde support for JSON AST output - html/parser.rs: HTML → IR parser using scraper/html5ever with configurable include/exclude CSS selectors, depth limiting (200), whitespace normalization, and non-content element stripping - render/markdown.rs: IR → Markdown renderer supporting headings, paragraphs, inline formatting, links, images, lists (nested), blockquotes, code blocks, tables (GFM), and thematic breaks - extract.rs: Wires parser + renderer together with sync/async APIs and separate parse_html() for IR-only access 92 tests across 4 test suites: - Unit tests (9): IR serialization, format detection, whitespace handling - Integration tests (58): every HTML element mapping, selectors, full documents, edge cases, determinism - Security tests (18): XSS payloads, deep nesting, malformed HTML, resource limits, unicode, null bytes - Property tests (6): fuzz with arbitrary input, determinism, no dangerous tags in output
Specs: - specs/ir-spec.md: canonical IR specification with node semantics - specs/threat-model.md: comprehensive threat model with attack surfaces - specs/examples-spec.md: examples directory structure spec - specs/html-extraction.md: expanded with whitespace rules, depth limits Test fixtures (9 paired HTML/MD files): - basic_headings, inline_formatting, links_and_images, lists, code_blocks, tables, blockquotes, full_article, malformed - html_fixtures.rs test runner validates each pair Examples (4 standalone binaries): - basic, custom_selectors, ir_access, configuration CLI enhancements: - --stdin flag and "-" source for piped input - 9 integration tests (help, stdin, file, JSON, selectors) Language bindings: - Python: extract_html_sync, extract_html_sync_with_options (kwargs) - JS: extract_html_sync
- Fix div/section/article transparent containers to emit block children directly into parent node list (previously dropped multi-block content) - Add is_transparent_container() to handle div, section, article, main, figure, figcaption as passthrough in block context - Add 21 new integration tests: tables edge cases, nested lists, renderer edge cases, config builder, format detection, extract_bytes - Add 2 new security tests: depth limit boundary (200/201) - Update CHANGELOG.md with all HTML pipeline additions - Fix formatting across all files (cargo fmt)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the core Parse → Normalize → Render pipeline for HTML-to-Markdown conversion.
ir.rs):Document, 16Nodevariants,ListItemwith serde JSON supporthtml/parser.rs): scraper/html5ever, CSS include/exclude selectors, depth limiting (200), whitespace normalization, transparent container unwrapping, non-content element strippingrender/markdown.rs): GFM tables, nested lists, blockquotes, fenced code blocks, inline formatting, hard breaks--stdinflag and-source convention for piped inputextract_html_sync()+extract_html_sync_with_options()with keyword argsextract_html_sync()synchronous functionbasic.rs,custom_selectors.rs,ir_access.rs,configuration.rsTest Coverage
135 tests across 8 suites:
Quality Gates
cargo fmt --checkcleancargo clippy --workspace -- -D warningscleanTest plan
cargo test --workspacepasses (135 tests)cargo clippy --workspace --all-targets -- -D warningscleancargo fmt --all -- --checkcleanecho '<h1>Hi</h1>' | markkit extract --stdin