Repository navigation
refactor(documents): one extractor per format, Chunker takes a SourceDocument - #15
Merged
Merged
Conversation
Maintainer-only read side under /api/admin: request list and inspector, summary with SQL percentiles, feedback list, corpus health, currently failed ingestions, a metadata-only list of private documents and read-only configuration. IDocumentTitles returns shared titles only, so the inspector never names a private document. The RagAssistant.Chat meter is recorded from the finished trace. /admin becomes tabs with a request side panel and a ?request= deep link; AppHost sets Admin:TraceUrlTemplate. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Pin every passage of representative Markdown, plain-text and PDF inputs to files under Golden/, so the coming extractor refactoring can prove that chunking output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Document Extraction and chunking were tangled: the run processor branched on the source format and held the PDF timeout and store access, and the Chunker parsed Markdown itself next to a second entry point for PDF blocks. - ISourceExtractor per SourceFormat (Markdown, PlainText, Pdf) produces a SourceDocument: explicit sections (path, heading line) of blocks with Markdown-flavoured text and pages. - Chunker.Chunk(SourceDocument, options) is the single entry point and never sees the format; Markdown syntax helpers are shared. - The PDF timeout, store access and extraction metrics move into PdfSourceExtractor; the processor keeps run events, progress and the pages-without-text update. Behaviour is unchanged: the golden chunking files are untouched and the existing chunking tests only changed their input lines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Extraction and chunking were tangled.
IngestionRunProcessorbranched on the source format and held the PDF timeout and file-store access.Chunkerparsed Markdown itself and had a second entry point for PDF blocks. Adding a format (DOCX, OCR) would have meant touching all of them.What
ISourceExtractorperSourceFormat:MarkdownSourceExtractor,PlainTextSourceExtractorandPdfSourceExtractoreach produce aSourceDocument. Sections are explicit (path plus raw heading line, since one heading can appear twice), and blocks carry Markdown-flavoured text and pages.Chunker.Chunk(SourceDocument, options)is the only entry point and never sees the format. The Markdown syntax helpers move toMarkdownSyntax, shared by the extractor and the Chunker.PdfSourceExtractor. The processor keeps the run events, the progress channel and the pages-without-text update.AddDocumentsModule, and a test checks that everySourceFormathas exactly one.Behaviour unchanged
GoldenChunkingTests, which pins every passage of representative Markdown, plain-text and PDF inputs inGolden/*.txt. It was generated from the code before the refactoring.ChunkerTests/ChunkerPageTestsonly changed their input lines, not their assertions.Tests
FileMissing,ExtractionTimeout) and the extractor registration.dotnet test --solution RagAssistant.slnx: 1,176 passed, 0 failed.dotnet format --verify-no-changesis clean.Follow-up (not in this PR)
A deliberate behaviour change for PDFs (no minimum-size merge across a section boundary) sits on
claude/sweet-kalam-5f70b6, on top of this branch.🤖 Generated with Claude Code