Repository navigation
Fix chunk tokenizer mismatch and L2/cosine index (#4, #5) - #6
Merged
Merged
Conversation
Chunks were sized with tiktoken (cl100k_base, 400 tokens) but all-MiniLM-L6-v2 truncates its input at 256 WordPiece tokens, so about a third of every chunk was never embedded. - count_tokens now uses the model's WordPiece tokenizer via a cached get_tokenizer(); long sentences are hard-split by character offsets so the original text is preserved - default CHUNK_SIZE=250 / CHUNK_OVERLAP=40 (254 usable tokens after [CLS]/[SEP]) - add EMBED_MAX_SEQ_LENGTH and validate_chunk_lengths(); chunk_data now raises before writing chunks.json if any chunk would be truncated - drop the tiktoken dependency and update the README - update and extend tests/ingestion/test_chunk.py Existing chunks and the vector DB are not rebuilt in this commit. Refs #4
Collections were created with Chroma's default l2 space, so cosign_simularity returned squared-L2 distances (lower is better) under a name that promised cosine similarity, and nothing recorded which embedding model built a collection. - create collections with the cosine space and store embed_model in their metadata; get_collection verifies existing collections and raises CollectionMismatchError on a wrong space or model - build_index checks the collection before Process_data, so a failed check cannot leave a manifest that makes the retry skip the rebuild - cosign_simularity returns 1 - distance, a similarity where higher is better - verify_data compares embed_model, chunk_size and chunk_overlap with the manifest and reprocesses every file when any of them changed - add tests for each of the above Existing collections are rejected until rebuilt; re-ingest required. Refs #5
When a chunk was flushed, the trailing sentences that fit in the overlap were carried into the next chunk without checking that the next sentence still fit alongside them. A short sentence in the overlap plus a near-full sentence produced a chunk over chunk_size (278 tokens vs 250), which the new ingest-time guard rejects. Drop sentences from the front of the overlap until the next sentence fits. Add regression tests, one of them reproducing the failing case. Refs #4
, #5) Re-ran both benchmarks on the re-ingested corpus. Retrieval metrics moved by 0.5 points or less and the conclusions hold. Added the bug write-up and issue links, corrected statements that no longer match the data, and simplified the wording.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4
Closes #5