Skip to content

Fix chunk tokenizer mismatch and L2/cosine index (#4, #5) - #6

Merged
Nabil-Haddad merged 4 commits into
mainfrom
fix/chunk-tockenizer-and-cosine-index
Oct 27, 2026
Merged

Nabil-Haddad merged 4 commits into
mainfrom
fix/chunk-tockenizer-and-cosine-index

Conversation

@Nabil-Haddad

Copy link
Copy Markdown
Owner

Closes #4
Closes #5

  • Chunk size is now counted with the embedding model's own tokenizer, with an ingest-time guard against overflow.
  • Collections use cosine space and refuse to open if built by a different embedding model.
  • The manifest reprocesses everything when the model or chunk settings change.
  • Fixed a chunker bug where carried-over overlap could push a chunk over budget.
  • Re-ran both benchmarks. The results moved by 0.5 points or less.

Chunks were sized with tiktoken (cl100k_base, 400 tokens) but
all-MiniLM-L6-v2 truncates its input at 256 WordPiece tokens, so
about a third of every chunk was never embedded.

- count_tokens now uses the model's WordPiece tokenizer via a cached
  get_tokenizer(); long sentences are hard-split by character offsets
  so the original text is preserved
- default CHUNK_SIZE=250 / CHUNK_OVERLAP=40 (254 usable tokens after
  [CLS]/[SEP])
- add EMBED_MAX_SEQ_LENGTH and validate_chunk_lengths(); chunk_data
  now raises before writing chunks.json if any chunk would be truncated
- drop the tiktoken dependency and update the README
- update and extend tests/ingestion/test_chunk.py

Existing chunks and the vector DB are not rebuilt in this commit.

Refs #4
Collections were created with Chroma's default l2 space, so
cosign_simularity returned squared-L2 distances (lower is better)
under a name that promised cosine similarity, and nothing recorded
which embedding model built a collection.

- create collections with the cosine space and store embed_model in
  their metadata; get_collection verifies existing collections and
  raises CollectionMismatchError on a wrong space or model
- build_index checks the collection before Process_data, so a failed
  check cannot leave a manifest that makes the retry skip the rebuild
- cosign_simularity returns 1 - distance, a similarity where higher
  is better
- verify_data compares embed_model, chunk_size and chunk_overlap with
  the manifest and reprocesses every file when any of them changed
- add tests for each of the above

Existing collections are rejected until rebuilt; re-ingest required.

Refs #5
When a chunk was flushed, the trailing sentences that fit in the
overlap were carried into the next chunk without checking that the
next sentence still fit alongside them. A short sentence in the
overlap plus a near-full sentence produced a chunk over chunk_size
(278 tokens vs 250), which the new ingest-time guard rejects.

Drop sentences from the front of the overlap until the next sentence
fits. Add regression tests, one of them reproducing the failing case.

Refs #4
, #5)

Re-ran both benchmarks on the re-ingested corpus. Retrieval metrics moved by 0.5 points or less and the conclusions hold. Added the bug write-up and issue links, corrected statements that no longer match the data, and simplified the wording.
@Nabil-Haddad
Nabil-Haddad merged commit a7940ec into main Oct 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant