[shaunpatterson]

Pre-size the terms slice to len(tokens) instead of growing from nil. Also un-skips BenchmarkTermTokenizer with a real workload.

BenchmarkTermTokenizer: allocs -24% geomean, B/op -14% geomean.

---

Independent change; branched from 4b9b399d. Verified: package unit tests pass + Go benchmark (benchstat, p<0.01). Single-thread microbenchmarks; not yet run through the Docker integration suite.

Summary by CodeRabbit

• Performance
  • Term collection now reserves space based on the number of tokens, while preserving the results and duplicate-removal behavior.
• Tests
  • Added allocation-reporting benchmarks for tokenization across three text samples. The benchmarks report errors if tokenization fails.

[coderabbitai]

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 60f74f73-af83-475e-a647-5460c3edb08e

📥 Commits

Reviewing files that changed from the base of the PR and between 04c09bf and b340bc4.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 62dbe2ca-50c3-4f07-a5f6-3bccbadb918f

📥 Commits

Reviewing files that changed from the base of the PR and between 2c125bd and 04c09bf.

📒 Files selected for processing: 2

• tok/bleve.go
• tok/tok_test.go

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

📝 Walkthrough

uniqueTerms now preallocates its result slice based on the input token count. The tokenizer benchmark now runs three text samples, reports allocations, and fails if tokenization returns an error.

Changes: Tokenizer performance

Layer / File(s)	Summary
Term collection and benchmarks tok/bleve.go, tok/tok_test.go	uniqueTerms preallocates result capacity based on the number of input tokens. The benchmark runs three text samples as sub-benchmarks and reports allocations.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~5 minutes

Change: Refactor

Suggested reviewers: mlwelles

Merge Risk: ⚪ Minimal · up to 04c09

The preallocation does not increase retained capacity compared with the prior implementation. No actionable merge-blocking risk remains beyond normal checks.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 3 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: pre-sizing the terms slice in uniqueTerms for performance.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}