[euforic]

Summary

• align the Go library with the official openai/harmony HEAD fixture set by mirroring upstream fixtures, adding parity coverage, and tightening parser/render/token behavior
• add upstream-compatible parsing and encoding APIs, built-in browser/python namespace helpers, CLI parse role-hint fixes, and CI jobs for blocking parity plus non-blocking upstream drift detection
• optimize the parse and stream hot paths and document the benchmark workflow, current benchmark snapshot, and local regression-check commands

Benchmark Notes

• full benchmark suite run with TIKTOKEN_OFFLINE=1 go test -run '^$' -bench '^Benchmark' -benchmem -benchtime=200x ./benchmarks/go
• parse benchmark comparison run with python3 scripts/compare_parse_bench.py
• current parse/stream benchmarks beat the checked-in baseline in benchmarks/python/results/go_bench.txt on ns/op, B/op, and allocs/op
• latest full-suite snapshot:
  • BenchmarkParseToolCall: 2056 ns/op, 2040 B/op, 52 allocs/op
  • BenchmarkStreamParseToolCall: 1842 ns/op, 2040 B/op, 52 allocs/op
  • BenchmarkParseLargeCompletion: 17073 ns/op, 26184 B/op, 465 allocs/op
  • BenchmarkStreamParseLargeCompletion: 20120 ns/op, 26184 B/op, 465 allocs/op

Test Plan

• go test ./...
• python3 scripts/check_upstream_fixture_drift.py
• python3 scripts/compare_parse_bench.py
• TIKTOKEN_OFFLINE=1 go test -run '^$' -bench '^Benchmark' -benchmem -benchtime=200x ./benchmarks/go

[Copilot]

Pull request overview

This PR aligns harmony-go with the upstream openai/harmony HEAD by mirroring upstream fixtures, expanding parity coverage, and adding upstream-compatible parsing/encoding APIs while optimizing parse/stream hot paths.

Changes:

• Added upstream-compatible Parse/Encode APIs (ParseOptions, EncodeOptions, Encode, Decode, IsSpecialToken, AllSpecialTokens) plus built-in browser/python tool namespace helpers.
• Refactored stream parsing to decode incrementally into byte buffers, added strict vs recovery parsing behavior, and tightened UTF-8 validation behavior in decoding.
• Mirrored upstream HEAD fixtures, added parity tests + benchmark comparison/drift scripts, and extended CI with parity + non-blocking upstream drift detection.

Reviewed changes

Copilot reviewed 44 out of 45 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
types.go	Adds ParseOptions and EncodeOptions types to support new parser/encoder APIs.
tokenizer/segmenter.go	Refactors prefix-letter rule into a helper for upstream parity.
tokenizer/bpe.go	Adds UTF-8 validation for DecodeUTF8 and introduces per-token decode helper to reduce allocations.
parser.go	Adds options-aware stream parser, incremental byte decoding, and UTF-8 recovery logic.
encoding.go	Adds upstream-compatible Encode, lossy Decode, IsSpecialToken, AllSpecialTokens, and options-based parsing entrypoints.
builtins.go	Introduces built-in BrowserToolNamespace / PythonToolNamespace helpers matching official prompts.
cmd/harmony-go/main.go	Makes parse -role optional to match upstream/fixtures behavior.
capability_parity_test.go	Adds fixture-driven rendering parity, encode/decode parity, parser recovery tests, and CLI parse role behavior test.
scripts/sync_upstream_fixtures.py	Adds a script to copy upstream fixtures and write pinned metadata.
scripts/compare_parse_bench.py	Adds a script to compare current parse benchmarks vs a checked-in baseline.
scripts/check_upstream_fixture_drift.py	Adds a script to detect drift vs live upstream HEAD fixtures.
docs/python_go_performance.md	Documents local parse regression check workflow.
docs/CI.md	Documents local parity/drift checks in CI docs.
.github/workflows/go.yml	Adds a blocking parity job and a non-blocking upstream drift job; wires parity into dependent jobs.
.gitignore	Ignores .worktrees/ directory.
README.md	Documents new APIs, built-ins, benchmark snapshot, and parity/drift workflows.
CONTRIBUTING.md	Updates contributor guidance with new benchmark/parity/drift commands.
benchmarks/go/golden/testdata/tool_call.tokens.json	Updates golden tokens to match upstream/aligned tokenizer behavior.
benchmarks/go/golden/testdata/system_developer.tokens.json	Updates golden tokens to match upstream/aligned tokenizer behavior.
benchmarks/go/golden/testdata/autodrop_true.tokens.json	Updates golden tokens to match upstream/aligned tokenizer behavior.
benchmarks/go/golden/testdata/autodrop_false.tokens.json	Updates golden tokens to match upstream/aligned tokenizer behavior.
testdata/upstream-head/test_tool_response_parsing.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_streamable_parser.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_tool_call.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_reasoning_response.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo_medium_effort_no_instruction.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo_medium_effort.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo_low_effort_no_instruction.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo_low_effort.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo_high_effort_no_instruction.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo_high_effort.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_simple_convo.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_render_functions_with_parameters.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_reasoning_system_message_with_dates.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_reasoning_system_message_no_instruction.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_reasoning_system_message.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_preserve_cot.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_no_tools.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_keep_analysis_between_finals.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_dropping_cot_by_default.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_does_not_drop_if_ongoing_analysis.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_browser_tool_only.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_browser_and_python_tool.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/test_browser_and_function_tool.txt	Adds mirrored upstream HEAD fixture.
testdata/upstream-head/metadata.json	Records upstream repo/commit/date/subject for the mirrored fixture set.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a65ea4431

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".