Repository navigation
test: assert hook payloads against lich's contract fixtures - #7
Merged
Merged
Conversation
The two sides of every hook live in different repositories: lich owns the endpoints, this repository owns the scripts that build the payloads. A renamed field went red in neither. lich now publishes each contract as bytes (docs/hooks/fixtures/*.jsonl there, one case per line: the payloads it accepts and the ones it refuses) and asserts its endpoints against them; this is the other half. node --test tests/*.test.mjs runs every hook script as a real subprocess — the command line taken from the registration each harness actually reads, quoting included — against a stub HTTP server, and asserts the body it POSTs: its key set and value classes match some accept case, it matches no reject case, it lands on the right endpoint with the right ?token=, and it never sends the deprecated claude_session_id. The accepted states and providers are derived from the fixtures rather than copied beside them, so a value lich stops accepting stops being accepted here. A reject case with no client-side rule fails its own test, so a contract that grows a refusal cannot slip through unmodelled. The client rules are pinned too: no report without the full lich environment, and exit 0 with an empty stdout when lich answers 500 or refuses the connection. Silence is load-bearing on Codex, where a hook's own exit code answers a PermissionRequest. The suite found one payload that disagreed. report-title.sh checked that it had extracted something before posting, which a title of nothing but spaces passes, and lich rejects a blank title with a 400. Codex made it reachable: it names a thread after its first user message, so a prompt whose first line is indentation produced exactly that payload. The title is trimmed before the check now, which also matches what lich stores. The fixtures are vendored in tests/fixtures/ so the suite never needs the network — network in a test is flake, drift is not. tests/refresh-fixtures.sh re-pulls them and CI diffs the copies against lich on every run, so a contract that moves upstream goes red here without any test reaching out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lich publishes every hook contract as bytes now —
docs/hooks/fixtures/*.jsonl, one case per line, each naming a payload and whether lich accepts (204) or refuses (400) it — and asserts its own endpoints against them ininternal/terminal/fixtures_test.go. This is the other half: the scripts in this repository build those payloads, so until now a renamed field went red only on lich's side.node --test tests/*.test.mjsdrives every hook script as a real subprocess, using the command line taken from the registration each harness actually reads (hooks/hooks.json,hooks/codex-hooks.json— quoting included), pointed at a stub HTTP server, and asserts the body it POSTs.What is asserted
POST,application/json, key set and value classes match someacceptcase'sbody, violates norejectcase, noclaude_session_id/hook,/session-start,/session-title,/session-touched, each with?token=$LICH_TOKENclaude/codexonly)"provider":"codex", Claude's sends"claude", a bare invocation defaults toclaudeai-titleline, Codex's firstuser_message, trimmed, JSON-escaped (quotes, backslash, tab), plus six cases that must stay silentsedfallbackjqremoved fromPATH(Windows)Stdout is asserted empty everywhere. That is load-bearing on Codex, where a hook's own exit code answers a
PermissionRequest.One payload disagreed with the contract
report-title.shchecked that it had extracted something before posting:which violates the
blank titlecase insession-title.jsonl— lich answers 400. Codex made it reachable: it names a thread after its first user message, so a prompt whose first line is indentation produced exactly that payload. Fixed in the script by trimming before the check, which also matches what lich stores. The fixture was not touched.Nothing else disagreed.
Vendored, not fetched
The fixtures live in
tests/fixtures/as byte-identical copies, so no test reaches the network — network in a test is flake, drift is not.tests/refresh-fixtures.shre-pulls them, and the newfixtures-driftCI job re-pulls andgit diff --exit-codes the copies againstomartelo/lich@main, so a contract that moves upstream goes red here on the next run.Test plan
node --test tests/*.test.mjs— 61 pass, 0 failstate:"Busy"→ 10 fail,claude_session_idre-added → 4 fail, path/session-touch→ 2 fail,session_iddropped from the title body → 4 failtests/refresh-fixtures.shfollowed bygit diff --exit-code -- tests/fixturesis clean