Skip to content

test: assert hook payloads against lich's contract fixtures - #7

Merged
omartelo merged 1 commit into
mainfrom
test/hook-contract-fixtures
Aug 10, 2026
Merged

omartelo merged 1 commit into
mainfrom
test/hook-contract-fixtures

Conversation

@omartelo

@omartelo omartelo commented Aug 10, 2026 •

Copy link
Copy Markdown
Collaborator

lich publishes every hook contract as bytes now — docs/hooks/fixtures/*.jsonl, one case per line, each naming a payload and whether lich accepts (204) or refuses (400) it — and asserts its own endpoints against them in internal/terminal/fixtures_test.go. This is the other half: the scripts in this repository build those payloads, so until now a renamed field went red only on lich's side.

node --test tests/*.test.mjs drives every hook script as a real subprocess, using the command line taken from the registration each harness actually reads (hooks/hooks.json, hooks/codex-hooks.json — quoting included), pointed at a stub HTTP server, and asserts the body it POSTs.

What is asserted

Assertion Covers
Registration sweep (18 cases) every script × both harnesses: exactly one request, POST, application/json, key set and value classes match some accept case's body, violates no reject case, no claude_session_id
Endpoint + token /hook, /session-start, /session-title, /session-touched, each with ?token=$LICH_TOKEN
Reject rules one predicate per rejected case; a rejected case with no rule fails its own test, so a contract that grows a refusal cannot slip through unmodelled
Enum pinning accepted states and providers are derived from the fixtures, not copied beside them; registration arguments are checked against them (all four states registered by both harnesses, claude/codex only)
Provider the Codex registration sends "provider":"codex", Claude's sends "claude", a bare invocation defaults to claude
Title extraction Claude's ai-title line, Codex's first user_message, trimmed, JSON-escaped (quotes, backslash, tab), plus six cases that must stay silent
sed fallback session-start still parses the payload with jq removed from PATH (Windows)
Client rules (24 cases) five partial-environment shapes report nothing and exit 0; a 500 and a refused connection both exit 0 with an empty stdout — per script

Stdout is asserted empty everywhere. That is load-bearing on Codex, where a hook's own exit code answers a PermissionRequest.

One payload disagreed with the contract

report-title.sh checked that it had extracted something before posting:

POST /session-title  {"session_id":"…","title":"   "}

which violates the blank title case in session-title.jsonl — lich answers 400. Codex made it reachable: it names a thread after its first user message, so a prompt whose first line is indentation produced exactly that payload. Fixed in the script by trimming before the check, which also matches what lich stores. The fixture was not touched.

Nothing else disagreed.

Vendored, not fetched

The fixtures live in tests/fixtures/ as byte-identical copies, so no test reaches the network — network in a test is flake, drift is not. tests/refresh-fixtures.sh re-pulls them, and the new fixtures-drift CI job re-pulls and git diff --exit-codes the copies against omartelo/lich@main, so a contract that moves upstream goes red here on the next run.

Test plan

  • node --test tests/*.test.mjs — 61 pass, 0 fail
  • Mutation-checked that the assertions bite: state:"Busy" → 10 fail, claude_session_id re-added → 4 fail, path /session-touch → 2 fail, session_id dropped from the title body → 4 fail
  • tests/refresh-fixtures.sh followed by git diff --exit-code -- tests/fixtures is clean

The two sides of every hook live in different repositories: lich owns the
endpoints, this repository owns the scripts that build the payloads. A renamed
field went red in neither. lich now publishes each contract as bytes
(docs/hooks/fixtures/*.jsonl there, one case per line: the payloads it accepts
and the ones it refuses) and asserts its endpoints against them; this is the
other half.

node --test tests/*.test.mjs runs every hook script as a real subprocess — the
command line taken from the registration each harness actually reads, quoting
included — against a stub HTTP server, and asserts the body it POSTs: its key
set and value classes match some accept case, it matches no reject case, it
lands on the right endpoint with the right ?token=, and it never sends the
deprecated claude_session_id. The accepted states and providers are derived
from the fixtures rather than copied beside them, so a value lich stops
accepting stops being accepted here. A reject case with no client-side rule
fails its own test, so a contract that grows a refusal cannot slip through
unmodelled.

The client rules are pinned too: no report without the full lich environment,
and exit 0 with an empty stdout when lich answers 500 or refuses the
connection. Silence is load-bearing on Codex, where a hook's own exit code
answers a PermissionRequest.

The suite found one payload that disagreed. report-title.sh checked that it had
extracted something before posting, which a title of nothing but spaces passes,
and lich rejects a blank title with a 400. Codex made it reachable: it names a
thread after its first user message, so a prompt whose first line is
indentation produced exactly that payload. The title is trimmed before the
check now, which also matches what lich stores.

The fixtures are vendored in tests/fixtures/ so the suite never needs the
network — network in a test is flake, drift is not. tests/refresh-fixtures.sh
re-pulls them and CI diffs the copies against lich on every run, so a contract
that moves upstream goes red here without any test reaching out.
@omartelo
omartelo merged commit 5ecd483 into main Aug 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant