build: make a failing golden readable and approvable - #99
Open
MPV wants to merge 2 commits into
Open
Conversation
go-approval-tests v1.13.0 removed the reporter that printed the diff, so since
the v1.14.0 bump a failing golden locally reports nothing actionable — no file
names, no contents:
Finding JetBrains IDE...
kir_test.go:49: Failed Approval: received does not match approved.
v1.14.0 added APPROVAL_TESTS_USE_REPORTER; pointing it at the Systemout reporter
in `make test` restores a usable failure. Same broken golden, after:
approval files did not match
approved: .../kir_test.TestKind.Pod.stdout.approved.txt
received: .../kir_test.TestKind.Pod.stdout.received.txt
Received content:
nginx
gcr.io/google-containers/sidecar
busybox:1.28
Approved content:
wrong:image
CI is deliberately left alone: it runs `go test -race ./...` directly, and
GitHub Actions sets CI=true, which already selects an equivalent reporter — the
diff is in the failing run on #24. Note ContinuousIntegrationReporter is not the
name to use outside CI: it no-ops unless it detects a CI environment and lets
the auto-detection chain take over.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MBt5arMEZnxpjgh3GREnri
Nothing in CONTRIBUTING.md or AGENTS.md said how to update a golden, so the only
way to find out was to read the library. That is how the newline change got
approved: seventeen received files moved one at a time.
go-approval-tests ships approve_all.py for this, but it is the wrong thing to
point contributors at. It is fetched from raw.githubusercontent.com by a
fire-and-forget goroutine that races process exit, so whether it exists depends
on how long the run took. From a wiped .approval_tests_temp:
go test ./approvals/ (~0.02s) logs only, no scripts
go test -race ./approvals/ (~1s) scripts present
Its download path also fails silently by design, so offline it never arrives at
all. A make target is deterministic, needs no network or Python, and sits next to
the `make test` that produced the files.
The rename is safe to do wholesale because a passing test deletes its own
received file (approvals.go: `_ = os.Remove(receivedFile)`), so what is left over
after a full run is exactly the failures. That does not hold after a filtered
`go test -run ...`, which is called out in the Makefile and CONTRIBUTING.md.
Verified: no-op with no failures, and with kir_test.TestKind.Pod.stdout and
kir_test.TestList.exitcode deliberately broken, `make test && make approve`
restores both and the suite passes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MBt5arMEZnxpjgh3GREnri
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the
go-approval-testsv1.14.0 bump (#24). Two Makefile targets, no test or production code touched.1. A failing golden says nothing (
build: print the approval diff on a failed golden)v1.13.0 removed the reporter that printed the diff, so since #24 landed,
make test— the command CONTRIBUTING.md and AGENTS.md tell every contributor to run before a PR — reports a golden failure like this:No file names, no contents. v1.14.0 added
APPROVAL_TESTS_USE_REPORTER; pointing it at the Systemout reporter restores a usable failure. Same broken golden, after:CI is deliberately untouched: it runs
go test -race ./...directly, and GitHub Actions setsCI=true, which already selects an equivalent reporter. (ContinuousIntegrationReporteris not the name to use outside CI — it no-ops unless it detects a CI environment and lets auto-detection take over.)2. Nothing documented how to approve one (
build: add make approve)There were no instructions anywhere for updating a golden. That is how the newline change got approved: seventeen received files moved one at a time.
go-approval-testsshipsapprove_all.pyfor this, but it is the wrong thing to point contributors at — it is fetched fromraw.githubusercontent.comby a fire-and-forget goroutine racing process exit, so whether it exists depends on how long the run took. From a wiped.approval_tests_temp:go test ./approvals/go test -race ./approvals/Its download path also fails silently by design, so offline it never arrives. A make target is deterministic, needs no network or Python, and sits beside the
make testthat produced the files.Wholesale renaming is safe because a passing test deletes its own received file (
approvals.go:_ = os.Remove(receivedFile)), so what remains after a full run is exactly the failures. That does not hold after a filteredgo test -run ..., which is called out in both the Makefile comment and CONTRIBUTING.md.Verification
make approveis a no-op when nothing failed (exit 0, no output).kir_test.TestKind.Pod.stdoutandkir_test.TestList.exitcodedeliberately broken,make test && make approverestores both and the suite passes.make test,go vet ./...,gofmt -l .(empty),go test -race ./...all pass.Both commits are
build:, so neither cuts a release.Generated by Claude Code