Skip to content

fix(validate): report results, and exit 1 under -strict on a failure - #55

Merged
jravani merged 2 commits into
mainfrom
fix/validate-report-results
Oct 2, 2026
Merged

jravani merged 2 commits into
mainfrom
fix/validate-report-results

Conversation

@jravani

@jravani jravani commented Oct 2, 2026

Copy link
Copy Markdown
Member

Closes #51.

validate now reports what it measures. A plain run prints the results and exits 0; -strict exits 1 when any building is outside ±2.5%.

Output

Against a database seeded from data/tabula-calculator-lite.xlsx at current main:

Pass rate per table (tolerance 2.5%):
  austria          165/165   100.0%
  ...
  spain             49/72     68.1%
  ...
Failures (23):
  spain           ES.ME.TH.06.Gen.ReEx.001.003        calculated       2.13  expected       2.20  error 2.96%
  ...
Total: 2124/2147 passed (98.9%)
Run Exit code
./bin/validate 0
./bin/validate -strict 1 (Spain, #47)

Changes

  • report prints the per-table counts, every failing building (or its pipeline error) and the total, and returns the failure count. exitCode returns 1 only under -strict with a failure. Results are sorted by row ID, so output is stable between runs.
  • The configuration is loaded in main. It was loaded at package init, which panicked without DB_PASSWORD and made the package untestable.
  • The commented-out summary block and printTestResult, which only that block called, are removed.
  • docs/validation.md and docs/getting-started.md document -strict and the output.
  • Separate commit: .gitignore anchors app, build_db and validate to the repository root. Unanchored they matched cmd/build_db/ and cmd/validate/, so the new cmd/validate/main_test.go was ignored.

validate compares the unrounded pipeline result; POST /api/v1/calculate/{code} rounds to two decimals. That is why validate reports 49 of 72 Spanish rows passing where the API gives 48: ES.ME.TH.01.Gen.ReEx.001.003 (reference 4.0) passes unrounded and lands exactly on 2.5% once rounded to 3.90. The validation docs now state this.

Checks

  • TestReport_countsPerTableAndListsFailures and TestExitCode_nonZeroOnlyWhenStrictAndFailing failed to build on main (undefined: report, undefined: exitCode) and pass here.
  • go vet ./... and go test ./... pass; mkdocs build --strict passes.
  • End to end against a seeded Postgres: the output above, exit 0 without the flag and 1 with it.

The bare entries app, build_db and validate were meant for binaries
built at the repository root, but they also matched the cmd/build_db
and cmd/validate directories. A new file there was ignored, and
`git add` on a tracked one warned. Anchoring them with a leading slash
keeps the root binaries ignored and the source directories visible.
validate computed a pass or fail for every building but printed none
of it: the summary and the failure listing were commented out, and the
tool exited 0 whatever the results. It now prints the pass count per
country, one line per failing building with its calculated and
expected q_h_nd and percentage error (or the pipeline error), and the
overall count. Results are ordered by row ID so two runs print the
same output.

A plain run exits 0. With -strict it exits 1 when any building is
outside the tolerance. The default stays 0 because Spain currently
fails 23 rows (#47), which would otherwise fail every run.

The configuration is now loaded in main instead of at package init,
which panicked without DB_PASSWORD and made the package untestable.
printTestResult is removed; it was only called from the commented-out
block.

Closes #51
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 15 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
cmd/validate/main.go 66.66% 14 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@jravani
jravani merged commit 2398063 into main Oct 2, 2026
6 checks passed
@jravani
jravani deleted the fix/validate-report-results branch October 2, 2026 12:52
@jravani jravani added this to the v0.8.0 milestone Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

validate reports no pass or fail and always exits 0

2 participants