Add academic_year period type to the source-package contract - #136
Conversation
…yEngine#133) Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025: Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data tables - all four status rows (in receipt / affected / not receiving a child element / exception granted) x headline, family-size and disability columns, 92 facts at month 2025-04. uk-data hand-transcribed only the 'not receiving' row and models the values at 2026, a populace-side timing declaration; the exception-granted rows are new coverage. Regional and time-series sheets are backfill-lane candidates. This PR is now content-complete for review: 9 packages, 912 facts. Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the Stat-Xplore extracts after an API key, and the uk_national profile in its own profile-lane PR) land in successor PRs after this merges, so the alias map and bundle constants stay conflict-free. Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107, 64 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Adversarial review complete — verdict: ready-with-nits, all findings addressed in What was attacked and survived: schema copies byte-identical with self-deriving sha (rejection of old-sha manifests reproduced at runtime); no other period-type enumeration anywhere in the repo; the UK FY2024-25→2024 precedent verified against the committed hmrc package; lane compliance clean on both judges. What the review caught and is now fixed:
🤖 Generated with Claude Code |
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Small and well-shaped: one vocabulary entry, the same description added to both copies of the schema (good — those two files drifting apart would be a nasty bug), a positive and a negative test on the validator, and a round-trip test through the consumer artifact including the PeriodContractError on a mismatched type. The convention itself — store the opening year of a split label, keep the label year when the publisher uses one — is the right call and is now written down in the place people will actually read it.
Two things:
1. The ALLOWED_PERIOD_TYPES comment cautions about a helper that doesn't exist.
Caution: the EES helper _academic_year_end (ledger/sources/rows.py) names value
COLUMNS by the academic year's END year
I can't find _academic_year_end anywhere — not on main, and not in #137, #139, or #141. If it's landing in a later PR, the forward reference will read as stale until then; if it was renamed or never merged, the caution points at nothing and the reader can't check the claim. Either way the most valuable half of that comment (column naming uses the END year, fact periods must not) survives without the symbol name, so I'd either drop the identifier or link the PR that introduces it.
2. The round-trip test asserts the convention on a fact that couldn't exist. test_academic_year_rows_round_trip_through_consumer_artifact takes _soi_agi_fact() — IRS SOI adjusted gross income — and gives it academic_year periods, with source_record_id strings like irs_soi.ay2023.table_1_1.all.adjusted_gross_income. It exercises the plumbing correctly, but as documentation it teaches the wrong thing: AGI is never an academic-year quantity, and someone reading this test to learn when to reach for the new type gets no signal. A neutral synthetic fact, or one shaped like the SLC/EES student-support rows this vocabulary is actually for, would carry the same coverage and explain itself.
Landing the vocabulary ahead of any producer is the right sequencing for a contract change, so no objection there — just worth a line in the description saying which package will be the first consumer (the SLC rows in #131, I assume, per the checklist row in #141).
UK student-finance publications (SLC/DfE EES) assert academic-year
periods; the vocabulary had no way to state that without re-labelling
AY 2024/25 as a calendar or fiscal year. Adds academic_year to
ledger.core.ALLOWED_PERIOD_TYPES, documents the opening-year integer
convention (FY2024-25 -> 2024, AY 2024/25 -> 2024) beside the
vocabulary and in the consumer schema's period.type description (both
byte-identical schema copies), and proves the contract end to end:
validate_fact accepts academic_year and rejects unknown types, and an
academic_year fact set builds into a consumer artifact, loads, and
resolves at {type: academic_year, value: 2024}.
The JSON schema deliberately gains a description, not an enum: the
single enforcement point for the vocabulary stays validate_fact.
Closes PolicyEngine#131.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three review findings fixed:
- The round-trip test now exercises the vocabulary gate itself
(validate_facts on the academic_year facts) rather than only the
period-type-agnostic export/build/load/resolve plumbing.
- Added the missing negative: resolving the academic_year artifact at
{fiscal_year, 2024} raises PeriodContractError - the cross-type
silent-misuse family (populace#212) UK sources now make likely, since
fiscal, tax and academic years share integer values.
- The core.py comment cites an attested US example (cbo FY2024 -> 2024)
instead of the unattested FY2025, and warns that the EES helper
_academic_year_end names value columns by END year - source-layout
naming that must not leak into fact periods (relevant to the SLC
packages this change unblocks). The schema description (both
byte-identical copies) gains the single-label-year clause.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2b2dab1 to
6b34e84
Compare
…yEngine#133) Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025: Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data tables - all four status rows (in receipt / affected / not receiving a child element / exception granted) x headline, family-size and disability columns, 92 facts at month 2025-04. uk-data hand-transcribed only the 'not receiving' row and models the values at 2026, a populace-side timing declaration; the exception-granted rows are new coverage. Regional and time-series sheets are backfill-lane candidates. This PR is now content-complete for review: 9 packages, 912 facts. Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the Stat-Xplore extracts after an API key, and the uk_national profile in its own profile-lane PR) land in successor PRs after this merges, so the alias map and bundle constants stay conflict-free. Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107, 64 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… with an SLC-shaped fact The _academic_year_end caution pointed at the pre-rename path (ledger/sources/rows.py), which made the symbol look nonexistent - it has lived in the harness since 376d4cc and now sits at chronicle/sources/rows.py:971. The comment now cites the current path and shows the behavior inline ("2024/25" -> 2025). The round-trip test no longer dresses an IRS AGI fact in academic-year periods: the fixture is now shaped like the SLC/EES student-support rows this vocabulary exists for (maintenance-loan recipients, England, person entity, count unit, administrative provenance), so the test documents when to reach for the type as well as proving the gate and the cross-type PeriodContractError. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Both review points addressed in 1. 2. The round-trip test now teaches with the fact it was built for. The fixture is no longer AGI-in-costume: it's shaped like the SLC/EES student-support rows ( 🤖 Generated with Claude Code |
…yEngine#133) Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025: Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data tables - all four status rows (in receipt / affected / not receiving a child element / exception granted) x headline, family-size and disability columns, 92 facts at month 2025-04. uk-data hand-transcribed only the 'not receiving' row and models the values at 2026, a populace-side timing declaration; the exception-granted rows are new coverage. Regional and time-series sheets are backfill-lane candidates. This PR is now content-complete for review: 9 packages, 912 facts. Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the Stat-Xplore extracts after an API key, and the uk_national profile in its own profile-lane PR) land in successor PRs after this merges, so the alias map and bundle constants stay conflict-free. Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107, 64 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…PolicyEngine#132) Six packages, 199 facts - the wave-1 remainder unblocked by PolicyEngine#136, and the first consumer of the academic_year period type: - slc-student-loan-borrower-forecasts-england-2025 (116): EES Table 6a borrowers liable to repay / earning above threshold by product group, plan and status, AY2024/25-2029/30 on the ees_permalink_table_html lane. Fact periods store the OPENING year while the EES value columns are END-year named (value_2025 = AY 2024/25) - the exact leak the ALLOWED_PERIOD_TYPES caution documents. All values are forecasts: model_output + assertion source_projection, so profiles opt in via assertion_policy (PolicyEngine#154). - slc-student-support-england-2025 (72): Table 3A maintenance loans (Grand-total recipients and GBP-million amounts) and Table 4C(i) Adult Dependents Grant + Parents Learning Allowance, AY2013/14-2024/25. uk-data's whole-pound roundings of the published GBP-million precision are enumerated; cohort rows and the other 4C products are named backfill exclusions. - slc-student-loan-repayments-england-2025 (8): the CORRECTED slcsp01 workbook's Table 1A net-repayments row, FY2024-25 block, all seven plan columns + total at publisher precision. uk-data's plan_2 and postgraduate targets are FT+PT / Masters+Doctoral sums of these columns - populace-side derivations. - slc-student-loan-repayments-{scotland,wales,northern-ireland}-2025 (1 each): the HTML-only country statistics pages, archived and parsed on the document-numbers lane (203.3m / 229.1m / 181.7m GBP, FY2024-25). Every uk-data snapshot value reconciles exactly (counts and England repayments byte-equal; GBP amounts equal at publisher precision). Bundle expectations re-measured under the disclosed-and-reverted sandbox: 39,418 facts / 66 packages on this branch, merged-bundle test passes. Manifests are upload-truthful. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes #131.
Lane:
ledger-contract-maintainer(.github/ledger-agents.yml) — judgesledger-contract+ledger-boundary. This is the one contract change the UK target migration needs, isolated per #131 so no data wave drags through contract review. Unblocks the SLC packages in #132 (wave 1) and #133 (slc_repayments).What changed
ledger/core.py—academic_yearjoinsALLOWED_PERIOD_TYPES, with a comment documenting the period-integer convention for split-label years: the stored integer is the opening year of the publisher's label — UK FY2024-25 →fiscal_year 2024(the existingpackages/hmrc/vat_firm_targets_2024_25precedent), AY 2024/25 →academic_year 2024. Publishers that label a fiscal year with a single year keep that label year (US federal FY2025 → 2025).docs/schemas/consumer_fact.v1.schema.json+policyengine_ledger/schemas/consumer_fact.v1.schema.json— byte-identicaldescriptionadded toperiod.typedocumenting the vocabulary and the opening-year convention. Deliberately a description, not an enum: the single enforcement point for the vocabulary staysvalidate_fact; the schema documents, it does not duplicate the gate.test_ledger_core.py:academic_yearfacts validate; unknown period types still fail withmalformed_period.test_ledger_consumer_contract.py: full round trip — twoacademic_yearfacts →write_consumer_facts_jsonl→build_consumer_artifact(profile shipped through the artifact) →load_consumer_artifact(schema-validates every row) →resolve_profile_targetsat{type: academic_year, value: 2024}resolves the exact fact onfactbasis.Behavior notes
CONSUMER_FACT_SCHEMA_SHA256derives from the packaged schema bytes, so it moves with this change. Newly built artifacts carry the new sha;load_consumer_artifactwill reject artifacts whose manifests pin the old sha — the designed exact-contract behavior. Existing consumer artifacts need a rebuild with the matching wheel.Checks
uv run ruff check ledger policyengine_ledger db scripts tests(repo-pinned, CI scope) — clean.uv run pytest -q— green on this branch (614 passed, 1 skipped).🤖 Generated with Claude Code