Skip to content

Add academic_year period type to the source-package contract - #136

Merged
juaristi22 merged 3 commits into
PolicyEngine:mainfrom
juaristi22:add-academic-year-period-type
Aug 10, 2026
Merged

Add academic_year period type to the source-package contract#136
juaristi22 merged 3 commits into
PolicyEngine:mainfrom
juaristi22:add-academic-year-period-type

Conversation

@juaristi22

Copy link
Copy Markdown
Collaborator

Closes #131.

Lane: ledger-contract-maintainer (.github/ledger-agents.yml) — judges ledger-contract + ledger-boundary. This is the one contract change the UK target migration needs, isolated per #131 so no data wave drags through contract review. Unblocks the SLC packages in #132 (wave 1) and #133 (slc_repayments).

What changed

  • ledger/core.pyacademic_year joins ALLOWED_PERIOD_TYPES, with a comment documenting the period-integer convention for split-label years: the stored integer is the opening year of the publisher's label — UK FY2024-25 → fiscal_year 2024 (the existing packages/hmrc/vat_firm_targets_2024_25 precedent), AY 2024/25 → academic_year 2024. Publishers that label a fiscal year with a single year keep that label year (US federal FY2025 → 2025).
  • docs/schemas/consumer_fact.v1.schema.json + policyengine_ledger/schemas/consumer_fact.v1.schema.json — byte-identical description added to period.type documenting the vocabulary and the opening-year convention. Deliberately a description, not an enum: the single enforcement point for the vocabulary stays validate_fact; the schema documents, it does not duplicate the gate.
  • Teststest_ledger_core.py: academic_year facts validate; unknown period types still fail with malformed_period. test_ledger_consumer_contract.py: full round trip — two academic_year facts → write_consumer_facts_jsonlbuild_consumer_artifact (profile shipped through the artifact) → load_consumer_artifact (schema-validates every row) → resolve_profile_targets at {type: academic_year, value: 2024} resolves the exact fact on fact basis.

Behavior notes

  • CONSUMER_FACT_SCHEMA_SHA256 derives from the packaged schema bytes, so it moves with this change. Newly built artifacts carry the new sha; load_consumer_artifact will reject artifacts whose manifests pin the old sha — the designed exact-contract behavior. Existing consumer artifacts need a rebuild with the matching wheel.
  • Period-integer alignment downstream (e.g. policyengine-uk-data's AY 2024/25 → calendar 2025 mapping) is a consumer concern; facts store what the publisher asserted.
  • No existing package or fact changes behavior: the vocabulary only widens, and the schema edit is documentation-only for row validation.

Checks

  • uv run ruff check ledger policyengine_ledger db scripts tests (repo-pinned, CI scope) — clean.
  • Full uv run pytest -q — green on this branch (614 passed, 1 skipped).
  • Deterministic checks per the lane: schema validation, consumer-contract validation, package import compatibility, raw-facts boundary validation — all exercised by the suite above.

🤖 Generated with Claude Code

@juaristi22
juaristi22 requested a review from MaxGhenis August 5, 2026 15:19
juaristi22 added a commit to juaristi22/ledger that referenced this pull request Aug 6, 2026
…yEngine#133)

Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025:
Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data
tables - all four status rows (in receipt / affected / not receiving a
child element / exception granted) x headline, family-size and
disability columns, 92 facts at month 2025-04. uk-data hand-transcribed
only the 'not receiving' row and models the values at 2026, a
populace-side timing declaration; the exception-granted rows are new
coverage. Regional and time-series sheets are backfill-lane candidates.

This PR is now content-complete for review: 9 packages, 912 facts.
Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics
Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the
Stat-Xplore extracts after an API key, and the uk_national profile in
its own profile-lane PR) land in successor PRs after this merges, so
the alias map and bundle constants stay conflict-free.

Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107,
64 tables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22
juaristi22 requested a review from vahid-ahmadi August 7, 2026 08:28
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Adversarial review complete — verdict: ready-with-nits, all findings addressed in 2b2dab1.

What was attacked and survived: schema copies byte-identical with self-deriving sha (rejection of old-sha manifests reproduced at runtime); no other period-type enumeration anywhere in the repo; the UK FY2024-25→2024 precedent verified against the committed hmrc package; lane compliance clean on both judges.

What the review caught and is now fixed:

  • The round-trip test proved period-type-agnostic plumbing, not the vocabulary gate — a probe with a bogus school_year type passed it end to end. The test now asserts validate_facts on the academic-year facts directly.
  • No negative existed for cross-type resolution anywhere in the repo; the test now proves resolving the academic-year artifact at {fiscal_year, 2024} raises PeriodContractError — the silent-misuse family (populace#212) that UK sources make likely, since fiscal/tax/academic years share integer values.
  • The core comment now cites an attested US example (cbo FY2024→2024) and warns that the EES helper _academic_year_end names value columns by end-year — source-layout naming that must not leak into fact periods when the SLC packages are written. Schema description gains the single-label-year clause (both copies, byte-identical).

🤖 Generated with Claude Code

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small and well-shaped: one vocabulary entry, the same description added to both copies of the schema (good — those two files drifting apart would be a nasty bug), a positive and a negative test on the validator, and a round-trip test through the consumer artifact including the PeriodContractError on a mismatched type. The convention itself — store the opening year of a split label, keep the label year when the publisher uses one — is the right call and is now written down in the place people will actually read it.

Two things:

1. The ALLOWED_PERIOD_TYPES comment cautions about a helper that doesn't exist.

Caution: the EES helper _academic_year_end (ledger/sources/rows.py) names value
COLUMNS by the academic year's END year

I can't find _academic_year_end anywhere — not on main, and not in #137, #139, or #141. If it's landing in a later PR, the forward reference will read as stale until then; if it was renamed or never merged, the caution points at nothing and the reader can't check the claim. Either way the most valuable half of that comment (column naming uses the END year, fact periods must not) survives without the symbol name, so I'd either drop the identifier or link the PR that introduces it.

2. The round-trip test asserts the convention on a fact that couldn't exist. test_academic_year_rows_round_trip_through_consumer_artifact takes _soi_agi_fact() — IRS SOI adjusted gross income — and gives it academic_year periods, with source_record_id strings like irs_soi.ay2023.table_1_1.all.adjusted_gross_income. It exercises the plumbing correctly, but as documentation it teaches the wrong thing: AGI is never an academic-year quantity, and someone reading this test to learn when to reach for the new type gets no signal. A neutral synthetic fact, or one shaped like the SLC/EES student-support rows this vocabulary is actually for, would carry the same coverage and explain itself.

Landing the vocabulary ahead of any producer is the right sequencing for a contract change, so no objection there — just worth a line in the description saying which package will be the first consumer (the SLC rows in #131, I assume, per the checklist row in #141).

juaristi22 and others added 2 commits August 10, 2026 10:11
UK student-finance publications (SLC/DfE EES) assert academic-year
periods; the vocabulary had no way to state that without re-labelling
AY 2024/25 as a calendar or fiscal year. Adds academic_year to
ledger.core.ALLOWED_PERIOD_TYPES, documents the opening-year integer
convention (FY2024-25 -> 2024, AY 2024/25 -> 2024) beside the
vocabulary and in the consumer schema's period.type description (both
byte-identical schema copies), and proves the contract end to end:
validate_fact accepts academic_year and rejects unknown types, and an
academic_year fact set builds into a consumer artifact, loads, and
resolves at {type: academic_year, value: 2024}.

The JSON schema deliberately gains a description, not an enum: the
single enforcement point for the vocabulary stays validate_fact.

Closes PolicyEngine#131.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three review findings fixed:
- The round-trip test now exercises the vocabulary gate itself
  (validate_facts on the academic_year facts) rather than only the
  period-type-agnostic export/build/load/resolve plumbing.
- Added the missing negative: resolving the academic_year artifact at
  {fiscal_year, 2024} raises PeriodContractError - the cross-type
  silent-misuse family (populace#212) UK sources now make likely, since
  fiscal, tax and academic years share integer values.
- The core.py comment cites an attested US example (cbo FY2024 -> 2024)
  instead of the unattested FY2025, and warns that the EES helper
  _academic_year_end names value columns by END year - source-layout
  naming that must not leak into fact periods (relevant to the SLC
  packages this change unblocks). The schema description (both
  byte-identical copies) gains the single-label-year clause.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22
juaristi22 force-pushed the add-academic-year-period-type branch from 2b2dab1 to 6b34e84 Compare August 10, 2026 08:12
juaristi22 added a commit to juaristi22/ledger that referenced this pull request Aug 10, 2026
…yEngine#133)

Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025:
Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data
tables - all four status rows (in receipt / affected / not receiving a
child element / exception granted) x headline, family-size and
disability columns, 92 facts at month 2025-04. uk-data hand-transcribed
only the 'not receiving' row and models the values at 2026, a
populace-side timing declaration; the exception-granted rows are new
coverage. Regional and time-series sheets are backfill-lane candidates.

This PR is now content-complete for review: 9 packages, 912 facts.
Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics
Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the
Stat-Xplore extracts after an API key, and the uk_national profile in
its own profile-lane PR) land in successor PRs after this merges, so
the alias map and bundle constants stay conflict-free.

Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107,
64 tables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… with an SLC-shaped fact

The _academic_year_end caution pointed at the pre-rename path
(ledger/sources/rows.py), which made the symbol look nonexistent - it has
lived in the harness since 376d4cc and now sits at
chronicle/sources/rows.py:971. The comment now cites the current path and
shows the behavior inline ("2024/25" -> 2025). The round-trip test no
longer dresses an IRS AGI fact in academic-year periods: the fixture is now
shaped like the SLC/EES student-support rows this vocabulary exists for
(maintenance-loan recipients, England, person entity, count unit,
administrative provenance), so the test documents when to reach for the
type as well as proving the gate and the cross-type PeriodContractError.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Both review points addressed in 60e0a8f — thank you, both were real.

1. _academic_year_end does exist — the comment was pointing at a dead path. The helper has been in the harness since the very first commit (376d4cc) and now lives at chronicle/sources/rows.py:971; the caution cited the pre-rename ledger/sources/rows.py, which is why searching came up empty. The comment now cites the current path and shows the behavior inline ("2024/25" → 2025), so the claim is checkable: the EES lane names value columns by the END year while this vocabulary stores the OPENING year — exactly the leak the caution exists to prevent when the SLC packages get written.

2. The round-trip test now teaches with the fact it was built for. The fixture is no longer AGI-in-costume: it's shaped like the SLC/EES student-support rows (slc.ay2023.student_support.maintenance_loan_recipients — maintenance-loan recipients, England, person entity, count unit, administrative provenance, AY 2023/24 → academic_year 2023), with the profile side switched to a uk/slc target. Same coverage as before — the vocabulary gate via validate_facts, the JSONL round trip, resolution at {academic_year, 2024}, and the cross-type PeriodContractError probe — but a reader now gets the when-to-use signal for free. 46 contract tests pass.

🤖 Generated with Claude Code

@juaristi22
juaristi22 merged commit 654336b into PolicyEngine:main Aug 10, 2026
1 check passed
juaristi22 added a commit to juaristi22/ledger that referenced this pull request Aug 10, 2026
…yEngine#133)

Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025:
Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data
tables - all four status rows (in receipt / affected / not receiving a
child element / exception granted) x headline, family-size and
disability columns, 92 facts at month 2025-04. uk-data hand-transcribed
only the 'not receiving' row and models the values at 2026, a
populace-side timing declaration; the exception-granted rows are new
coverage. Regional and time-series sheets are backfill-lane candidates.

This PR is now content-complete for review: 9 packages, 912 facts.
Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics
Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the
Stat-Xplore extracts after an API key, and the uk_national profile in
its own profile-lane PR) land in successor PRs after this merges, so
the alias map and bundle constants stay conflict-free.

Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107,
64 tables.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
juaristi22 added a commit to juaristi22/ledger that referenced this pull request Aug 10, 2026
…PolicyEngine#132)

Six packages, 199 facts - the wave-1 remainder unblocked by PolicyEngine#136, and the
first consumer of the academic_year period type:

- slc-student-loan-borrower-forecasts-england-2025 (116): EES Table 6a
  borrowers liable to repay / earning above threshold by product group,
  plan and status, AY2024/25-2029/30 on the ees_permalink_table_html lane.
  Fact periods store the OPENING year while the EES value columns are
  END-year named (value_2025 = AY 2024/25) - the exact leak the
  ALLOWED_PERIOD_TYPES caution documents. All values are forecasts:
  model_output + assertion source_projection, so profiles opt in via
  assertion_policy (PolicyEngine#154).
- slc-student-support-england-2025 (72): Table 3A maintenance loans
  (Grand-total recipients and GBP-million amounts) and Table 4C(i) Adult
  Dependents Grant + Parents Learning Allowance, AY2013/14-2024/25.
  uk-data's whole-pound roundings of the published GBP-million precision
  are enumerated; cohort rows and the other 4C products are named
  backfill exclusions.
- slc-student-loan-repayments-england-2025 (8): the CORRECTED slcsp01
  workbook's Table 1A net-repayments row, FY2024-25 block, all seven plan
  columns + total at publisher precision. uk-data's plan_2 and
  postgraduate targets are FT+PT / Masters+Doctoral sums of these columns
  - populace-side derivations.
- slc-student-loan-repayments-{scotland,wales,northern-ireland}-2025
  (1 each): the HTML-only country statistics pages, archived and parsed on
  the document-numbers lane (203.3m / 229.1m / 181.7m GBP, FY2024-25).

Every uk-data snapshot value reconciles exactly (counts and England
repayments byte-equal; GBP amounts equal at publisher precision). Bundle
expectations re-measured under the disclosed-and-reverted sandbox:
39,418 facts / 66 packages on this branch, merged-bundle test passes.
Manifests are upload-truthful.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add academic_year period type to the source-package contract

2 participants