Skip to content

UK wave 1: SLC student loan forecasts, student support and repayments (#132) - #155

Open
juaristi22 wants to merge 2 commits into
PolicyEngine:mainfrom
juaristi22:uk-wave1-slc
Open

UK wave 1: SLC student loan forecasts, student support and repayments (#132)#155
juaristi22 wants to merge 2 commits into
PolicyEngine:mainfrom
juaristi22:uk-wave1-slc

Conversation

@juaristi22

@juaristi22 juaristi22 commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

The wave-1 remainder, unblocked by #136's academic_year merge — with #137 (region×age population; ISC landed via #141 batch 4), this completes #132's package inventory.

Closes #132.

Lane: ledger-source-ingestor — judges ledger-source-fidelity + ledger-boundary.

What

Six packages, 199 facts, new slc source directory — and the first real consumers of two fresh mechanisms:

package facts mechanics
slc-student-loan-borrower-forecasts-england-2025 116 EES permalink Table 6a on the ees_permalink_table_html lane: borrowers liable to repay / earning above threshold, by product group × plan × status, AY2024/25–2029/30. First academic_year consumer: fact periods store the opening year while the EES value columns are END-year named (value_2025 = AY 2024/25) — the exact leak the ALLOWED_PERIOD_TYPES caution documents. All values are forecasts → model_output + assertion: source_projection, so profiles opt in via assertion_policy (#154). Null publisher cells skipped per year, not zero-filled.
slc-student-support-england-2025 72 slcsp052025 Tables 3A (maintenance-loan Grand-total recipients ×1e3 and amounts ×£1m) + 4C(i) (ADG + PLA recipients and amounts), AY2013/14–2024/25. 'Grand total ' carries a trailing space in the workbook — guards pin the exact text. Cohort rows and the other 4C products are named backfill exclusions.
slc-student-loan-repayments-england-2025 8 The corrected slcsp012025 workbook uk-data cites, Table 1A "Net repayments posted during the financial year", FY2024-25 block: all seven plan columns + Total HE at publisher £m precision, fiscal_year 2024. Earlier year blocks → backfill lane.
slc-student-loan-repayments-{scotland,wales,northern-ireland}-2025 1+1+1 The country publications are HTML-only; archived pages parsed on the document-numbers lane (£203.3m / £229.1m / £181.7m, FY2024-25), sentence + element-index pinned.

Fidelity vs policyengine-uk-data (every delta enumerated)

  • Exact matches: all EES Higher-education-total snapshot values (e.g. plan_2 above-threshold AY2024/25 = 3,985,000); Table 3A/4C counts; England repayments total 5,018,231,834.95 and plan 1; all three country figures.
  • uk-data derivations kept populace-side: its plan_2 (2,778,253,361.64) and postgraduate (346,409,713.95) England targets are FT+PT and Masters+Doctoral sums of the publisher's columns — the columns land here, the sums reconcile to the penny; its END-year calendar keying of academic years (its 2025 = AY 2024/25) is a timing declaration over these facts; its whole-pound roundings of published £m values (e.g. ADG 2013/14 £33,050,090.68 → 33,050,091) are transcription artifacts.
  • uk-data's EES scrape is runtime-only with hardcoded TESTING=1 snapshots; the archived permalink page is now the pinned artifact.

Gates

🤖 Generated with Claude Code

juaristi22 and others added 2 commits August 10, 2026 16:40
…PolicyEngine#132)

Six packages, 199 facts - the wave-1 remainder unblocked by PolicyEngine#136, and the
first consumer of the academic_year period type:

- slc-student-loan-borrower-forecasts-england-2025 (116): EES Table 6a
  borrowers liable to repay / earning above threshold by product group,
  plan and status, AY2024/25-2029/30 on the ees_permalink_table_html lane.
  Fact periods store the OPENING year while the EES value columns are
  END-year named (value_2025 = AY 2024/25) - the exact leak the
  ALLOWED_PERIOD_TYPES caution documents. All values are forecasts:
  model_output + assertion source_projection, so profiles opt in via
  assertion_policy (PolicyEngine#154).
- slc-student-support-england-2025 (72): Table 3A maintenance loans
  (Grand-total recipients and GBP-million amounts) and Table 4C(i) Adult
  Dependents Grant + Parents Learning Allowance, AY2013/14-2024/25.
  uk-data's whole-pound roundings of the published GBP-million precision
  are enumerated; cohort rows and the other 4C products are named
  backfill exclusions.
- slc-student-loan-repayments-england-2025 (8): the CORRECTED slcsp01
  workbook's Table 1A net-repayments row, FY2024-25 block, all seven plan
  columns + total at publisher precision. uk-data's plan_2 and
  postgraduate targets are FT+PT / Masters+Doctoral sums of these columns
  - populace-side derivations.
- slc-student-loan-repayments-{scotland,wales,northern-ireland}-2025
  (1 each): the HTML-only country statistics pages, archived and parsed on
  the document-numbers lane (203.3m / 229.1m / 181.7m GBP, FY2024-25).

Every uk-data snapshot value reconciles exactly (counts and England
repayments byte-equal; GBP amounts equal at publisher precision). Bundle
expectations re-measured under the disclosed-and-reverted sandbox:
39,418 facts / 66 packages on this branch, merged-bundle test passes.
Manifests are upload-truthful.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olicyEngine#132)

Six artifacts uploaded to ledger-raw under content-addressed keys via the
post-PolicyEngine#148/PolicyEngine#149 publish path. Flips the SLC bundle contract from designed-red
to green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@juaristi22
juaristi22 marked this pull request as ready for review August 10, 2026 14:55
juaristi22 added a commit to juaristi22/ledger that referenced this pull request Aug 11, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Six SLC packages covering all four nations with a registry-only code change — clean. Splitting England (which has the full forecast + support tables) from the three smaller repayment packages matches how SLC actually publishes.

1. academic_year looks like a new period type — is the vocabulary declared anywhere?

This PR introduces academic_year:2013 through academic_year:2029, which I don't see elsewhere in the existing by_period counts. If period types are a closed vocabulary validated at load, that needs a registration; if they're open, then the consumer side needs to know how academic_year:2024 aligns to a tax_year or calendar_year base period, or these facts will be unresolvable by any profile that asks for a normal year.

Related: academic_year:2025 through 2029 are borrower forecasts. #154 (open) adds the assertion: source_projection axis precisely for this, with observed_only as the default resolution policy. Without that field these forecast years look identical to the observed 2013-2024 history, and the first consumer to ask for "latest" quietly gets a projection. Worth sequencing against #154 — the SLC forecast table is a textbook case for it.

2. The golden-count test will collide with the other three open UK waves.

#137, #139, #141 and #155 all branch from the same base and rewrite the absolute totals in tests/test_chronicle_bundle.py to different values:

PR fact_count source_package_count period_count
#137 42895 64 19
#139 40131 69 61
#155 39418 66 33

Each is correct in isolation and stale the moment a sibling lands. That's four rebases with hand-recomputed constants, and resolving the conflict by picking a side gives you a green test that has stopped checking anything. Worth filing as its own issue — per-package delta assertions would compose where one whole-bundle snapshot can't.

3. Minor

  • The diff includes a couple of unrelated re-orderings in the by_source dict (hmrc/ici swapped, onem_rva_unemployment/ons swapped) that aren't caused by this PR's data. Harmless in a dict literal, but they add conflict surface against the three sibling branches that touch the same lines. Reverting them would make the rebase cheaper.
  • db/data/slc/student_loan_borrower_forecasts_england_2025/ees-table-6a-permalink.html is committed as a single line (+1). If that's a minified permalink response it's fine — just checking it isn't a truncated download.
  • Three of the four nations land as .html scrapes while England is .xlsx; if SLC publishes equivalent workbooks for Scotland/Wales/NI, those would be more stable inputs than parsed HTML.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UK source packages, wave 1: the production 149-target national surface (ONS SNPP, SLC, OBR)

2 participants