UK wave 2 continuation: DWP, VOA, scotgov, DfT and ONS national packages (#133) - #141
Open
juaristi22 wants to merge 14 commits into
Open
UK wave 2 continuation: DWP, VOA, scotgov, DfT and ONS national packages (#133)#141juaristi22 wants to merge 14 commits into
juaristi22 wants to merge 14 commits into
Conversation
…icyEngine#133) Wave-2 batch 1: policyengine-uk-data's OBR calibration surface from publisher truth. obr-efo-receipts-march-2026 (63 facts: income tax and NIC classes from sheet 3.4 accrued, NI total/VAT/fuel duties/CGT/SDLT from the cash-basis current-receipts sheet 3.8) and obr-efo-expenditure-march-2026 (133 facts: council tax by nation plus total from 4.1, 13 welfare lines with UC in/outside cap disambiguated by row from 4.9, BBC licence fee from 4.19). 196 source_projection facts, fiscal_year periods 2024-2030 storing the opening year, GBP billion value_scale 1e9, all rows table_record_kind total (each line is a total over its program population). Artifacts fetched directly from obr.uk (residential passes the 403 wall that forces uk-data's committed fallback and forced CBO's curated-CSV compromise), double-fetched byte-identical. Explicit per-year record sets rather than {year} templating so the full forecast surface lands in every build (PolicyEngine#135 year handling). Excluded as uk-data-computed: salary-sacrifice uprated series (SPP-PDF base values become their own package); private_school_students is ISC data in wave-1 scope. Registers both aliases and updates bundle expectations to the measured post-publication truth (39,415 facts, 62 packages, 22 periods, source obr=196, 57 tables). Consumer export awaits ledger publish-raw. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…icyEngine#133) Wave-2 batch 2. ons-savings-interest-income: UKEA HAXV annual observations 2018-2025 (GBP million, value_scale 1e6, 30-06-2026 release); policyengine-uk-data's flat-fill of the last actual through 2029 is a computed extension the facts-only ADR excludes. ons-families-households-2025: Table 7 household counts by type, 10 types x 2018-2025 (thousands, value_scale 1e3), LFS-derived so provenance_class survey_aggregate with survey_instrument named; row guards are exact cell text including indentation and note markers. Deliberately not ported: ons_tenure England totals (the SPREE workbook is LA-grain only; uk-data sums ~300 LA rows in code, and Ledger stores only publisher-emitted aggregation levels - the workbook becomes the wave-3 LA package). Salary sacrifice moves to batch 3: uk-data's pinned asset now returns status gone (its runtime download fails silently); current asset is media/6a673e0e5e87122783093901 (Private pension statistics, 2026-07-30). Bundle expectations re-measured with all four wave-2 packages: 39,503 facts, 64 packages, 24 periods, ons 1,334, obr 196, 59 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…licyEngine#133) Wave-2 batch 3. Six facts from the Private pension statistics tidy CSV (updated 2026-07-30): income tax relief on salary sacrificed pension contributions, total and by rate band, plus Class 1 employee and employer NICs relief. tax_year period 2024 (opening year). Six guard cells per row; [z]-suppressed by-rate NICs splits not ported. policyengine-uk-data's pinned asset for these tables returns status gone, so its runtime download silently drops the targets today, and its 3 percent p.a. uprating from the old 2023-24 base is a computed series the facts-only ADR excludes either way. Correction recorded on the PR: the resolve_pending first-print lane referenced in earlier issue comments does not exist in this repo; hand-transcribed uk-data families will be packaged from their actual publication files. HMRC CGT scoped: Table 1 is ODS, so CGT joins SPI behind the ods_used_range parser (next infrastructure item). Bundle expectations: 39,509 facts, 65 packages, hmrc 199, 60 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…consumer (PolicyEngine#133) Wave-2 batch 4. hmrc-cgt-statistics-2025: the full published Table 1 history, 37 disposal years (1987-88 to 2023-24) x 9 measures = 333 facts - taxpayer counts (thousands), chargeable gains and tax liabilities (GBP million), each for individuals, trusts, and the publisher's own Total columns. tax_year periods store the opening year; 2023-24 is provisional per the publication notes; percentage columns are derivable and not ported. Correction recorded on the PR: the ods_used_range lane already existed (dispatch + repeat-aware reader) with zero consumers - the earlier 'missing ODS parser' scoping mistook no-usage for no-capability. This package proves the lane end to end; SPI and the ODS-format DWP tables were never parser-blocked. Fidelity: policyengine-uk-data's hardcoded 378k taxpayers / GBP 65.9bn gains are exactly the publisher's Total columns for 2023-24 (verbatim); its AEA gating and PE-parameter uprating are populace-side measurement semantics. populace's native UK_CGT_TARGET_SPECS single-homes onto this package in populace#622. Bundle expectations: 39,842 facts, 66 packages, 59 periods, hmrc 532, 61 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ngine#133) Wave-2 batch 5, both on the ods_used_range lane. hmrc-spi-income-bands-2023: SPI collated tables 2023-24, Tables 3.6 and 3.7 - 13 total-income bands x 7 income types, counts (thousands) and amounts (GBP million), 182 facts at tax_year 2023. survey_aggregate with the Survey of Personal Incomes named as instrument. Numeric ODS band cells make numeric row-header guards workable here. uk-data's 1.9x property-income scale and its uprated projections are populace-side adjustments; means are derivable and not ported. Same artifact populace's national HMRC stage parses. dwp-benefit-cap-november-2025: benefit-cap tables to November 2025, Table 4 point-in-time - UC households capped at Nov-2025, GB Total (110,637) plus the 13 amount-capped bands, 15 facts at month 2025-11 (first UK month-period facts; first dwp source). uk-data's 110,637 is exactly this GB Total; its GBP 320.9m annualized reduction is a computed band-midpoint construction, excluded - populace can declare its own annualization over the ported distribution. Stat-Xplore families (UC by children, Scotland UC) stay parked pending an API key; two-child limit, PIP and DWP benefit-statistics publications are the next batch. Bundle expectations: 40,039 facts, 68 packages, 60 periods, 31 sources, hmrc 714, dwp 15, geography 1,054, 63 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…yEngine#133) Wave-2 batch 6 (final batch on this PR). dwp-uc-two-child-limit-2025: Tables 01, 03A/03B and 04A/04B of the April 2025 two-child limit data tables - all four status rows (in receipt / affected / not receiving a child element / exception granted) x headline, family-size and disability columns, 92 facts at month 2025-04. uk-data hand-transcribed only the 'not receiving' row and models the values at 2026, a populace-side timing declaration; the exception-granted rows are new coverage. Regional and time-series sheets are backfill-lane candidates. This PR is now content-complete for review: 9 packages, 912 facts. Remaining wave-2 families (PIP Jan-2026, DWP benefit statistics Feb-2026, VOA, NTS, ScotGov budget, land values, SLC after PolicyEngine#136, the Stat-Xplore extracts after an API key, and the uk_national profile in its own profile-lane PR) land in successor PRs after this merges, so the alias map and bundle constants stay conflict-free. Bundle expectations: 40,131 facts, 69 packages, 61 periods, dwp 107, 64 tables. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…citations, complete bundle expectations (PolicyEngine#133) Review verdict was needs-fixes; all findings resolved: - Entity correction (medium): DWP two-child-limit children counts were typed entity household. Table 01 is split into a household record set (column B) and a person record set (columns C-D); tables 03B/04B flip to entity person, role child_in_uc_household. DWP's captions count children, and 03B values are exact child-multiples of 03A households (891,930 = 3 x 297,310). Fact count unchanged (92). - Concept ids (low): all 23 TCL concepts hand-curated, replacing 48-char slug truncations (one with a trailing underscore) - concepts are durable join keys and renames after first publish are migrations. - SPI citations (medium): the income-tax-summarised-accounts landing URL now 404s; manifest source_page and all measure evidence URLs point at the live personal-incomes-statistics-2023-24 page. - OBR assertions (low): FY2024-25 columns are labelled Outturn in the workbooks; all 28 fy2024 record sets now carry assertion observation while fy2025-2030 remain source_projection. - Named exclusions (low): TCL and SPI head NOTEs now enumerate every within-sheet scope drop (04x columns H-J, exception-reason and family-type rows; SPI Other-income pair and total column); salary-sacrifice NOTE cites where the GBP-million unit is asserted. - Bundle expectations (high): the deep assertion blocks (by_period, by_geography length, K02000001, by_entity) are set to post-publication truth - 61 period keys, geography 1,054 with K03000001 at 107, K02000001 2,244, entities household 209 / person 4,453 (reflecting the entity split) - and PROVEN by running the bundle test under the disclosed, reverted R2 sandbox: 1 passed. Not changed, deliberately: OBR row kind stays total (flipping to detail would retrigger the acceptance battery's constraints requirement, which unfiltered aggregate lines cannot satisfy); commit 88d2ebb's message says 13 amount-capped bands where the artifact has 14 - the yaml and counts were always right. Review confirmations recorded: ~1,500 pins independently re-derived from artifact bytes with zero mismatches across all nine packages; all 912 facts distinct at fact/aggregate/semantic key levels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…icyEngine#133) Fixes the dead pe-uk-source-checklist.md reference in the harness doc and records, per uk-data target family at the pinned ref ebf733c: ported / gated / parked / wave-3 / excluded-as-computed status, which Ledger PR carries it, and whether the package reuses uk-data's exact source bytes or had to re-pin (runtime downloads, link rot, transcriptions). Also queues the harness commands for the two new wave-2 packages without entries. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…olicyEngine#133) 270 facts across the remaining unblocked wave-2 families: - dwp-benefit-statistics-february-2026 (4): ESA total/contributions/income and JSA claimants at Aug-2025 from the archived HTML statistical summary (html_tables_and_text document-numbers lane; the release is HTML-only). The ESA total lands as the publisher-emitted 1,000,000 ("1 million"); uk-data's 999,000 is Stat-Xplore precision for the same series. - voa-council-tax-bands-2025 (110): CTSOP2.0 stock by band at 31 Mar 2025, publisher national + regional rows; Wales-only band I split into its own record set ([z]-suppressed for England and regions). - scotgov-council-tax-bands-2025 (9): CTAXBASE Sep-2025 Scotland chargeable dwellings by band; the publisher's cached SUM row (total 2,623,149 - byte-identical to uk-data's hardcoded fallback total). - scotgov-scottish-budget-social-security-assistance-2026 (54): full Table 5.08 by line x FY2024-26 from the archived Chapter 5 HTML; uk-data's 2027-29 3% uprating excluded per the facts-only ADR. - dft-nts-vehicle-ownership-2024 (81): NTS0205a car-availability shares at publisher precision, 27 single-year rows; combined-period rows are a named exclusion. First dft source. - ons-public-sector-employment-2026 (12): PSE March-2026 edition Table 1 headcount by sector, Dec-2023 + Dec-2024, CDID column guards; ONS reclassifications revised the series above uk-data's transcription. Manifests are upload-truthful (no storage.r2 until publish-raw runs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…kages (PolicyEngine#133) Measured under the disclosed-and-reverted local storage.r2 sandbox (same protocol as PolicyEngine#137/PolicyEngine#139), where the merged-bundle test passes: 40,401 facts, 75 packages, 34 sources, 9 entities (dwelling is new), 84 periods, 1,067 geographies. All deep blocks (by_period, by_source, by_geography spot keys, by_entity, table count) re-measured together so the test converges with zero further commits once `ledger publish-raw --root db/data` runs on the branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ine#133) 2 facts: PIP claimants in receipt of the enhanced (1,608,000) and standard (1,283,000) daily living component, caseload at end January 2025 - the true publisher of policyengine-uk-data's pip_dl targets (its cited Jan-2026 PIP statistics publication carries clearance/journey tables only; correction recorded in the checklist). First pdf_text_numbers UK consumer. Publisher scope carried as first-class constraints: DWP policy ownership (England, Wales and Abroad), working-age, normal rules only; volumes publisher-rounded to the nearest 1,000. The FOI's less-than-4-points percentages are a named exclusion (withdrawn UC/PIP Bill analysis). Artifact provenance: the FOI host blocks automated fetches, so the response PDF was downloaded manually in a browser and registered from the local file; the manifest source_url is the canonical public attachment URL. Bundle expectations re-measured under the disclosed-and-reverted sandbox: 40,403 facts / 76 packages, test passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Batch 2: 2 facts from DWP FOI response FOI2025/24990 (caseload at end January 2025): enhanced daily living 1,608,000, standard 1,283,000 — the true publisher of uk-data's Bundle expectations re-measured under the same disclosed-and-reverted sandbox: 40,403 facts / 76 packages, merged-bundle test passes. 🤖 Generated with Claude Code |
…Engine#133) 176 facts from the 'Households on Universal Credit' dataset at May 2025, closing the parked Stat-Xplore lane now that an API key exists: - dwp-uc-payment-distribution-may-2025 (140): Monthly Award Amount (bands) x Family Type, GB. Publisher band labels are the categorical identities (incl. 'No payment' and the overlapping '£1500.01 or over' summary band); the x12 annualization uk-data applies is populace-side, and the open '£2500.01 or over' band - which uk-data's parser drops to NaN - is an ordinary category here. - dwp-uc-households-children-may-2025 (8): integer number_of_children constraints for digit categories, categorical for '5 or more' and tails. - dwp-uc-households-family-type-may-2025 (5): raw counts only; uk-data's undercount_relative inflation stays populace-side per the facts-only ADR. - dwp-uc-scotland-youngest-child-may-2025 (23): Scotland by single year of age of youngest child; age 0 = 14,412, the precise value behind uk-data's rounded 14,000. Single-year cells differ from the published 0-4 band by 4 households (Stat-Xplore disclosure-control rounding). Each artifact is the raw API response with its query echoed inside it (api_response precedent), on a new generic statxplore_table_json_rows lane (ledger/sources/rows.py + dispatch): one source row per cube cell, columns = measure, value, and a label/URI pair per dimension; publisher value URIs carry the codes (GSS S92000003 guard) and serve as row headers for digit categories, which int-coerce in the guard contract (mye precedent). Vintage note: uk-data's transcriptions are the provisional May 2025 (p) extract; DWP has since revised the month, so these are the current revised series (deltas roughly 3-5%, enumerated in the PR). Bundle expectations re-measured under the disclosed-and-reverted sandbox: 40,579 facts / 80 packages, merged-bundle test passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Batch 3: the four parked Stat-Xplore UC families — PR totals now 11 packages, 448 facts. An API key is now available, so the parked lane closes: New lane for review: Fidelity notes:
Gates: all four Remaining #133 scope after this batch: SLC (gated on #131/#136), land values, SPP Review PDF bases, ISC census (#132 remainder), and the 🤖 Generated with Claude Code |
…ngine#133, PolicyEngine#132) 95 facts across the last unblocked wave-2 families plus the wave-1 remainder: - ons-national-balance-sheet-land-2025 (90): Land (AN.211) at current prices, 1995-2024, whole economy / households / non-financial corporations (E44R/E44N/E43B) from the 2025-edition NBS reference tables. uk-data's hand-transcribed totals match exactly for 2021-2023; its 2024 value approximated the published 7,117,771 and its 2020 sector split is an earlier, since-revised vintage. Flat-fills, the fixed share split and the regional split stay populace-side per the facts-only ADR. - hmrc-salary-sacrifice-reform-2029-headcounts (3): 7.7m employees using salary sacrifice for pensions, 3.3m above / 4.3m within the GBP 2,000 threshold, from the archived HMRC policy paper (publisher relabel from uk-data's obr tag; the x1.024 growth extension is excluded). - isc-annual-census-2023 / isc-annual-census-2024 (1 each): 554,243 and 556,551 pupils at ISC member schools - the true publisher behind obr/private_school_students (557,000 = the 2024 figure rounded), closing the last wave-1 row of PolicyEngine#132. New isc source directory. Also recorded in the checklist: the 2025 SPP Review PDF behind the NI-relief bases is link-rotted with no Wayback snapshot - blocked until the document resurfaces; the ported TY2024-25 relief package is the anchor. Bundle expectations re-measured under the disclosed-and-reverted sandbox: 40,674 facts / 84 packages / 35 sources, merged-bundle test passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Batch 4 (final for this PR's reachable scope): land values, salary-sacrifice headcounts, ISC census — PR totals now 15 packages, 543 facts.
Blocked, recorded in the checklist: the 2025 SPP Review PDF behind the £1.2bn/£2.9bn NI-relief bases is link-rotted ( Gates: all four green through What remains on #133 after this PR: SLC (gated on #131/#136), the housing.py base facts (needs its own scoping — PRHI + EHS tables), the SPP Review bases (blocked above), and the 🤖 Generated with Claude Code |
… summary-band kind, caveat notes (PolicyEngine#133) Fidelity core: every value across the 15 packages independently re-derived from committed artifact bytes (2,120 checks - manifest shas, cell geometry, sentence pins, cube walks - zero mismatches), plus cross-package additivity checks. Fixed from findings: - VOA (from additivity): the publisher's UNMD 'Unmatched' row (note 15, properties unmatched to any constituency) is now ported with a geography_match == unmatched identity - England + Wales + Unmatched reconciles to the England-and-Wales row within nearest-10 cell rounding (band I exactly). +10 facts. Comment also corrects '[z]' to not-applicable (England has no Band I), not suppression. - NTS (major): the workbook's own note 3 says pre-1989 rows are Great Britain - the 1951/1961/1971/1981 record sets now carry K03000001, and per note 2 name the pre-NTS household surveys as their instrument. - Scottish budget (major): the FY2025-26 Two Child Limit Payment GBP 11.3m allocation is no longer ported - the publisher's same-table footnote withdraws it at Spring Budget Revision; now a named exclusion. -1 fact. - UC payment distribution: the five overlapping '£1500.01 or over' rows are table_record_kind total (machine-visible subtotal; detail rows are now disjoint), and the three UC cubes document cross-cube non-additivity from DWP cell-level adjustment. - DWP benefit statistics: the JSA 71,000 is flagged post-methodology-break (inactive-claims removal at Aug-2025), and the ESA notes state that the claim-type categories do not partition the total. - Salary-sacrifice headcounts: provenance_class corrected to model_output - the paper's counts are unnamed-source policy-costing estimates. - Checklist: family-type provisional-vintage deltas corrected to the measured +2.6%..+15.0% range; wrong PR link and a stale cross-reference fixed. PR body refreshed (crash-key name, current totals). Bundle expectations re-measured under the disclosed-and-reverted sandbox: 40,683 facts / 84 packages (552 on this branch), merged-bundle test passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
Adversarial review complete — all findings addressed in Fidelity core: every value independently re-derived from the committed artifact bytes across all 15 packages — 2,120 checks (manifest shas/sizes, xlsx/zip cell geometry and guards via direct openpyxl, prose facts via page-pinned sentence checks, Stat-Xplore facts via an independent cube walk with label-identity verification), zero mismatches. All 552 fact keys distinct. Cross-package arithmetic: Scottish budget lines sum exactly to the published totals in all three years; UC cross-cube marginals and VOA additivity produced the review's best catches below. Fixed from findings:
Accepted-as-disclosed (no change): PIP's K04000001 footprint with the FOI's "Abroad" tail carried via the Bundle expectations re-measured under the disclosed-and-reverted sandbox: 40,683 facts / 84 packages / 35 sources, merged-bundle test passes — converges with zero further commits after 🤖 Generated with Claude Code |
juaristi22
marked this pull request as ready for review
August 7, 2026 16:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #133 (and closes the last #132 package row). Stacked on #139 (branched from its head
a335f37) so the two drafts shareSOURCE_PACKAGE_ALIASESand one bundle-constants lineage; the diff collapses to this branch's commits once #139 merges.Lane:
ledger-source-ingestor— judgesledger-source-fidelity+ledger-boundary.What (current state, after batches 1-4 + adversarial review)
15 packages, 552 facts, five new source directories (
voa,scotgov,dft,isc, plus the firstdwpStat-Xplore extracts), and one new parser lane (statxplore_table_json_rows— raw Open Data API responses unpivoted to one row per cube cell; the censusapi_responseprecedent generalized):dwp-benefit-statistics-february-2026,voa-council-tax-bands-2025(incl. the UNMD unmatched row),scotgov-council-tax-bands-2025,scotgov-scottish-budget-social-security-assistance-2026,dft-nts-vehicle-ownership-2024,ons-public-sector-employment-2026dwp-pip-daily-living-foi-2025(FOI2025/24990, manually-downloaded PDF, provenance in the package comment)dwp-uc-payment-distribution-may-2025,dwp-uc-households-children-may-2025,dwp-uc-households-family-type-may-2025,dwp-uc-scotland-youngest-child-may-2025(Stat-Xplore Open Data API, key-authenticated, query echoed in each artifact)ons-national-balance-sheet-land-2025,hmrc-salary-sacrifice-reform-2029-headcounts,isc-annual-census-2023,isc-annual-census-2024Also adds
docs/pe-uk-source-checklist.md— the per-family uk-data→Ledger migration tracker (fixes a dead harness-doc link).Key fidelity notes
Gates
validate-package+build-suite+agent_acceptancegreen for all 15; adversarial review re-derived every value independently from committed artifact bytes (2,100 checks, zero mismatches) plus cross-package additivity checks (VOA E&W = England+Wales+Unmatched within publisher rounding; Scottish budget lines sum exactly to the published totals).storage.r2sandbox. Committed manifests remain upload-truthful.uv run ledger publish-raw --root db/databy a credentialed maintainer (15 artifacts). Note: a whole-tree run crashes oncms_nhe/historical_service_source's non-yearhistorical_service_sourcemanifest key (andssa/ssi_monthly_statistics_2024_12carries a similarsource_capturekey) — run per-directory roots, or fix those manifests first.Remaining #133 scope after this PR
SLC (gated on #131/#136), housing.py base facts (needs scoping — PRHI + EHS tables), the SPP Review bases (blocked, artifact lost), and the
uk_nationalprofile PR. Wave 3 (#134) untouched.🤖 Generated with Claude Code