Add opt-in flags to estimate MTRs against engine data and to carry NICs as payroll

[vahid-ahmadi]

Addresses #80. Both changes are behind flags defaulting to today's behaviour, so no existing result moves.

Why

The engine's MTRs are computed and then discarded. calibrate() estimates a Gouveia-Strauss function on the ETR only, then reuses those parameters verbatim as mtrx_params and mtry_params. The mtr_labinc / mtr_capinc columns cost an extra Simulation per income type plus the _person_mtr incidence correction, and never enter the objective — they affect the fit only through the cleaning quantile filters. Labour and capital therefore share an identical marginal schedule, which is wrong for the UK: dividend rates, the savings allowance and the PSA all differ from earned-income treatment.

Payroll revenue is mis-attributed. frac_tax_payroll = 0.0, so the revenue decomposition assigns all NICs revenue to income tax.

What changes after merging

Nothing, by default. Two opt-in flags become available:

• estimate_mtrs=True — mtrx_params fitted against mtr_labinc, mtry_params against mtr_capinc, so labour and capital can diverge. Also threaded into _estimate_bracket_tax_functions, so age_specific="brackets"/"each" honour it rather than silently ignoring it.
• separate_payroll=True — payroll_tax_liab carries person-level NICs, total_tax_liab comes from the engine columns rather than the clipped-ETR imputation, and frac_tax_payroll is computed as Σw·NICs / Σw·(IT+NICs).

Two booleans rather than one enum: the changes are independent and each needs to be attributable on its own.

Important: NICs are NOT removed from the ETR numerator

#80 originally proposed stripping NICs out of the ETR base. That would be wrong, and I've corrected the issue.

OG-Core's estimated ETR is a combined income-tax-and-payroll schedule by design (aggregates.py:497-498: "Payroll taxes are embedded in the income and payroll tax functions, so revenue is a fraction of the combined revenue"). frac_tax_payroll is used only in aggregates.revenue to split combined revenue for reporting, and OG-Core's own pipeline computes the same ratio with payroll inside the denominator (txfunc.py:1083). Removing NICs from the numerator would delete them from the household budget constraint unless tau_payroll were separately calibrated — and the UK NIC schedule is not flat, so one rate cannot represent it.

So separate_payroll changes the split, not the base. _payroll_split's docstring states what the fraction is of, over which sample, for which year.

Change

oguk/api.py (+286/−69). _MicroData gains adult-masked income_tax and national_insurance. New helpers, all on the production path: _payroll_split, _fit_gs, _fit_gs_triple, _liability_columns, plus TXFUNC_COLS / GS_NUMPARAMS. These also remove the duplicated frac / column / txfunc_est blocks between the pooled and bracket estimators. CHANGELOG entry included.

Evidence

oguk/tests/test_tax_function_flags.py — 14 tests in 8.2s, no calibrate() or _build_specs() call. Flag presence and defaults on all four entry points; _liability_columns in both modes; _payroll_split computed-not-hardcoded, zero under default columns, zero-total guard; both estimators end-to-end on a 1,200-row synthetic frame with deliberately steeper labour than capital MTRs.

The divergence, which is the point of the first change:

flag OFF -> etr == mtrx == mtry            (True)
flag ON   etr : [0.99856 0.48104 0.00088]
flag ON   mtrx: [0.99995 0.66611 0.00023]
flag ON   mtry: [0.99999 0.47645 0.00039]

ruff format / ruff check clean. Full suite: 25 passed, 3 skipped, plus 2 pre-existing test_get_micro_data.py failures (DatasetMaterializationError, missing HUGGING_FACE_TOKEN) — confirmed present on unmodified main.

What is not verified

• The flag-on paths are exercised only on synthetic data. A prior docstring warned that separate MTR estimation "introduces instability (the optimiser can land in distant basins)". DE with OG-Core's seed=1 is deterministic, but I cannot show that the real mtr_capinc distribution — heavy zero mass, clipped to [0,1] by _person_mtr — gives a sensible GS fit. Anyone publishing numbers from estimate_mtrs=True should eyeball the fitted schedules first.
• Real-data frac_tax_payroll is uncomputed (no HF access here). Expect roughly NI/(IT+NI) ≈ 0.2–0.25; the definition cannot exceed OG-Core's validator cap of 1.0.
• No SS/TPI solve was run, so flag-off equivalence is by construction rather than measured: _liability_columns(md, False) returns exactly etr*market_income and zeros, and _fit_gs_triple(..., False) returns the same ETR array three times, matching the previous triple-return.

[vahid-ahmadi]

@jdebacker This one is also ready for review whenever you have time.

It adds two opt-in flags, both defaulting to today's behaviour, so nothing existing moves:

• estimate_mtrs — currently the engine's mtr_labinc/mtr_capinc columns are computed at real cost (an extra Simulation per income type, plus the person-level incidence correction) and then never enter the objective; the ETR fit is reused verbatim for both marginal schedules, so labour and capital face identical marginal rates. With the flag on they are estimated separately.
• separate_payroll — frac_tax_payroll is currently 0.0, so the revenue decomposition attributes all NICs revenue to income tax.

One thing worth stating plainly, because my own issue #80 had it wrong: this does not remove NICs from the ETR numerator. OG-Core's estimated ETR is a combined income-tax-and-payroll schedule by design (aggregates.py:497-498), and frac_tax_payroll only splits combined revenue for reporting. Stripping NICs out would delete them from the household budget constraint unless tau_payroll were separately calibrated, and the UK NIC schedule is not flat. So the flag changes the split, not the base.

Flag-off equivalence is measured, not just argued: bit-identical to main through both the pooled and bracket estimators, including the bracket-to-age mapping refactor.

The caveat I would want a maintainer's eye on: the flag-on paths are exercised only against synthetic data, and there is a prior warning in the code that separate MTR estimation can land the optimiser in distant basins. Anyone publishing numbers from estimate_mtrs=True should eyeball the fitted schedules first — that is in the PR body too.

Test is red repo-wide for an unrelated token issue; see my note on #75.