feat(optimizer): flatten cost-neutral grid peaks and report service shortfalls

[frahlg]

Inspired by evcc-io/optimizer: take the two things that are worth stealing, and leave the rest.

evcc’s planner is a cloud MILP that currently only visualises a schedule. Ours is already on the dispatch path. Their useful ideas are (1) a cost-neutral second solve that flattens grid peaks among equally priced schedules, and (2) telling the caller when a goal was missed. Soft grid limits, EV-as-battery, CBC/PuLP, and cloud-by-default stay out.

Accepted text proposal

This work was requested in the agent thread: compare evcc-io/optimizer with FTW, take the useful bits, and implement them. Maintainer (Fredrik) asked to take the comparison into a build plan and then implement.

What changed

After the economic solve, a preference stage keeps that cost (plus numerical slack, not 0.1 öre) and minimises the horizon’s import and export peaks. Site fuse limits stay hard. If the extra solve is late, fractional, or fails, the economic schedule is kept. Both the direct HiGHS champion and the CVXPY fallback do this. A one-slot horizon skips the second solve (preference_stage: single_slot) because there is no tie to break across time.

The solver payload now carries preference_stage, import_peak_w, export_peak_w, and an optional service_report with remaining flex/storage/thermal shortfall. The HiGHS champion path reports storage-target shortfall; flex and thermal still go through CVXPY (direct HiGHS is ineligible for those assets). Core unmarshals those onto SolverInfo so /api already exposes them on active_solver. No UI copy in this PR.

settings.flatten_peaks defaults to true. false turns the stage off. Handshake feature: preference_flatten_peaks. Protocol version is unchanged.

Why this implementation

Putting a tiny peak weight in the same objective is what evcc had to abandon: the MIP gap swallows it. They freeze money, then break the tie. Same here, as a third stage after the existing service-then-economic split. That is the right place: service stays lexicographic, money stays lexicographic, peak shape is only a tie-break.

0.1 öre of cost slack was real money: at 50 öre/kWh it buys 2 W of peak reduction. That failed unique-export and arbitrage-spread tests. Slack is now 1e-8 öre (or 1e-12 of |cost|), which is solver noise, not a watt budget.

Boundaries and safety

• Hard import/export limits are not softened.
• Recourse and multistage challengers are unchanged (diagnostic only).
• Go-DP fallback does not flatten.
• No operator YAML knob yet.
• Planner output is still validated in core before dispatch.
• fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 (leftover PV + battery grid-charge) still owns the surplus-only constraint in model.py. This PR’s model.py delta is the post-economic hook only; rebase onto fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 if they land together. Both also touch go/internal/mpc/mpc.go / external_optimizer_test.go. fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 has no reviews yet.

Dual release

Old optimizer images omit the new fields; Core ignores unknown JSON today and understands the fields after this. Flattening only happens once the optimizer image includes this commit. Ship a new ftw-optimizer image with this Core.

Verification

optimizer/.venv/bin/pytest -q optimizer/tests
go test ./internal/mpc -count=1

• 185 optimizer tests passed locally
• two equal-price cheap hours then an expensive load: flatten charges both cheap hours instead of one spike
• flattening does not spend money versus flatten_peaks: false (abs 1e-3 öre)
• profitable fuse-limited export is not nibbled to lower the peak
• one-slot horizons report single_slot and keep the economic plan
• an expired preference deadline reports no_time and does not start the second solve
• an impossible EV deadline still returns a plan and names the kWh shortfall (CVXPY)
• an impossible storage energy target is named on both HiGHS and CVXPY

Checklist

• The diff implements one accepted scope and does not add follow-on work.
• I checked open pull requests that touch the same files. fix(mpc): surplus-only EV can take leftover PV while the home battery grid-charges #957 has right of way on leftover-PV in model.py; this hook is post-economic only.
• Tests cover the changed behaviour and its failure path.
• A human reviewed every changed web/UI view in a browser, or no UI changed.
• A Changeset is included, or the change is exempt.
• Every commit has a DCO sign-off.