Repository navigation
feat(pace): add empirical slot-remainder forecasting and identifiability gating - #1414
FelixIsaac wants to merge 1 commit into
Conversation
|
Thanks for your interest in contributing to OpenUsage! External pull requests must reference an open issue that:
Please discuss the change on an issue first, wait for a maintainer to approve and assign it to you, then reopen this pull request with |
There was a problem hiding this comment.
Important
Two forecasting issues need attention before merging: the empirical non-red gate is undone by the meter renderer, and incomplete historical observations can produce falsely reassuring forecasts.
Reviewed changes across the forecasting engine, local history store, meter and notification integration, tests, and documentation.
- Forecasting: Historical remainder medians blend with linear projections, with empirical hit-rate classification and elapsed/usage identifiability gates.
- Local History: Identity-scoped quota samples and weekly rollups are persisted locally and supplied through
WidgetDataStore. - Presentation And Alerts: Meter severity and pace notifications defer early-window warnings while preserving absolute remaining-quota alerts.
- Verification And Docs: New regression fixtures, behavior documentation, and research/simulation artifacts accompany the feature. Isolated Swift checks reproduced both findings; the complete macOS app test suite could not run on Linux.
gpt-6.1-sol | 𝕏
| else { status = .behind } | ||
| else if let hitRate = empiricalHitRate(used: used, limit: limit, remainders: remainders), | ||
| remainders.count >= 2, hitRate < hitRateRedThreshold { | ||
| status = .onTrack // projected over, but comparable slots rarely actually hit |
There was a problem hiding this comment.
[P2] The empirical non-red gate is reversed by the meter consumer. With 80/100 used halfway through a week and remainders [15, 22, 40], this returns .onTrack with a projection of 136.49, but WidgetData.meterState calculates negative spare and returns .runningOut anyway, so the bar remains red and can still trigger “Will Run Out.”
Technical details
# Preserve Empirical Classification Through Presentation
## Affected sites
- Sources/OpenUsage/Support/Pace.swift:146-148 returns onTrack without changing the over-limit projectedUsage.
- Sources/OpenUsage/Models/WidgetData.swift:529-531 maps every onTrack projection with spare below 1% back to runningOut.
- QuotaNotificationEvaluator consumes that meter state, so the mismatch affects alerts as well as color.
## Required outcome
- Preserve the intended empirical non-red verdict in the meter and notification path without fabricating positive spare from an over-limit projection.
- Add an integration regression using the existing testProjectedOverLimitStaysNonRedWhenHitRateBelowThreshold fixture, checking meterState and notification classification, not just Pace.Result.status.
## Reproduction
- used = 80, limit = 100, elapsed = 0.5 * week, remainders = [15, 22, 40].
- Actual Pace result: onTrack, projectedUsage = 136.48700633507892.
- Meter spare = -36, so the onTrack branch returns runningOut.| } | ||
| .max(by: { $0.recordedAt < $1.recordedAt }) | ||
| guard let atSlot else { continue } | ||
| collected.append(max(0, last.used - atSlot.used)) |
There was a problem hiding this comment.
[P2] A past reset timestamp does not establish that the cycle's final usage was observed. If the app records one mid-cycle reading and is then closed until the next cycle, last and atSlot can be the same reading, producing a false zero remainder; two such cycles are enough to turn a current half-week 60% reading from a 120% linear forecast into a reassuring 95.68% forecast.
Technical details
# Exclude Unobserved Cycle Remainders
## Affected sites
- Sources/OpenUsage/Support/QuotaSampleStore.swift:263-268 considers every past-reset, non-current cluster completed without observation-coverage requirements.
- Sources/OpenUsage/Support/QuotaSampleStore.swift:274-294 uses its latest observed sample as final usage, even when that observation occurred well before reset.
- Sources/OpenUsage/Support/Pace.swift:98-104 and 146-148 trust those remainders for projection and hit-rate gating.
## Required outcome
- Treat a missing remainder as unknown rather than measured zero when the cycle's end was not adequately observed; retain the no-history fallback for unusable cycles.
- Add a regression covering ordinary downtime across resets, with sparse prior cycles that must not supply zero remainders.
## Reproduction
- Construct two prior weekly windows, each containing only one sample at 50% elapsed.
- Query at 50% elapsed in a new week: remainders returns [0, 0].
- projectUsage(used: 60, elapsed: week / 2, periodDuration: week, remainders: [0, 0]) returns 95.67621345008163, versus 120 with no history.
- Actual unobserved usage may have continued on another device or while OpenUsage was closed, so these zero remainders are not evidence of zero remaining consumption.
Fixes #1415
TL;DR
Replaces fragile linear quota extrapolation with a robust, mathematically grounded pace forecasting engine combining physical burst decay (Modified Omori's Law / Damped ETS) for session windows, empirical slot-remainder blending for weekly cycles, and identifiability gating. Slashes session false-red alarms from 33.1% to 1.4%, reduces session MAE by 45.2%, and cuts weekly token error by 36.8% with zero measurable CPU overhead (<100 nanoseconds per evaluation).
What was happening
used / elapsed * period) assumes instantaneous burn rate persists uniformly forever. When a user runs 2–3 agent prompts in the first 15 minutes of a 5-hour session or on Monday morning, linear multiplies that burst across the entire period.What this changes
Adopts physical aftershock decay based on Modified Omori's Law and Holt-Winters Damped-Trend Exponential Smoothing (
Implements an adaptive power blend
Pace verdicts (yellow/red) and "Cutting It Close" / "Will Run Out" notifications are strictly gated until a window is identifiable (
QuotaSampleStore:Stores an atomic, 35-day rolling local ring buffer (
~/Library/Application Support/OpenUsage/quota-samples/) coalesced to 1-hour (weekly) and 10-minute (session) buckets. Resilient to provider clock jitter (Heads-up
Tests
PaceForecastTests.swiftlocking in adaptive power-law blending, empirical hit-rate gates, and horizon projections.QuotaSampleStoreTests.swifttesting identity isolation, rolling 35-day truncation, jitter clustering, and mid-window usage wipes.PaceTests.swift,MeterSeverityTests.swift,PaceNotificationLogicTests.swift, andWidgetDataStoreTests.swift(100% passing across the entire suite).simulate.pyand research scripts.