Local research dashboard for official energy-data signals. The first MVP focuses on the EIA Weekly Natural Gas Storage Report, read-only event-market discovery, deterministic signal generation, paper-trade thesis records, and in-app/Telegram research alerts.
python -m pip install -r requirements.txtOptional local config:
Copy-Item config.example.json config.jsonMarkets that need an exact or accepted settlement source can be unlocked with a local CSV feed when you have legal access to that data. Add the CSV as a commodity_prices.series item, then point market_solver_v02.target_sources at the imported benchmark:
{
"commodity_prices": {
"series": [
{
"name": "Configured WTI active-month daily high",
"source": "configured_settlement_csv",
"series_id": "WTI.ACTIVE_HIGH.D",
"commodity": "crude_oil",
"benchmark": "wti_active_month_high",
"frequency": "daily",
"unit": "usd_per_barrel",
"csv_path": "data/manual/wti-active-high.csv",
"date_column": "trading_day",
"price_column": "high"
},
{
"name": "Configured WTI active-month daily low",
"source": "configured_settlement_csv",
"series_id": "WTI.ACTIVE_LOW.D",
"commodity": "crude_oil",
"benchmark": "wti_active_month_low",
"frequency": "daily",
"unit": "usd_per_barrel",
"csv_path": "data/manual/wti-active-low.csv",
"date_column": "trading_day",
"price_column": "low"
},
{
"name": "Configured WTI active-month daily close",
"source": "configured_settlement_csv",
"series_id": "WTI.ACTIVE_CLOSE.D",
"commodity": "crude_oil",
"benchmark": "wti_active_month_close",
"frequency": "daily",
"unit": "usd_per_barrel",
"csv_path": "data/manual/wti-active-close.csv",
"date_column": "trading_day",
"price_column": "close"
}
]
},
"market_solver_v02": {
"target_sources": {
"wti": [
{
"benchmark": "wti_active_month_high",
"frequency": "daily",
"source_tier": "exact_settlement",
"market_status": "executable",
"label": "Configured WTI active-month high CSV",
"source_requirement": "configured_active_month_high_csv",
"price_field": "high",
"accepted_settlement_source": true
},
{
"benchmark": "wti_active_month_low",
"frequency": "daily",
"source_tier": "exact_settlement",
"market_status": "executable",
"label": "Configured WTI active-month low CSV",
"source_requirement": "configured_active_month_low_csv",
"price_field": "low",
"accepted_settlement_source": true
},
{
"benchmark": "wti_active_month_close",
"frequency": "daily",
"source_tier": "exact_settlement",
"market_status": "executable",
"label": "Configured WTI active-month close CSV",
"source_requirement": "configured_active_month_close_csv",
"price_field": "close",
"accepted_settlement_source": true
}
]
}
}
}CSV paths are resolved relative to the project root unless absolute. Imported rows keep raw-document archives and append-only value versions like other price feeds.
Use price_field values such as high, low, close, or ohlc on market_solver_v02.target_sources. Field-specific benchmark or source names such as wti_active_month_high, wti_active_month_low, and wti_active_month_close are still useful, but price_field is the explicit contract. The v02 solver filters accepted settlement feeds by market contract so close-threshold and Up/Down markets use close data, while intraperiod high/low touch markets use the matching high or low field.
For official or legally usable public APIs that return JSON, add a json_api commodity-price series. The imported observations use the same raw archive, revision tracking, and market_solver_v02.target_sources mapping as CSV feeds:
{
"commodity_prices": {
"series": [
{
"name": "Configured macro index API",
"source": "configured_api_json",
"adapter": "json_api",
"series_id": "MACRO.API.D",
"commodity": "macro",
"benchmark": "macro_index",
"frequency": "daily",
"unit": "index",
"url": "https://api.example.com/macro-index",
"query_params": { "series": "macro_index" },
"data_path": "payload.observations",
"date_column": "asOf",
"price_column": "metric.value"
}
]
}
}data_path, date_column, and price_column accept dot paths for nested JSON. APIs that need a key can use api_key_env with either api_key_param or api_key_header; headers containing keys or tokens are redacted in raw-document request metadata.
For legally usable public pages that publish a simple HTML table, use the html_table adapter. This is intended for official/reference pages where a CSV or JSON endpoint is not available:
{
"commodity_prices": {
"series": [
{
"name": "Configured retail gasoline table",
"source": "configured_html_table",
"adapter": "html_table",
"series_id": "FUEL.TABLE.D",
"commodity": "gasoline",
"benchmark": "configured_retail_gasoline",
"frequency": "daily",
"unit": "usd_per_gallon",
"url": "https://example.com/fuel-table",
"table_index": 0,
"date_column": "Report Date",
"price_column": "National Average"
}
]
}
}The adapter stores raw HTML in the archive and writes normalized observations through the same versioned price tables as CSV and JSON feeds. Use table_index when the page has more than one table.
When market_solver_v02.refresh_sources_on_market_refresh is enabled, the selected-market refresh endpoint now filters commodity-price ingestion to series that match the market target or configured target_sources entry. This keeps a single market refresh from crawling unrelated configured feeds; if no target-specific series can be matched, the app falls back to the broader commodity-price refresh. Official feeds such as NWS/NHC/EPA also use their own news_official_feeds freshness gate, so source-backed disruption context can update even when GDELT/news search was refreshed recently.
Manual watchlist markets can declare a normalized target directly when the parser does not know the market type yet. This keeps the market-centric dashboard flexible for non-energy or newly configured datasets:
{
"markets": {
"manual_watchlist": {
"enabled": true,
"markets": [
{
"venue": "manual",
"ticker": "MACRO-INDEX-320",
"title": "Will the configured macro index clear the target?",
"probability": 0.44,
"market_target": {
"commodity": "macro",
"benchmark": "macro_index",
"contract_type": "price_threshold",
"threshold": 320,
"direction": "above",
"target_period": "2025-07-31",
"source_requirement": "configured_macro_index_csv"
}
}
]
}
}
}CSV watchlists can use the same fields with target_ prefixes, for example target_benchmark, target_contract_type, target_threshold, target_direction, target_period, and target_source_requirement. Add a matching commodity_prices.series import and a market_solver_v02.target_sources.<benchmark> entry to make the market executable. Without that source, the market is still discoverable but marked Needs data.
For arbitrary event markets, use contract_type: "event_probability" and a configured source with price_field: "probability" and unit: "probability". The CSV/API value can be either 0.62 or 62; v02 normalizes both to a 62% event probability. This pattern is intended for legal model outputs, official event-probability APIs, or crawler-derived forecasts that have their own verification logic. Plain commodity prices are rejected for this contract unless the source is explicitly marked as a probability feed.
Multiple-choice event markets use the same contract. Put the market labels in answer_options, then configure one probability source per option with answer_option matching the label:
{
"market_target": {
"commodity": "event",
"benchmark": "event_option_probability_model",
"contract_type": "event_probability",
"target_period": "2026-08-31",
"answer_options": ["Alpha", "Beta", "Gamma"],
"source_requirement": "configured_event_option_probability_csv"
},
"market_solver_v02": {
"target_sources": {
"event_option_probability_model": [
{
"benchmark": "event_option_probability_model",
"series_id": "EVENT.OPT.ALPHA.D",
"frequency": "daily",
"source_tier": "exact_settlement",
"market_status": "executable",
"price_field": "probability",
"answer_option": "Alpha",
"accepted_settlement_source": true
}
]
}
}
}All configured answer options must have a probability feed before the market is executable. v02 normalizes the latest option values into a distribution and selects the highest-probability answer.
For average-over-period contracts, use contract_type: "period_average_threshold" or phrase the market as an average price/settlement threshold. v02 uses observations inside the target period when available and otherwise models the average with lower variance than a terminal close. This is distinct from price_threshold, which represents terminal level, and from touch contracts, which represent a benchmark reaching a level before expiry.
For price-touch contracts, use contract_type: "price_touch_threshold" with threshold, direction, and a target period, or phrase the market with wording such as "hit", "reach", or "touch". v02 applies a deterministic weighted path simulation to configured public price history for modelable proxy markets, using distance to threshold, days remaining, realized volatility, price momentum, petroleum/gas fundamentals, and any capped event adjustment; the feature snapshot exposes those components for audit. Explicit active-month, high/low, intraday, or 1-minute-candle contracts still require exact high/low settlement feeds.
The selected-market graph stores moving averages, threshold/range/baseline lines, a projected settlement-date marker, a one-sigma volatility band, and touch-state metadata such as observed extreme, first touch, current threshold gap, threshold-touch probability, and path counts when the contract resolves on a touch.
The dashboard applies a selected-market section profile so benchmark prices, gas fundamentals, petroleum fundamentals, COT/reference imports, and active paper trades are filtered to the selected market commodity/contract and hidden when they are not relevant.
For baseline-direction contracts, use contract_type: "settlement_direction" or directional_move with direction: "up" or "down" and a target period. v02 prices the direction from configured public history and stores baseline_value, baseline_period, and baseline_label in the graph payload so the selected-market graph can show what Up/Down is measured against. Active-month Pyth/CME Up/Down markets remain active_month_settlement_direction and still require exact accepted close data.
Selected-market news is filtered before it appears in the market forecast: v02 keeps articles matching the market commodity, target horizon, and relevant event types such as OPEC policy, refinery outages, weather-driven demand, LNG outages, or grid stress. The detail view surfaces this as event_context with one to three concise bullets; generic news remains evidence/context unless a scored lab model promotes it.
For threshold-touch markets, matched disruption context can add a capped event_adjustment to the model probability. This is a bounded jump-risk input from source-backed events, not a replacement for price history, fundamentals, or market odds.
Official source ingestion supports RSS/Atom feeds, configured html_page sources, KML/GIS feeds, and official JSON alert feeds such as NWS active alerts. The default official feeds include NOAA/NWS weather alerts, NOAA/NHC tropical outlook plus active KML feeds, and the EPA fuel waivers page so weather and gasoline policy shocks can be stored as source-backed context without a paid API.
For binary range contracts, use contract_type: "price_range" with lower_threshold, upper_threshold, direction: "inside" or direction: "outside", and a target period. The parser also recognizes clear wording such as "between $70 and $80". v02 estimates the terminal settlement probability inside the band and the selected-market graph shades the target range.
For spread/difference contracts, use contract_type: "spread_threshold" with left_benchmark, right_benchmark, threshold, direction, and a target period. v02 accepts an exact configured spread series first; otherwise it derives a proxy history from synchronized same-unit leg series, for example Brent minus WTI for brent_wti_spread. Direct comparison wording such as "Brent higher than WTI" is represented as the same spread contract with threshold: 0.
For ratio contracts, use contract_type: "ratio_threshold" with left_benchmark, right_benchmark, threshold, direction, and a target period. v02 accepts an exact configured ratio series first; otherwise it derives a dimensionless ratio from synchronized same-unit leg series, for example Brent divided by WTI for brent_wti_ratio.
For percentage-return contracts, use contract_type: "percentage_move_threshold" with percent_threshold, direction: "up" or "down", and a target period. The parser recognizes wording such as "rise at least 5%" or "drop at least 10%"; v02 prices the terminal return with the same benchmark history used for directional markets.
For duration/count threshold contracts, use contract_type: "observation_count_threshold" with threshold, count_threshold, direction: "above" or "below", and a target period. The parser recognizes wording such as "above $75 on at least 5 days"; v02 uses observed in-period days when available and otherwise estimates the remaining daily hit count from benchmark history.
Bootstrap live EIA data, discover markets, create signals, and draft paper trades:
python -m forecaster bootstrapIn the dashboard, Data Freshness -> Refresh Data is source/discovery-only: it refreshes feeds and market discovery without running legacy signal generation, paper-trade scoring, Telegram dispatch, or per-market forecast runs. Use a market card's Refresh Data action to refresh relevant sources and create a durable selected-market forecast run.
Start the local dashboard:
python -m forecaster serveOpen:
http://127.0.0.1:8000
python -m forecaster ingest eia-ng-storage
python -m forecaster ingest eia-ng-storage-history
python -m forecaster ingest gas-fundamentals
python -m forecaster ingest weather
python -m forecaster ingest petroleum
python -m forecaster ingest cot
python -m forecaster ingest reference-markets
python -m forecaster ingest commodity-prices
python -m forecaster ingest expectations
python -m forecaster ingest news
python -m forecaster ingest official-feeds
python -m forecaster discover-markets
python -m forecaster run-signals
python -m forecaster backtest signals
python -m forecaster lab replay --as-of 2026-06-19
python -m forecaster lab replay --as-of 2026-06-20 --commodity crude_oil
python -m forecaster lab walk-forward --start 2026-01-01 --end 2026-06-30
python -m forecaster lab walk-forward --start 2026-01-01 --end 2026-06-30 --commodity gasoline
python -m forecaster lab score
python -m forecaster score-paper-trades- Live trading and authenticated order placement are intentionally out of scope.
- Polymarket is discovery-only until eligibility is confirmed.
- LLMs are optional and limited to summaries/rationale. Numeric signals are deterministic and source-backed.
- Backtests and paper-trade outcomes are research calibration tools, not trading claims.
- The Prediction Lab replays stored public data as of a chosen date, stores feature snapshots, and scores 1d/7d/30d direction and above/below-threshold outcomes for Henry Hub, WTI, RBOB gasoline, and ULSD/heating oil without mutating live signals. Lab snapshots include as-of executable market candidates using the same derived use-status filter shown in Markets. Commodity price observations keep append-only value versions so replay entry/context prices do not inherit later corrections.
- Gas fundamentals add EIA API v2 public-data context for production, LNG/export proxy, electric-power consumption, Lower 48 demand forecast, nuclear outages, and NWS weather-demand degree-day context. Fundamental and weather observations keep append-only value versions for strict historical replay, and candidate features stay contextual until scored walk-forward lab evidence meets the live promotion gate.
- Petroleum fundamentals add EIA API v2 public-data context for crude inventories, Cushing/SPR, imports/exports, refinery utilization, crude inputs, gasoline/distillate inventories, and product supplied. The first deterministic petroleum thesis flags crude-headline moves that contradict product subseries. Petroleum observations keep append-only value versions so historical lab replay uses the value available at the replay cutoff instead of later revisions.
- Reference-market context adds CFTC COT positioning and manual CSV import for futures curves, options-implied probabilities, and ETF proxies. COT and manual reference rows keep append-only versions for strict as-of replay; COT is used as a regime hint and does not alter live confidence until calibrated.
- Market discovery includes read-only Kalshi/Polymarket, ForecastEx public data, Manifold crowd-belief context, optional authenticated Metaculus/IBKR/ForecastTrader context, and manual event-contract watchlists or CSV imports for venues without reliable public APIs.
- News ingestion includes GDELT, configurable official EIA/NOAA RSS feeds, and optional credential-gated Alpha Vantage News Sentiment, Currents, Marketaux, NewsAPI, and Finnhub providers; article rows keep append-only versions for strict as-of replay. News supports audit and review but does not directly change confidence. Prediction Lab contribution rows segment news by provider, source, and event type so news can be reviewed only after scored walk-forward evidence exists.
- Advanced power-market layers remain planned extensions.
Supplemental storage-surprise features can affect live direction/confidence only when prior scored walk-forward lab predictions for the matching segment meet prediction_lab.live_promotion thresholds. Defaults require at least 20 scored predictions, 55% hit rate, positive average return, and positive risk-adjusted edge. Replay-only runs are ignored for promotion evidence.
Prediction Lab adaptive refinement is separate from live promotion. prediction_lab.adaptive_refinement defaults to at least 20 prior resolved walk-forward samples, 58% hit rate, positive average return, and 0.05 risk-adjusted edge before a replay prediction can be calibrated or inverted. Walk-forward runs are evaluated online: later replay dates can use only earlier predictions whose target dates have already resolved. prediction_lab.actionable_gate is stricter: defaults require at least 30 samples, 75% hit rate, positive average return, 0.30 risk-adjusted edge, recent holdout confirmation when at least 5 recent samples exist, an allowed feature family, and no inversion before a prediction is marked as a model candidate. Feature contribution rows are grouped by model, commodity, target instrument, feature family, and segment, and include conservative feature_combo segments such as news-source plus price-direction, confidence plus price direction, inventory change plus horizon, and executable-market edge plus horizon or price direction. Contribution rows also report a recent holdout hit rate and stability delta so deteriorating segments are visible before promotion. Executable market snapshots parse threshold contracts and only compute side-specific YES/NO model edge when the contract threshold is close enough to the current benchmark price and the market expiration lines up with the prediction horizon; defaults cap threshold distance at 2% for 1-day forecasts, 5% for 7-day forecasts, and 10% for 30-day forecasts through prediction_lab.executable_markets.max_threshold_distance_by_horizon_pct, with max_threshold_distance_pct as the absolute cap. Farther thresholds or mismatched expirations remain context only.
Add these to config.json under external_data when available:
eia_api_key: EIA API v2 historical routes for natural gas, petroleum, electricity, LNG, and production data.fred_api_key: FRED Henry Hub spot and related macro/energy price series.nasdaq_data_link_api_key: Historical futures/ETF datasets where available through Nasdaq Data Link.cme_datamine_api_id: CME DataMine historical futures data, if you have purchased/entitled access.ibkr_api_token,forecasttrader_api_key: Optional event-contract discovery credentials. The example config usesIBKR_API_TOKENandFORECASTTRADER_API_KEYenvironment variables.alpha_vantage_api_key,currents_api_key,marketaux_api_token,newsapi_api_key,finnhub_api_key: Optional news-provider credentials. The example config usesALPHA_VANTAGE_API_KEY,CURRENTS_API_KEY,MARKETAUX_API_TOKEN,NEWSAPI_API_KEY, andFINNHUB_API_KEYenvironment variables.kalshi_api_key_idandkalshi_private_key_path: authenticated Kalshi portfolio/order data later; not needed for current public market browsing.polymarket_clob_api_key,polymarket_clob_secret, andpolymarket_clob_passphrase: Polymarket L2 CLOB credentials for eligible users only. Current app behavior remains discovery/read-only.polymarket_private_key_env: environment variable name used by the optional credential helper. Never paste or commit a private key.polymarket_deposit_wallet_addressandpolymarket_signature_type: later authenticated CLOB setup if eligible; not used for order placement in this app.nws_user_agent: NWS weather API requires an identifying User-Agent rather than an API key.
Optional Polymarket credential derivation for eligible users:
$env:PRIVATE_KEY = "0x..."
python scripts/derive_polymarket_credentials.py --i-am-eligibleThe helper prints credentials only; it does not save them and it does not place orders.
python -m pytest -q