Reinforcement-learning agents for home battery dispatch — trained, evaluated, and served end-to-end.
A research dashboard for a Home Energy Management System (HEMS) trained on real Australian household solar + load data (Ausgrid, 30-min intervals, 300 households). Four families of RL agents (12 agents × 5 seeds = 60 checkpoints) are compared on annual electricity cost savings against a no-battery baseline. A live simulation tab lets you run any agent against any household and date window and see the battery dispatch, SoC trace, and TOU cost breakdown in real time.
Australian households with rooftop solar and a home battery still pay significant grid import costs because naive charge/discharge rules ignore time-of-use (TOU) pricing and forecast uncertainty. This project trains TD3, SAC, and DDPG agents with different forecast information levels and measures whether richer forecasts actually improve real-world robustness.
| Prefix | Observation | Trained with |
|---|---|---|
| I-* | Interval forecasts (PV, load, price — low/high bounds) | Noise injection |
| N-* | Point forecasts | Gaussian noise during training |
| V-* | Point forecasts | No noise (vanilla) |
| R-* | No forecasts | Reactive only — no future information |
Each variant × {TD3, SAC, DDPG} × 5 seeds = 60 checkpoints total. The Robustness tab shows how each agent degrades under bias, drift, and noise forecast errors.
Ausgrid CSV (962 MB)
│
▼
train.py ──► MLflow experiment tracking
│ │
│ model registry (hems-{variant}-{alg})
│ │
└─────────────▼
FastAPI backend (:8000)
├── /simulate ← loads checkpoint, runs rollout
├── /explain ← streams LLM summary (Claude)
├── /health
└── /household-range
│
▼
React + Recharts dashboard
├── Overview tab (aggregate results)
├── Household tab (per-household cost curves)
├── Cluster tab (radar chart by cluster)
├── Robustness tab (heatmap under forecast errors)
└── Simulation tab (live rollout — calls backend)
Requirements: Docker Desktop, the Ausgrid dataset at data/full_dataset.csv.
# First time — build the image and start both services
docker compose up --build
# After that — start without rebuilding (much faster)
docker compose up
# Run in the background
docker compose up -d
# Stop
docker compose down| Service | URL |
|---|---|
| Dashboard + API | http://localhost:8000 |
| MLflow tracking UI | http://localhost:5001 |
Ollama (local LLM for /explain) |
http://localhost:11434 |
The dataset is mounted as a read-only volume — it is never baked into the image.
Agent checkpoints are copied into the image at build time (agent/*.pt).
On the first docker compose up, the ollama-pull service downloads the explanation model
(qwen2.5:7b, ~4.7 GB) into a named volume — follow progress with
docker compose logs -f ollama-pull. Until it finishes, the Simulation tab's Explanation panel
shows a "still computing" spinner. The download happens once; later starts reuse the volume.
The raw Ausgrid download is three annual CSV files in wide format (one row per customer × date × consumption category, 48 time-of-day columns). data-preprocess/preprocess.py converts them to the normalised long format:
cd data-preprocess
pip install pandas numpy # one-time
python preprocess.py # writes ../data/full_dataset.csvOr call the function directly from Python:
from preprocess import build_full_dataset
df = build_full_dataset(
raw_paths=["ausgrid/2010-2011 Solar home electricity data.csv", ...],
output_path="../data/full_dataset.csv",
)What the script does:
- load = sum of
GC(general consumption) +CL(controlled load) rows per customer × time step - pv =
GG(gross solar generation) rows - price = 3-tier NSW residential TOU tariff computed from time of day: off-peak 0.2332 AUD/kWh (10:30–14:00), peak 0.4957 (15:30–21:00), shoulder 0.3276 (all other times)
- Wide → long reshape: 48 half-hour columns become individual rows with
dt= interval start timestamp
After building the dataset, run data-preprocess/data_preprocess.ipynb to cluster the 200 training households and regenerate test_clus_full.json (household → cluster assignments used by data_config.py).
Source: Ausgrid Solar Home Electricity Data — 300 households, 3 years (2010-07-01 → 2013-06-30), 30-minute intervals.
| Column | Type | Unit | Description |
|---|---|---|---|
customer_id |
int | — | Household identifier (1–300) |
dt |
datetime | — | Interval start timestamp (YYYY-MM-DD HH:MM:SS) |
load |
float | kW | Average household electricity demand over the 30-min step |
pv |
float | kW | Average rooftop solar generation over the 30-min step |
price |
float | AUD/kWh | Time-of-use electricity import tariff |
Sample rows:
customer_id,dt,load,pv,price
1,2010-07-01 00:00:00,3.106,0.000,0.3276 ← winter midnight, no solar
1,2011-01-15 12:00:00,0.440,1.888,0.2332 ← summer noon, solar > load
1,2011-01-15 13:00:00,0.308,3.038,0.2332 ← solar peak (~3 kW)
1,2011-01-15 18:00:00,1.620,0.000,0.4957 ← evening peak, no solar, high tariff
Key statistics:
| Value | |
|---|---|
| Households | 300 |
| Total rows | ~15.8 million |
| Date range | 2010-07-01 → 2013-06-30 (3 years) |
| Step size | 30 minutes (48 steps/day) |
| Load range | 0 – 15.6 kW |
| PV range | 0 – 8.8 kW |
| Price range | 0.23 – 0.50 AUD/kWh |
Note: The raw CSV contains two leading index columns that the backend drops automatically via
usecols. You do not need to pre-process the file.
AI-generated simulation explanations. The stack ships a local LLM (the ollama service,
qwen2.5:7b) so the Simulation tab streams a real explanation with no API key. The panel has a
Model dropdown to choose the engine per run — it lists each local model plus Claude (if a key
is set) and defaults to the local qwen2.5:7b. Switching engines re-generates the explanation for
the same run, and a badge shows which one produced the text. Under the hood /explain takes
engine + model; when none is given it falls back to the priority order ANTHROPIC_API_KEY
(Claude) → OLLAMA_URL (local) → plain-text template.
# Add a Claude key to enable the Claude option in the dropdown (optional, fastest):
echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .env
# Offer more local models in the dropdown (comma-separated; first = default).
# Larger models like qwen3.5:9b are only practical on a GPU host — see below.
echo 'OLLAMA_MODEL_CHOICES=qwen2.5:7b,qwen3.5:9b' >> .env- GPU: Docker-on-Mac is CPU-only (no Metal passthrough), so
qwen2.5:7btakes ~25 s warm; a bigger model likeqwen3.5:9bwould be ~1-4 min (use the Claude option for speed). On an NVIDIA Linux host, uncomment thedeploy.resourcesblock in theollamaservice for GPU acceleration — then larger models inOLLAMA_MODEL_CHOICESbecome practical. - Memory:
qwen2.5:7bneeds ~5-6 GB. If theollamacontainer is OOM-killed, raise Docker Desktop's memory limit or use a smaller model (e.g.qwen2.5:3b, ~3 GB).
hems-project/
├── Dockerfile multi-stage build (Node → Python)
├── docker-compose.yml api + mlflow services
├── .github/workflows/ci.yml GitHub Actions CI pipeline
├── requirements-test.txt light deps for CI tests (no torch)
├── ruff.toml Python linter config
│
├── hems-frontend/ Vite + React dashboard
│ └── src/
│ ├── components/ Overview, Household, Cluster, Robustness, Simulation
│ └── data/dashboardData.js baked-in chart data (no CSV at runtime)
│
├── hems-backend/
│ ├── main.py FastAPI app + API routes
│ ├── agent_loader.py checkpoint loading + inference (Tianshou 1.2.0)
│ ├── hems_env.py Gymnasium environment
│ ├── data_config.py household splits + date constants
│ ├── requirements.txt loose deps (development)
│ └── requirements-lock.txt pinned deps from pip freeze (Docker)
│
├── train.py unified training script (all variants + algorithms)
├── train_helper.py shared utilities: env builder, policy factory, samplers
├── test.py evaluation script (error sweep, MLflow logging)
├── register_existing_checkpoints.py one-time migration: register .pt files to MLflow
│
├── agent/ 60 checkpoints: {VARIANT}-{ALG}-{SEED}.pt
├── data/ full_dataset.csv (Ausgrid — not in repo, volume mount)
├── data-preprocess/
│ ├── preprocess.py build_full_dataset() — raw Ausgrid → full_dataset.csv
│ ├── data_preprocess.ipynb customer clustering (KMeans/GMM/Birch/Agglomerative)
│ └── ausgrid/ raw annual CSVs (not in repo — download from Ausgrid)
└── tests/ pytest test suite (21 tests, no dataset needed)
├── test_api.py
├── test_data_config.py
└── test_env_basic.py
Every push and pull request runs three jobs in parallel:
| Job | What it checks | Time |
|---|---|---|
lint |
ruff check on all Python files |
~30 s |
frontend-build |
npm ci && npm run build |
~2 min |
backend-test |
21 pytest tests (no torch needed) | ~3 min |
On push to main only, a fourth job runs:
| Job | What it checks | Time |
|---|---|---|
docker-build |
Full multi-stage image build with layer cache | ~2 min (cached) |
Tests run without the 962 MB dataset or torch by using a synthetic 200-row DataFrame for the environment tests and FastAPI TestClient for the API tests.
Training registers every checkpoint under hems-{variant}-{alg} (e.g. hems-N-TD3) and tags each version with seed, horizon_hours, reward_mode, and run_dt.
# Train a new agent (registers automatically)
python train.py \
--data_path data/full_dataset.csv \
--clusters clusters.json \
--forecast_style point \
--alg TD3 \
--seed 1
# Register the 60 existing checkpoints (one-time migration)
python register_existing_checkpoints.py
# Browse the registry
mlflow ui --backend-store-uri mlruns/ # http://localhost:5001The backend loads checkpoints from the registry when MLFLOW_TRACKING_URI is set,
falling back to local agent/*.pt files otherwise.
cd hems-backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload --port 8000cd hems-frontend
nvm use 22 # Node >= 18 required
npm install
npm run dev # :5173, proxies /simulate /health /explain to :8000The Simulation tab can stream a 3-5 sentence explanation after each run. Two backends are supported — the first one found wins:
Option A — Claude (Anthropic API)
echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .envOption B — Local LLM via Ollama (no API key needed)
# Install Ollama: https://ollama.com
ollama pull qwen2.5:7b # the shipped default
echo 'OLLAMA_URL=http://localhost:11434' >> .env
echo 'OLLAMA_MODEL_CHOICES=qwen2.5:7b' >> .env # comma-separate to add moreThe dropdown offers each model in OLLAMA_MODEL_CHOICES plus Claude (if a key is set); the first
listed model is the default. When /explain gets no explicit choice it falls back to the priority
ANTHROPIC_API_KEY → OLLAMA_URL → plain-text fallback (always works).
The model loads into memory on first use (cold start) and stays resident for ~5 min of idle, so
subsequent explanations are faster — qwen2.5:7b is ~25 s warm on CPU. Add larger models (e.g.
qwen3.5:9b) to the list only on a GPU host, where they run in seconds rather than minutes.
pip install pytest httpx ruff
PYTHONPATH=hems-backend pytest tests/ -vruff check hems-backend/ train.py train_helper.py test.py tests/npm run build --prefix hems-frontend
uvicorn main:app --host 0.0.0.0 --port 8000 --app-dir hems-backend &
cloudflared tunnel run hems-twin- Forecast style is per-agent:
agent_loader.pyreadsAGENT_STYLE_MAPand builds the correctForecastConfigautomatically.I-*→ interval (obs_dim 168),N-*/V-*→ point (obs_dim 104),R-*→ no forecast (obs_dim smaller, episode length = full window). - Error injection is disabled for R-*: The no-forecast agents observe nothing about the future, so injecting forecast errors has no effect.
build_sim_envforcesForecastErrorConfig(enabled=False)regardless of the UI selection. - Dataset is loaded once:
get_df()inmain.pycaches the 962 MB CSV in memory on first call./healthand/household-rangeare importable without the ML stack — torch/tianshou are deferred inside/simulate. - Policy cache:
agent_loader._policy_cachestores loaded policies byagent-seedkey, so repeated simulations with the same agent don't reload the checkpoint. - MLflow naming: model names use the agent prefix (
I/N/V/R) not the style string to avoid collision between N-* and V-* (both use point style but are distinct agents).
This project was developed with the assistance of Claude Code (Anthropic), an AI coding assistant. Claude Code was used throughout the development process — from scaffolding the FastAPI backend and React frontend, to wiring up the Docker Compose stack, CI pipeline, and AI-generated explanation feature.