An end-to-end, automated machine learning system that ingests real-world European electricity load data, engineers time-series features, trains and tunes a LightGBM forecasting model, serves predictions via a REST API, and continuously monitors for data drift — all built within free-tier cloud infrastructure.
Architecture · Features · Quick Start · API · Project Structure · Tech Stack
| Category | Detail |
|---|---|
| Data Pipeline | Automated ingestion from ENTSO-E (Kaggle mirror) → schema validation (Great Expectations) → Azure Blob Storage |
| Feature Engineering | 8 lag features, rolling statistics (24h/168h), calendar + German holiday detection — all with strict leakage prevention |
| Model Training | LightGBM with Optuna hyperparameter tuning (50 trials), MLflow experiment tracking, early stopping |
| Model Governance | Champion/challenger framework with configurable promotion threshold — only better models get deployed |
| Serving | FastAPI REST API with on-the-fly feature computation, background model hot-reload, Pydantic v2 validation |
| Monitoring | Evidently AI data drift detection with per-feature PSI scoring and automated retrain triggers |
| CI/CD | Full GitHub Actions pipeline: lint → test → Docker build → deploy → weekly retrain → weekly drift check |
| Explainability | SHAP TreeExplainer for global feature importance (beeswarm + bar plots) logged to MLflow |
| Infrastructure | Built entirely on free-tier: Azure for Students, Databricks CE, App Service F1, GitHub Actions |
- Python 3.11+
- Azure Storage Account with containers:
raw,processed,features,models - Kaggle API credentials
# Clone the repo
git clone https://github.com/knight22-21/GridCast.git
cd GridCast
# Create virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Configure environment
cp .env.example .env
# Edit .env with your AZURE_STORAGE_CONNECTION_STRING, KAGGLE_USERNAME, KAGGLE_KEY
# Setup pre-commit hooks
pre-commit install# 1. Ingest raw data from Kaggle → Azure Blob
python src/ingestion/download_entsoe.py
# 2. Engineer features (lag, rolling, calendar)
python pipelines/feature_pipeline.py
# 3. Train model with hyperparameter tuning
python src/models/train.py # Full Optuna tuning (50 trials)
python src/models/train.py --skip-tuning # Quick run with defaults
# 4. Start the prediction API
uvicorn src.serving.app:app --reload --port 8000
# 5. Run drift monitoring
python src/monitoring/drift_report.py# All tests
pytest tests/ -v
# With coverage report
pytest tests/ --cov=src --cov-report=term-missing
# Individual test suites
pytest tests/unit/ -v # Unit tests
pytest tests/integration/ -v # API integration tests
pytest tests/data_validation/ -v # Schema validation tests| Method | Endpoint | Description |
|---|---|---|
GET |
/health |
Health check — returns model readiness and uptime |
GET |
/model-info |
Champion model metadata, metrics (RMSE, MAE, MAPE) |
POST |
/predict |
Generate 1–168 hour demand forecast from recent load history |
POST |
/admin/reload-model |
Force immediate model reload from Azure Blob |
GET |
/docs |
Interactive Swagger UI documentation |
curl -X POST http://localhost:8000/predict \
-H "Content-Type: application/json" \
-d '{
"recent_load_values": [42000, 41500, ..., 43000],
"start_datetime": "2024-01-01T00:00:00Z",
"horizon_hours": 24,
"area_name": "DE_LU"
}'GridCast/
├── src/
│ ├── ingestion/ # Kaggle download + schema validation
│ ├── features/ # Lag, rolling, calendar feature engineering
│ ├── models/ # Training (LightGBM+Optuna), evaluation (SHAP), champion/challenger
│ ├── serving/ # FastAPI app + Pydantic schemas
│ └── monitoring/ # Evidently drift detection + auto-retrain trigger
├── pipelines/ # CLI orchestrators (feature, retrain, batch inference)
├── tests/
│ ├── unit/ # Feature, model, monitoring tests
│ ├── integration/ # API endpoint tests
│ └── data_validation/ # Great Expectations schema tests
├── notebooks/ # Databricks-ready Jupyter notebooks
├── docker/ # Dockerfile + serving requirements
├── .github/workflows/ # CI, CD, retrain, drift check
├── monitoring/ # Alert config + drift report output
├── deployment/ # Azure App Service deploy scripts
├── params.yaml # Central hyperparameter configuration
└── data/ # Local data directory (gitignored)
├── raw/ # Raw CSV from Kaggle
├── features/ # Engineered Parquet files
├── models/ # Champion model + metrics
└── drift_baselines/ # Training baseline for drift comparison
| Layer | Technology |
|---|---|
| Language | Python 3.11 |
| ML Model | LightGBM |
| Tuning | Optuna (Bayesian hyperparameter optimization) |
| Experiment Tracking | MLflow |
| Explainability | SHAP (TreeExplainer) |
| Feature Validation | Great Expectations |
| API Framework | FastAPI + Uvicorn |
| Data Validation | Pydantic v2 |
| Drift Monitoring | Evidently AI |
| Cloud Storage | Azure Blob Storage |
| Compute | Databricks Community Edition |
| Serving Infra | Docker + Azure App Service F1 |
| CI/CD | GitHub Actions |
| Linting | Ruff |
| Testing | pytest + pytest-cov |
| Pipeline Config | DVC, YAML |
The LightGBM model is evaluated on a chronological validation split (last 20% of data):
| Metric | Value |
|---|---|
| RMSE | ~1,200 MW |
| MAE | ~800 MW |
| MAPE | ~3.5% |
Performance varies based on the country/zone and time period. The champion/challenger framework ensures only models that improve RMSE by >1% are promoted.
- API Docs — Interactive Swagger UI (when running locally)
| Workflow | Schedule | What it does |
|---|---|---|
| CI | Every push/PR | Ruff lint → unit tests → integration tests → format check |
| CD | Push to main (serving changes) |
Docker build → push to ACR → deploy to App Service |
| Weekly Retrain | Monday 2:00 AM UTC | Feature rebuild → Optuna tuning → champion/challenger → artifacts |
| Weekly Drift Check | Monday 3:00 AM UTC | Evidently report → PSI threshold check → auto-trigger retrain |
This project is licensed under the Apache License 2.0 — see the LICENSE file for details.