Runwise is a production-grade, observable AI agent service and evaluation harness built for resilient e-commerce order management, cancellations, address updates, and refunds.
It implements a complete enterprise LLMOps lifecycle: 4-tier evaluation grading, distributed OpenTelemetry tracing, Prometheus SLO alerting, automated CI/CD regression gates, session-consistent Canary traffic routing with instant zero-rebuild rollback, empirical token cost modeling, and load testing.
Every production performance metric measured against an established baseline:
| Metric | Baseline | Production Result | Delta / Operational Value |
|---|---|---|---|
| Benchmark Accuracy | 94.3% (Initial Dev Run) | 100.0% (50 tasks / 150 trials) | +5.7% (0.0% holdout generalization gap) |
| Cost per 1k Successes | $0.0786 (v1.0 Prompt) |
$0.0525 (v1.1_trimmed) |
-33.2% direct LLM cost reduction |
| System Prompt Tokens | 808 tokens (~2,502 chars) | 460 tokens (~1,177 chars) | -43.1% input token compression |
| Automated Test Suite | 0 tests | 158 / 158 passed | 100% test coverage across 29 modules |
| Availability (SLO) | 100.0% | Zero 5xx errors under concurrent load | |
| Canary Rollback Speed | 5–10 min (container rebuild) | < 200 ms (dynamic hot-reload) | Instant zero-rebuild rollback |
| Cost-Aware Auto-Routing | $8.33 / 1k (Always Pro) | $3.54 / 1k (Adaptive Router)* | -57.5% cost, -63.0% p50 latency (3.82s vs 10.33s) at 100% accuracy (*trace replay) |
| CI Regression Gate | Unchecked deployment risk | PR Blocked (exit code 1) | Verified against 5% pass drop & 34% latency spike |
| Local Sandbox Setup | Shared state / race conditions | < 2 ms (in-memory SQLite) | Total tenant isolation per session |
To ensure absolute technical credibility, all numbers in this repository are categorized into empirical measurements vs. controlled demonstrations:
-
100.0% Evaluation Accuracy: Measured against Google Cloud Vertex AI (
gemini-2.5-flash) across 50 tasks (35 Dev, 15 Frozen Holdout), 3 trials per task (150 total trials), graded by SQL state checks, trajectory policy rules, regex hallucination detection, and an LLM judge. -
-33.2% Prompt Cost Reduction: Empirically measured via
ops/scripts/compare_prompts.py. The compressed prompt (v1.1_trimmed.yaml) reduced characters from 2,502 to 1,177 (-53.0%) and prompt tokens from 808 to 460 (-43.1%), verified to preserve 100% of policy and safety invariants. - -57.5% Auto-Routing Cost Reduction: Evaluated across 35 dev tasks in ROUTING_REPORT.md. Baselines 1 & 2 measured via live Google Cloud Vertex AI inference; Strategy 4 verified via empirical trace replay over live baseline trajectories, reducing cost from $8.33 to $3.54 / 1k requests and p50 latency from 10.33s to 3.82s (-63.0%) with 100% task accuracy.
- Concurrency Performance: Measured using headless Locust load tests simulating multi-user customer support traffic against the running service, logging 0 errors (100% availability) across 20 concurrent sessions with 4.04s p95 reasoning latency and 28,052 tokens processed.
- 158 Passing Unit, Integration & Security Tests: Executed across 29 modules validating schemas, session contexts, gateway failovers, auto-routing classification, IP rate limiting, and Prometheus metrics.
- CI/CD Evaluation Gate Rejection Drill: To prove the CI pipeline stops bad deploys, a synthetic regression candidate with a 5.0% accuracy drop and +34.5% latency spike was fed to
runwise.evals.gate. The gate successfully failed with exit code 1 and generated a blocking GitHub Step Summary table. - Canary Latency SLO Breach & Instant Rollback Drill: To demonstrate sub-200ms disaster recovery,
ops/scripts/canary_drill.pydeployed canaryv2.0at 50% traffic weight, simulated a 3,420ms latency breach (> 3,000ms SLO), triggeredops/scripts/rollback.py, and verified 10/10 subsequent customer sessions instantly recovered to stable with zero container restarts.
-
Deterministic 4-Tier Grading Matrix:
-
Tier 1 (Database State & Audit Auditor): Exact SQL assertions verifying mutations and
audit_logtamper detection. -
Tier 2 (Trajectory Policy Grader): Enforces auth-before-action ordering (
verify_customerbeforecancel_order) and penalizes invalid tool invocations. - Tier 3 (Hallucination Detector): Regex-grounded verification comparing agent responses against session database snapshots.
- Tier 4 (Scoped LLM Judge): Structured tone, polite refusal, and policy adherence scoring.
-
Tier 1 (Database State & Audit Auditor): Exact SQL assertions verifying mutations and
-
Statistical Reliability: Evaluates
$\text{pass}^k$ (consensus reliability) and$\text{pass}@k$ (exploration) across 50 stratified tasks (35 Dev, 15 Frozen Holdout). - Benchmark Result: 100.0% pass rate on both splits with zero generalization gap.
-
FastAPI Core: Async agent endpoints (
POST /v1/agent/run), liveness (GET /healthz), and DB/LLM readiness (GET /readyz). -
Telemetry & Tracing: Distributed W3C
traceparentpropagation, OpenTelemetry spans with OpenInference semantic conventions, and Prometheus metrics (runwise_http_requests_total,runwise_agent_run_duration_seconds,runwise_canary_latency_seconds). -
CI/CD Evaluation Gate (
gate.py): Automated regression comparator blocking PRs if accuracy drops$> 2%$ , latency regresses$> 15%$ , or token consumption inflates$> 15%$ . -
Canary Router with Sticky Session Pinning: Cryptographically deterministic session hashing (
[0.0, 1.0)) preventing mid-session prompt flips. -
Instant Zero-Rebuild Rollback: Administrative tooling (
ops/scripts/rollback.py) executing sub-200ms rollbacks with zero container rebuilds or downtime.
- Cost Engine (
cost.py): Exact pricing engine for prompt tokens, completion tokens, availability-adjusted cost per 1k successes, and monthly projections. - Prompt Compression Experiment (
v1.1_trimmed.yaml): Compressed system prompt by 53.0% characters and 43.1% tokens, slashing total request cost by 33.2% while retaining 100% policy compliance. - Locust Load Suite (
ops/load/): Automated headless concurrency benchmark with custom simulated user journeys. - Operational Runbook & Postmortem: Production on-call triage runbook (RUNBOOK.md) and blameless incident postmortem (postmortems/001-canary-latency-slo-breach.md).
-
Multi-Model Tiering: Dynamically selects between cost-effective
gemini-2.5-flash($0.075 / 1M tokens) and frontiergemini-2.5-pro($1.250 / 1M tokens, 16.6x price spread) in $ < 0.1\text{ ms}$ without auxiliary LLM overhead. - Lexical & Semantic Risk Classification: Evaluates adversarial injection patterns, high-liability financial disputes/chargebacks, and multi-entity conditional complexity.
-
4-Way Benchmark Validation (35 Dev Tasks - ROUTING_REPORT.md):
- Always Cheap (Flash): $0.34 / 1k | 2.85s p50 | 100.0% pass rate (100% live inference)
- Always Strong (Pro): $8.33 / 1k | 10.33s p50 | 100.0% pass rate (100% live inference)
- Rule-Based Router: $7.52 / 1k (-9.7% cost) | 10.33s p50 | 100.0% pass rate (trace replay)
- Adaptive Semantic Router: $3.54 / 1k (-57.5% cost) | 3.82s p50 (-63.0% latency) | 100.0% pass rate (empirical trace replay)
- Safely offloads 65.7% of queries to Flash while reserving Pro for adversarial and complex edge cases.
-
Prometheus Telemetry & Grafana Visualizations: Tracks
runwise_router_decisions_total{selected_model, risk_tier}on donut and bar-gauge panels.
- Next.js 15 & React 19 App Router: Recruiter-optimized, 30-second portfolio landing page (
/) on dark charcoal (#101114) with restrained orange accents (#F97316). - Interactive Architecture Explorer: 3 tabbed system architectural diagrams:
- Execution Architecture: Request → Semantic AutoRouter → Sandboxed AgentLoop → Isolated SQLite & Tools.
- Reliability Architecture: 50-Task Harness → Tiers 1-3 Deterministic Graders → CI/CD Regression Gate.
- Operations Architecture: ModelGateway Fallback → OpenTelemetry & Prometheus → Grafana & Alerting.
- Dual-Pane Live Playground (
/demo):- Scenario Mode: 5 canonical scenarios backed by isolated in-memory SQLite sandbox fixtures and verified with deterministic graders.
- Conversation Mode: Free-form multi-turn conversational agent turns against the live backend, strictly labeling ad-hoc turns as
NOT_EVALUATED. - 4-Tab Execution Inspector: Chronological Execution Timeline, Allowlisted Tool Calls, Deterministic Evaluation Matrix, and Real-time Routing Diagnostics.
- Offline Resilient: Embedded versioned benchmark replay artifacts allow zero-dependency demonstration if the live backend is offline.
- Python 3.11+
- Node.js 18+ and npm
- uv package manager
- Google Cloud Vertex AI ADC or Gemini API Key (
GOOGLE_API_KEYin.env) - Docker & Docker Compose (for local container stack)
# Clone repository
git clone https://github.com/ameynarwadkar/runwise.git
cd runwise
# Install backend dependencies using uv
uv sync
# Install frontend dependencies
cd frontend
npm install
cd ..
# Run all 158 unit, integration, and security tests
uv run pytest tests/ -v
# Start FastAPI service with hot-reload
uv run uvicorn runwise.service.app:app --host 127.0.0.1 --port 8000 --reload
cd frontend
npm run dev
- Portfolio Landing Page: http://localhost:3000
- Live Agent Playground: http://localhost:3000/demo
Verify health:
curl.exe -i http://127.0.0.1:8000/healthz
curl.exe -i http://127.0.0.1:8000/readyz
Runwise is configured for zero-cost, serverless-edge deployment:
- Frontend (Vercel): Next.js 15 App Router on Vercel edge network, configured via
frontend/vercel.json. - Backend (Render): FastAPI Python 3.11 web service deployed via
render.yamlon Render Free Tier with in-memory IP rate limiting and safe proxy header handling. - Replay-Only Public Demo Mode (
RUNWISE_DEMO_MODE=replay): Public demo mode serves genuine recorded benchmark trajectories (exact token economics, reasoning latencies, and Tier 1-3 deterministic grader verdicts). Arbitrary live model inference and administrative endpoints are disabled to guarantee zero cloud spend.
To deploy your own instance:
- Deploy Backend to Render: Connect your GitHub repository to Render using
render.yaml. SetCORS_ALLOWED_ORIGINSto your Vercel URL. - Deploy Frontend to Vercel: Import the
frontend/directory into Vercel. SetFASTAPI_URLto your Render service URL andNEXT_PUBLIC_DEMO_MODE=replay.
# Run evaluation gate comparing candidate against main baseline
uv run python -m runwise.evals.gate --candidate results/eval_candidate/summary.json --baseline data/baseline_eval.json
# Executes full lifecycle: canary rollout, session pinning, anomaly detection, instant rollback, and recovery check
uv run python ops/scripts/canary_drill.py
# Evaluates token reduction and cost delta between v1.0 and v1.1_trimmed
uv run python ops/scripts/compare_prompts.py
# Runs 4 concurrent users at 2 users/sec for 15s against active service
uv run python ops/load/run_bench.py --users 4 --spawn-rate 2 --run-time 15s
# Boots App, Prometheus, and Grafana
docker compose up -d
# Check container status
docker compose ps
- Grafana Dashboard: http://localhost:3000 (admin / admin)
- Prometheus Dashboard: http://localhost:9090
| Document | Purpose |
|---|---|
| RUNBOOK.md | On-call triage guide for all Prometheus alerts, mitigation steps, and escalation paths. |
| postmortems/001-canary-latency-slo-breach.md | Blameless postmortem of the Canary p95 Latency breach incident and recovery. |
| docs/BENCHMARK_REPORT.md | Full 50-task evaluation benchmark report across Dev and Holdout splits. |
| docs/ROUTING_REPORT.md | 4-Way auto-routing benchmark analysis, empirical trace replay, and cost evaluation. |
| docs/CAPACITY_AND_COST.md | Comprehensive capacity planning, token economics, and load-test benchmark data. |
| docs/specs/agent-eval-harness-design.md | Customer Support Agent & 4-tier evaluation harness design specification. |
| docs/specs/semantic-model-router-design.md | Cost-aware semantic model router architecture & calibration design. |
| docs/specs/public-website-design.md | Public website & interactive playground architecture & defense-in-depth design. |
| docs/resources.md | Curated literature and engineering references (LLMOps, evals, routing). |
| render.yaml | Render Web Service Infrastructure-as-Code specification. |
| frontend/vercel.json | Vercel edge deployment and security header configuration. |
| ops/prometheus/alerts.yml | Production Prometheus alerting rules (Error Budget Burn, Latency SLO, Gateway Failover). |
| ops/grafana/dashboard.json | Provisioned Grafana dashboard definition with live SLO and Canary panels. |
Apache 2.0. Built with engineering rigor by Amey Narwadkar.
