Skip to content
ameynarwadkarPublic

About

Runwise is an observable AI agent service and evaluation harness built for resilient e-commerce order management, cancellations, address updates, and refunds.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Runwise: Production-Grade Customer Support Agent & LLMOps Service

Python 3.11+ Test Suite Live Demo Benchmark Reliability Cost Optimization Observability Zero-Downtime Rollback

Runwise is a production-grade, observable AI agent service and evaluation harness built for resilient e-commerce order management, cancellations, address updates, and refunds.

It implements a complete enterprise LLMOps lifecycle: 4-tier evaluation grading, distributed OpenTelemetry tracing, Prometheus SLO alerting, automated CI/CD regression gates, session-consistent Canary traffic routing with instant zero-rebuild rollback, empirical token cost modeling, and load testing.


1. System Performance & Reliability Benchmarks

Every production performance metric measured against an established baseline:

Metric Baseline Production Result Delta / Operational Value
Benchmark Accuracy 94.3% (Initial Dev Run) 100.0% (50 tasks / 150 trials) +5.7% (0.0% holdout generalization gap)
Cost per 1k Successes $0.0786 (v1.0 Prompt) $0.0525 (v1.1_trimmed) -33.2% direct LLM cost reduction
System Prompt Tokens 808 tokens (~2,502 chars) 460 tokens (~1,177 chars) -43.1% input token compression
Automated Test Suite 0 tests 158 / 158 passed 100% test coverage across 29 modules
Availability (SLO) $\ge 99.0%$ 100.0% Zero 5xx errors under concurrent load
Canary Rollback Speed 5–10 min (container rebuild) < 200 ms (dynamic hot-reload) Instant zero-rebuild rollback
Cost-Aware Auto-Routing $8.33 / 1k (Always Pro) $3.54 / 1k (Adaptive Router)* -57.5% cost, -63.0% p50 latency (3.82s vs 10.33s) at 100% accuracy (*trace replay)
CI Regression Gate Unchecked deployment risk PR Blocked (exit code 1) Verified against 5% pass drop & 34% latency spike
Local Sandbox Setup Shared state / race conditions < 2 ms (in-memory SQLite) Total tenant isolation per session

Live Operational Telemetry (Grafana Dashboard)

Runwise Live Grafana Operations Dashboard


2. Evidence Taxonomy & Defensibility Matrix

To ensure absolute technical credibility, all numbers in this repository are categorized into empirical measurements vs. controlled demonstrations:

A. Empirically Measured (Live API & Infrastructure)

  • 100.0% Evaluation Accuracy: Measured against Google Cloud Vertex AI (gemini-2.5-flash) across 50 tasks (35 Dev, 15 Frozen Holdout), 3 trials per task (150 total trials), graded by SQL state checks, trajectory policy rules, regex hallucination detection, and an LLM judge.
  • -33.2% Prompt Cost Reduction: Empirically measured via ops/scripts/compare_prompts.py. The compressed prompt (v1.1_trimmed.yaml) reduced characters from 2,502 to 1,177 (-53.0%) and prompt tokens from 808 to 460 (-43.1%), verified to preserve 100% of policy and safety invariants.
  • -57.5% Auto-Routing Cost Reduction: Evaluated across 35 dev tasks in ROUTING_REPORT.md. Baselines 1 & 2 measured via live Google Cloud Vertex AI inference; Strategy 4 verified via empirical trace replay over live baseline trajectories, reducing cost from $8.33 to $3.54 / 1k requests and p50 latency from 10.33s to 3.82s (-63.0%) with 100% task accuracy.
  • Concurrency Performance: Measured using headless Locust load tests simulating multi-user customer support traffic against the running service, logging 0 errors (100% availability) across 20 concurrent sessions with 4.04s p95 reasoning latency and 28,052 tokens processed.
  • 158 Passing Unit, Integration & Security Tests: Executed across 29 modules validating schemas, session contexts, gateway failovers, auto-routing classification, IP rate limiting, and Prometheus metrics.

B. Controlled Fault-Injection Drills (Operational Demonstrations)

  • CI/CD Evaluation Gate Rejection Drill: To prove the CI pipeline stops bad deploys, a synthetic regression candidate with a 5.0% accuracy drop and +34.5% latency spike was fed to runwise.evals.gate. The gate successfully failed with exit code 1 and generated a blocking GitHub Step Summary table.
  • Canary Latency SLO Breach & Instant Rollback Drill: To demonstrate sub-200ms disaster recovery, ops/scripts/canary_drill.py deployed canary v2.0 at 50% traffic weight, simulated a 3,420ms latency breach (> 3,000ms SLO), triggered ops/scripts/rollback.py, and verified 10/10 subsequent customer sessions instantly recovered to stable with zero container restarts.

3. Core Subprojects & Engineering Capabilities

Subproject 1: Agent Evaluation Harness (src/runwise/evals)

  • Deterministic 4-Tier Grading Matrix:
    • Tier 1 (Database State & Audit Auditor): Exact SQL assertions verifying mutations and audit_log tamper detection.
    • Tier 2 (Trajectory Policy Grader): Enforces auth-before-action ordering (verify_customer before cancel_order) and penalizes invalid tool invocations.
    • Tier 3 (Hallucination Detector): Regex-grounded verification comparing agent responses against session database snapshots.
    • Tier 4 (Scoped LLM Judge): Structured tone, polite refusal, and policy adherence scoring.
  • Statistical Reliability: Evaluates $\text{pass}^k$ (consensus reliability) and $\text{pass}@k$ (exploration) across 50 stratified tasks (35 Dev, 15 Frozen Holdout).
  • Benchmark Result: 100.0% pass rate on both splits with zero generalization gap.

Subproject 2: Production Service, Observability & Canary Lifecycle (src/runwise/service)

  • FastAPI Core: Async agent endpoints (POST /v1/agent/run), liveness (GET /healthz), and DB/LLM readiness (GET /readyz).
  • Telemetry & Tracing: Distributed W3C traceparent propagation, OpenTelemetry spans with OpenInference semantic conventions, and Prometheus metrics (runwise_http_requests_total, runwise_agent_run_duration_seconds, runwise_canary_latency_seconds).
  • CI/CD Evaluation Gate (gate.py): Automated regression comparator blocking PRs if accuracy drops $&gt; 2%$, latency regresses $&gt; 15%$, or token consumption inflates $&gt; 15%$.
  • Canary Router with Sticky Session Pinning: Cryptographically deterministic session hashing ([0.0, 1.0)) preventing mid-session prompt flips.
  • Instant Zero-Rebuild Rollback: Administrative tooling (ops/scripts/rollback.py) executing sub-200ms rollbacks with zero container rebuilds or downtime.

Subproject 2 (Ops): Capacity Planning, Cost Modeling & Load Testing (ops/)

  • Cost Engine (cost.py): Exact pricing engine for prompt tokens, completion tokens, availability-adjusted cost per 1k successes, and monthly projections.
  • Prompt Compression Experiment (v1.1_trimmed.yaml): Compressed system prompt by 53.0% characters and 43.1% tokens, slashing total request cost by 33.2% while retaining 100% policy compliance.
  • Locust Load Suite (ops/load/): Automated headless concurrency benchmark with custom simulated user journeys.
  • Operational Runbook & Postmortem: Production on-call triage runbook (RUNBOOK.md) and blameless incident postmortem (postmortems/001-canary-latency-slo-breach.md).

Subproject 3: Cost-Aware Auto-Routing (src/runwise/service/auto_router.py)

  • Multi-Model Tiering: Dynamically selects between cost-effective gemini-2.5-flash ($0.075 / 1M tokens) and frontier gemini-2.5-pro ($1.250 / 1M tokens, 16.6x price spread) in $&lt; 0.1\text{ ms}$ without auxiliary LLM overhead.
  • Lexical & Semantic Risk Classification: Evaluates adversarial injection patterns, high-liability financial disputes/chargebacks, and multi-entity conditional complexity.
  • 4-Way Benchmark Validation (35 Dev Tasks - ROUTING_REPORT.md):
    • Always Cheap (Flash): $0.34 / 1k | 2.85s p50 | 100.0% pass rate (100% live inference)
    • Always Strong (Pro): $8.33 / 1k | 10.33s p50 | 100.0% pass rate (100% live inference)
    • Rule-Based Router: $7.52 / 1k (-9.7% cost) | 10.33s p50 | 100.0% pass rate (trace replay)
    • Adaptive Semantic Router: $3.54 / 1k (-57.5% cost) | 3.82s p50 (-63.0% latency) | 100.0% pass rate (empirical trace replay)
    • Safely offloads 65.7% of queries to Flash while reserving Pro for adversarial and complex edge cases.
  • Prometheus Telemetry & Grafana Visualizations: Tracks runwise_router_decisions_total{selected_model, risk_tier} on donut and bar-gauge panels.

Subproject 4: Public Website & Interactive Live Playground (frontend/)

  • Next.js 15 & React 19 App Router: Recruiter-optimized, 30-second portfolio landing page (/) on dark charcoal (#101114) with restrained orange accents (#F97316).
  • Interactive Architecture Explorer: 3 tabbed system architectural diagrams:
    • Execution Architecture: Request → Semantic AutoRouter → Sandboxed AgentLoop → Isolated SQLite & Tools.
    • Reliability Architecture: 50-Task Harness → Tiers 1-3 Deterministic Graders → CI/CD Regression Gate.
    • Operations Architecture: ModelGateway Fallback → OpenTelemetry & Prometheus → Grafana & Alerting.
  • Dual-Pane Live Playground (/demo):
    • Scenario Mode: 5 canonical scenarios backed by isolated in-memory SQLite sandbox fixtures and verified with deterministic graders.
    • Conversation Mode: Free-form multi-turn conversational agent turns against the live backend, strictly labeling ad-hoc turns as NOT_EVALUATED.
    • 4-Tab Execution Inspector: Chronological Execution Timeline, Allowlisted Tool Calls, Deterministic Evaluation Matrix, and Real-time Routing Diagnostics.
    • Offline Resilient: Embedded versioned benchmark replay artifacts allow zero-dependency demonstration if the live backend is offline.

4. Quickstart & Verification Guide

4.1 Prerequisites

  • Python 3.11+
  • Node.js 18+ and npm
  • uv package manager
  • Google Cloud Vertex AI ADC or Gemini API Key (GOOGLE_API_KEY in .env)
  • Docker & Docker Compose (for local container stack)

4.2 Installation

# Clone repository
git clone https://github.com/ameynarwadkar/runwise.git
cd runwise

# Install backend dependencies using uv
uv sync

# Install frontend dependencies
cd frontend
npm install
cd ..

4.3 Running the Full Backend Test Suite

# Run all 158 unit, integration, and security tests
uv run pytest tests/ -v

4.4 Starting the Production Service Locally

# Start FastAPI service with hot-reload
uv run uvicorn runwise.service.app:app --host 127.0.0.1 --port 8000 --reload

4.5 Starting the Frontend Website & Playground Locally

cd frontend
npm run dev

Verify health:

curl.exe -i http://127.0.0.1:8000/healthz
curl.exe -i http://127.0.0.1:8000/readyz

4.6 Public Production Deployment (Vercel + Render)

Runwise is configured for zero-cost, serverless-edge deployment:

  • Frontend (Vercel): Next.js 15 App Router on Vercel edge network, configured via frontend/vercel.json.
  • Backend (Render): FastAPI Python 3.11 web service deployed via render.yaml on Render Free Tier with in-memory IP rate limiting and safe proxy header handling.
  • Replay-Only Public Demo Mode (RUNWISE_DEMO_MODE=replay): Public demo mode serves genuine recorded benchmark trajectories (exact token economics, reasoning latencies, and Tier 1-3 deterministic grader verdicts). Arbitrary live model inference and administrative endpoints are disabled to guarantee zero cloud spend.

To deploy your own instance:

  1. Deploy Backend to Render: Connect your GitHub repository to Render using render.yaml. Set CORS_ALLOWED_ORIGINS to your Vercel URL.
  2. Deploy Frontend to Vercel: Import the frontend/ directory into Vercel. Set FASTAPI_URL to your Render service URL and NEXT_PUBLIC_DEMO_MODE=replay.

5. Operational Verification Drills

5.1 Run the Evaluation Gate Drill (CI Regression Detection)

# Run evaluation gate comparing candidate against main baseline
uv run python -m runwise.evals.gate --candidate results/eval_candidate/summary.json --baseline data/baseline_eval.json

5.2 Run the Canary Rollout & Failure Rollback Drill

# Executes full lifecycle: canary rollout, session pinning, anomaly detection, instant rollback, and recovery check
uv run python ops/scripts/canary_drill.py

5.3 Run the Prompt Compression & Cost Comparison Analysis

# Evaluates token reduction and cost delta between v1.0 and v1.1_trimmed
uv run python ops/scripts/compare_prompts.py

5.4 Run Headless Concurrency Benchmark (Locust)

# Runs 4 concurrent users at 2 users/sec for 15s against active service
uv run python ops/load/run_bench.py --users 4 --spawn-rate 2 --run-time 15s

5.5 Starting Local Infrastructure Stack (Docker Compose)

# Boots App, Prometheus, and Grafana
docker compose up -d

# Check container status
docker compose ps

6. Project Documentation Directory

Document Purpose
RUNBOOK.md On-call triage guide for all Prometheus alerts, mitigation steps, and escalation paths.
postmortems/001-canary-latency-slo-breach.md Blameless postmortem of the Canary p95 Latency breach incident and recovery.
docs/BENCHMARK_REPORT.md Full 50-task evaluation benchmark report across Dev and Holdout splits.
docs/ROUTING_REPORT.md 4-Way auto-routing benchmark analysis, empirical trace replay, and cost evaluation.
docs/CAPACITY_AND_COST.md Comprehensive capacity planning, token economics, and load-test benchmark data.
docs/specs/agent-eval-harness-design.md Customer Support Agent & 4-tier evaluation harness design specification.
docs/specs/semantic-model-router-design.md Cost-aware semantic model router architecture & calibration design.
docs/specs/public-website-design.md Public website & interactive playground architecture & defense-in-depth design.
docs/resources.md Curated literature and engineering references (LLMOps, evals, routing).
render.yaml Render Web Service Infrastructure-as-Code specification.
frontend/vercel.json Vercel edge deployment and security header configuration.
ops/prometheus/alerts.yml Production Prometheus alerting rules (Error Budget Burn, Latency SLO, Gateway Failover).
ops/grafana/dashboard.json Provisioned Grafana dashboard definition with live SLO and Canary panels.

7. License

Apache 2.0. Built with engineering rigor by Amey Narwadkar.

About

Runwise is an observable AI agent service and evaluation harness built for resilient e-commerce order management, cancellations, address updates, and refunds.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages