Everything you need to understand, run, and extend devops-bench. New here? Start with Getting started, then skim the architecture and glossary.
- Getting started — set up your dev environment, understand how evals run, and meet the skills in this repo.
- Architecture — the eval lifecycle and how the components fit together.
- Glossary — the vocabulary, plus an annotated map of the codebase.
- Model providers — the LLMs that power agents and judges, and how to configure them.
- Infrastructure — deployers and cloud providers, and how to configure infra for a task.
- Agents — the agent harnesses under test and their capabilities.
- Metrics — the scoring framework and how to read results.
- Bastion — the VM that runs the harness in-VPC.
- vCluster provider — virtual clusters as a cheap, fast backend for parallel evals.
- Run evals — single runs and parallel matrices.
- Add a task — author a new benchmark task.
- Add a model provider — wire up a new LLM backend.
- Add an agent harness — plug in a new agent under test.
- Leaderboard — how the leaderboard works and how to ingest results.
- Known issues — a recovery router for eval failures, plus the current known hacks.
- Migration runbook — moving devops-bench to its upstream home at kubernetes-sigs/devops-bench (for maintainers).