OpsPilot

OpsPilot is a local incident-diagnosis baseline for an agentic systems capstone. It uses gpt-oss:20b through Ollama to investigate noisy Kubernetes-style incidents, choose which evidence source to inspect, and submit a minimal root-cause diagnosis.

The repository uses deterministic incident snapshots, so it runs without cloud credentials or a Kubernetes cluster. Ground-truth labels are kept separate from the agent and are used only by the scorer.

What makes this agentic?

The LLM does not receive all evidence at once and does not follow a fixed workflow.

At each decision turn it can choose one of these actions:

• get_metrics()
• get_logs()
• get_recent_deployments()
• get_topology()
• submit_diagnosis(...) when it believes it has enough evidence

The loop is therefore:

incident -> LLM decides action -> tool observation -> LLM replans
                                               |
                                               +-> submit_diagnosis -> scorer

The agent may stop without calling every tool. This makes it possible to measure both diagnostic accuracy and investigation efficiency.

Evidence-calibrated policy

The current agent adds a few safeguards to a basic tool-use loop:

• It prevents repeated reads of the same evidence source.
• It requires direct evidence from at least two sources before accepting a diagnosis.
• It requires the model to record an alternative hypothesis and why the collected evidence rejects it.
• It normalizes free-form model names to canonical resource identifiers only when the collected evidence supports that mapping.

The result JSON includes the submitted evidence, alternative hypothesis, raw model prediction, and any canonicalization that was applied.

Requirements

• Python 3.10+
• uv
• Ollama, with the local gpt-oss:20b model downloaded
• No API key is required.

If uv is not installed:

python -m pip install uv

Setup

From the repository root:

uv sync

Download the model:

ollama pull gpt-oss:20b

Ollama normally starts its local server automatically. If it is not running, start it with:

ollama serve

The defaults are OPSPILOT_PROVIDER=ollama, OPSPILOT_MODEL=gpt-oss:20b, OLLAMA_BASE_URL=http://localhost:11434, and a 180-second request timeout. Use .env.example as a reference and set these variables in your shell; --model also overrides the model. If local inference is slow, set OLLAMA_TIMEOUT_SECONDS=300 or higher.

Verify the configured server without running inference:

uv run python run_baseline.py --check-ollama

Run one scenario

Run one scenario first:

uv run python run_baseline.py --agent llm --scenario s01_db_pool --verbose

Input file:

data/scenarios/s01_db_pool.json

You should see the agent:

• the LLM autonomously chooses which tools to inspect;
• it identifies ConfigMap/checkout-db-pool as the direct cause;
• it avoids blaming the unrelated recommendation/v7.2 deployment;
• it calls submit_diagnosis when it believes evidence is sufficient;
• the scorer compares the submitted root-cause set against evaluator-only ground truth.

The exact model trajectory can vary between runs. The observed trajectory, prediction, tool-call count, and score are written to the output JSON.

Run all scenarios

uv run python run_baseline.py --agent llm --scenario all

To inspect every trajectory in the terminal:

uv run python run_baseline.py --agent llm --scenario all --verbose

Detailed results are written to:

outputs/baseline_results.json

Run the heuristic reference

A non-LLM heuristic baseline is retained only as a reproducibility/reference check:

uv run python run_baseline.py --agent heuristic --scenario all

This is not the primary agent. It is a deterministic comparison point that also lets you validate the scenario and scoring harness without Ollama.

Its verified reference metrics are:

Scenarios:                     8
Avg precision @ full recall:   0.625
Exact-match accuracy:          37.5%
Full-recall rate:              87.5%
Average tool calls:            4.00
False positives:               5
False negatives:               1

Inputs

Eight noisy incident snapshots are in:

data/scenarios/

Each includes only observable incident evidence:

• incident description and alerts
• metrics
• logs
• recent deployments/configuration changes
• dependency/topology state

Ground truth is not embedded in these files.

Evaluator-only labels are stored separately in:

data/ground_truth.json

The LLM agent never receives this file.

Scoring

Let G be the ground-truth root-cause entity set and P the agent's submitted set.

• If the agent misses any ground-truth entity, precision-at-full-recall = 0.
• If full recall is achieved, score = |G ∩ P| / |P|.

This penalizes an agent that names every suspicious symptom/entity instead of committing to the minimal supported root cause.

Additional metrics:

• exact-match accuracy
• full-recall rate
• false positives / false negatives
• average investigation tool calls

Safety boundary

All four investigation tools are read-only. The model can recommend remediation, but it cannot execute changes. Any rollback, credential rotation, configuration change, or restart requires human approval.

Run tests

uv run python -m unittest discover -s tests -v

The tests include:

• scorer correctness;
• a mocked LLM-tool loop proving that the agent can autonomously request a tool observation and then submit a structured diagnosis without any network/API dependency.

Repository structure

opspilot/
├── README.md
├── .env.example
├── pyproject.toml
├── uv.lock
├── run_baseline.py
├── opspilot/
│   ├── agent.py
│   ├── llm_agent.py
│   ├── scorer.py
│   └── tools.py
├── data/
│   ├── scenarios/
│   │   └── eight noisy incident JSON files
│   └── ground_truth.json
├── outputs/              # created when a run finishes
└── tests/
    ├── test_llm_agent.py
    └── test_scorer.py

Limitations and next steps

• The scenarios are synthetic and intentionally small.
• Local model outputs can vary between runs.
• Canonical entity resolution is rule-based and limited to the evidence types in this repository.
• Remediation remains recommendation-only.

The next step is to test the same pipeline against richer incident data or a live local demo service, then compare repeated runs for accuracy and tool-use efficiency.