Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -511,18 +511,21 @@ just pre-pr # Pre-PR checks

Bashkit includes a [mira eval study](crates/bashkit-eval/) that measures how well LLMs use bashkit: one tagged 88-task dataset (`bashkit_bash`) covering agent tool use (58 basic, 8 multi-turn repo, 10 hard) and 12 runtime tasks (python3, sqlite3, state on the VFS), plus a 15-task one-shot script generation eval (`bashkit_generate`).

_Latest run: 2026-10-09, on the mira eval framework. GPT-6.1 Sol and GPT-6 Luna are pending a rerun on the 88-task set (their 58-task scores are in the [detailed results](crates/bashkit-eval/README.md#results))._
_Latest run: 2026-10-09 (GPT-6 models added 2026-10-12), on the mira eval framework._

| Model | Score | Tasks Passed | Basic | Repo | Hard | Runtime | Generate |
|-------|-------|-------------|-------|------|------|---------|----------|
| Claude Opus 5.5 | **99%** | **87/88** | **58/58** | 7/8 | **10/10** | **12/12** | 13/15 |
| Kimi K3 | 98% | 86/88 | 57/58 | **8/8** | 9/10 | **12/12** | **15/15** |
| Claude Sonnet 5.5 | 97% | 85/88 | 56/58 | **8/8** | 9/10 | **12/12** | **15/15** |
| Muse Spark 1.3 Contributor | 95% | 84/88 | **58/58** | 5/8 | 9/10 | **12/12** | **15/15** |
| GPT-6.1 Sol | 86% | 76/88 | 54/58 | 3/8 | 8/10 | 11/12 | 14/15 |
| Gemini 3.8 Flash | 84% | 74/88 | 55/58 | 4/8 | 6/10 | 9/12 | 13/15 |
| GPT-6 Luna | 81% | 71/88 | 53/58 | 1/8 | 5/10 | **12/12** | 14/15 |

Repo and hard tasks carry most of the spread; the new runtime tasks (python3
and sqlite3 inside the sandbox) separate only Gemini so far. See the [detailed analysis](crates/bashkit-eval/README.md#results).
and sqlite3 inside the sandbox) are near-saturated, with only Gemini and GPT-6.1
Sol missing any. See the [detailed analysis](crates/bashkit-eval/README.md#results).

```bash
cargo install mira-cli # one-time: the `mira` host CLI
Expand Down
41 changes: 22 additions & 19 deletions crates/bashkit-eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,35 +99,38 @@ First run of the merged, tagged `bashkit_bash` dataset (mode=agent|runtime,
difficulty=basic|repo|hard) and of `bashkit_generate`. Run folders:
`results/mira/20261009T173925Z-a0a7` (`bashkit_bash`) and
`results/mira/20261009T171610Z-2942` (`bashkit_generate`). GPT-6.1 Sol and
GPT-6 Luna are not in this run: the OpenAI key ran out of credit mid-run, so
resume both run folders with `mira run ... --resume <run_id>` once it is
topped up.

| Metric | Opus 5.5 | Kimi K3 | Sonnet 5.5 | Muse Spark 1.3 | Gemini 3.8 Flash |
|--------|----------|---------|------------|----------------|------------------|
| bashkit_bash (88) | **87/88** | 86/88 | 85/88 | 84/88 | 74/88 |
| basic (58) | **58/58** | 57/58 | 56/58 | **58/58** | 55/58 |
| repo (8) | 7/8 | **8/8** | **8/8** | 5/8 | 4/8 |
| hard (10) | **10/10** | 9/10 | 9/10 | 9/10 | 6/10 |
| runtime (12) | **12/12** | **12/12** | **12/12** | **12/12** | 9/12 |
| bashkit_generate (15) | 13/15 | **15/15** | **15/15** | **15/15** | 13/15 |
| Tool-call success | 92% | 92% | 92% | **97%** | 94% |
| Tokens (in/out) | 628K / 102K | 974K / 192K | **519K / 79K** | 2639K / 679K | 1619K / 127K |
| Duration | 18.3 min | 40.9 min | **13.1 min** | 119.3 min | 56.1 min |
GPT-6 Luna were added on 2026-10-12 with `mira run ... --resume <run_id>`
(the OpenAI key ran out of credit during the first pass).

| Metric | Opus 5.5 | Kimi K3 | Sonnet 5.5 | Muse Spark 1.3 | GPT-6.1 Sol | Gemini 3.8 Flash | GPT-6 Luna |
|--------|----------|---------|------------|----------------|-------------|------------------|------------|
| bashkit_bash (88) | **87/88** | 86/88 | 85/88 | 84/88 | 76/88 | 74/88 | 71/88 |
| basic (58) | **58/58** | 57/58 | 56/58 | **58/58** | 54/58 | 55/58 | 53/58 |
| repo (8) | 7/8 | **8/8** | **8/8** | 5/8 | 3/8 | 4/8 | 1/8 |
| hard (10) | **10/10** | 9/10 | 9/10 | 9/10 | 8/10 | 6/10 | 5/10 |
| runtime (12) | **12/12** | **12/12** | **12/12** | **12/12** | 11/12 | 9/12 | **12/12** |
| bashkit_generate (15) | 13/15 | **15/15** | **15/15** | **15/15** | 14/15 | 13/15 | 14/15 |
| Tool-call success | 92% | 92% | 92% | **97%** | 88% | 94% | 84% |
| Tokens (in/out) | 628K / 102K | 974K / 192K | 519K / 79K | 2639K / 679K | **275K / 42K** | 1619K / 127K | 497K / 82K |
| Duration | 18.3 min | 40.9 min | **13.1 min** | 119.3 min | 17.8 min | 56.1 min | 18.8 min |

Duration is the sum of per-task wall-clock time for `bashkit_bash`.

#### Highlights

1. **`hard_makefile_deps` is the hardest task**: Sonnet, Kimi, Muse and Gemini
all miss it; only Opus 5.5 solves it.
1. **`hard_makefile_deps` is the hardest task**: only Opus 5.5 and GPT-6.1 Sol
solve it.
2. **Runtime tasks are near-saturated** for the top four (12/12). Gemini misses
`rt_sqlite_hr_report`, `rt_state_csvq_tool` and `rt_state_ledger_ingest`.
3. **Repo tasks still split the field**: `repo_parallel_runner_masks_failure`
(masked background-job failures) defeats Opus, Muse and Gemini.
(masked background-job failures) defeats Opus, Muse, Gemini and both GPT-6
models; GPT-6 Luna passes 1 of 8 repo tasks, GPT-6.1 Sol 3.
5. **GPT-6.1 Sol is the most token-frugal** (275K in, about half of Sonnet)
but has the lowest tool-call success after GPT-6 Luna.
4. **Generate**: Opus misses `gen_assoc_inventory` and `gen_retry_backoff`
(one reply had no script fence); Gemini misses `gen_csv_to_jsonl` and
`gen_semver_bump`.
`gen_semver_bump`; GPT-6.1 Sol misses `gen_batch_rename`, GPT-6 Luna
`gen_config_template`.

### 2026-10-09, 7-model lineup (58 tasks + 8 repo + 10 hard)

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"eval": "bashkit_generate",
"sample": "gen_assoc_inventory",
"target": "openai/gpt-6-luna",
"input": [
"Write a bash script (use bash associative arrays) that aggregates an inventory.\n\nStarting files:\n--- /home/eval/data/inventory.txt ---\n# warehouse sku qty\nberlin bolt 120\nparis nut 40\nberlin nut 15\nlyon bolt 30\nparis bolt 5\nberlin washer 300\nlyon nut 0\n--- end ---\n\nLines starting with `#` are comments; fields are separated by one or more spaces.\n\n1. Write /home/eval/data/sku_totals.txt: one line `SKU TOTAL_QTY` per SKU, sorted by SKU.\n2. Write /home/eval/data/warehouses.txt: one line `WAREHOUSE N` per warehouse, sorted by warehouse, where N is the number of distinct SKUs that warehouse holds with qty > 0.\nExit 0."
],
"passed": true,
"aggregate": 1.0,
"scores": [
{
"scorer": "bashkit_expectations",
"value": 1.0,
"pass": true,
"reason": "3/3 checks passed (weighted rate 100%)"
}
],
"transcript": {
"final_response": "```bash\n#!/usr/bin/env bash\n\ninput=/home/eval/data/inventory.txt\ndata_dir=/home/eval/data\nsku_tmp=\"$data_dir/.sku_totals.tmp\"\nwarehouse_tmp=\"$data_dir/.warehouses.tmp\"\n\ndeclare -A totals=()\ndeclare -A warehouse_counts=()\ndeclare -A positive_pairs=()\n\nwhile read -r warehouse sku qty _; do\n [[ -z $warehouse || $warehouse == \\#* ]] && continue\n [[ -z $sku || -z $qty ]] && continue\n\n qty_num=$((10#$qty))\n totals[\"$sku\"]=$((${totals[$sku]:-0} + qty_num))\n\n if (( qty_num > 0 )); then\n pair=\"$warehouse|$sku\"\n if [[ -z ${positive_pairs[$pair]+present} ]]; then\n positive_pairs[\"$pair\"]=1\n warehouse_counts[\"$warehouse\"]=$((${warehouse_counts[$warehouse]:-0} + 1))\n fi\n fi\ndone < \"$input\"\n\n: > \"$sku_tmp\"\nfor sku in \"${!totals[@]}\"; do\n printf '%s %s\\n' \"$sku\" \"${totals[$sku]}\" >> \"$sku_tmp\"\ndone\nsort -k1,1 \"$sku_tmp\" > \"$data_dir/sku_totals.txt\"\n\n: > \"$warehouse_tmp\"\nfor warehouse in \"${!warehouse_counts[@]}\"; do\n printf '%s %s\\n' \"$warehouse\" \"${warehouse_counts[$warehouse]}\" >> \"$warehouse_tmp\"\ndone\nsort -k1,1 \"$warehouse_tmp\" > \"$data_dir/warehouses.txt\"\n\nrm -f \"$sku_tmp\" \"$warehouse_tmp\"\nexit 0\n```",
"iterations": 1,
"tool_calls_count": 1,
"tool_calls": [
"#!/usr/bin/env bash\n\ninput=/home/eval/data/inventory.txt\ndata_dir=/home/eval/data\nsku_tmp=\"$data_dir/.sku_totals.tmp\"\nwarehouse_tmp=\"$data_dir/.warehouses.tmp\"\n\ndeclare -A totals=()\ndeclare -A warehouse_counts=()\ndeclare -A positive_pairs=()\n\nwhile read -r warehouse sku qty _; do\n [[ -z $warehouse || $warehouse == \\#* ]] && continue\n [[ -z $sku || -z $qty ]] && continue\n\n qty_num=$((10#$qty))\n totals[\"$sku\"]=$((${totals[$sku]:-0} + qty_num))\n\n if (( qty_num > 0 )); then\n pair=\"$warehouse|$sku\"\n if [[ -z ${positive_pairs[$pair]+present} ]]; then\n positive_pairs[\"$pair\"]=1\n warehouse_counts[\"$warehouse\"]=$((${warehouse_counts[$warehouse]:-0} + 1))\n fi\n fi\ndone < \"$input\"\n\n: > \"$sku_tmp\"\nfor sku in \"${!totals[@]}\"; do\n printf '%s %s\\n' \"$sku\" \"${totals[$sku]}\" >> \"$sku_tmp\"\ndone\nsort -k1,1 \"$sku_tmp\" > \"$data_dir/sku_totals.txt\"\n\n: > \"$warehouse_tmp\"\nfor warehouse in \"${!warehouse_counts[@]}\"; do\n printf '%s %s\\n' \"$warehouse\" \"${warehouse_counts[$warehouse]}\" >> \"$warehouse_tmp\"\ndone\nsort -k1,1 \"$warehouse_tmp\" > \"$data_dir/warehouses.txt\"\n\nrm -f \"$sku_tmp\" \"$warehouse_tmp\"\nexit 0\n"
],
"usage": {
"input_tokens": 469,
"output_tokens": 1125,
"cost_usd": 0.0
},
"timing": {
"duration_ms": 12288
},
"metrics": {
"exec_ms": 10.0,
"exit_code": 0.0,
"extracted": 1.0,
"generate_ms": 12278.0,
"script_bytes": 1163.0,
"script_found": 1.0,
"timed_out": 0.0
},
"metadata": {
"bashkit": {
"dirs": [],
"last_exit_code": 0,
"links": {},
"tool_outputs": [
{
"commands": "#!/usr/bin/env bash\n\ninput=/home/eval/data/inventory.txt\ndata_dir=/home/eval/data\nsku_tmp=\"$data_dir/.sku_totals.tmp\"\nwarehouse_tmp=\"$data_dir/.warehouses.tmp\"\n\ndeclare -A totals=()\ndeclare -A warehouse_counts=()\ndeclare -A positive_pairs=()\n\nwhile read -r warehouse sku qty _; do\n [[ -z $warehouse || $warehouse == \\#* ]] && continue\n [[ -z $sku || -z $qty ]] && continue\n\n qty_num=$((10#$qty))\n totals[\"$sku\"]=$((${totals[$sku]:-0} + qty_num))\n\n if (( qty_num > 0 )); then\n pair=\"$warehouse|$sku\"\n if [[ -z ${positive_pairs[$pair]+present} ]]; then\n positive_pairs[\"$pair\"]=1\n warehouse_counts[\"$warehouse\"]=$((${warehouse_counts[$warehouse]:-0} + 1))\n fi\n fi\ndone < \"$input\"\n\n: > \"$sku_tmp\"\nfor sku in \"${!totals[@]}\"; do\n printf '%s %s\\n' \"$sku\" \"${totals[$sku]}\" >> \"$sku_tmp\"\ndone\nsort -k1,1 \"$sku_tmp\" > \"$data_dir/sku_totals.txt\"\n\n: > \"$warehouse_tmp\"\nfor warehouse in \"${!warehouse_counts[@]}\"; do\n printf '%s %s\\n' \"$warehouse\" \"${warehouse_counts[$warehouse]}\" >> \"$warehouse_tmp\"\ndone\nsort -k1,1 \"$warehouse_tmp\" > \"$data_dir/warehouses.txt\"\n\nrm -f \"$sku_tmp\" \"$warehouse_tmp\"\nexit 0\n",
"exit_code": 0,
"stderr": "",
"stdout": ""
}
]
},
"generate": {
"extraction": "fenced",
"timed_out": false
}
}
},
"skipped": false
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"eval": "bashkit_generate",
"sample": "gen_assoc_inventory",
"target": "openai/gpt-6.1-sol",
"input": [
"Write a bash script (use bash associative arrays) that aggregates an inventory.\n\nStarting files:\n--- /home/eval/data/inventory.txt ---\n# warehouse sku qty\nberlin bolt 120\nparis nut 40\nberlin nut 15\nlyon bolt 30\nparis bolt 5\nberlin washer 300\nlyon nut 0\n--- end ---\n\nLines starting with `#` are comments; fields are separated by one or more spaces.\n\n1. Write /home/eval/data/sku_totals.txt: one line `SKU TOTAL_QTY` per SKU, sorted by SKU.\n2. Write /home/eval/data/warehouses.txt: one line `WAREHOUSE N` per warehouse, sorted by warehouse, where N is the number of distinct SKUs that warehouse holds with qty > 0.\nExit 0."
],
"passed": true,
"aggregate": 1.0,
"scores": [
{
"scorer": "bashkit_expectations",
"value": 1.0,
"pass": true,
"reason": "3/3 checks passed (weighted rate 100%)"
}
],
"transcript": {
"final_response": "```bash\n#!/usr/bin/env bash\n\ndeclare -A sku_totals=()\ndeclare -A warehouse_counts=()\ndeclare -A positive_pairs=()\n\nwhile read -r warehouse sku qty; do\n [[ -z \"$warehouse\" || \"$warehouse\" == \\#* ]] && continue\n\n sku_totals[\"$sku\"]=$(( ${sku_totals[\"$sku\"]:-0} + qty ))\n warehouse_counts[\"$warehouse\"]=${warehouse_counts[\"$warehouse\"]:-0}\n\n pair=\"$warehouse|$sku\"\n if (( qty > 0 )) && [[ -z \"${positive_pairs[\"$pair\"]:-}\" ]]; then\n positive_pairs[\"$pair\"]=1\n warehouse_counts[\"$warehouse\"]=$(( ${warehouse_counts[\"$warehouse\"]} + 1 ))\n fi\ndone < /home/eval/data/inventory.txt\n\nexport LC_ALL=C\n\n{\n for sku in \"${!sku_totals[@]}\"; do\n printf '%s %s\\n' \"$sku\" \"${sku_totals[\"$sku\"]}\"\n done\n} | sort > /home/eval/data/sku_totals.txt\n\n{\n for warehouse in \"${!warehouse_counts[@]}\"; do\n printf '%s %s\\n' \"$warehouse\" \"${warehouse_counts[\"$warehouse\"]}\"\n done\n} | sort > /home/eval/data/warehouses.txt\n\nexit 0\n```",
"iterations": 1,
"tool_calls_count": 1,
"tool_calls": [
"#!/usr/bin/env bash\n\ndeclare -A sku_totals=()\ndeclare -A warehouse_counts=()\ndeclare -A positive_pairs=()\n\nwhile read -r warehouse sku qty; do\n [[ -z \"$warehouse\" || \"$warehouse\" == \\#* ]] && continue\n\n sku_totals[\"$sku\"]=$(( ${sku_totals[\"$sku\"]:-0} + qty ))\n warehouse_counts[\"$warehouse\"]=${warehouse_counts[\"$warehouse\"]:-0}\n\n pair=\"$warehouse|$sku\"\n if (( qty > 0 )) && [[ -z \"${positive_pairs[\"$pair\"]:-}\" ]]; then\n positive_pairs[\"$pair\"]=1\n warehouse_counts[\"$warehouse\"]=$(( ${warehouse_counts[\"$warehouse\"]} + 1 ))\n fi\ndone < /home/eval/data/inventory.txt\n\nexport LC_ALL=C\n\n{\n for sku in \"${!sku_totals[@]}\"; do\n printf '%s %s\\n' \"$sku\" \"${sku_totals[\"$sku\"]}\"\n done\n} | sort > /home/eval/data/sku_totals.txt\n\n{\n for warehouse in \"${!warehouse_counts[@]}\"; do\n printf '%s %s\\n' \"$warehouse\" \"${warehouse_counts[\"$warehouse\"]}\"\n done\n} | sort > /home/eval/data/warehouses.txt\n\nexit 0\n"
],
"usage": {
"input_tokens": 469,
"output_tokens": 395,
"cost_usd": 0.0
},
"timing": {
"duration_ms": 8415
},
"metrics": {
"exec_ms": 8.0,
"exit_code": 0.0,
"extracted": 1.0,
"generate_ms": 8407.0,
"script_bytes": 951.0,
"script_found": 1.0,
"timed_out": 0.0
},
"metadata": {
"bashkit": {
"dirs": [],
"last_exit_code": 0,
"links": {},
"tool_outputs": [
{
"commands": "#!/usr/bin/env bash\n\ndeclare -A sku_totals=()\ndeclare -A warehouse_counts=()\ndeclare -A positive_pairs=()\n\nwhile read -r warehouse sku qty; do\n [[ -z \"$warehouse\" || \"$warehouse\" == \\#* ]] && continue\n\n sku_totals[\"$sku\"]=$(( ${sku_totals[\"$sku\"]:-0} + qty ))\n warehouse_counts[\"$warehouse\"]=${warehouse_counts[\"$warehouse\"]:-0}\n\n pair=\"$warehouse|$sku\"\n if (( qty > 0 )) && [[ -z \"${positive_pairs[\"$pair\"]:-}\" ]]; then\n positive_pairs[\"$pair\"]=1\n warehouse_counts[\"$warehouse\"]=$(( ${warehouse_counts[\"$warehouse\"]} + 1 ))\n fi\ndone < /home/eval/data/inventory.txt\n\nexport LC_ALL=C\n\n{\n for sku in \"${!sku_totals[@]}\"; do\n printf '%s %s\\n' \"$sku\" \"${sku_totals[\"$sku\"]}\"\n done\n} | sort > /home/eval/data/sku_totals.txt\n\n{\n for warehouse in \"${!warehouse_counts[@]}\"; do\n printf '%s %s\\n' \"$warehouse\" \"${warehouse_counts[\"$warehouse\"]}\"\n done\n} | sort > /home/eval/data/warehouses.txt\n\nexit 0\n",
"exit_code": 0,
"stderr": "",
"stdout": ""
}
]
},
"generate": {
"extraction": "fenced",
"timed_out": false
}
}
},
"skipped": false
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
{
"eval": "bashkit_generate",
"sample": "gen_batch_rename",
"target": "openai/gpt-6-luna",
"input": [
"Write a bash script that normalises photo file names.\n\nStarting files (contents shown; names contain spaces where shown):\n--- /home/eval/photos/IMG_0001.JPG ---\na\n--- end ---\n--- /home/eval/photos/IMG_0002.JPG ---\nb\n--- end ---\n--- /home/eval/photos/IMG 0003.JPG ---\nc\n--- end ---\n--- /home/eval/photos/notes.txt ---\nn\n--- end ---\n--- /home/eval/photos/Raw/IMG_0004.JPG ---\nd\n--- end ---\n--- /home/eval/photos/Raw/Old Shots/DSC 9.JPG ---\ne\n--- end ---\n\nFor every file whose name ends in `.JPG`, anywhere under /home/eval/photos (recursively), rename it in place (same directory) so that its file name is lowercase and every space in the file name becomes `_`. Directory names must NOT change (`Raw` and `Old Shots` keep their names). Other files are left alone. Exit 0."
],
"passed": true,
"aggregate": 1.0,
"scores": [
{
"scorer": "bashkit_expectations",
"value": 1.0,
"pass": true,
"reason": "7/7 checks passed (weighted rate 100%)"
}
],
"transcript": {
"final_response": "```bash\n#!/usr/bin/env bash\n\nfind /home/eval/photos -type f -name '*.JPG' -print |\nwhile IFS= read -r path; do\n dir=${path%/*}\n name=${path##*/}\n normalized=$(printf '%s' \"$name\" | tr '[:upper:]' '[:lower:]')\n normalized=${normalized// /_}\n mv -- \"$path\" \"$dir/$normalized\"\ndone\n\nexit 0\n```",
"iterations": 1,
"tool_calls_count": 1,
"tool_calls": [
"#!/usr/bin/env bash\n\nfind /home/eval/photos -type f -name '*.JPG' -print |\nwhile IFS= read -r path; do\n dir=${path%/*}\n name=${path##*/}\n normalized=$(printf '%s' \"$name\" | tr '[:upper:]' '[:lower:]')\n normalized=${normalized// /_}\n mv -- \"$path\" \"$dir/$normalized\"\ndone\n\nexit 0\n"
],
"usage": {
"input_tokens": 503,
"output_tokens": 431,
"cost_usd": 0.0
},
"timing": {
"duration_ms": 4893
},
"metrics": {
"exec_ms": 7.0,
"exit_code": 0.0,
"extracted": 1.0,
"generate_ms": 4886.0,
"script_bytes": 294.0,
"script_found": 1.0,
"timed_out": 0.0
},
"metadata": {
"bashkit": {
"dirs": [],
"last_exit_code": 0,
"links": {},
"tool_outputs": [
{
"commands": "#!/usr/bin/env bash\n\nfind /home/eval/photos -type f -name '*.JPG' -print |\nwhile IFS= read -r path; do\n dir=${path%/*}\n name=${path##*/}\n normalized=$(printf '%s' \"$name\" | tr '[:upper:]' '[:lower:]')\n normalized=${normalized// /_}\n mv -- \"$path\" \"$dir/$normalized\"\ndone\n\nexit 0\n",
"exit_code": 0,
"stderr": "",
"stdout": ""
}
]
},
"generate": {
"extraction": "fenced",
"timed_out": false
}
}
},
"skipped": false
}
Loading
Loading