Skip to content

Support llm-bench - #2060

Open
Mattbusel wants to merge 2 commits into
taiki-e:mainfrom
Mattbusel:add-llm-bench
Open

Mattbusel wants to merge 2 commits into
taiki-e:mainfrom
Mattbusel:add-llm-bench

Conversation

@Mattbusel

Copy link
Copy Markdown

Adds llm-bench, a CLI I maintain that benchmarks LLM APIs (OpenAI, Anthropic, Groq, Gemini, Ollama or any OpenAI-compatible URL) on your own prompts and reports time to first token, p50/p95 latency, tokens per second, error rate and cost. It is meant to run in CI as a latency or cost gate, so tool: llm-bench is the natural way to get it into a workflow.

Releases publish prebuilt archives for x86_64/aarch64 linux-musl (static), x86_64/aarch64 macOS and x86_64 Windows, with SHA256SUMS and GitHub build provenance attestations, so the base manifest sets gh-attestation signing. version_range starts at 0.4.0, the first release with this asset layout.

rust_crate is left out on purpose: the crates.io metadata for llm-bench points at the GitLab mirror (gitlab.com/mattbusel/llm-bench), so the generator's repository check would fail. Asset names use literal target triples instead, like tombi.

Generated with tools/manifest.sh llm-bench and tools/update-markdown.sh. I also ran main.sh locally with INPUT_TOOL=llm-bench on Windows x64: it downloaded the 0.4.0 zip, verified the checksum and the installed binary reports llm-bench 0.4.0.

@Mattbusel

Copy link
Copy Markdown
Author

Pushed the missing 0.4 major.minor alias, which is what the major.minor test jobs looked up. The manifest / gen failure is a GitHub API 403 while fetching terrastruct/d2 releases (fork runs have no token), not llm-bench.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant