Repository navigation
Support llm-bench - #2060
Open
Mattbusel wants to merge 2 commits into
Open
Support llm-bench#2060Mattbusel wants to merge 2 commits into
Mattbusel wants to merge 2 commits into
Conversation
Author
|
Pushed the missing |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds llm-bench, a CLI I maintain that benchmarks LLM APIs (OpenAI, Anthropic, Groq, Gemini, Ollama or any OpenAI-compatible URL) on your own prompts and reports time to first token, p50/p95 latency, tokens per second, error rate and cost. It is meant to run in CI as a latency or cost gate, so
tool: llm-benchis the natural way to get it into a workflow.Releases publish prebuilt archives for x86_64/aarch64 linux-musl (static), x86_64/aarch64 macOS and x86_64 Windows, with SHA256SUMS and GitHub build provenance attestations, so the base manifest sets
gh-attestationsigning.version_rangestarts at 0.4.0, the first release with this asset layout.rust_crateis left out on purpose: the crates.io metadata for llm-bench points at the GitLab mirror (gitlab.com/mattbusel/llm-bench), so the generator's repository check would fail. Asset names use literal target triples instead, like tombi.Generated with
tools/manifest.sh llm-benchandtools/update-markdown.sh. I also ranmain.shlocally withINPUT_TOOL=llm-benchon Windows x64: it downloaded the 0.4.0 zip, verified the checksum and the installed binary reportsllm-bench 0.4.0.