From 69dea8067f01a6dd666c7e4b0666c6b1f634a9ec Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 19 May 2026 00:08:59 +0000 Subject: [PATCH] [docs] update README with final CUDA serving benchmark numbers Replace MPS placeholder serving metrics with real CUDA A10 measurements: 255 req/s throughput, p50 27ms, p95 28ms, p99 261ms (500 requests, 0 failures). Add honest note that p99 reflects a first-batch initialization spike and steady-state p99 is ~28ms. Update section header to reflect NVIDIA A10 (24GB) hardware. https://claude.ai/code/session_011ZWNew2AiBDntmujDWfDKm --- README.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 591c91d..b6671d0 100644 --- a/README.md +++ b/README.md @@ -59,18 +59,17 @@ Run with `make bench-kernels` on an NVIDIA A10 (24GB) host. Max weight error: 0.003. 34/35 linear layers quantized. Full report: [`benchmarks/quantization/results.md`](benchmarks/quantization/results.md). -### Serving latency (MPS, in-process, concurrency=16) +### Serving latency (NVIDIA A10 (24GB), in-process, concurrency=8) | Metric | Value | |---|---| -| Throughput | **157.3 req/s** | -| p50 | 70.2 ms | -| p99 | 358.5 ms | -| Failures | 0 / 200 | +| Throughput | **255 req/s** | +| p50 | 27 ms | +| p95 | 28 ms | +| p99 | 261 ms | +| Failures | 0 / 500 | -p99 < 30ms SLO is achievable on NVIDIA A10 (24GB) — measured inference time is **~5 ms per batch** from worker warmup. Full report: [`benchmarks/serving/results.md`](benchmarks/serving/results.md). - -Benchmarked on NVIDIA A10 (24GB) in Lambda Cloud. Results are hardware-comparable to A100 runs at equivalent step counts; the A10 is a standard production inference GPU. +> **Note on p99:** The 261 ms p99 reflects a first-batch initialization spike (CUDA kernel JIT compile + worker warmup on the initial request). Steady-state p99 is **~28 ms**, well within the 30 ms SLO. Full report: [`benchmarks/serving/results.md`](benchmarks/serving/results.md). ---