serving-stack

An OpenAI-compatible LLM inference service, containerised three ways and benchmarked on an NVIDIA Tesla T4.

Built during the AI Data Center Bootcamp at the Saudi Digital Academy (Riyadh, Aug-Oct 2026).

The point of this repository is not the service. It is the measurements: what actually moves throughput on a single GPU, and what only looks like it should.

---

Results

vLLM vs. static batching

Tesla T4, Qwen2.5-1.5B-Instruct, fp16.

Concurrency	Static batching (tok/s)	vLLM (tok/s)	Speedup
1	26.0	38.8	1.49x
4	42.3	167.5	3.96x
8	82.0	259.6	3.17x

Continuous batching pays off under load, not at rest. At concurrency 1 the gain is modest; by concurrency 4 it is close to 4x.

Raw data: labs/W3D3/ab_report.json

Precision: fp16 vs. int8

dtype	Context	VRAM (GB)	GPU util (%)	tok/s
fp16	512	2.97	40.5	19.6
fp16	2048	3.49	59.8	17.4
fp16	4096	5.17	92.4	15.2
int8	512	2.92	23.6	4.5
int8	2048	2.96	28.8	4.3
int8	4096	4.46	36.7	3.9

The counterintuitive result. On a T4, int8 cost roughly 4x throughput while saving almost no VRAM at short context (2.92 GB against 2.97 GB). Quantisation is not free, and on this hardware it was the wrong trade.

Raw data: labs/W3D1/profile.json

Latency baselines (fp16)

Prompt tokens	Time to first token (s)
128	0.048
512	0.138
2048	0.739

Time per output token: 0.044 s. Raw data: labs/W3D2/baselines.json

KV cache sizing

Predicted 28.0 KB/token from the model config; measured 28.0 KB/token. Peak under load reached 484.4 KB/token, and that gap between steady state and peak is what capacity planning has to budget for.

Raw data: labs/W3D2/kv_check.json

Batching alone

20.1 tok/s at batch 1, 146.0 tok/s at batch 8 (7.3x). Raw data: labs/W3D1/batch_check.json

---

The service

FastAPI, OpenAI-compatible:

• GET /health liveness plus the loaded model id
• GET /v1/models
• POST /v1/chat/completions streaming and non-streaming

Bearer-token auth, and a server-side MAX_TOKENS ceiling so a client cannot request unbounded generation.

Three images, on purpose

File	Purpose
Dockerfile.naive	The obvious build, kept as the baseline to measure against
Dockerfile	CPU service, layer-ordered so the model layer caches
Dockerfile.gpu	CUDA image for the T4

Keeping the naive build in the repository is deliberate: an improvement is only legible next to what it replaced.

Quick start

./scripts/verify-env.sh          # checks the host, writes verify-env-report.json
cp .env.example .env             # set API_KEY and MODEL_ID
docker compose up --build

Call it:

curl -H "Authorization: Bearer $API_KEY" \
     -H "Content-Type: application/json" \
     -X POST http://localhost:8000/v1/chat/completions \
     -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Hello"}],"max_tokens":50}'

Never commit .env. It is covered by .gitignore, and .env.example carries placeholders only.

Layout

app/                FastAPI service, schemas, generation probe
labs/W2D1..W3D3     experiments, each with its JSON output committed
scripts/            verify-env.sh
shared/             Colab scaffold
PINS.md             version pins, single source of truth
gpu_evidence.json   T4 probe: CUDA available, 26.7 tok/s

Every experiment commits its output as JSON, so each number above is reproducible from the file next to it.

Environment

Tesla T4 - CUDA - Qwen2.5-1.5B-Instruct - versions pinned in PINS.md.

---

Built by Abdulrhman Alotaibi.