Repository navigation
Add consumer gateway: an API gateway in front of the endpoint (one planned reload) #27
Description
Intent
Add one new consumer key, gateway, for an API gateway that will sit in front of this endpoint for a trial. The gateway gives its own clients per-client keys, attribution and fallbacks; from this box's point of view it is one more consumer with its own revocable key, like any other.
Nothing else changes: every existing consumer keeps its key and its direct connection, and the request profiles are untouched.
Change
llama-swap.yaml: oneapiKeysline,"${env.LLM_KEY_GATEWAY}", with a naming comment.secrets.env.example: the new variable.- README "Keys and rotation": a
gatewayrow. - On the box, at the window: the variable in
~/ai-stack/secrets/api-key.env(freshopenssl rand -hex 32) and the same line in~/ai-stack/llama-swap.yaml.
Cost
Adding a key is a config change, so it is one ~13-minute model reload for every consumer (README, "Adding or removing a consumer"). The standby key is not used for this, so it stays in reserve.
Window
To be announced before it happens; the time will be recorded here. Procedure: stop llama-swap, edit the secrets file and llama-swap.yaml, start, wait for Application startup complete, then scripts/check-keys.sh for every consumer plus gateway (all 200; a wrong key 401).
While the window is still announced, one cold long-prompt request (~60k tokens) goes through the gateway, to confirm its timeouts survive a cold prefill. That is a single request, not a load test.
Undo
Remove the line and the variable at a later planned reload (same procedure, same cost).
Activity
DustinTrap commentedon Oct 6, 2026
Status: the gateway side is deployed and working with its other tiers; this key is the last piece. The reload window is on hold at the operator's request: the box was serving 1–2 requests continuously when checked, so it was not interrupted. The time will be recorded here when the window runs.
DustinTrap commentedon Oct 6, 2026
Window started 2026-10-06 17:53 UTC at the operator's go. Expect ~13 minutes of downtime for every consumer while the model reloads. Completion and the key check results will follow here.
DustinTrap commentedon Oct 6, 2026
Window done.
LLM_KEY_GATEWAYadded to the secrets file; oneapiKeysline added to the deployedllama-swap.yaml(-validate: "config is valid: 1 model(s)")Application startup complete) — ~13.5 minscripts/check-keys.sh http://127.0.0.1:9292 …gateway→ 200; deliberately wrong key → 401; no key → 401Backups on the box:
~/ai-stack/changes/llama-swap.yaml.before-gateway-20261006-1753Zand the secrets file's copy alongside it insecrets/; log~/ai-stack/changes/gateway-key-20261006-1753Z.log.After the window, one cold long-prompt request went through the gateway to confirm its timeouts survive a long prefill: first byte at 133.5 s, delivered. That prompt came out larger than intended (~100k+ tokens rather than ~60k), so it was ~2 minutes of prefill on the box.
The deployed config now matches PR #28; the issue can close when that merges.