feat: let an alias point at several models and fail over between them

[tito]

Why

An alias points at exactly one model on one provider. When that provider is down, rate limited, or returns 5xx, every request to the alias fails, and the operator must edit the alias by hand to move traffic. Two deployments of the same model (for example on two inference hosts, or on a self-hosted node and a paid API) cannot share one name. The LiteLLM compat endpoint returns 409 for a second deployment under one model_name, so tooling that relied on LiteLLM load balancing breaks.

What

An alias keeps its current meaning, but its target becomes a list: it can point at one model or at several. There is no separate "group" object in the UI, the admin API, or the docs, because the user wants one concept: an alias. When an alias has two or more targets, it has a strategy that picks which target serves a request: failover (listed order, the default), round_robin (start target rotates), weighted (random start target by weight), or least_busy (fewest in-flight requests on this replica). With one target, there is no strategy and the alias behaves exactly as today. If the chosen target fails before any response byte reaches the caller (connection error, timeout before headers, 429, or 5xx), the proxy tries the remaining targets in listed order. A failure in the middle of a stream is not retried, because bytes already reached the caller. A target that fails is skipped for a fixed cooldown on that replica, because otherwise every request pays the failed attempt again. If every target is in cooldown, the proxy tries them all anyway. Each failed attempt is recorded as its own usage event with its real outcome and a failed_over flag, because the user wants recovered upstream failures to stay visible on the errors dashboard. Request counts exclude flagged events. The LiteLLM compat endpoint adds a second distinct deployment under one model_name as another target of that alias, because the user wants existing registration tooling to keep working.

Request	Before	After
POST /admin/models with {"alias": "ha", "target": "m-ok"}	one-target alias	same, target is accepted as a one-item targets
POST /admin/models with {"alias": "ha", "targets": ["m-dead", "m-ok"], "strategy": "failover"}	400, unknown field	alias with two targets and a strategy
POST /v1/chat/completions with model: ha while m-dead is down	502 or 503 from the dead target	200 from m-ok, x-llmproxy-provider names it, x-llmproxy-attempts: 2
Same call within LLMPROXY_FAILOVER_COOLDOWN (30s)	same error	200, x-llmproxy-attempts: 1, dead target skipped
Upstream returns 400	400 to the caller	400 to the caller, no failover, x-llmproxy-attempts: 1
LiteLLM POST /model/new with an existing model_name and a new api_base	409	200, a new target on that alias, strategy weighted
DELETE /admin/models/{id} on a model that is one of several targets	n/a	409 model_in_use, as for a single target

Where to look, in reading order:

• internal/store/schema.go, migrate.go, types.go: table model_binding_target, column model_binding.strategy, column usage_event.failed_over.
• internal/store/providers.go, targets_test.go: the target list is written in the binding's transaction. One target keeps target_id, two or more use the new table. ListBindingsTargeting reads both forms for the delete guard.
• internal/catalog/catalog.go: Resolve returns a plan of routes with the intersected capabilities of the enabled targets.
• internal/catalog/balance.go, balance_test.go: per-replica ordering by strategy, round-robin counter, in-flight counter, cooldown.
• internal/server/openai.go, ratelimit.go, usagerec.go, failover_test.go: the attempt loop around the single credential site, header writes moved after the relay decision, x-llmproxy-attempts.
• internal/store/usage.go: request-count aggregates add failed_over = 0; the errors dashboard keeps flagged rows.
• internal/server/admin.go, aliastargets_test.go: targets and strategy on create and patch, each target checked for the one-hop rule.
• internal/server/litellm.go, litellm_test.go: a second deployment under one model_name becomes a target, /model/info and /model/delete work per deployment.
• internal/metrics/metrics.go, internal/config/config.go: llmproxy_failover_total{provider, model, reason} and LLMPROXY_FAILOVER_COOLDOWN.
• ui/src/pages/Models.tsx, ModelsCatalog.tsx, Requests.tsx, lib/strategies.ts: several targets in the alias form, strategy select with two or more, weights and order buttons, failed-over flag in the request log.
• docs/concepts/architecture.md, guides/models.md, reference/api.md, reference/openapi.yaml, reference/configuration.md.

Design changes

An alias target becomes a list. The target_id column holds one id, and the one-row join in bindingSource cannot express an ordered list with weights. One-target aliases keep target_id unchanged, so no existing row migrates and the shared join stays as it is. An alias with two or more targets stores them in model_binding_target and leaves target_id empty. This split is internal: the API and the UI only show targets.

	Before	After
Alias target	target_id points at one direct binding	target_id for one target, rows in model_binding_target for 2+
API	target: "x"	targets: ["x", "y"], strategy; target still accepted
Hop rule	target must be direct	each target must be direct
Delete guard	refused while an alias targets it	same, over both storage forms

catalog.Resolve returns a plan instead of one route. The server needs the full candidate list to fail over without a second lookup.

	Before	After
Return	*Route	plan: []*Route + strategy
Cache entry	one route	one plan

Response headers are written after the attempt decision. Before, observeRateLimit wrote x-ratelimit-* headers on the first upstream response. A failed attempt must not leak its headers into the final response.

	Before	After
x-ratelimit-* headers	written on the first upstream response	written only for the attempt that is relayed
x-llmproxy-provider, x-llmproxy-model	the resolved route	the target that answered
x-llmproxy-attempts	absent	attempt count

usage_event gains failed_over. Without the flag, request counts per alias double-count retried requests.

	Before	After
Failed attempt followed by another	not possible	its own event, real outcome, failed_over = 1
Request counts	all events	events with failed_over = 0
Errors dashboard	all events	all events, flag shown

Results

The alias form with two targets, the strategy select, weights, and order buttons (after; the form had one target combobox before): 3-edit-ha.png, closeup 2-form-targets.png.

The usage models table with an alias that lists its targets and strategy, and a request count that excludes the failed attempts (after): 4-catalog.png.

The screenshots come from the seeded local server. gh cannot attach images, so they are attached through the web UI after the PR opens.

Left alone

• Retry inside a stream after bytes reached the caller: not possible without rewriting the stream.
• Fallback to a different model name on context-window or content-policy errors: a later feature.
• Latency-based, usage-based, or cost-based strategies: add when someone asks.
• Shared cooldown and in-flight counters across replicas: state stays per replica, a database round trip per request costs more than it saves. Two replicas balance independently.
• Several targets on the transparent Anthropic relay: the relay does not use the catalog.
• Active health checks: cooldown is passive only.
• Worst-case latency on a cold start is the sum of the targets' connect timeouts. The cooldown bounds it after the first failure. The guide states this.
• An upstream that processed a request and then returned 5xx is billed twice. The failed attempt's usage, if the upstream reported any, is recorded, so spend stays correct.
• LiteLLM compat creates direct bindings named <model_name>@<provider> for extra deployments. Operators see rows they did not create by hand. Targets that are not bindings would break the one-hop rule and every listing.
• The architecture page said "no silent multi-provider resolution". An alias with several targets is configured on purpose and the serving provider is in a header. The page is amended in this PR.

How to verify

just lint
just race
just build
just ui && git diff --exit-code -- internal/server/uidist
uvx --with mkdocs==1.6.1 --with mkdocs-material==9.5.49 --with mkdocs-redirects==1.2.2 mkdocs build --strict
go test ./internal/server -run 'TestFailover|TestAliasTargets|TestLiteLLM'
go test ./internal/store -run TestTargets
go test ./internal/catalog

1. Create two providers, one pointing at a stopped port and one at a working upstream. Bind a model on each. In the UI, create alias ha and pick both models as targets, dead one first. The strategy select appears, set to failover. Call POST /v1/chat/completions with model: ha. The call succeeds, x-llmproxy-provider names the working provider, and x-llmproxy-attempts is 2.
2. Repeat the call within 30s. x-llmproxy-attempts is 1, because the dead target is in cooldown.
3. Open the Requests log and the errors dashboard. The dead target shows one unreachable event flagged as failed over. The request count for ha is 2, not 3.
4. Make the working upstream return 400. The caller receives the 400 and x-llmproxy-attempts is 1.
5. Set strategy round_robin with two working targets and send 4 requests. Each provider serves 2.
6. Remove one target from ha in the UI. The strategy select disappears, and the alias behaves as a plain alias.
7. Run LiteLLM POST /model/new twice with the same model_name and two different api_base. Both return success. GET /model/info lists two deployments. Calls to the name succeed while one api_base is down.