Feature: multi-endpoint upstream fallback (retry a secondary LLM endpoint when the primary is rate-limited or down)

Problem

The proxy forwards every request to exactly one upstream LLM endpoint (llm.api_base + llm.api_key + llm.model_id). When that endpoint fails, _forward_to_llm_openai (skillclaw/api_server.py) retries the same endpoint up to 6 times with exponential backoff and then returns 502 Bad Gateway to the agent.

This hurts most with account-level quotas, where retrying is pointless by definition. Real example (z.ai / GLM coding endpoint):

08:00:14  429 {"error":{"code":"1308","message":"Usage limit reached for 5 hour.
               Your limit will reset at 2026-08-30 09:16:13"}}
08:00:14  upstream LLM error (attempt 1/6), retrying in 1.3s
08:00:16  attempt 2/6 ... 429 (same body)
...
08:00:48  attempt 6/6 ... 429 (same body)
08:00:48  upstream LLM error → 502 Bad Gateway

Six identical retries burn ~34 seconds and the caller still ends up with a 502 — a scheduled job (e.g. a cron-driven post) simply fails even though a second key/endpoint was available. This is the same failure family as #24 (transient upstream errors), which was closed as "not our bug", but with a fallback endpoint it would have been a non-event.

Current behavior

• skillclaw/api_server.py, _forward_to_llm_openai: single api_base, max_retries = 6, on exhaustion raises HTTPException(502).
• The only built-in failover is OpenRouter-specific: openrouter.fallback_models adds models: [primary, ...] to the request body, which delegates failover to OpenRouter itself. Nothing equivalent exists for a plain OpenAI-compatible upstream (z.ai, DeepSeek, vLLM, etc.), and it can't express "another endpoint/key for the same or different provider".
• Config plumbing is a fixed 1:1 bridge: ConfigStore.to_skillclaw_config() (skillclaw/config_store.py) maps llm.api_base/api_key/model_id onto SkillClawConfig.llm_* fields (skillclaw/config.py).

Proposal

Support an ordered list of upstream endpoints. Try the primary; when it is exhausted (or when the error is a quota/limit error where retrying the same account is meaningless), fail over to the next entry; only after the last entry fails return 502.

Config sketch (YAML, consumed by ~/.skillclaw/config.yaml / skillclaw config llm.fallbacks[0].api_base ...):

llm:
  provider: custom
  api_base: https://api.z.ai/api/coding/paas/v4
  api_key: sk-primary
  model_id: glm-5.2
  fallbacks:
    - api_base: https://api.z.ai/api/coding/paas/v4   # same provider, second account
      api_key: sk-backup
      model_id: glm-5.2
    - api_base: https://openrouter.ai/api/v1          # or a completely different provider
      api_key: sk-or
      model_id: vendor/model

Implementation notes:

• The change is naturally contained: the (api_base, key, model_id) triple is already assembled inside _forward_to_llm_openai; the single-endpoint loop becomes the inner loop, and the endpoint list becomes the outer loop. Streaming does not need separate handling — the proxy always sends stream: false upstream and re-assembles SSE itself.
• New config keys must be added in three places: SkillClawConfig dataclass, the _DEFAULTS/bridge in ConfigStore, and the retry loop.
• Quota-aware fast-fail: on HTTP 429 whose body matches a usage-limit error (or that carries a Retry-After far in the future), skip the remaining retries for that endpoint and move to the next one immediately. Retrying the same account inside the limit window only adds latency (34 s in the example above).
• Respect Retry-After for the final fallback decision only if cheap; otherwise current exponential backoff is fine.
• The response's model field should reflect the endpoint that actually served the request (agents normally use proxy.served_model_name, so this is cosmetic but useful for debugging).
• Optional follow-ups (same pattern): evolve_server/core/llm_client.py on the server side, and the Codex Responses path (_forward_to_llm_responses). The chat path alone would already cover the Hermes/OpenClaw integrations.

Alternatives considered

• OpenRouter as primary with fallback_models: works, but locks you into OpenRouter and only switches models, not endpoints/accounts.
• LiteLLM router in front of the proxy (fallbacks config): works, but adds a whole extra service to deploy next to skillclaw, and the fallback logic is ~30 lines away from where the request already is.

Environment

• Source: main as of 2026-08-31 (client proxy, Docker deployment, Hermes integration).
• Upstream in the incident: api.z.ai OpenAI-compatible endpoint, 429 / code 1308 (5-hour usage quota).