Repository navigation
skills: land litellm-gateway runbook and prod-deploy-via-release-only - #61
Conversation
…ash chain) The skill existed only as an orphan directory under ~/.omp/agent/skills, never installed from this repo (make install only walks repo skills/*/), so it could not be reviewed or versioned. Adds it as the canonical source with the current chain: OpenCode Go deepseek-flash -> Muse -> DeepSeek direct (z.ai/GLM removed 2026-09-10). Also records the two rollout traps found while making that change: - normalize with swap.sh status first: an installed blue side is LIVE, flip to green (content-identical) before editing; - test.sh derives the cascade probe's kill target from the config head's api_base, so it cannot pass vacuously after a head change.
Adds the missing promotion step: nothing converged the two sides after a successful rollover, so green kept an older chain and a later rollback would silently revert the chain the user had accepted. swap.sh gains promote, which runs the same test.sh gate, saves the outgoing chain to config.green.yaml.prev, copies blue over green with a cmp assertion, and returns live to GREEN. Also makes the gates mechanical rather than remembered: - activate/promote dry-run by default and print the rollback command; they act only with --yes. - activate refuses when the side being edited is live. - activate/promote poll health instead of sleeping, then prove the chain with a real 'primary' call. - rollback keeps depending on nothing but the config files and systemd, and shares its flip+restart code path with promote, so it is exercised on every promote.
…r-rules branch The skill existed only on docs/skills-over-rules, which never had a PR, so make install left a dangling symlink for it under ~/.omp/agent/skills/. Content unchanged (authored in 54d5d25, 'docs: default to skills over rules; prod-deploy guidance as a skill'). The branch's other commits (semver-release rule, writing-documentation restructure, lucidlint baseline note) overlap the open lucidlint-baseline work and are deliberately not included here.
PR Reviewer Guide 🔍(Review updated until commit 228d274)Here are some key observations to aid the review process:
✅ Resolved findingsskills/litellm-gateway/SKILL.md:1-146Documentation-standard violation The skill references external scripts ( skills/prod-deploy-via-release-only/SKILL.md:1-45Missing post-change steps Same as above: the PR description lacks the required mention that skills/litellm-gateway/SKILL.md:1Missing required statement The PR does not explicitly mention that the change needs |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
…omotion undo single-use Review findings: the cascade probe section named the kill target without saying how the script picks it (the first api_base line in the file, patched to an unreachable host), and the promotion undo did not warn that a second promotion overwrites the saved chain.
Standalone PR ReviewPR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
…out shell config
Review finding: $EDITOR may be unset, leaving the command to fail silently; ${EDITOR:-vi} keeps the reader's editor when set and still works when it is not.
Standalone PR ReviewPR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
The change cycle — exact commandscd ~/.paseo/litellm
swap.sh status # confirm: live side GREEN, blue free to edit
${EDITOR:-vi} config.blue.yaml # order, models, keys, timeouts — blue ONLY
swap.sh test # gate: 3 probes must PASS (≈20 s, :4001 only)
swap.sh activate # dry run: prints rollback + plan, switches nothing
swap.sh activate --yes # re-runs the gate, then live -> blue, then verifies
# ... soak: use the system normally ...
swap.sh promote --yes # gate, green.prev <- green, green <- blue, live -> greenExpected ends: activate prints
What each part guarantees
ROLLBACK FIRST — non-negotiableBefore any command that changes the live side, print the rollback command and Traps (each cost a real incident)
Clients
Paseo sessions pin the model per session — restarts never migrate themEach session stores its model in
New sessions follow Diagnostics
~/.paseo/litellm/ Expected ends: activate prints
What each part guarantees
ROLLBACK FIRST — non-negotiableBefore any command that changes the live side, print the rollback command and Traps (each cost a real incident)
Clients
Paseo sessions pin the model per session — restarts never migrate themEach session stores its model in
New sessions follow Diagnostics
Compliance: writing-documentation
|
…nce check Three review findings: the live-side trap now says what actually guards it (a symlink cannot be locked by permissions), the Diagnostics section no longer points at references/adding-providers.md, which does not exist in the repo, and prod-deploy gains a checkable acceptance criterion for its permissions guard.
Standalone PR ReviewPR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
…e cost concretely Final review round: the live-side trap no longer leans on a symlink-permission claim (the real point is that nothing on the filesystem stops the write), and the promote trap names the cost — green overwritten with the bad chain, rollback gone, recovery only via the one-step .prev restore or the manual Cloudflare switch.
Standalone PR ReviewPR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
…grep matches nothing GNU xargs runs the command once with no operands when the input is empty, which hands sed a stdin read with no files; -r makes the zero-match case an honest no-op. The step already says to expect 0 hits afterwards, so the empty case is a real one.
Standalone PR ReviewPR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
PR Code Suggestions ✨No code suggestions found for the PR. |
The gate asked only whether the ALIAS answered, so a dead head hidden behind a working fallback validated clean while every request was served by the last resort. test.sh now asserts content (not just 200), streamed content deltas, and each chain member serving ALONE with fallbacks disabled; its member probe fails the config that was live today. The skill states the rule and the mechanism: chain membership requires the member probe, OpenCode Go needs x-opencode-session plus its own user-agent (extra_headers), and the pin migration moved to skill://litellm-gateway/examples/ per the documentation standard instead of an inline script.
Standalone PR ReviewPR-Agent could not safely update the persistent review. This standalone result will not replace the canonical review. PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
…ent-shaped traffic Muse Spark is unreachable on OpenCode Go's /chat/completions route (500 at the provider edge) but serves normally via /responses, which LiteLLM bridges to chat completions including streaming — so the deployment is openai/responses/muse-spark-1.3-contributor. Probe traffic is agent-shaped on a reasoning-sized budget: a trivial reply-OK prompt makes a reasoning model spend its budget thinking and answer 200 with empty content, which the gate would misread as a broken model.
…ting them The skill duplicated config and code contents — the chain order and model ids, the subcommand list, the probe inventory, the command syntax. Those live in config.blue.yaml, swap.sh and test.sh; the skill now says where to read them and keeps only what is not in any file: the invariant that the live side is never edited, the gate rule, rollback-first, the traps, and the client wiring.
|
Persistent review updated to latest commit 54e9fcb |
PR Code Suggestions ✨No code suggestions found for the PR. |
|
Persistent review updated to latest commit 43ce1fe |
PR Code Suggestions ✨No code suggestions found for the PR. |
… the single top-level pin
|
Persistent review updated to latest commit 2a6949a |
…istory strings untouched
|
Persistent review updated to latest commit b3a0ecb |
|
Persistent review updated to latest commit b3a0ecb |
PR Code Suggestions ✨No code suggestions found for the PR. |
|
Addressing the ACTIVE finding on b3a0ecb:
For the earlier compliance findings: the description now states the |
…y-and-prod-deploy
|
Persistent review updated to latest commit 9cad28a |
PR Code Suggestions ✨No code suggestions found for the PR. |
…y-and-prod-deploy
|
Persistent review updated to latest commit 228d274 |
PR Code Suggestions ✨No code suggestions found for the PR. |
User description
Theme
Land two skills that lived outside the repo's
skills/tree:litellm-gateway(previously installed directly under
~/.omp/agent/skills/, so never versioned,reviewed, or refreshed by
make install) andprod-deploy-via-release-only(authored on an unmerged branch, which left
make installwith a danglingsymlink).
Key changes
skills/litellm-gateway/SKILL.md(canonical source)The runbook for the local LiteLLM proxy at
http://localhost:4000/v1, rewrittenso an agent does not have to reason about blue-green state:
config.current.yamlsymlinks the live one, and "sides identical" meansnothing is staged.
swap.sh statusand execute thenext:line it prints; no decision table to keep in sync by hand.test.shprints each failingmember's HTTP status, error type and message plus which members ride instead;
whether an exhausted-credit member is acceptable is the operator's call.
rollbackreverts a soak; the undo for apromotion comes from
swap.sh promote's own dry-run output.~/.paseo/config.jsononly when the user explicitly asks for that session.The gateway tooling lives in
ashbywinch/bluegreen-config(swap.sh,test.sh,snapshot-live.sh) and is deployed into~/.paseo/litellm/by thatrepo's
make install;make check-installedfails on drift. Config files,.gate-stateand backups stay unversioned.skills/prod-deploy-via-release-only/SKILL.md(new)Deployment by release only, with the mechanical guard written as instructions:
restrict the deploy user's elevated access to the release scripts, keep
read-only diagnostics, and check it with
sudo -l.Verification
make test: 61 docs scanned, 42 skills well-formed, links resolve, 18 tests OK.migrate-paseo-pins.shexercised on scratch fixtures: three pin spacingsrewritten, a two-match file skipped, invalid JSON skipped, and the run exits 1
while any file still pins the old model.
make install(symlinks refreshed); restart omp to load the newskills — required for any skill change to take effect.
PR Type
Documentation, Enhancement
Description
Add litellm-gateway skill with deterministic blue-green runbook
Add prod-deploy-via-release-only skill enforcing release-only deployments
Both skills were previously unversioned or on unmerged branches
Diagram Walkthrough
File Walkthrough
SKILL.md
Add litellm-gateway skill with blue-green runbookskills/litellm-gateway/SKILL.md
health
SKILL.md
Add prod-deploy-via-release-only skillskills/prod-deploy-via-release-only/SKILL.md
process
opportunity