feat: add observer_cli for inspecting a node from a remote shell

[zorn]

Closes #300

This adds observer_cli 2.x to the release, so a remote IEx shell on production can show a live, top-style view of the node. Prometheus and Tempo show trends, but they don't show which processes hold memory or which mailboxes are backing up. The change also documents the remote shell itself in CLAUDE.md, since nothing described it before.

Risk is in how the node starts, not in the app

observer_cli starts no processes at boot. Its only dependency, recon 2.5.6, already ships in prod through Paraxial, so the lock gains one package.

The larger change is rel/env.sh.eex. Before this, EPMD and the distribution port listened on every interface, so other services on Render's private network could reach them, guarded only by the cookie. They now bind to loopback, and the node is named flick@localhost. If that breaks boot on Render, the deploy fails its health check, and a revert undoes it completely. The start command's eval "Flick.Release.migrate" still runs, since eval starts no distribution.

No app behavior changes, and mix precommit passes with the suite unchanged at 190 passing. This deploys, because the release itself changes.

How to demo it

Locally, as a prod release. If epmd -names shows no nodes, stop any running EPMD with epmd -kill first. An EPMD started earlier keeps listening on every interface, so the binding check below would mislead. Build the release, then start it against your dev database on port 4010. The basic-auth and metrics credentials are checked at compile time, so they must match the dev defaults or boot aborts.

MIX_ENV=prod mix release --overwrite

DATABASE_URL=ecto://postgres:postgres@localhost/flick_dev \
SECRET_KEY_BASE=$(openssl rand -hex 64) \
BASIC_AUTH_ADMIN_USERNAME=flick-admin BASIC_AUTH_ADMIN_PASSWORD=unsafe-password \
METRICS_AUTH_USERNAME=flick-metrics METRICS_AUTH_PASSWORD=unsafe-metrics-password \
PHX_SERVER=true PORT=4010 \
_build/prod/rel/flick/bin/flick start

In a second terminal, attach and start it:

_build/prod/rel/flick/bin/flick remote

:observer_cli.start()

The home page shows system, memory, and a table of the busiest processes. Press m to rank by memory. Press q and then Enter to quit, then Ctrl+C and a to leave the shell. http://localhost:4010 still answers afterward. To check the binding, run lsof -nP -iTCP -sTCP:LISTEN | grep -E 'epmd|beam'. EPMD on 4369 and the distribution port show 127.0.0.1, and EPMD may also show [::1]. Stop the node with _build/prod/rel/flick/bin/flick stop.

In production, after merge. Open the Shell tab of the flick service in the Render dashboard and run the same remote command and :observer_cli.start(). I have run the local path but not this one, so this step is the real check that Render's Shell works on the starter plan.

Worth a reviewer's attention

• It uses the in-node start, not the 2.0 escript controller. observer_cli 2.0 recommends an escript on your laptop that connects over Erlang distribution. That needs network access to the node, while Render Shell already runs inside the container.
• The requirement admits 2.1. The issue wanted 2.1 kept out, but 2.1 breaks only the escript CLI, which Flick does not use. mix.lock holds 2.0.0, so 2.1 arrives only through a reviewed update.
• Loopback distribution rules out clustering. DNS_CLUSTER_QUERY is unset on Render, so nothing clusters today. A second instance would need this binding revisited.
• The docs forbid a custom release cookie and exposing distribution. The issue listed these as non-goals, and the review flagged writing them down as scope creep. I kept them as standing rules, so a later change does not open the node to the network.

[Copilot]

🟡 Changes recommended

The dependency constraint allows 2.1 despite the issue’s explicit intent to exclude it.

1 open issue

• Restrict dependency constraint to 2.0.x to exclude breaking 2.1 · New

What changed in this PR

Adds observer_cli to Flick’s release for live node inspection through a remote IEx shell, complementing metrics and traces.

Changes:

• Adds and locks observer_cli 2.0.0.
• Documents remote-shell access, inspection commands, and safe exit procedures.

File	Description
mix.lock	Locks observer_cli and its existing recon dependency.
mix.exs	Includes observer_cli in the release.
AGENTS.md	Documents node inspection and distribution safety.

_{Review effort: Balanced}

[zorn]

Verified in production after the deploy went live at 14:47 UTC. From the Shell tab of flick in the Render dashboard, _build/prod/rel/flick/bin/flick remote attached to the running release, and :observer_cli.start() rendered the home page.

• The node reports itself as flick@localhost, so rel/env.sh.eex took effect on Render, and remote found the node through the loopback-only EPMD.
• Uptime matched the deploy, so attaching restarted nothing.
• The node used 161 MiB of the 512 MB instance, with loaded code the largest share at 63 MiB.

Render Shell works on the starter plan, so the production step in the demo path holds.

One new line appeared in the boot log: "No open ports detected on 0.0.0.0, continuing to scan..." It came 6 seconds after Bandit bound port 10000, and the service went live 5 seconds later. The previous deploy had no such line. The new loopback-only listeners are a plausible cause, but that is unconfirmed. Worth checking whether it recurs on the next deploy.