core: status page exceeds the API rate limit, leaving cards permanently loading

[andre8244]

Component and version

ns8-core main (d4435d58) — affects api-server, the cluster-admin UI, and the Python agent.

The rate limiter that triggers this landed in dcc0a3fc (2026-07-16).

Steps to reproduce

1. Install a single-node NS8 cluster from main, with the shipped defaults (GLOBAL_RATE_LIMIT_AVERAGE=25, GLOBAL_RATE_LIMIT_BURST=100).
2. Log in to https://<leader>/cluster-admin/ and open the browser DevTools Network panel.
3. Navigate to #/status and hard-reload the page.

Expected behavior

Every API request succeeds and all Status cards render.

Actual behavior

Some API requests return 429, and the cards whose task-status call was throttled stay in the loading state indefinitely.

A single cold load of #/status issues 146 same-origin requests — 79 static assets in a 422 ms burst, then 67 API calls. The first 429 lands on the ~96th request of the page load, i.e. precisely when the 100-token burst is exhausted:

static burst          : 79 requests in 0.42s
API before first 429  : 16 requests
                        ---------------------
cumulative            : 95 requests
first 429             : the ~96th request

So the burst ceiling is the binding constraint, not the sustained rate — refill contributes only ~50 tokens across the ~2 s window. Burst 100 was mirrored from nethsecurity-controller PR #282 rather than derived from cluster-admin's real request profile, and a legitimate cold load needs ~150.

Contributing request volume, all measured on a live load:

• 47 prefetch-only URLs, all actually requested, fired at t=45 ms and finished within 174 ms — Vue CLI's default prefetch is not deferred to browser idle time.
• 34 /context requests for only 13 distinct task IDs (21 wasted); one task was fetched 4×. notification.js checks the cache, awaits the HTTP call, and only then writes the cache, while websocket.js handleTaskMessage calls the async handler without awaiting — so all 3-4 progress frames per task enter the cache-miss branch concurrently.
• Static assets are served with no Cache-Control and no ETag (only Last-Modified), so 55 of 78 assets were full downloads even on a warm reload.

Two defects turn a transient 429 into a permanently broken page:

• core/ui/src/mixins/notification.js — the getTaskStatus error branch creates a notification but has no return, then dereferences statusResponse.data.data on an undefined response → TypeError → the *-completed event never fires and the card spins forever.
• core/imageroot/usr/local/agent/pypkg/agent/tasks/apiclient.py — http_temporary_errors = [500, 502, 503, 504] omits 429, so _retry_request re-raises instead of using its existing exponential backoff. Since nested tasks call the same rate-limited HTTP API (agent/tasks/run.py defaults to http://cluster-leader:9311), throttling fails whole tasks.

The 429 response also carries no Retry-After header, so no client can back off intelligently.

Suggested fix or workaround

Workaround: set GLOBAL_RATE_LIMIT_AVERAGE=0 in /etc/nethserver/api-server.env to disable the limiter.

Efficacy below is judged against the arithmetic above: a fix either removes requests from the first ~2 s, or raises the ceiling, or it does nothing for this bug. "first" / "repeat" = cold vs. warm browser cache.

#	Fix	Feasibility	Complexity	Efficacy on the 429	Verdict
A	Differentiated limits: strict bucket on /api/login, raise global burst	High	Low	Decisive — first + repeat	Do first (needs author review)
B	Add missing return in the getTaskStatus error branch (notification.js)	High	1 line	None on 429; removes the permanent hang	Do
C	Add 429 to http_temporary_errors (apiclient.py)	High	1 line	None on browser 429; stops task failures	Do
D	Coalesce in-flight /context requests	High	Low	High — −21 req, first + repeat	Do
E	Send Retry-After on the 429	High	Low	None alone; enables H	Do
F	Don't re-arm the poll timer on 429 (notification.js)	High	Low	Medium — stops amplification	Do
G	Cache-Control on static assets	High	Medium	High on repeat, zero on first	Do (after A–F)
H	429 retry with backoff in the axios interceptor	Medium	Medium	High on user-visible outcome	Do, carefully
I	Disable or trim webpack prefetch	High	1 line	High — −47 req, first + repeat	Judgement call
J	De-duplicate redundant task creation	Medium	Medium	Medium — ~−8 req	Optional
K	Serve precompressed assets	Medium	High	Zero — CPU only, not request count	Defer

Notes on the non-obvious rows:

• A — needs review by the author of dcc0a3fc before anyone implements it. Raising the burst is close to free in flood-resistance terms, because sustained throughput is what bounds a flood: over 60 s an attacker lands 100 + 25×60 = 1600 requests at burst 100 vs. 300 + 25×60 = 1800 at burst 300 (+12%). What a bigger burst does increase is instantaneous concurrency on the one genuinely expensive pre-auth path, /api/login (password hashing) — so the proposal is not a blanket bump but a stricter per-route bucket on login (e.g. 5 rps / burst 20, stronger than today) plus a global burst of 300. RateLimiter is already a self-contained closure with its own visitors map, so it works as per-route middleware with no refactor.
• G does nothing on a first-ever load — it only removes repeat-load requests. Requires a carve-out: index.html and config/config.production.js must stay no-cache, because install-coreimage unlinks content-hashed chunks that disappear on a core update, so a stale index.html would reference files that no longer exist.
• H must retry GET only. POST /cluster/tasks is not idempotent and blind retry would create duplicate tasks.
• I is stock Vue CLI behaviour, not a misconfiguration — removing it trades slower first navigation to lazy routes for 47 fewer requests. With A in place it is no longer needed to fix the bug, so it is a product decision.
• K has zero efficacy here because the limiter counts requests, not bytes or CPU. It is a genuine CPU improvement (gzip.Gzip wraps static.Serve, recompressing every asset on every request) but gin-contrib/gzip is pinned at v0.0.6, which has no precompressed support — and neither does v1.2.6. Better as a standalone performance PR; G largely obviates it.

Relevant logs or output

Throttled response (no Retry-After):

HTTP/2 429
content-type: application/json; charset=utf-8

{"code":429,"data":null,"message":"too many requests"}

Limiter parameters confirmed empirically — 800 parallel GETs completing over 25.2 s yielded 729 successes, matching burst + rate × elapsed = 100 + 25 × 25.2 = 730.

Duplicate /context fetches for a single task, from the Network panel:

GET /api/node/1/task/33c2edbb-.../context  200
GET /api/node/1/task/33c2edbb-.../context  200
GET /api/node/1/task/33c2edbb-.../context  200
GET /api/node/1/task/33c2edbb-.../context  200