Add API rate limiting

[bencap]

Rate limiting is already enforced at the load balancer's firewall (AWS WAF). What's left is writing the setup down in the deployment repo, giving staging the same rule, deciding on a stricter limit for CSV downloads, and documenting the limit for API users.

Status

Prod enforces a per-IP rate limit at the WAF. The regional web ACL mavedb-prod-rate-limit, on the prod-api ALB, has:

• rate-limit-ip: Block at 1500 requests per 5 minutes per IP, returning 429 with Retry-After: 300.
• block-ai-crawlers-on-csv: Block. deployment#33 replaces it.
• shadow-csv-endpoints: Count at 100 per 5 minutes on the CSV endpoints.
• shadow-requests: Count at 300 per minute.

The limit was sized from a 90-day per-IP baseline, in which the mavedb.org UI peaked at 739 requests per IP per minute and browser embeds at 840.

Remaining scope

• Record the web ACL and its rules in mavedb-deployment. It's managed in the console today and isn't in procedure.md.
• Give staging the same enforced rule, so a change can be tested there first.
• Decide whether to enforce the CSV limit, using the shadow-csv-endpoints counts.
• Document the limit and the 429 response in the API Quickstart docs.

Acceptance criteria

• The prod and staging web ACLs are described in mavedb-deployment, and the two match apart from thresholds.
• The API docs state the per-IP limit and the Retry-After behavior.
• The CSV-limit decision and its numbers are recorded in this issue.

[bencap]

WAF-Based Rate Limiting on Staging & Prod

Instead of rate limits in app middleware, I went with WAF-level rate limiting for the initial implementation. This should be cheap in AWS and is well suited to adding blanket protection for abuse.

What was done:

• Enabled WAF rate limit counting on both staging and prod environments
• Thresholds are currently set to observe/count traffic, not enforce hard limits

Next steps:

• Monitor traffic patterns over the next ~2 weeks (through ~2026-05-14)
• Use observed request volumes to set appropriate WAF thresholds that block abuse without impacting legitimate users

Future consideration:
If we need more granular, business-logic-aware limits (e.g. cap score set publications per hour to avoid saturating the background worker), we can layer slowapi on top of the WAF limit. This would let us define endpoint- or operation-specific quotas in the application layer without replacing the WAF protection.

[bencap]

WAF rate limiting is enforced on prod and staging

mavedb-prod-rate-limit and mavedb-staging-rate-limit carry identical rules:

Rule	Action	Limit (per IP)	Scope
rate-limit-ip	Block: 429, Retry-After: 300	1500 / 5 min	all requests
block-ai-crawlers-on-csv	Block: 403	—	declared AI agents on the bulk endpoints
shadow-csv-endpoints	Count	100 / 5 min	/scores, /counts, /variants/data, /mapped-variants
shadow-requests	Count	300 / 1 min	all requests

Sizing. In 90 days of prod WAF logs (3.2M requests from 53.7k IPs), the mavedb.org UI peaked at 739 requests per IP per minute and browser embeds (MobiDetails, ClinGen) at 840. Scripts peaked at 8,156. At 1500 / 5 min, one browser IP in that window would have been limited. On 2026-09-22 a single client sending 950–1,250 requests/min pushed prod to 33–43% 5xx for about 20 minutes. The limit holds any one client to about a quarter of that rate. Since enforcement, rate-limit-ip has blocked 46,131 requests from 4 IPs, nearly all from that client.

CSV limit stays a shadow rule. At 100 / 5 min it would have blocked 216k requests from 195 IPs over the 90 days. Almost all were scripted research downloads, plus 3,321 requests from 4 browser IPs. Since it went in on 2026-09-23 it has matched 5,388 requests from 11 IPs, none from browsers. The per-IP limit already bounds any single client. Revisit after CSV streaming (#770).

Monitoring and tuning are documented in mavedb-deployment docs/api-observability.md (VariantEffect/mavedb-deployment#40). That covers:

• a dashboard widget for enforced blocks vs. shadow would-blocks;
• saved Logs Insights queries for rate-limited clients, browser traffic caught by a rate rule, and per-IP peak rate;
• a tuning procedure that trials thresholds as shadow rules and changes staging before prod.

API docs. The Quickstart's Rate Limits section documents the limit and Retry-After handling, with Python, R, and curl examples (VariantEffect/mavedb-ui#725).

[bencap]

CSV limit is now enforced on prod and staging

shadow-csv-endpoints is now rate-limit-csv-endpoints and blocks. The limit is unchanged: 100 requests per IP per 5 minutes on /scores, /counts, /variants/data and /mapped-variants. Requests over it get a 429 with Retry-After: 300 and a body that points to the bulk archive. The CloudWatch metric is still rateLimitCsv.

Two related changes:

• Both rules' 429 responses now carry Access-Control-Allow-Origin: * and Access-Control-Expose-Headers: Retry-After, so a browser can read it instead of reporting a network error.
• The CSV rule ignores OPTIONS requests. Logged-in users' requests send a CORS preflight first, so before this change those preflights counted toward the limit, and once the limit tripped the preflight itself was blocked.

Why now instead of after #770. The per-IP limit counts requests, but the CSV stalls come from what each request costs. The API request logs from the last week show two patterns:

• On 2026-09-28 from 15:30 to 17:30 UTC, one Python-urllib client downloaded the full /scores CSV for 1,706 score sets. That's about 180 requests per 5 minutes, well under the 1,500 per-IP limit, but it used about 321 server-minutes in two hours and stalled the API.
• Several clients paged deep into urn:mavedb:00000053-a-1 (648k variants) with start/limit. Each page took 5–23 minutes, because every page sorted the whole score set.

Streaming (#770) won't land in the next release, and this rule is the cheapest protection we have until then.

Browsers. In the last 7 days, CSV traffic from mavedb.org peaked at 80 requests per 5 minutes across all users combined. We're treating the 4 browser IPs from the 90-day analysis as scripted clients, and limiting them is fine.

Also going into the next release:

• Database statement, lock and pool timeouts that return 503 instead of hanging (fix(db): bound API database waits and stop per-request login writes #897, for API handlers block the event loop, and database calls have no timeout #881).
• An index for the variant-number sort, so deep pages no longer sort the whole score set (perf(db): index variants by score set and variant number #899).
• A per-process cap on concurrent large CSV builds, which returns 503 with Retry-After when it's full (perf(csv): cap concurrent large CSV builds per process #898).
• The Quickstart's Rate Limits section documents the CSV limit (docs(api-quickstart): document the CSV download limit and 503 retries mavedb-ui#730).