Repository navigation
Stream the auction house instead of collecting it - #22
Conversation
The Lowest BIN refresh fetched every page of the Hypixel auction house, collected all ~100k auctions in one list and only then indexed them. The payloads carry base64 NBT blobs, so the process peaked around 700 MiB for an index that is a few hundred kilobytes. On the 2 GiB production host the kernel OOM-killed uvicorn - PID 1 in the container - roughly every five minutes; the uptime monitor reported that as hours of downtime. Pages are now folded into the index as they arrive and dropped immediately. Peak allocation follows the pages in flight rather than the size of the house: measured flat at 8.1 MiB from 4k to 32k auctions, where collecting grew 9.5 -> 20.9 MiB over the same range. Fetching stays concurrent, but as a worker pool rather than one task per page. Scheduling every page up front bounds the concurrent requests and nothing else: completed tasks hold their pages until the consumer gets to them, and since decoding is the slow part they queue up anyway. A worker only fetches its next page once it has handed the current one over, which is what actually applies back pressure. Consumption is serialised behind a lock because all workers fold into one shared index. Drift handling is unchanged in behaviour: pages are still checked against page 0, and the client now resets the consumer before each attempt so a retry cannot inherit what the discarded attempt already folded in. build_lowestbin_index() is kept for callers that already hold every auction; it now delegates to the same incremental builder. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a0b077278b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for worker in workers: | ||
| if not worker.done(): | ||
| worker.cancel() | ||
| await asyncio.gather(*workers, return_exceptions=True) |
There was a problem hiding this comment.
When one worker detects snapshot drift while another is awaiting _IndexingPageConsumer.add_page, cancelling that worker does not stop the function submitted through asyncio.to_thread; gather(..., return_exceptions=True) can therefore finish and the retry can call consumer.reset() while the old fold is still running. The old fold subsequently executes accumulator.total_bin_auctions += page_bin_auctions against the reset builder—and can overlap the next attempt—so a successfully retried snapshot may persist counts from the discarded attempt. Wait for any active consumer operation to actually finish before resetting and retrying, while cancelling only outstanding fetch work.
AGENTS.md reference: AGENTS.md:L76-L80
Useful? React with 👍 / 👎.
1 participant
Fixes the cause of the repeated multi-hour outages.
The incident
The host runs on 2 GiB with no swap (a swapfile existed since June but was never in
fstab, so it disappeared at the next reboot).scamscreener-apisat at ~700 MiB, and the kernel OOM-killer tookuvicorn— PID 1 in the container — 33 times in one day, roughly every 3–9 minutes.restart: unless-stoppedbrought it straight back, it reloaded the snapshot, and it died again. From outside that reads as hours of downtime rather than a crash loop.Why the API needed 700 MiB
_fetch_consistent_snapshotfetched every page of the auction house, collected all ~100k auctions into one list, and only then indexed them. Each auction carries a base64 NBT blob initem_bytes, whichresolve_auction_itemneeds — so the payloads cannot simply be trimmed. The result of all that isdict[str, float], a few hundred kilobytes.What changed
Pages are folded into the index as they arrive and dropped immediately. Peak allocation now follows the pages in flight instead of the size of the house:
Fetching stays concurrent, but as a worker pool rather than one task per page. That distinction is the whole fix, and the first attempt got it wrong: scheduling every page up front bounds the concurrent requests and nothing else — completed tasks hold their pages until the consumer reaches them, and since decoding is the slow part they queue up regardless. Measured at 1.0x, no improvement at all. A worker only fetching its next page once it has handed the current one over is what actually applies back pressure. Consumption is serialised behind a lock, because all workers fold into one shared index.
Drift handling behaves as before — pages are still validated against page 0 — with one addition: the client resets the consumer before each attempt, so a retry cannot inherit what the discarded attempt already folded in.
build_lowestbin_index()stays for callers that already hold every auction and delegates to the same incremental builder.Tests
129 pass. Four new ones:
test_auction_snapshot_stream_releases_every_page— weak references prove no page outlives the fetchtest_stream_snapshot_keeps_only_the_workers_pages_in_flight— the in-flight bound; verified by mutation, it fails when the pool is unbounded, which is exactly the bug the first attempt hadtest_auction_snapshot_drift_discards_the_partial_index— a retry does not double-counttest_lowestbin_refresh_indexes_every_page— folding indexes the same auctions collecting didThe existing off-the-event-loop regression test moved to the new seam (
fold_auctions_into_index); its assertions on thread identity and loop liveness are unchanged.Not covered here
Swap is now active and persisted on the host, but that is a safety net, not the fix: the 700 MiB were touched during indexing, so they could not stay paged out. Per-container memory limits are still worth adding — without them the OOM-killer picks its own victim, and it once picked
systemd-journald.🤖 Generated with Claude Code