Repository navigation
Read stale pickled tagging-cache entries as misses, not 500s - #974
Merged
Merged
Conversation
Codex 2.5.3 (comicbox 5.2.2) pickled imagehash's numpy.float64 cover scores into pending online-tag prompts. 2.5.4 (comicbox 5.3.1) ships without numpy, so every read of those prompts raised ModuleNotFoundError: the daemon's startup prune (logged under "Clearing stale online tag scan marker on startup"), /admin/tag-prompts and /admin/tag-sessions/snapshot. - ResilientFileBasedCache also treats ImportError and AttributeError on load as an unreadable entry: delete the file, report a miss. A pickle naming a module or class that no longer exists is as unloadable as a truncated one. The discard now logs a warning, so an admin whose review queue emptied after an upgrade can see why. - serialize_candidate casts score, metadata_score and cover_score to plain float, so a library's number type can't end up in the cache again. Prompts already stored with numpy scores are dropped; those comics need another online-tag pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The discard deletes the file, so the log line is the only record of what was dropped and why. Name the cache key (the file name is a hash) and attach the traceback, so an error class caught there by mistake can be traced from the log alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After upgrading to the 2.5.4 image, a user with online-tag matches pending review sees
ModuleNotFoundError: No module named 'numpy'at startup (logged under "Clearing stale online tag scan marker on startup") and as 500s on/api/v4/admin/tag-promptsand/api/v4/admin/tag-sessions/snapshot.Codex 2.5.3 (comicbox 5.2.2) scored covers with imagehash, whose hash distance is
numpy.int64, socover_scoreand the blendedscorewerenumpy.float64.serialize_candidatecopied them into each pending prompt, which is pickled into the file-basedtaggingcache and kept across restarts. 2.5.4 (comicbox 5.3.1, which uses a pure-Python pHash) no longer installs numpy, so every read of those prompts fails while unpickling.ResilientFileBasedCacheonly treatedzlib.error,UnpicklingErrorandEOFErroras unreadable, so the error reached the caller.Changes
ResilientFileBasedCachealso treatsImportErrorandAttributeErrorraised while loading an entry as unreadable: delete the file, report a miss. A pickle naming a module or class that no longer exists is as unloadable as a truncated one.serialize_candidatecastsscore,metadata_scoreandcover_scoreto plainfloat, so a library's number type can't end up in the cache again.Prompts already stored with numpy scores are dropped on first read; those comics need another online-tag pass.
Testing
tests/test_resilient_cache.pysaves a value whose class comes from a temporary module, then removes the module (ModuleNotFoundError) or just the class (AttributeError). It coversget,touch, the warning's content and the startup prompt prune. All of them failed before the fix with the sameModuleNotFoundErrorfrom Django'sfilebased.pythat the user reported.tests/test_onlinetag_serialize_candidate.py.make fix,make lint,make tyclean;make test: 1668 passed, 1 xfailed (Python), 994 passed (frontend).make lintstill shows two existingreportUnreachablewarnings incodex/views/browser/annotate/cover.py:63,69; this PR doesn't touch that file.🤖 Generated with Claude Code