Repository navigation
store: remove orphaned session index entries during GC listing - #495
CapThunder19 wants to merge 1 commit into
Conversation
Index members whose session key no longer exists were skipped but never removed. They keep the oldest scores, so once 100 build up they fill the GC candidate window and GC stops collecting expired and idle sandboxes. Remove them from both indexes with a Lua script that re-checks the key is still missing. Signed-off-by: Anirudh <anirudhpatwal19@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #495 +/- ##
==========================================
+ Coverage 60.45% 61.68% +1.23%
==========================================
Files 35 35
Lines 3616 3693 +77
==========================================
+ Hits 2186 2278 +92
+ Misses 1211 1187 -24
- Partials 219 228 +9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| return sandboxResults, nil | ||
| } | ||
|
|
||
| var removeOrphanedIndexEntriesValkeyScript = valkey.NewLuaScript(removeOrphanedIndexEntriesLua) |
There was a problem hiding this comment.
| var removeOrphanedIndexEntriesValkeyScript = valkey.NewLuaScript(removeOrphanedIndexEntriesLua) | |
| // In cluster mode, all keys accessed by this Lua script must hash to the same slot. | |
| var removeOrphanedIndexEntriesValkeyScript = valkey.NewLuaScript(removeOrphanedIndexEntriesLua) |
As noted in #75 (comment), we currently only support master/slave or single redis/valkey. Add a comment here to flag this constraint for future cluster support. The same as:
agentcube/pkg/store/store_valkey.go
Lines 113 to 114 in 1017f4d
What type of PR is this?
/kind bug
What this PR does / why we need it
When an entry in
session:expiry/session:last_activityhas lost itssession:<id>key,loadSandboxesBySessionIDsskips it but never removes it. These orphans always have the oldest scores, so they sit at the front of every GC query and take up candidate slots for good. Once 100 have built up, GC returns nothing on every cycle and expired or idle sandboxes stop being cleaned up. CodeInterpreter claims are hit hardest, since GC is the only thing that removes them.This PR removes orphans from both indexes when they're found, using a small Lua script that first re-checks the key is still missing, so a session created at that moment can't lose its entries. Redis and Valkey both get the fix. If the cleanup fails, it only logs a warning and GC carries on as before.
Which issue(s) this PR fixes
Fixes #491
Special notes for your reviewer
garbageCollector.once(): without the fix the claim was still alive after 10 cycles; with it, it was deleted in the first cycle.MGETin the same function already assumes one slot.Does this PR introduce a user-facing change?