HBASE-30285 BucketCache shutdown/disable does not release backingMap-owned BucketEntry references

[mwkang]

See HBASE-30285.

BucketCache.backingMap owns the initial reference of each BucketEntry.
The existing shutdown and disable paths discard map entries without releasing these references, so their reference counts may never reach zero.

This change:

• disables the cache and interrupts writer threads before cleanup
• waits for in-flight writers before releasing backingMap entries
• stops the cache persister before final serialization
• persists metadata only for explicit persistent-cache shutdown
• releases each backingMap owner reference under its offset write lock
• preserves references held by in-flight callers until those callers release them
• runs cleanup on a one-shot thread to avoid writer self-join and read-to-write lock upgrade deadlocks
• removes corrupt persistent-cache entries through blockEvicted, preserving file and region cache metadata
• does not add synchronization to the normal cache read, write, or writer queue paths

The tests cover:

• non-persistent shutdown with a retained caller reference
• cache disable after repeated IO errors
• entries published by an in-flight writer
• concurrent and repeated shutdown
• persistent-cache serialization and recovery
• corrupt persistent-cache reads through direct and reference-file lookups

[liuxiaocs7]

We’d better keep the Jira title and commit message consistent, thanks!

[wchevreuil]

Thank you for submitting this fix. I have some concerns about this, however.

Considering we call BucketCache.shutdown normally when stopping a RS process, the reference count leak here wouldn't be a big problem (the jvm process will be over soon). The introduction of locks on a hot code path, though, could have a performance impact.

Have we done any performance benchmark comparison for validating this change is not introducing a noticeable performance impact?

[mwkang]

The benchmark concern was valid for the previous revision, which added lifecycle locking around cacheBlock and writer-queue publication.
That hot-path locking and the queue-polling change have been removed.

Compared with master, cacheBlock and the normal writer-queue path are unchanged.
The successful getBlock path adds no new lock or backingMap scan. In getBlock, the new scan and offset write-lock path runs only after an HBaseIOException. Shutdown/disable cleanup runs after the cache is disabled and acquires per-entry offset write locks there.
Therefore, I did not run a benchmark for this revision.

The leak is also reachable when IOEngine read, write, or sync errors persist beyond the configured tolerance duration: BucketCache disables itself while the RegionServer JVM continues running. It is therefore not limited to process shutdown.

[wchevreuil]

We are modifying behaviour with this. Now, in the event of an io read error, even if checkIOErrorIsTolerated ignores the error, we will evict the block?

[wchevreuil]

Rather than iterating the whole backingMap, can we use similar logic from getBlockForReference to resolve the reference key? Please make the common code reusable.

[wchevreuil]

Why we wouldn't want to persist the backingMap here?

[wchevreuil]

nit: Let's call this 'cleanupBackingMap' or 'cleanupCacheIndex' to avoid confusion.

[wchevreuil]

The leak is also reachable when IOEngine read, write, or sync errors persist beyond the configured tolerance duration: BucketCache disables itself while the RegionServer JVM continues running. It is therefore not limited to process shutdown.

For file based bucket cache, backingMap can grow quite large and traversing it to release references can be costly. Should we worry about this only when shutting down BC after exceeding IO errors thresholds, and ignoring it when doing RS shutdown?