Skip to content

fix(storage): decouple details context, invalidate cache on refresh, and add singleflight - #2950

Open
Vurliy wants to merge 2 commits into
OpenListTeam:mainfrom
Vurliy:fix/storage-details-async-and-cooldown
Open

fix(storage): decouple details context, invalidate cache on refresh, and add singleflight#2950
Vurliy wants to merge 2 commits into
OpenListTeam:mainfrom
Vurliy:fix/storage-details-async-and-cooldown

Conversation

@Vurliy

@Vurliy Vurliy commented Aug 19, 2026

Copy link
Copy Markdown

Summary / 摘要

1. Problem & Symptoms / 现状与问题现象

  • Symptom 1 (Multi-drive quota missing on home/manage pages) / 现象 1(多网盘容量在首页和管理页显示为 -
    When multiple cloud storages (such as OneDrive, Google Drive, Baidu Netdisk, or high-latency remote storage) are mounted, opening the home page (/) or the storage management list often displays - instead of the storage capacity/quota progress bar. Users have to manually navigate into each specific folder and refresh to temporarily trigger a quota query (as reported in OpenList无法显示各个网盘的容量大小了 #1815, [BUG] 别名容量显示错误 #2633, [BUG] 别名容量显示错误 #2635).
    挂载多个云盘(如 OneDrive、Google Drive、百度网盘或跨国高延迟存储)时,进入首页或管理后台存储列表,大部分网盘在“大小”列中均显示为 - 而非容量进度条。用户必须手动点进各个子文件夹并刷新才能临时触发容量获取(参见 OpenList无法显示各个网盘的容量大小了 #1815[BUG] 别名容量显示错误 #2633[BUG] 别名容量显示错误 #2635)。

  • Symptom 2 (Force refresh abortion & missing background caching) / 现象 2(强刷时后台任务被强行中断且无法落入缓存)
    When clicking the bottom-right refresh button on the root directory (refresh: true), drives that take longer than 1 second to respond return -. More importantly, waiting does not result in the cache being populated because the background tasks are terminated immediately after the frontend request finishes.
    在根目录点击右下角强刷时,响应超过 1 秒的网盘返回 -。但更严重的是,等待后再次刷新依然无法获得数据,因为后台请求在前端响应结束的瞬间就被直接中断了。

  • Symptom 3 (Stale cache confusion on force refresh & immediate F5) / 现象 3(强刷后立即刷新页面导致旧缓存混淆与数据不一致)
    When clicking the bottom-right force refresh button, unfinished drives initially return -. If the user immediately right-clicks / refreshes the page (F5) while background requests are still in-flight, the backend (lacking explicit cache invalidation) serves the old stale cache from before the refresh. The user cannot distinguish whether the displayed quota is fresh or old cached data, violating data consistency expectations.
    用户点击右下角强刷按钮后,尚未完成探测的网盘会先返回 -。若用户在后台请求仍在进行时立即右键刷新(F5),由于缺乏显式的缓存失效机制,后端会直接返回刷新前的旧缓存数据。用户根本无法分辨当前看到的容量是最新探测到的结果还是之前的历史旧缓存,违背了数据一致性预期。


2. Root Cause Analysis / 核心根因深度分析

  1. Premature Context Cancellation / 请求 Context 过早取消
    In internal/op/storage.go (GetStorageVirtualFilesWithDetailsByPath), the background goroutines executing GetStorageDetails(ctx, dri, refresh) directly inherit the incoming HTTP request's c.Request.Context(). To ensure fast initial page load, the frontend only waits up to 1 second (time.After(time.Second)). As soon as the 1-second deadline elapses and the HTTP response completes, the web framework automatically cancels c.Request.Context(). Any in-flight network requests (e.g., to Microsoft Graph API or remote storage) are immediately aborted with context canceled, preventing the background goroutines from completing and writing the fresh quota into detailCache.
    internal/op/storage.goGetStorageVirtualFilesWithDetailsByPath 中,并发后台 Goroutine 调用 GetStorageDetails 时直接继承了 HTTP 请求的 c.Request.Context()。为了保证首屏秒开,前台最多等待 1 秒。一旦 1 秒超时到达,HTTP 响应结束,框架立即自动发出 cancel() 信号,导致后台正在进行的跨国网络请求被强制以 context canceled 中断,未能跑完并写入 detailCache

  2. OneDrive Token Refresh Blocked by noRetry: true / OneDrive 驱动禁止自动刷新令牌
    In drivers/onedrive/util.go (getDrive) and drivers/onedrive_app/util.go (getDrive), getDrive() calls d.Request(api, http.MethodGet, ..., &resp, true). The 4th argument noRetry = true explicitly disables automatic token renewal. When an account has been idle for more than 1 hour (Microsoft OAuth AccessToken expired), Microsoft Graph API responds with 401 InvalidAuthenticationToken. Because noRetry is true, the driver immediately fails and refuses to invoke refreshToken(), causing all quota queries for idle OneDrive accounts to fail permanently.
    drivers/onedrive/util.gogetDrivedrivers/onedrive_app/util.gogetDrive 中,请求调用传递了 noRetry = true。当账号闲置超过 1 小时(微软 AccessToken 过期)时,微软返回 401 InvalidAuthenticationToken,驱动因禁止重试而直接报错放弃,根本不会去调用 refreshToken() 换取新令牌,导致闲置账号的容量探测全部失败。

  3. Lack of Explicit Cache Invalidation on Force Refresh / 强刷时缺少显式缓存失效机制
    When a force refresh (refresh: true) is triggered, the existing detailCache was not explicitly purged. If an immediate follow-up non-force request (e.g., F5) arrived while the background probing was still in progress, GetStorageDetails would hit the stale detailCache and return outdated numbers instead of indicating that probing was underway.
    当触发强刷(refresh: true)时,系统未显式清除原有的 detailCache。若用户在后台探测期间发起常规请求(如 F5 刷新),GetStorageDetails 会直接命中未失效的旧缓存,导致返回陈旧数据,而非反映当前正在探测的状态。


3. What Was Fixed / 修复与改造内容

  1. Context Decoupling with Bounded Timeout / 上下文解耦与独立超时保护
    Decoupled the context in GetStorageVirtualFilesWithDetailsByPath using bgCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), timeoutSec) (storage_details_timeout_seconds, default 15s). The foreground retains the 1-second fast response deadline (zero UI blocking), while the background goroutine is granted an independent 15-second lifetime to finish fetching data and write into detailCache.
    使用 context.WithoutCancel 配合独立超时保护(默认 15 秒)解绑上下文。前台保持 1 秒快速响应(保证页面不卡顿),后台 Goroutine 拥有独立生命周期平稳跑完并落入 30 分钟缓存。

  2. Automatic OAuth Token Refresh in OneDrive / 修复 OneDrive 驱动自动换 Token
    Removed noRetry = true in onedrive.getDrive() and onedrive_app.getDrive(), allowing d.Request() to automatically capture InvalidAuthenticationToken, invoke refreshToken(), and retry the quota query seamlessly.
    移除了 getDrive() 中的 noRetry: true 参数,使底层自动捕获 401 错误并自动换取新 Token 重试。

  3. Strict Invalidate-On-Refresh & Singleflight Cooldown / 强刷显式失效与 Singleflight 保护
    Explicitly invalidate stale detailCache on force refresh (Cache.InvalidateStorageDetails) to ensure data consistency, combined with detailsG singleflight.Group to coalesce concurrent requests, and added a configurable setting storage_details_cooldown_seconds (default 0s, preserving native behavior).
    强刷时主动失效旧缓存(Cache.InvalidateStorageDetails)以保证数据强一致性,配合 Singleflight 并发去重,并增加了可配置项 storage_details_cooldown_seconds(默认 0 秒,完全兼容官方原生预期)。

  4. Unit Testing / 完整单元测试
    Added internal/op/storage_details_test.go covering Singleflight coalescing, Invalidate-On-Refresh, and Cooldown logic.
    添加了完整的 Go 单元测试。


4. Behavior After Fix / 修复后完整表现行为

  1. Zero UI Blocking & Reliable Background Caching / 首屏秒开与后台平稳落盘
    Opening the home page or storage list responds in under 1 second. Fast storage quota is displayed instantly; slower cloud storages continue probing in the background without cancellation and write into the 30-minute cache upon completion.
    首屏与列表请求在 1 秒内迅速返回,零页面卡顿;慢速网盘在后台平稳完成探测并写入 30 分钟缓存。
  2. Automatic Token Recovery / OneDrive 闲置账号无感恢复
    Idle OneDrive accounts automatically refresh expired tokens on demand, eliminating permanent - displays.
    长期闲置的 OneDrive 账号在探测时自动静默刷新令牌,彻底消除永久 - 现象。
  3. Strict Consistency & Zero Stale Data Leak / 强一致性与杜绝旧缓存泄露
    Clicking the bottom-right refresh button purges the old cache immediately. In-flight drives consistently return - (even during rapid F5 page refreshes). Once the background probe finishes (typically 2-3s), subsequent page visits immediately display 100% fresh and accurate storage quota.
    点击右下角强刷立即清除旧缓存;在后台探测完成前,任何常规刷新均一致性返回 -,绝不回退返回旧缓存;探测完成后,后续刷新立即秒级呈现 100% 真实最新的容量数据。
  4. Singleflight Deduplication & Configurable Protection / 请求并发合并与防刷保护
    High-frequency concurrent requests are merged into a single upstream call, protecting external cloud storage APIs.
    高频并发请求自动合并为单个底层探测,有效保护外部网盘 API。
  • This PR has breaking changes. / 此 PR 包含破坏性变更。
  • This PR changes public API, config, storage format, or migration behavior. / 此 PR 修改了公开 API、配置、存储格式或迁移行为。
  • This PR requires corresponding changes in related repositories. / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

Related Issues / 关联 Issue

Testing / 测试

  • go test ./internal/op -run TestGetStorageDetails
  • Manual test on live instance with 30+ OneDrive and WebDAV storage mounts.

Checklist / 检查清单

  • I have read CONTRIBUTING. / 我已阅读 CONTRIBUTING
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct. / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt / go fmt.
  • I have requested review from relevant maintainers or code owners where applicable. / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content. / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • Gemini

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Tests / 测试

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR. / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution. / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools. / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenList无法显示各个网盘的容量大小了

1 participant