[PaulHax]

Issue

Girder folder browsing checks subtree loadability before showing Open Folder in VolView. This can generate substantial Python and MongoDB work without opening VolView. This addresses avoidable work in #110 after the filtering and 1,000-file limit added in #111.

The largest gains are for many rejected candidates across separate items, including large-image TIFFs, SM DICOMs, job outputs and staged processing inputs.

Sparse image trees can also benefit when browsing one or two levels above the leaf folders. Full descendant discovery still happens.

Videos and unsupported formats are already filtered before the expensive metadata checks, so folders containing only those files benefit little.

Ordinary CT folders usually succeed on the first file and see little change.

Rejected slices sharing one item already reuse cached metadata, so they also see little benefit.

Diagnosis

Each newly examined non-session item requires an item read and an inherited folder permission check. Distinct items repeatedly fetch the same folder, so rejecting many candidates can cause thousands of reads. Parent documents include unrelated metadata; cursor batches can perform joins and transfer files that an early success never needs.

The browser can also request eligibility before its VolView button exists, leaving no button to update.

Changes

• Project required parent fields, check the first candidate alone, then batch parent reads in groups of 128. Small helpers separate item queries, readable-folder caching and parent caching. Permissions and positive/negative caches remain request-scoped.
• Request one initial file candidate, then cursor batches of 128 to bound subsequent read-ahead.
• Skip eligibility requests when no VolView button exists; retain checks after item-list updates.
• Exclude unused job logs from output-folder correlation lookups; this separate change was not timed.
• Add two cases to the existing full-stack suite against deployed Girder, MongoDB and VolView: upload files, browse folders, revisit metadata changes and open the resulting image.

Evidence

Final measurements compare 38892f0257bb2a39ea02524e6b90824658aa8bbe with remote main eeee766, using synthetic metadata, real Girder/MongoDB routes and nonadmin users.

Measurement	Before	After
512 rejected SM slices in separate items, route ms	626.70	32.42
512 CT slices sharing one item, route ms	17.32	16.60
512 rejected SM slices sharing one item, route ms	21.58	25.64
1,000 rejected TIFF/SM candidates, Mongo commands/request	2,008	32
1,000 rejected items with 64 KiB extra metadata each, returned BSON MB/request	66.35	0.21
Cursor-only comparison: 100 sparse leaves, route ms	30.42	17.16

Only the cursor row holds parent checks fixed; other rows combine backend changes. Command/BSON accounting is separate from timing. Padded metadata is a stress fixture.

A ten-second mixed workload used five users, three tabs and five active jobs each. Before/after runs completed 374/374 status polls and 5/5 browsing checks, with 0/0 errors. Girder CPU changed 4.104 → 1.379 seconds, and Mongo CPU 2.058 → 0.530 seconds. One phase per revision does not establish polling-latency improvement or production capacity.

Methodology, scripts and raw samples include reproduction commands.

Tradeoffs and coverage

Smaller cursor batches add round trips: the isolated 1,000-SM-item scan takes 43.57 → 45.64 ms. Initial size one can leave a one-file cursor open for deferred cleanup, whose costs are not fully measured. Parent batching can read ahead after an early rejection.

The aggregation pipeline, permissions and 1,000-candidate limit are preserved. Full descendant discovery remains; descendant order is unspecified. These measurements do not establish the production incident's cause or actual import layout.

Regression coverage includes permissions, batch boundaries, metadata changes, button visibility, loading an image through the browser UI, and job-log preservation.

[manthey]

This is a strict improvement. If there is a very large tree that isn't volview loadable, I think the aggregation will still be slow. I'll dig into what else can help.

[manthey]

@PaulHax As far as I am concerns, this is good to merge.