feat: Handle all forms of range requests in fsspec

[gschulze]

Closes #259.

Adds support for all forms of range requests documented by fsspec, without changing obstore's own API.

Implemented:

• cat_file and cat_ranges accept every range form fsspec documents: either bound alone, and either counting back from the end of the object.
• cat_ranges additionally takes a scalar or None broadcast across all paths, and None elements. It also honors max_gap, batch_size, and on_error, which were previously ignored.
• Unless on_error="raise", failures now come back in the returned list.
• Suffix requests fall back to a size lookup and a bounded read on stores that reject them (Azure). A suffix larger than the object becomes a plain get.
• test_cat_ranges_mixed is no longer xfail.
• New tests were added to cover the new functionality.

Request cost: A plain start/end goes through get_range; start-only and negative-start become get with {"offset": n} and {"suffix": n}, so they stay single requests. A zero or absent start with no end sends no range header. cat_ranges splits its input the same way _get_partial_values does in the zarr PR: bounded ranges batch per object through get_ranges so nearby ones are merged, and each open-ended range is a separate get request. The object size is only needed for a negative end, or a negative start paired with an end.

Degenerate ranges: Zero-length, inverted, and start-past-the-end ranges raise an exception rather than returning empty. The first two are rejected by validate_range in obstore/src/get.rs, the third by object_store itself.

This only touches fsspec.py and its tests. get_range and get_ranges are unchanged.

[kylebarron]

This seems unnecessarily complicated when you could just add a parameter into a dict

[gschulze]

Agree on _CoalesceKwarg being ugly, it was added solely to satisfy the type checker. To make your suggestion work, I had to pass lengths explicitly, otherwise Pyright would complain.

[kylebarron]

Resolving a range should never need to know the size of the object (except on Azure, which doesn't support suffix requests).

I'd strongly recommend taking a similar approach to the Zarr-Python obstore adapter. https://github.com/zarr-developers/zarr-python/blob/d44f9f92ab4f12a8008de2553a7c9988669e3910/src/zarr/storage/_obstore.py#L440-L491

(For Azure, you can have a config parameter for the fsspec adapter for whether to avoid suffix requests, which then would make a HEAD request to always know the size)

[gschulze]

There are two cases where we need to know the size of the object: when start is negative and the backend does not support suffix requests, and when the bounds do not map to an HTTP range request, which occurs if end is negative, or when a negative start is paired with an end.

For the first case, I went with the zarr adapter's runtime fallback instead of a config parameter, which looks up the size after the store refuses the suffix.

[kylebarron]

Can you link to source code in fsspec that supports this typing? I.e. are there tests or a code path where we know that starts and ends can take None, either in isolation or as an element in a sequence?

[gschulze]

Added a comment with a permalink in _cat_ranges.

[kylebarron]

Overall this PR still seems to be overly complex for what it does (I'd say it seems to be written too much by Claude), and needs to be simplified a lot before merge

[kylebarron]

Context from a Claude review on my side, if you're interested. From a quick read of this summary, I think all of these are valid points that would need to be addressed before merging.

---

Thanks for the rework — going the adapter-only route is right, and the overall shape is what I had in mind: bounded ranges batched per object through get_ranges_async, open-ended ones through get_async with {"offset"}/{"suffix"}, mirroring _get_partial_values.

I built this branch and ran the suite against minio: 28 passed, 1 xfail, and all the new tests pass. ruff is clean; pyright shows only the pre-existing missing-stub errors for the fsspec imports.

Some follow-ups on my inline comments, including one where I was wrong.

Follow-ups on the inline comments

_CoalesceKwarg (fsspec.py:126)

Stands — and it's a one-liner. Both pyright and mypy accept the plain form:

kw: dict[str, int] = {} if max_gap is None else {"coalesce": max_gap}
store.get_ranges_async(path, starts=..., ends=..., **kw)

I see where the TypedDict came from: ebca428 refactor: Accept None for coalesce so the default lives in one place got reverted wholesale in 275e3c6 along with the get_opts change. My objection was only to get_range routing through get_opts — coalesce: int | None = None on the binding is independently reasonable and I'd take it as a follow-up if you'd rather have the default live in one place.

Needing the object size (fsspec.py:131)

I was too absolute here. A negative end genuinely has no HTTP range equivalent: data[:-10] is bytes 0..size-10, and a suffix request gives the last N bytes, not all but the last N. fsspec's own reference implementation does exactly what this PR does:

# fsspec/spec.py, AbstractFileSystem.cat_file
if end is not None:
    if end < 0:
        end = f.size + end

And test_cat_ranges_mixed — the test this issue exists to un-xfail — passes ends=[None, -10, -10]. The zarr adapter avoids the problem only because zarr's ByteRequest is a closed set (Range/Offset/Suffix) with no negative end, so it doesn't port directly. Fetching the size for a negative end is fine.

There is a real bug in the neighborhood, though. _resolve_inexpressible_bounds applies the fetched size to every row, not just the rows that needed it:

fs.cat_ranges([path, path], starts=[-5, 0], ends=[None, -10])
# range actually sent for row 0: {'offset': 9995}   ← should be {"suffix": 5}

Row 0 is an exactly-expressible suffix, but it gets rewritten into a size-dependent offset because row 1 happened to need a HEAD. So a pure suffix silently becomes size-dependent based on unrelated entries in the same call. Please gate _apply_object_size on _needs_object_size per row so rows that don't need the size are left alone.

Sequence[int | None] typing (fsspec.py:479)

You're right, and here's the source I was asking for — worth putting a pointer to one of these in a comment:

• AbstractFileSystem.cat_ranges (fsspec/spec.py) forwards each element unchanged: out.append(self.cat_file(p, s, e)). cat_file's documented contract is "start, end: int — If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively."
• AsyncFileSystem._cat_ranges (fsspec/asyn.py) does the same: self._cat_file(p, start=s, end=e) for p, s, e in zip(paths, starts, ends).
• fsspec.utils.merge_offset_ranges normalizes starts = [s or 0 for s in starts] and guards e is not None — None elements are expected there.
• Scalar None broadcast follows from if not isinstance(starts, Iterable), since None isn't Iterable.

Other things I found

on_error diverges from upstream, and the comment justifying it is wrong. The code says "Like fsspec's own _cat_ranges, on_error is ignored, so failures propagate." Upstream ignores the on_error argument but hardcodes return_exceptions=True, so it always behaves as the default on_error="return":

upstream memory fs: [b'abc', FileNotFoundError('/nope')]
this branch:        raises FileNotFoundError

Not a regression — the old asyncio.gather also raised — but the comment should be corrected either way. If we want parity it's one kwarg plus widening output_buffers to list[bytes | BaseException].

A fully-unbounded element in cat_ranges still sends a range header. _cat_file special-cases start in (0, None), end=None into a plain get, but _cat_ranges turns it into {"offset": 0} → Range: bytes=0-, which 416s on a zero-length object where cat_file returns b"". Cheap to mirror the special case.

Degenerate ranges. Agreed with the choice to keep raising, but note upstream isn't merely inconsistent between backends — the reference cat_file does f.read(end - f.tell()), and AbstractBufferedFile.read treats a negative length as "rest of file", so an inverted range silently returns the tail. Raising is better behavior; it just deserves a line in the cat_file/cat_ranges docstring since it deviates from the documented slice semantics.

fsspec.asyn._run_coros_in_chunks is private API. It's the only way to honor batch_size and the adapter already leans on fsspec internals, so I'm fine with it — just noting the coupling.

Azure. This introduces suffix requests where the adapter previously made none, and Azure doesn't support them. Out of scope for this PR — I'll open a separate issue. (The zarr adapter catches the failure and falls back to HEAD + bounded range.)

[gschulze]

Thanks for your detailed feedback, this is all fair. Everything you raised is implemented locally. Regarding complexity, I'm afraid the PR has not gotten smaller after incorporating your points, but I think the code quality has improved. I still need some time to check whether everything is consistent now, and whether I can simplify it any further. Will ping when ready for another look.

[gschulze]

I've addressed all the points you raised and updated the PR description.