Skip to content

Use off_t instead of the glibc-specific off64_t in FileStreamer - #228

Open
hjmjohnson wants to merge 1 commit into
malaterre:masterfrom
BRAINSia:fix/off64_t-portability
Open

hjmjohnson wants to merge 1 commit into
malaterre:masterfrom
BRAINSia:fix/off64_t-portability

Conversation

@hjmjohnson

@hjmjohnson hjmjohnson commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

gdcmFileStreamer.cxx fails to build on musl-based distributions (Alpine) because off64_t is a glibc LFS64 name. This replaces it with a GDCM-owned gdcm::offset_t, deleting the hand-maintained platform list without adding to it.

It also fixes a latent large-file bug found while investigating: the file's _FILE_OFFSET_BITS 64 definition is placed after the include block, which makes it a no-op, so writes past 2 GB are silently broken on 32-bit glibc today.

Reported downstream as InsightSoftwareConsortium/ITK#6779.

Question for you before this is worth reviewing in detail: defining _FILE_OFFSET_BITS in a single TU changes struct stat layout and off_t width for that TU only. It is safe here (the only struct stat use is a local fstat(), and offset_t does not appear in gdcmFileStreamer.h), but if you would rather set it project-wide in CMake I am happy to redo it that way; say the word.

Root cause

musl gates its LFS64 names strictly on _LARGEFILE64_SOURCE and, unlike glibc, does not imply that from _GNU_SOURCE:

/* musl 1.2.4, sys/types.h */
#if defined(_LARGEFILE64_SOURCE)
#define off64_t off_t
#endif

musl upstream is also removing these aliases, so enabling _LARGEFILE64_SOURCE is not a durable answer. musl's off_t is unconditionally 64-bit, so off_t was always the right type there.

Extending the platform list would fix Alpine but not the class of bug. off64_t is a typedef, so the preprocessor can never test for it, so every guard is necessarily a proxy for "which libc is this", which is why the list has grown one platform at a time.

The glibc build works only because the C++ frontend implicitly defines _GNU_SOURCE, which turns on __USE_LARGEFILE64. Nothing in GDCM requests it, and the same block does not compile as C:

$ g++ -dM -E -x c++ /dev/null | grep _GNU_SOURCE
#define _GNU_SOURCE 1
$ gcc -dM -E -x c   /dev/null | grep _GNU_SOURCE      # nothing

Separately, #define _FILE_OFFSET_BITS 64 must precede every libc header. In this file it sits after nine gdcm*.h includes, all of which transitively pull <string> → <features.h>. Once that include guard is set, a later _FILE_OFFSET_BITS is ignored. __GNU_LIBRARY__ is likewise only defined after a libc header has been read, so the guard is order-dependent in the same way.

What changed
  • Move #define _FILE_OFFSET_BITS 64 ahead of all includes, unconditionally and #ifndef-guarded. It has no effect on macOS, the BSDs, musl, or MSVC, where off_t is already 64-bit or unused for file offsets. mingw-w64 honors it, which is harmless because offset_t is int64_t on the Windows CRT.
  • Replace off64_t with gdcm::offset_t: off_t on POSIX, int64_t on the Windows CRT. This deletes the __APPLE__ || __FreeBSD__ || __OpenBSD__ || __NetBSD__ || __EMSCRIPTEN__ list without adding anything.
  • Add static_assert(sizeof(offset_t) >= 8, ...) so a 32-bit off_t is a build error rather than silent truncation.
  • Add an explicit <cstdint> for int64_t, which was previously reached only transitively.

The one remaining #if tests "is this the Windows CRT", a genuine API difference. offset_t does not appear in gdcmFileStreamer.h, so this is contained to one translation unit and changes no public API.

Property before after
Platform names in the guard 5 + Windows Windows only
Breaks on the next unlisted libc yes no
Depends on include order yes no
Depends on implicit _GNU_SOURCE yes no
Uses a libc-reserved name as a macro yes no
32-bit off_t outcome silent truncation build error
_FILE_OFFSET_BITS takes effect no yes
Testing

Synthetic probes, three libc/arch combinations:

musl x86_64 glibc x86_64 glibc i386
LFS define, current placement dead, off_t=8 dead, off_t=8 dead, off_t=4
LFS define, patched placement dead, off_t=8 active active, off_t=8
current off64_t logic does not compile compiles compiles
patched offset_t logic clean, 8 clean, 8 clean, 8

Against the real source rather than probes:

  • Alpine/musl, unpatched: error: 'off64_t' has not been declared at gdcmFileStreamer.cxx:55, :68, :81, reproducing the report.
  • Alpine/musl, patched: TU compiles clean at -Wall -Wextra.
  • i386 glibc, patched: TU compiles, so the static_assert passing is itself proof that off_t is now 64-bit in GDCM's own TU.
  • i386 glibc, patched but with the LFS define restored to its current placement: static_assert failed: GDCM requires 64-bit file offsets, evidence that 32-bit builds truncate today.

macOS arm64, full build, -DGDCM_BUILD_TESTING=ON with GDCM_DATA_ROOT set, -Wall -Wextra:

  • Builds clean; the only warning is a pre-existing one in gdcmPrivateTag.h.
  • TestFileStreamer1..6: 6/6 pass.
  • Full suite 206/230. The 24 failures are identical on unpatched master (verified by reverting only this file and re-running the same set), so they are pre-existing and unrelated.

Not covered: MSVC, MinGW-w64, and a >2 GB FileStreamer round-trip: no such fixture exists in gdcmData, so the 32-bit claim rests on the static_assert and the probes above rather than a runtime test.

Verification, 2026-10-26 (after review fixups; gdcmData and gdcmDataExtra)
Platform Result
glibc x86_64, Ubuntu 24.04, gcc 13 failure set identical to master (7 network/server tests and TestInvalidDICOMFiles)
i386 Debian bookworm (32-bit off_t without the define) builds; static_assert passes; failure set identical to master
musl x86_64, Alpine builds (master: 67 errors); only the 6 network/server tests fail
macOS arm64, AppleClang failure set identical to master

The review fixups only shortened two comments; the code is unchanged.

Follow-ups, deliberately not bundled here
  1. Hoist _FILE_OFFSET_BITS=64 to CMake (target_compile_definitions(gdcmMSFF PRIVATE ...) or global) and drop the per-file define. This is what AC_SYS_LARGEFILE does and removes the single-TU caveat above entirely. Happy to open it if you prefer that shape.

  2. MinGW-w64 truncates at 2 GB. All three wrappers take 32-bit paths under __MINGW32__ (fseek, ftell, _chsize with a long), and mingw-w64 also defines __MINGW32__, so every mingw-w64 build silently caps at 2 GB. mingw-w64 provides _fseeki64, _ftelli64 and _chsize_s, so those branches can likely fold into the MSVC path. The static_assert here does not catch it: the type is fine, the calls are wrong. Needs a real mingw-w64 build to confirm; untouched by this PR.

  3. Testing/Source/Common/Cxx/TestSystem2.cxx has the same dead _FILE_OFFSET_BITS placement (after gdcmSystem.h, guarded on __GNU_LIBRARY__), so its sizeof(off_t) check still sees 4 on i386 glibc.

  4. 32-bit Android below API 24 likely loses fseeko/ftello under _FILE_OFFSET_BITS=64 (bionic gates the 64-bit variants on API 24). Unverified; a guard is easy if anyone builds that target.

@hjmjohnson
hjmjohnson force-pushed the fix/off64_t-portability branch 2 times, most recently from 21cb6bd to f122877 Compare August 21, 2026 20:02
@hjmjohnson
hjmjohnson marked this pull request as ready for review August 21, 2026 20:07
@hjmjohnson

Copy link
Copy Markdown
Contributor Author

@malaterre Improved code compilation.

@hjmjohnson

Copy link
Copy Markdown
Contributor Author

Full-toolkit musl results, since "it builds" deserves numbers.

Pristine master on Alpine fails with 67 compiler errors, and all 67 are in gdcmFileStreamer.cxx — this one file is the sole blocker for the entire toolkit on musl. With this PR applied: 0 errors, 36 artifacts, and the test suite matches the macOS run exactly apart from three tests that need an unreachable external server.

pristine master with this PR
compiler errors 67 (all in gdcmFileStreamer.cxx) 0
warnings 7 7 (unchanged)
ninja exit 1 0
artifacts in bin/ — 36 (6 test drivers + 30 tools)
ctest not reachable 206 / 233 pass (88%)
Environment and exact method

Alpine 3.24.1, gcc 15.2.0, musl x86_64 (/lib/ld-musl-x86_64.so.1), under Docker on an arm64 host via --platform linux/amd64.

The source tree is a pristine git archive of master — not a working checkout — so nothing else can leak in. The only difference between the two columns is this PR's patch:

git archive gitupstream/master | tar -x -C ./pristine

docker run --rm --platform linux/amd64 \
    -v "$PWD/pristine:/src:ro" \
    -v /path/to/gdcmData:/data:ro \
    alpine:latest sh -c '
  apk add -q build-base cmake ninja patch
  cp -r /src /work && cd /work
  # omit the next line for the baseline column
  patch -p1 < /sp/pr228.patch
  cmake -G Ninja -S /work -B /bld \
      -DCMAKE_BUILD_TYPE=Release \
      -DGDCM_BUILD_TESTING=ON \
      -DGDCM_BUILD_APPLICATIONS=ON \
      -DGDCM_DATA_ROOT=/data
  ninja -C /bld -j"$(nproc)" -k 0     # -k 0 so the baseline reports ALL errors, not the first
  cd /bld && ctest -j"$(nproc)"
'

-k 0 matters for the baseline: without it ninja stops at the first failure and you cannot tell whether one file or twenty are broken.

Baseline error breakdown

All 67 errors come from a single file. Grouped by source file:

     67 Source/MediaStorageAndFileFormat/gdcmFileStreamer.cxx

The first three are the reported symptom; the remaining 64 are cascade from the three wrapper signatures failing to parse:

gdcmFileStreamer.cxx:55:40: error: 'off64_t' has not been declared
gdcmFileStreamer.cxx:68:15: error: 'off64_t' does not name a type
gdcmFileStreamer.cxx:81:51: error: 'off64_t' does not name a type
Test results, and why 27 fail

206 / 233 pass. The 27 failures split cleanly:

  • 24 are exactly the set that fails on macOS arm64 on the same tree, verified by comm on the two sorted failure lists — zero symmetric difference. Pre-existing and unrelated to this PR (TestMD5, the Test*ChangeTransferSyntax* family, TestImageReader, TestPrinter1/2, TestCleaner4, …). They also fail identically on unpatched master where it compiles, so they are not musl-specific and not caused by this change.
  • 3 are gdcmscu-echo-dicomserver, gdcmscu-store-dicomserver, gdcmscu-find-dicomserver, which contact a public DICOM server over the network. The container has no route to it, so they time out. These exist only in the musl column because I set GDCM_BUILD_APPLICATIONS=ON there; the macOS run did not build the applications.

So: no test regresses on musl relative to macOS, and nothing in the failure set is attributable to this patch.

Reproduced twice from scratch with identical results.

The 32-bit _FILE_OFFSET_BITS half of this PR is not exercised by any of the above — musl's off_t is unconditionally 64-bit. That part was verified separately on i386 glibc, as described in the PR body.

@hjmjohnson

Copy link
Copy Markdown
Contributor Author

Corroboration: Alpine has been carrying a downstream patch for this, 30-off_t.patch in testing/gdcm (currently at 3.2.5). So this is an existing packaging burden, not a hypothetical one, and the distro independently reached the same conclusion — off_t is the right type.

Their patch also makes the case for using a distinct alias name rather than renaming in place.

What the Alpine patch does, and why this PR does it differently

30-off_t.patch is a blanket off64_t → off_t rename. Same semantic conclusion as this PR, but applied textually, which produces three artefacts:

 #if defined(_WIN32) && (defined(_MSC_VER) || defined(__MINGW32__))
 #include <io.h>
-typedef int64_t off64_t;
+typedef int64_t off_t;          // redefines the system off_t
 #else
 #if defined(__APPLE__) || ...
-#  define off64_t off_t
+#  define off_t off_t           // no-op self-definition
 #endif
-// Basically enforce use of off64_t over off_t since on windows off_t is pretty
+// Basically enforce use of off_t over off_t since on windows off_t is pretty

typedef int64_t off_t; on the Windows CRT conflicts with the platform's own off_t, and #define off_t off_t does nothing. Neither bites Alpine, because the patch only ever runs on musl/Linux where both branches are dead — it is a scoped downstream unblock and fine as that. But it is not upstreamable as-is, and it is a good illustration of why this PR introduces gdcm::offset_t instead of reusing the name off_t: a GDCM-owned alias cannot collide with a libc type on any platform, so the same change becomes safe everywhere rather than on Linux only.

If this lands, 30-off_t.patch can be dropped from aports at the next version bump.

Separately, on whether this was already reported: I searched the SourceForge tracker via its REST API (/rest/p/gdcm/bugs/search) for musl, alpine and off64_t — no existing ticket. The one off64_t hit is #585, which is about the Testing/Data submodule and unrelated. So this PR is not duplicating anything filed there.

@hjmjohnson

Copy link
Copy Markdown
Contributor Author

@malaterre A few PR's for issues identified from ITK buildings.

@hjmjohnson

Copy link
Copy Markdown
Contributor Author

@malaterre Is there anything I can do to assist with the GDCM reviews?

@hjmjohnson

Copy link
Copy Markdown
Contributor Author

@malaterre Could the ITK team be of assistance in reviewing and updating GDCM? We have several upstream patches outstanding, and would like to avoid local vendering of separate code paths if possible.

gdcmFileStreamer.cxx used off64_t and hand-maintained a list of platforms
on which it must be #defined to off_t. musl-based distributions (Alpine)
were not on that list and fail with "off64_t has not been declared": musl
gates its LFS64 names on _LARGEFILE64_SOURCE and, unlike glibc, does not
imply that from _GNU_SOURCE. musl is also removing those aliases, so
enabling _LARGEFILE64_SOURCE is not a durable answer.

Extending the list is not a fix either. off64_t is a typedef, so the
preprocessor cannot test for it; every guard is a proxy for "which libc is
this", which is why the list grew one platform at a time. The glibc build
worked only because the C++ driver implicitly defines _GNU_SOURCE, which
turns on __USE_LARGEFILE64; the same block does not compile as C.

Separately, the _FILE_OFFSET_BITS 64 definition in this file was a no-op.
It sat after the gdcm*.h include block, which transitively pulls
<features.h>, so it was ignored. On 32-bit glibc off_t therefore stayed
32-bit and FSeeko/FTello/FTruncate silently narrowed, breaking writes past
2 GB.

Move the definition ahead of all includes and replace off64_t with a
GDCM-owned gdcm::offset_t: off_t on POSIX, int64_t on the Windows CRT.
This deletes the __APPLE__ || __FreeBSD__ || __OpenBSD__ || __NetBSD__ ||
__EMSCRIPTEN__ list without adding anything. A static_assert makes a
32-bit off_t a build error rather than silent truncation.

offset_t does not appear in gdcmFileStreamer.h, so the change is contained
to one translation unit and alters no public API.

Verified: Alpine/musl reproduces the reported error before the change and
compiles clean after; on 32-bit glibc the static_assert fires with the old
define placement and passes with the new one; macOS arm64 full build at
-Wall -Wextra with all six TestFileStreamer tests passing and no change to
the pre-existing failure set.

Reported downstream as InsightSoftwareConsortium/ITK#6779.

Assisted-by: Claude Code -- cross-platform instrumentation and patch drafting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant