Only write layer files whose content actually changed - #169
Draft
viktordick wants to merge 2 commits into
Draft
viktordick wants to merge 2 commits into
viktordick wants to merge 2 commits into
Conversation
GNU tar before 1.35 (e.g. Ubuntu 22.04) tries to remove the extraction
root itself if the tarball contains "." as a member, which fails with
tar: .: Cannot unlink: Invalid argument
and makes the whole layer-init/layer-update run fail. Clear the target
directory in Python instead. This is equivalent: --recursive-unlink
emptied the same hierarchy, and tar rewrites every member on extraction
in any case, so no additional disk load.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Unpacking a layer source rewrote every file in the tarball, even if only a handful of them changed. With a layer of several hundred MB that is a lot of needless disk traffic, and since the timestamps are not taken from the archive, git had to rehash every single file afterwards. Replace the call to tar by helpers.unpack_tar, which compares each member to the file that is already there and only writes it if it differs. Superfluous elements in the target are removed afterwards, so the result is the same as before. Each member is expected to occur only once in the archive, but no specific order is required - anything that is in the way of a member is cleared when that member is unpacked, including parent elements that are not directories. As before, timestamps from the archive are not applied. Permissions now are applied exactly, which is what the sources intend: they set 2775 on directories and 664 on files so the workdir becomes a group repository. tar dropped the setgid bit (it only restores it with -p) and masked the group write permission with the umask, so this only worked by accident of the umask, if at all. The permissions of the extraction root itself, which the archives carry as ".", are applied as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Builds on top of #168 |
viktordick
marked this pull request as draft
August 4, 2026 09:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unpacking a layer source rewrote every file in the tarball, even if only
a handful of them changed. With a layer of several hundred MB that is a
lot of needless disk traffic, and since the timestamps are not taken from
the archive, git had to rehash every single file afterwards.
Replace the call to tar by helpers.unpack_tar, which compares each member
to the file that is already there and only writes it if it differs.
Superfluous elements in the target are removed afterwards, so the result
is the same as before. Each member is expected to occur only once in the
archive, but no specific order is required - anything that is in the way
of a member is cleared when that member is unpacked, including parent
elements that are not directories.
As before, timestamps from the archive are not applied. Permissions now
are applied exactly, which is what the sources intend: they set 2775 on
directories and 664 on files so the workdir becomes a group repository.
tar dropped the setgid bit (it only restores it with -p) and masked the
group write permission with the umask, so this only worked by accident of
the umask, if at all. The permissions of the extraction root itself, which
the archives carry as ".", are applied as well.