Skip to content

evetest: hand the linuxkit cache back to the user after a test - #7127

Open
milan-zededa wants to merge 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-linuxkit-cache-perms
Open

milan-zededa wants to merge 1 commit into
lf-edge:masterfrom
milan-zededa:evetest-linuxkit-cache-perms

Conversation

@milan-zededa

Copy link
Copy Markdown
Contributor

Description

Since #7089, make evetest mounts the host's ~/.linuxkit/cache into the
evetest container, which runs as root (to create the SDN tunnel, TAP devices,
bridges and dummy interfaces, and to use /dev/kvm and the Docker socket).
When TestVGAPassthroughNoHostAccess compiles its bpftrace tracer there
(only on a miss in the bpftrace-compiler cache, e.g. for a new EVE kernel),
bpftrace-compiler creates cache/blobs/sha256 and rewrites index.json as
root. The user's own linuxkit can then neither add blobs to the cache nor
update its index.

On CI this took a runner out of service: the next job's
rm -fr ~/.linuxkit in "Clean stale state from previous runs" failed with
Permission denied, so farfalle-runner-evetest-04 failed 42 suites within
seconds each across the last three evetest-nightly runs and one evetest-pr
run.

This PR:

  • chowns the mounted linuxkit and bpftrace-compiler caches back to the host
    user in evetest/entrypoint.sh, at the same points where the artifacts
    already get their ownership fixed (after go test exits and in the signal
    handler), through a new set_host_ownership wrapper;
  • removes ~/.linuxkit with sudo in the evetest-nightly and evetest-pr
    cleanup steps, as a backstop for a container killed before it could chown
    the cache (as is already done for the workspace and the artifacts).

How to test and validate this PR

  1. Start from an empty linuxkit cache and bpftrace-compiler cache, so that
    the tracer has to be compiled:

    sudo rm -rf ~/.linuxkit ~/.evetest/bpftrace-compiler-cache/*
  2. Run the test against a broker whose EVE VM has a vIOMMU and a VGA
    controller (otherwise the test skips before compiling):

    make evetest NAME=TestVGAPassthroughNoHostAccess

    The log must show could not find compiled script in cache, compiling now ....

  3. After the test exits, nothing in the caches must be owned by root:

    find ~/.linuxkit ~/.evetest/bpftrace-compiler-cache ! -user "$(id -un)"

    Without this PR, ~/.linuxkit/cache/blobs, blobs/sha256 and
    index.json are owned by root and rm -fr ~/.linuxkit fails with
    Permission denied.

Verified locally both ways (reproduced without the fix, clean with it).

Changelog notes

None (test framework and CI only).

PR Backports

Not to be backported as a separate PR. Instead, this commit should be added
to the open backports of #7089, which introduced the issue:

Checklist

  • I've provided a proper description
  • I've added the proper documentation (not needed, internal fix)
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device (the tracer is amd64-only)
  • I've written the test verification instructions
  • I've set the proper labels to this PR

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

🤖 Generated with Claude Code

make evetest mounts ~/.linuxkit/cache into the evetest container, which
runs as root (to create the SDN tunnel, TAP devices, bridges and dummy
interfaces, and to use /dev/kvm and the Docker socket). When
TestVGAPassthroughNoHostAccess compiles its bpftrace tracer there,
bpftrace-compiler creates cache/blobs/sha256 and rewrites index.json as
root. The user's linuxkit can then neither add blobs to the cache nor
update its index, and on CI the next job's "rm -fr ~/.linuxkit" fails
with "Permission denied", so the runner fails every suite it picks up
within seconds.

Chown the mounted linuxkit and bpftrace-compiler caches to the host
user whenever the entrypoint fixes the ownership of the artifacts. A
container killed before it gets to that still leaves root-owned files
behind, so CI also removes the cache with sudo, as it already does for
the workspace and the collected artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Milan Lenco <milan@zededa.com>
@milan-zededa milan-zededa added the testing New test or test framework changes label Nov 16, 2026
@github-actions
github-actions Bot requested a review from europaul November 16, 2026 11:02
@milan-zededa

Copy link
Copy Markdown
Contributor Author

@christoph-zededa Please review, it addresses permission errors that we got over the weekend on the nightly runs after the #7089 change, for example: https://github.com/lf-edge/eve/actions/runs/43766754157/job/132984959897

@rene

rene commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

@claude confirm the origin of the linuxkit's cache at /root and evaluate if execution under proper user isn't possible to fix the issue.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @rene's task in 1m 32s —— View job


Answer

1. Origin of the cache at /root

/root/.linuxkit/cache is the host user's own ~/.linuxkit/cache, bind‑mounted by make evetest:

# evetest/Makefile:168-172
BPFTRACE_COMPILER_MOUNTS := \
    -v .../eve-tools/bpftrace-compiler:/eve-tools/bpftrace-compiler:ro \
    -v $(HOME)/.linuxkit/cache:/root/.linuxkit/cache \
    -v $(EVETEST_HOME_DIR)/bpftrace-compiler-cache:/root/.bpftrace-compiler/cache

The mount target is /root/... simply because $HOME inside the container is /root (the container runs as root), and bpftrace-compiler invokes linuxkit, which writes its kernel‑image pull cache under $HOME/.linuxkit/cache. Because a bind mount shares inodes/uids with the host, blobs and the rewritten index.json created by the root process end up root‑owned on the host. Same mechanism for /root/.bpftrace-compiler/cache ← ~/.evetest/bpftrace-compiler-cache (Makefile:171). So this is not a cache that "lives at /root" — it is the user's cache temporarily exposed as root's $HOME.

2. Could "proper use" (running non‑root) fix it instead of chown‑ing back?

Not realistically, and the PR's approach is the right one. The evetest container is required to run as root for its core job, independent of bpftrace — see DOCKER_COMMON_ARGS (Makefile:226‑252) and entrypoint.sh:

  • --cap-add=NET_ADMIN + --device=/dev/net/tun — the harness/SDN creates the tunnel, TAP devices, bridges and dummy interfaces;
  • -v /var/run/docker.sock:/var/run/docker.sock — issues Docker builds/bind‑mounts (also used by bpftrace-compiler);
  • --device=/dev/kvm — QEMU/broker and the compiled tracer run under KVM.

Options for running as the host user, and why they don't apply here:

  • Run the whole container / go test as host uid (--user, or gosu in the entrypoint): loses the capabilities and the Docker‑socket//dev/kvm access the harness needs; entrypoint.sh even runs chown -R itself, which requires root.
  • Drop privileges only for the bpftrace-compiler sub‑process: it is spawned mid‑test by the root go test process and still needs the Docker socket and /dev/kvm; the host uid inside the container would not be in the docker group, so this just moves the permission problem.
  • User‑namespace remapping (userns-remap, container‑root → host‑user): a daemon‑wide Docker setting, not something a test target can assume, and it conflicts with the direct /dev/kvm, docker.sock and NET_ADMIN access this container relies on.

So the container must create these files as root, and the only clean way to hand them back is to chown them to EVETEST_HOST_UID:GID — which is exactly what the entrypoint already does for /artifacts. This PR generalizes that same, proven pattern to the two mounted caches (set_host_ownership = artifacts + caches) and adds a sudo rm backstop in CI for a container SIGKILL‑ed before it can chown. This is consistent and appropriate; I don't see a better "run‑as‑user" fix available in this setup.

· branch evetest-linuxkit-cache-perms

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing New test or test framework changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants