Skip to content

[amd64-v6.18.63-generic] build: prune and kill the linuxkit builder by its configured name - #341

Open
NopeCharon wants to merge 1 commit into
lf-edge:eve-kernel-amd64-v6.18.63-genericfrom
NopeCharon:prune-kernel-linuxkit-builder
Open

NopeCharon wants to merge 1 commit into
lf-edge:eve-kernel-amd64-v6.18.63-genericfrom
NopeCharon:prune-kernel-linuxkit-builder

Conversation

@NopeCharon

Copy link
Copy Markdown

Summary

Since the builder container was renamed to LINUXKIT_BUILDER_NAME (linuxkit-builder-kernel-$(USER), commit f6671e0), the CI Clean step no longer prunes the kernel's BuildKit cache, and make -f Makefile.eve clean always fails. This PR moves the prune into a Makefile.eve target that uses the configured name, and fixes clean.

Fixes #339

What was wrong

  • pr-build.yml / publish.yml Clean step: docker ps -f name=linuxkit-builder is a substring match, so it finds the renamed container. The step then runs docker exec linuxkit-builder ... against a container that usually does not exist. In publish run 43034763427 the core, rt and ai legs print Error response from daemon: No such container: linuxkit-builder three times each. The hwe leg's runner does have a container named exactly linuxkit-builder (31.66 GB), so the step pruned that one instead. Because of the | tail, the step passes in every case.
  • clean: docker kill $(BUILD_KIT_BUILDER) refers to a variable that does not exist, so it runs docker kill with no argument and fails.

Changes

  • New prune-builder target in Makefile.eve. It finds the container with an anchored name=^$(LINUXKIT_BUILDER_NAME)$ filter and runs the same du / prune --keep-storage / du sequence as before. BUILDKIT_KEEP_STORAGE_MB can still be overridden from the environment and still defaults to 32000. The target is listed in help.
  • Both workflows' Clean steps call make -f Makefile.eve prune-builder, so the builder name is defined only in Makefile.eve.
  • clean kills $(LINUXKIT_BUILDER_NAME). Like clean-image-%, it ignores the error when there is nothing to kill.

Validation

  • Real Docker (29.5): with containers named X and X-kernel-runner running, the old name=X filter matches both, and name=^X$ / name=^X-kernel-runner$ each match only their own container. With the builder absent, prune-builder prints the skip message and exits 0. The test containers were removed afterwards.
  • With a docker stub on PATH:
    • When linuxkit-builder-kernel-runner and linuxkit-builder are both running, prune-builder runs du, prune --keep-storage 32000 and du on linuxkit-builder-kernel-runner only.
    • When only linuxkit-builder is running, it skips.
    • BUILDKIT_KEEP_STORAGE_MB and LINUXKIT_BUILDER_NAME from the environment are honoured.
    • make clean now runs docker kill linuxkit-builder-kernel-runner and exits 0. On the base branch it fails with "docker kill" requires at least 1 argument.
  • Both workflow files still parse as YAML.

Note: the 31.66 GB linuxkit-builder container on the hwe runner of that run is no longer touched by CI. If nothing else on that host uses it, it can be removed by hand.

Commit f6671e0 ("Makefile.eve: Bump Linuxkit and name builder
container") made the kernel's linuxkit builder run as
LINUXKIT_BUILDER_NAME, linuxkit-builder-kernel-$(USER), so it does not
clash with an EVE build on the same host. Two places still use the old
name:

- The Clean step of pr-build.yml and publish.yml tests for the builder
  with `docker ps -f name=linuxkit-builder`, a substring match that the
  renamed container satisfies, and then runs `docker exec
  linuxkit-builder buildctl ...`. That container does not exist on most
  runners ("No such container: linuxkit-builder" on the core, rt and ai
  legs of run 43034763427), and the `| tail` keeps the step green. Where
  a container really is called linuxkit-builder, the step prunes that
  one instead (31.66 GB on the hwe leg of the same run). Either way the
  kernel's own cache is never trimmed to BUILDKIT_KEEP_STORAGE_MB and
  keeps growing on the self-hosted runners.

- `make -f Makefile.eve clean` ends with `docker kill
  $(BUILD_KIT_BUILDER)`. That variable is not defined, so the target
  always fails with '"docker kill" requires at least 1 argument'.

Move the prune into a new Makefile.eve target, prune-builder, which
looks the container up by LINUXKIT_BUILDER_NAME with an anchored
`name=^...$` filter, so it acts on that container and no other, and
keeps the existing du / prune --keep-storage / du sequence and the
BUILDKIT_KEEP_STORAGE_MB override with its 32000 MB default. Both
workflows call it, so the name is defined only in Makefile.eve. clean
now kills $(LINUXKIT_BUILDER_NAME) and, like clean-image-%, does not
fail when there is nothing to kill.

Fixes: lf-edge#339
Signed-off-by: NopeCharon <115794254+NopeCharon@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant