Build kernels from upstream tarballs + an IGLOO patch series, via Nix/kernelsmith - #59
Merged
Conversation
…eries Replaces the Docker + submodule build with a Nix flake sourcing the kernel as a pristine kernel.org tarball plus an explicit IGLOO patch series. Why: - Toolchains were unversioned `wget https://musl.cc/*-cross.tgz` downloads, so shipped kernels had no recorded compiler identity. kernelsmith resolves the (kernel version, arch) -> toolchain mapping from pinned Bootlin SDKs instead. - .gitmodules named the WRONG branches: it declared main_6.7 for linux/6.13 while the pin was the head of main_6.13 -- trees 89,775 commits apart. A `git submodule update --remote` would have silently regressed the kernel. Basing on a release tag makes that class of drift unrepresentable. - Taking the kernel as a derivation input is what makes MODVERSIONS CRC skew between vmlinux and igloo.ko unrepresentable rather than merely detectable. Series, verified by scripts/verify-series.sh against the branches they replace: - 4.10: 38 patches on v4.10; matches new_4.10 except four upstream export-ignore'd paths absent from every release tarball (.gitattributes, .get_maintainer.ignore, two arch/sh linker scripts) -- none build-affecting. - 6.13: 33 patches on v6.13; tree a1e70bc2 is byte-identical to main_6.13. The four non-IGLOO commits main_6.13 carried are all upstream: it forked immediately before the final x86_urgent merge and cherry-picked them back. Only 5 of the 23 same-subject patches are byte-identical across versions -- the new-file additions -- so core/ holds 5 and the rest are per-version. Shared files were never shared patches; hook-site code differs because its context does. configs/ is unchanged: the cpp -P -undef assembly runs inside the derivation, so no linuxManualConfig migration is needed. 18 of 20 cells build; loongarch64 and riscv64 await kernelsmith arch support and are listed as `missingArches` rather than silently dropped.
nixpkgs' multiple-outputs setup hook relocates include/ to `outputDev`, which falls back to "out" when no output is literally named "dev". Calling it "devel" therefore moved the kernel headers into $out and left a kernel-devel tree that looked complete in a directory listing but could not build a module. Found by building igloo.ko against the output rather than inspecting it; the dev closure goes 34M -> 91M once include/ actually lands there.
Kernel 4.10 validates its O= output directory with `cd $dir && /bin/pwd`, and
there is no /bin/pwd in the sandbox, so the build died as:
Makefile:141: *** failed to create output directory "/build/build"
which points nowhere near the cause. The Docker build only worked because the
Ubuntu image happens to ship coreutils at /bin. Also patchShebangs the kbuild
helpers, whose #!/usr/bin/awk-style shebangs otherwise fail "not found" and
cascade into Kconfig syntax errors.
Both fixes live in the builder, NOT the patch series: they are environment
adaptations rather than IGLOO changes, and the series has to remain a faithful
description of the fork branch for verify-series.sh to mean anything.
4.10/armel now builds end to end on kernelsmith's k4 band (Bootlin gcc 9.3.0):
240M vmlinux + 7.1M zImage + Module.symvers, ARM EABI5. This was the plan's
biggest unretired risk; the compiler era was never the problem.
"The series applies cleanly to the pinned upstream tarball" is the permanent invariant: without it enforced, someone edits a tree, forgets to export, and the committed series quietly stops describing reality. Comparing against the fork branch is a MIGRATION check -- once main_4.10 and main_6.13 are retired there is nothing to compare against -- so it is now an optional 3rd/4th argument rather than required. The workflow has NOT been exercised on the self-hosted runners; treat the runner and Cachix wiring as a first draft. It does push to Cachix from PRs, not only releases, since PR builds never populating the cache is why cross-toolchain closures get rebuilt from source elsewhere in this stack.
…ed vmlinux
Adds the two analysis artifacts shipped alongside every kernel, and the
release seam that reassembles the tarballs penguin already consumes.
The packaging was the easy half; the finding is provenance. Both extractors
are unpinned fetches of a default branch at image-build time:
* extract_kernelinfo -- wget'd from refs/heads/main of panda-re/panda-ng,
a repo this stack is retiring in favour of qemu/.
* dwarf2json -- `git clone --depth 1` of the default branch of
rehosting/dwarf2json, a FORK. nixpkgs ships UPSTREAM volatility's tool
under the same attribute name, so reaching for pkgs.dwarf2json would
silently produce different ISFs -- and penguin loads them at runtime
(pyplugins/apis/kffi.py reads cosi.<arch>.json.xz). The fork is packaged
with buildGoModule from a pinned rev instead.
Every osi.config and COSI ISF shipped to date was produced by whatever those
two branches said on image-build day.
kernel.nix gains a third output, `vmlinux`, holding the UNSTRIPPED image.
_in_container_build.sh extracts osi/cosi from the build-tree vmlinux and
strips the shipped copy afterwards; a derivation cannot reproduce that
ordering, so stripping in place left the analysis derivations reading a
vmlinux with no debug info on exactly the mips*/powerpc* targets that ship
one. $out now carries a vmlinux only where it is the deliverable, matching
build.sh more closely than before. Named `vmlinux` and not `debug`, which
nixpkgs' multiple-outputs machinery treats specially -- the same trap class
as the earlier dev/devel mistake.
VALIDATED against the profile shipped in rehosting/penguin:latest, for
6.13/armel: 61 lines both, identical key set, and every value identical
except two:
- task.per_cpu_offsets_addr, off by 8 bytes
- name, which embeds the kernel build timestamp; ours is '@0' because the
build sets KBUILD_BUILD_TIMESTAMP for reproducibility. Self-consistent,
since the kernel and its profile come from the same derivation.
Every other struct offset and address matches exactly, across a completely
different toolchain.
Known contract gap: perf.<target> is not reproduced. build.sh swallows its
failure, so the shipped tarball already contains whatever compiled that day;
no consumer was found in penguin. See nix/release.nix.
…toolchain
6.13/powerpc64le did not build: CONFIG_COMPAT (a defconfig default we never
chose -- configs/6.13/powerpc64le sets only CONFIG_CPU_LITTLE_ENDIAN) makes
kbuild compile a 32-bit vDSO, and the build died with
cc1: error: '-m32' not supported in this configuration
The cause is structural. get_cc compiles EVERY powerpc* target with a single
powerpc64 big-endian toolchain -- bitness and endianness come from the
kernel's own arch Makefile via Kconfig, not from the triple. kernelsmith
models the four variants as four independent arches, and the per-variant
Bootlin toolchains are not equivalent:
powerpc64 (BE, power8): -m32 OK -m64 OK -mlittle/-mbig OK
powerpc64le (LE, power8): -m32 FAIL
Production confirms the arrangement: the shipped ppc64le kernel in
rehosting/penguin:latest reports 'powerpc64-linux-musl-gcc (GCC) 11.2.1' --
a big-endian-triple compiler built the little-endian kernel. So this is a
fidelity fix, not a workaround.
Verified the obvious hazard -- a BE-default compiler emitting a big-endian
vDSO into an LE kernel -- does NOT occur: -mlittle-endian comes from
KBUILD_CPPFLAGS/cflags for CONFIG_CPU_LITTLE_ENDIAN and vdso32's
CC32FLAGSREMOVE strips only -mcmodel=/-mabi=/-mcall-aixdesc, so -m32 lands on
top of it. The built artifacts are ELF32 little endian:
sigtramp32-32.o Class: ELF32 Data: little endian
vdso32.so.dbg Class: ELF32 Data: little endian
All four powerpc cells now build. Bonus: powerpcle no longer needs a
from-source musl-cross-make toolchain (Bootlin ships no powerpcle at all),
so the matrix got cheaper.
TODO(kernelsmith): upstream this as a kernel-specific resolver
(kernelToolchainFor), NOT as a change to toolchainFor -- userland musl for
powerpc64le should still use the powerpc64le triple.
rehosting/kernelsmith#2 is merged, so the branch no longer needs --override-input to resolve a toolchain for those two arches. All 20 cells evaluate from the pinned input alone.
The shell build ran perf with `|| echo Warning` and copied it behind a
`[ -f ]` guard, so an arch whose perf did not compile shipped without one --
rehosting/penguin:latest carries perf for 3 of 13 targets as a result. perf.nix
already fails loudly instead; these are the cells that were failing, plus one
that was silently wrong.
4.10/x86_64 perf resolves its tools headers with
-I tools/arch/$(ARCH)/include/uapi, using ARCH verbatim
rather than the SRCARCH kbuild derives from it, and
tools/arch/x86_64/ does not exist. Reported as a missing
uapi/asm/mman.h, which names the wrong file entirely.
4.10/mips64{eb,el}
the musl ld defaults to the n32 emulation while the objects
are n64. LD= on the make command line does not fix it:
tools/perf/Makefile does `unexport MAKEFLAGS`, so
command-line variables never reach the nested tools/lib/*
builds -- which is exactly where the link fails. A PATH shim
survives that. Not LDEMULATION, which the HOST ld building
fixdep would also pick up.
6.13/loongarch64 the only glibc target. perf's cross-compile feature
detection fails, so it defines its own gettid and
pthread_attr_setaffinity_np, colliding with glibc's real
ones. Also needs the WRAPPED cross cc, for the sysroot, and
glibc's separate static output. Its dlfilters cannot link at
all -- binutils 2.41 segfaults on a .so against a
static-only glibc -- so drop dlfilters everywhere: they are
dlopen plugins, dead weight in a -static perf, and never
shipped.
The silently-wrong one: perf.nix had inherited kernel.nix's powerpc family
collapse, which routes all four variants through a single biarch powerpc64 BE
toolchain. Correct for a kernel -- freestanding, and arch/powerpc/Makefile
takes bitness and endianness from Kconfig -- and wrong for perf, which has no
Kconfig and links -static against a target libc. It produced four
byte-identical big-endian 64-bit binaries, three of which cannot run on their
guest. Every cell built, so nothing complained.
20/20 cells now build, and the powerpc four are 32-bit BE, 32-bit LE, 64-bit BE
and 64-bit LE respectively.
…cell outputs Slice 3. Two things: 1. Per-cell perf/osi/cosi/driver outputs. Without them a broken perf could only be reached through the whole release tarball, which rebuilds every cell to show you one compiler error. 2. nix/driver.nix builds igloo.ko for each cell from THIS FLAKE's kernel derivation, through kernelsmith's new buildModule, with no tarball round trip. That is the acceptance test for the kernel `dev` output: a build tree that cannot build the one module we care about is broken, and this repo is where that should be caught. The CRC property is the point. _in_container_build.sh extracts kernel-devel-all.tar.gz to a scratch directory and builds against whatever is there, so nothing structurally prevents pairing a module with a different kernel than the one it is inserted into. Here the kernel derivation is an input, so a mismatched pair is not representable. igloo_driver is added as a source-only input for this. That is not a claim about which repo should own the driver build long-term -- under the provider/consumer split both linux_builder and igloo_driver are consumers of kernelsmith, and igloo_driver owning its own build is the better end state.
Fixing the powerpc family collapse exposed a target that is not real.
arch/powerpc/platforms/Kconfig.cputype:
config CPU_LITTLE_ENDIAN
depends on PPC_BOOK3S_64
A 32-bit little-endian powerpc kernel cannot be expressed in mainline Linux.
configs/6.13/powerpcle sets CONFIG_CPU_LITTLE_ENDIAN=y, olddefconfig silently
drops it, and the built vmlinux is byte-identical to powerpc's (verified by
sha256 -- same hash, both 32-bit MSB). configs/4.10/powerpcle is already
retired as .unused.
So powerpcle's kernel is big-endian, and building perf 'as the target name
implies' would ship a little-endian userspace binary that cannot run on it.
Point perf at the powerpc toolchain instead, so the pair is at least
self-consistent.
The larger question -- whether this cell should exist at all, given it costs a
kernel, a perf, an osi/cosi extraction and a driver build per release to
duplicate 'powerpc' under a name that claims otherwise -- is for draft 34, not
for this commit.
The push step was guarded by `if: env.CACHIX_AUTH_TOKEN != ''` while the token was set in that same step's `env:` block. A step's own env is not in scope for its own `if:`, so the condition was always false and the cache was never populated -- the exact failure the workflow was written to prevent. Hoisted to job level, where the `if:` can see it. Also: - The gate now builds kernel + perf + module for its two cells, not just the kernel. perf and the module fail independently of the kernel, and both have done so recently. - A dispatch-only full-matrix job (20 cells + both release tarballs), with per-cell OK/FAIL lines so a single failure is attributable rather than buried. - Fixed the base.json path, which went through a pointless patches/<version>/../ indirection. Still unexercised on the self-hosted runners; the note at the top stands until a real run.
Two bugs shipped on this branch that every other check passed: perf emitted four byte-identical big-endian 64-bit binaries across the powerpc family, and configs/6.13/powerpcle asked for an endianness Kconfig cannot grant so the kernel came out big-endian. Both built cleanly and produced files of the right name in the right place. They are one failure class -- an ELF whose class or byte order disagrees with what its name promises -- and reading the ELF header catches it for nothing. Checks the unstripped `vmlinux` output rather than $out: only some targets deliver a vmlinux in $out (the rest ship zImage/bzImage/Image, compressed blobs with no readable class), so checking $out would silently skip armel, arm64, loongarch64, riscv64 and x86_64 -- five of thirteen, which is the same partial coverage the check exists to prevent. Missing artifacts now fail rather than being skipped. powerpcle is excluded, with the exclusion reported as SKIP rather than passing quietly: a 32-bit little-endian powerpc kernel is not expressible in mainline Linux, so there is no honest expectation to assert -- LSB would fail a build behaving as well as it can, MSB would bless a name that lies. Draft 34 carries the recommendation to retire the target.
arch/powerpc/platforms/Kconfig.cputype:
config CPU_LITTLE_ENDIAN
bool "Build little endian kernel"
depends on PPC_BOOK3S_64
PPC_BOOK3S_64 is 64-bit only, so on a 32-bit config the symbol is not
selectable. configs/6.13/powerpcle was '#include "powerpc"' plus
CONFIG_CPU_LITTLE_ENDIAN=y; olddefconfig dropped the line without a word and
built a big-endian kernel. vmlinux.powerpcle was verified BYTE-IDENTICAL
(sha256) to vmlinux.powerpc.
The cell cost a kernel, a static perf, an osi + cosi extraction and an igloo.ko
per release, to ship 'powerpc' twice under a name asserting the opposite -- and
had penguin ever selected it for genuinely little-endian firmware it would have
been handed a big-endian kernel silently. (powerpc LE hardware starts at
POWER8/ppc64le, so the likely device impact is zero, but that is luck.)
4.10 already retired this target. This brings 6.13 into line: 19 cells, not 20.
The .unused file keeps the reasoning and the original two lines, so nobody
rediscovers this from scratch.
Drops the powerpcle special case from perf.nix (toolchainArch is now the
identity for every cell in the matrix) and the exclusion from shape.nix, which
was only there because no honest expectation could be written for the target.
Also pins kernelsmith to the pushed buildmodule branch, so driver cells build
with no --override-input.
The branch pin was only ever a bridge while rehosting/kernelsmith#3 was in review. It merged rebased, so the commit hashes changed -- but the input's narHash did not, and all 19 driver cells resolve to the same store paths with no rebuild. Same content, permanent ref.
Every job in the previous draft died at "Set up job" with
Unable to resolve action rehosting/nix-setup, repository not found
before running a step -- the referenced action repo does not exist. It is now a
local composite action, version-locked to the workflow that uses it and unable
to 404, mirroring the install-nix-action setup that already works on
rehosting-arc in hyperfs.
Also sets the substituters in the runner's nix.conf rather than relying on
flake.nix's nixConfig, which is ignored for non-trusted users -- the failure
mode there is not an error but a silent full rebuild of every cross toolchain.
install-nix-action downloads the installer fine and then fails with
you do not have 'xz' installed, which I need to unpack the binary tarball
The rehosting-arc image has no xz. Guarded on command -v so it costs nothing on
a runner that already has it.
2729 lines -> 163. The real content is 109 insertions and 10 deletions; the
other ~2600 lines were an artifact.
Every context line in the patch carried a leading space that was never
stripped: it was generated by treating a unified diff's body as file content,
so the " " that marks a context line became part of the line. That is not
confined to the patch file -- it is in the tree we ship:
pristine 6.13 IGLOO (before)
# directives at column 0 120 9
# directives indented one space 0 119
struct x; forward decls indented 0 62
Indented preprocessor directives are legal C, which is why this shipped in
Aug 2025 and no compiler has complained since.
Repaired by reconstructing the header against pristine upstream: where a line
is equal modulo whitespace, upstream's formatting wins; IGLOO's genuinely new
lines are kept verbatim. The amended commit was then rebased under the
remaining 18 patches of the series, all of which applied clean.
Verified:
- `diff -w` old vs new syscalls.h is empty, and across the whole patched
tree syscalls.h is the ONLY file that differs.
- The committed series re-imports from scratch: 33 commits apply cleanly to
the pinned upstream tarball and yield the same header.
- kernel-6.13-armel rebuilt and compared section by section: .text, .rodata,
.data, .init.text and __ksymtab are byte-identical. The only differences
anywhere are .debug_line (-12 bytes), .debug_info, and the 20-byte GNU
build-id -- DWARF records column numbers, so de-indenting moves them, and
the build-id is a hash over the linked image. The raw boot image differs in
exactly those 20 contiguous build-id bytes out of 26.7 MB.
So: no change to the compiled kernel's behaviour, and the worst rebase hazard
in the series is gone. For comparison, 4.10's equivalent patch was already
clean at 141 lines; a scan of all 71 patches finds no other instance.
Note for anyone using the round-trip tooling: export-series.sh renumbers the
whole series (format-patch numbers 1..33, while the committed names come from
an older 37-commit export and have gaps where core/ patches sit), so running it
for a one-patch fix yields 33 renames. This change regenerated only 0019's body
and kept its original header. Worth making export stable against existing names.
The previous commit added a local composite action because `rehosting/nix-setup@v1` 404'd. The action does exist -- as `rehosting/ci/actions/nix-setup@v1`, alongside arc-registry-setup, pull-image and cleanup-pr-cache. Only the path was wrong, and nine workflows across penguin, fw2tar, qemu and vpnguin already consume it. The shared action is a superset of what was hand-rolled here: it installs the xz/curl the Arc pods lack (same fix, same reasoning), sets accept-flake-config, and exposes enable-kvm -- which matters on these runners, where install-nix-action's KVM/udevadm step fails on pods that expose /dev/kvm without a working udevd. Set to false; nothing here needs a VM. It also owns the Cachix wiring via cachix-action, so the manual `nix profile install nixpkgs#cachix` + `cachix push` steps are gone, and with them the job-level CACHIX_AUTH_TOKEN that only existed so a step-level `if:` could see it. One behaviour change, deliberate: nix-setup's skip-push default means PR runs PULL from rehosting-tools but do not push, with pushes to main populating it. That is the org convention (the rationale is fork PRs and pull-only tokens 403ing the post step) and it differs from what this workflow did before, which pushed from PRs too. Acceptable because the expensive closures -- the cross toolchains -- are already in the cache from the earlier run, so PRs now substitute them rather than rebuilding. Switches to secrets.CACHIX_REHOSTING, which is what the other nix workflows in the org use; CACHIX_AUTH_TOKEN also exists at org level but is not the convention.
nix-setup's Cachix step failed on both jobs with
$USER must be set. If running in a container, try setting USER=root.
The Arc pods run as a uid with no passwd entry, so $USER and $LOGNAME are
unset and HOME-relative cache paths are not writable. `cachix use` refuses to
run in that state.
rehosting/qemu's nix workflow already carries exactly this block; I dropped it
when adapting that workflow on the assumption it was incidental. It is a
precondition for running Nix on this runner pool.
TMPDIR and XDG_CACHE_HOME are created before Nix runs, since pointing at a
path that does not exist fails the same way slightly later.
Unrelated to this branch's contents -- infrastructure drift that happens to
show up as a red check on it.
setup-buildx-action's default buildkit image has moved onto a version carrying
the /proc/acpi masking regression, so build-image now dies with
can't mask dir "/proc/acpi": mount src=tmpfs, dst=/proc/acpi,
flags=MS_RDONLY, data=nr_blocks=1,nr_inodes=1: invalid argument
on the kernel-5.4 Arc runners. The workflow already knew not to pin
moby/buildkit:master for exactly this reason, but trusting the default is no
longer sufficient. main last ran 2026-07-28 and so has not hit it yet; it will
on its next run.
v0.30.0 is the same pin igloo-dev applies in its source-nightly publish for
this regression.
penguin's flake pins the release TARBALL (`inputs.kernels`, flake = false) and relies on Nix unpacking it. For a Nix consumer to take these kernels from this flake instead, going through `kernels-latest` would mean packing ~335 MB of archive purely so the consumer can unpack it again. `kernelsTree` is that payload as a plain directory -- `<version>/<artifacts>`, which is exactly the layout penguin lays down at /igloo_static/kernels/. It is exposed as the `kernels` package. No duplication of the assembly: kernelsTarball is now defined in terms of kernelsTree, so the two cannot drift. The README.txt and the reproducible-tar flags are unchanged, so `kernels-latest` is byte-identical to before.
Contributor
Author
|
Downstream consumer: rehosting/penguin#932 takes That PR is the concrete answer to "who consumes this and does the seam hold": It also surfaces what the switch changes for penguin's shipped image — |
Contributor
Author
|
Built Only in the shipped Docker tarball Only in the nix tarball Everything else is present in both — every kernel image, every To be precise about what this does and does not show: this is a manifest comparison, not a byte one. The file contents necessarily differ — the two are built by entirely different toolchains (unversioned |
Lets the nix path produce a real, downloadable release before it takes over the version line, so downstream consumers can be tested against an actual artifact rather than a branch. The tag prefix is deliberately NOT `dev_*`: build.yml already triggers on that, so a shared prefix would have both pipelines cut a release for the same tag and race version-increment for the same vX.Y.Z. `nixdev_*` does not match `dev_*` (the globs are anchored), so the two release paths stay independent while both exist. The tag itself is the version -- no version-increment here -- and the release is always a prerelease. The full 19-cell matrix now runs on such a tag as well as on dispatch, which has a second effect that matters downstream: a tag is a push event, so nix-setup's skip-push default does not apply and everything built lands in rehosting-tools. A consumer pinning this flake (rehosting/penguin#932) then substitutes all 19 kernels instead of cross-building them in its own CI. Also logs the tarball manifest and per-version perf coverage, so the difference from the Docker release is visible in the run without downloading anything.
lacraig2
added a commit
to rehosting/penguin
that referenced
this pull request
Aug 14, 2026
An immutable ref, so this PR's CI tests a fixed tree instead of whatever nix-patchset happens to point at when a job starts. The tag also runs linux_builder's full 19-cell matrix and, being a push event rather than a PR, pushes all of it to rehosting-tools. CI here therefore substitutes the kernels rather than cross-building 19 of them, which is what makes this testable at all in a normal CI budget. Repin to a release tag (or main) once rehosting/linux_builder#59 lands. Evaluates to the same igloo-static derivation as the branch pin did -- the tagged commit differs only in workflow YAML.
Contributor
Author
|
Cut One regression this surfaced, worth fixing before nix takes over the version line. find "$OUTDIR" -name '*.cmd' -delete
find "$OUTDIR" -name '*.o' ! -path '*/arch/powerpc/lib/crtsavres.o' \
! -path '*/scripts/*' ! -path '*/tools/*' -delete
find "$OUTDIR" -name '*.c' ! -path '*/scripts/*' ! -path '*/tools/*' -deleteAn Consumer is igloo_driver (its The fix belongs in kernelsmith's |
This was referenced Aug 14, 2026
Closed
Contributor
Author
nixdev_0.1.0 ships a 4.10/x86_64 kernel that does not bootFound while triaging rehosting/penguin#932, which switched to these kernels. It is not the config. The nix and Docker Booted both directly, no penguin involved: With Why x86_64 specifically
# ---- k4: gcc 9.3.0 (2020.08-1); x86_64 has no musl that old -> 2021.11-5 (gcc 10.3) ----
(t "x86_64" "k4" "2021.11-5" "sha256-KUNhf2U3...")x86_64 is the only arch in the k4 band that does not get the band's intended compiler. Every other 4.10 target builds with gcc 9.3.0 and boots; x86_64 falls forward to gcc 10.3.0 because Bootlin has no x86_64 musl toolchain that old. The Docker build used gcc 6.5.0 here. Confirmed by contrast — same nix tree, same run:
So this is 4.10 + x86_64 + gcc 10.3 only. 6.13/x86_64 is fine. Two things to fix
The fallback-forward is a reasonable compromise to have made — it is just not one that can be made without a boot test behind it. |
Contributor
Author
Correction: I over-attributed this to the gcc versionMy comment above concluded "it is the compiler," on the strength of same-config/different-compiler. I have since tested the compiler axis directly and that conclusion does not hold as stated. Built linux_builder's patched 4.10 source with the exact same
So the boundary is not between gcc 9 and gcc 10, and "route k4/x86_64 to the band's gcc 9.4 instead of Bootlin's 10.3" — my proposed fix — would very likely not have fixed it. Good thing to find out before pinning something and declaring victory. What still stands, unchanged and re-verified:
What is now open is which variable it is. Two candidates remain, and I am testing both:
Either way the second half of my earlier comment gets stronger, not weaker: nothing boots a kernel in this pipeline. kernelsmith's Will post the isolating result. |
nixdev_0.1.0 published a 4.10/x86_64 bzImage that does not boot. It
prints nothing -- not one kernel line. Every gate on this branch passed
it:
* the build succeeded
* shape-check passed (a bzImage's ELF class says nothing about whether
it runs; it is also the same 8.1 MB as the Docker build's)
* the series gate passed (it only proves the patches APPLY)
* kernelsmith's own boot.nix passed -- but its k4 cell boots 5.10.229,
and the k4 band spans 4.x AND 5.x, so 4.10 had never been booted by
anything, here or upstream
It was caught by a downstream consumer's integration tests
(rehosting/penguin#932), which is two repos and one release too late.
nix/boot.nix boots each shipped kernel headless under qemu with no
rootfs and asserts two things: the kernel banner appears, and it reached
the root-fs stage (a VFS/panic marker -- the whole pre-userspace path).
Verified against both outcomes before committing:
boot-6.13-armel ok
boot-4.10-x86_64 FAIL: no kernel banner -- it never started
The machine/console table is transcribed from penguin's
src/penguin/arch_registry.py, so each kernel is booted on the machine
penguin will actually run it on -- booting it on some other qemu model
is not the claim any consumer needs. 32-bit powerpc is unbootable by
declaration (arch_registry records qemu_machine=None for it); that is an
explicit SKIP entry rather than a silent omission, and an unknown target
is a hard error rather than a pass.
Wired into both CI jobs: the per-cell gate boots its cell, and the full
matrix boots all 19.
This does NOT fix 4.10/x86_64 -- it makes it impossible to ship. The
cause is still open; see the PR discussion. It is not the compiler
(reproduced with gcc 6.5/7.5/9.5/10.5) and the .config is byte-identical
to the Docker build's.
Contributor
Author
Second correction: it is not the source either, and my local harness was invalidI said above I was testing "a lower compiler boundary" vs "source divergence." Both lines are now closed, and neither hypothesis is supported. Posting the negative result because I put two wrong attributions on this PR already. Compiler: ruled outBuilt linux_builder's patched 4.10 source, same byte-identical
So my original "route k4/x86_64 off Bootlin's gcc 10.3 onto the band's 9.4" would not have fixed anything. Source: not shown, because the harness is brokenThe control that was supposed to isolate source — pristine upstream 4.10, zero IGLOO patches, same config, gcc 6.5 — is also dead, 285 bytes. That invalidates the harness, not the kernel. If my local native build cannot produce a booting 4.10/x86_64 from untouched upstream, it cannot discriminate anything, and every gcc result in the table above is void as evidence about the real build. (Disabling nixpkgs' cc-wrapper hardening did not change it; host binutils 2.41 being far too new for 4.10's x86 boot path is my next guess, but I am not going to assert a third cause I have not proven.) I should have validated the harness against a known-good build before drawing conclusions from it. That is the actual mistake here, and it is why this comment exists twice. What is actually establishedAll directly observed, same qemu invocation throughout:
With The cause is open. 4.10/x86_64 should be treated as broken in What I have landed (e707180)
Machine/console table transcribed from penguin's This does not fix 4.10/x86_64 — it makes it unshippable, which is the part that should not have been missing. One more gap worth fixing separatelyCI runs |
Wiring boot.nix up against the full matrix turned up three harness bugs.
All three reported a healthy kernel as dead, which is the failure mode a
boot gate can least afford -- it trains you to distrust the gate.
1. The image glob was not a glob. The comment claimed it globbed "rather
than tabulated" to avoid drift; the code hard-coded four names
(bzImage/zImage/Image/vmlinux). 6.13/loongarch64 ships
vmlinuz.efi.loongarch64 and failed as "no bootable image" while the
kernel was fine. Now ranks every *.<target> artifact, preferring the
arch's wrapped image over raw vmlinux -- the wrapped one is what
penguin boots -- and warns rather than fails on an unranked name.
2. loongarch64 needs two things nothing else here does, both of which
penguin already supplies (penguin_run.py:659):
* >1G RAM -- qemu's virt machine refuses to start below it
* EFI firmware -- kernel_fmt is vmlinuz.efi, a PE image, and the
built-in loader rejects it ("The image is not ELF")
Booting its ELF vmlinux instead would have passed this test while
testing an image penguin never runs.
The pinned nixpkgs (24.05, qemu 8.2.7) predates nixpkgs shipping
edk2-loongarch64-code.fd, so this adds a separate nixpkgs-qemu input
for the harness alone. Worth doing on its own terms: what the kernel
builds against and what we boot on should not be forced to move
together, and the latter should track what penguin runs.
3. Asserting on the "Linux version" banner alone was too strict. On
loongarch64 the EFI stub hands over after console setup, so the
earliest printks never reach the serial log -- a kernel that booted
to a root-fs panic was reported as never having started. Any
timestamped printk is the portable evidence; the genuinely dead
4.10/x86_64 image emitted none, so this still catches it.
Full matrix with kernelsmith#5: 18 boot, 1 declared SKIP (32-bit
powerpc, which arch_registry.py records as having no qemu machine).
4.10/x86_64 -- the cell that started all of this -- now reaches the
root-fs stage.
flake.lock pins kernelsmith ahead of main pending kernelsmith#5.
Contributor
Author
Root cause of 4.10/x86_64: binutils, not gccCorrecting my two earlier comments on this thread — the gcc attribution was wrong, and so was my subsequent claim that the native harness was invalid. That gcc sweep (6.5 / 7.5 / 9.5 / 10.5, all dead) was a well-controlled result ruling gcc out, and it left binutils as the one axis never varied — it sat at 2.41 throughout. binutils 2.31 began emitting Fix is in rehosting/kernelsmith#5 — resolve Isolation is clean: gcc held at 6.5.0 across both 4.10 runs, only the linker moved, and the kernel went from silent to reaching the root-fs stage. Full boot matrix now greenThree harness bugs this exposed (0cee4bf)Wiring
|
…landed Drops the temporary pin at the PR branch. Also picks up kernelsmith's $dev slimming (kernelsmith#4), which changes what an `M=` module build reads out of the kernel-devel tree -- igloo_driver is the consumer that exercises that, and its matrix is what confirms it. Boot matrix against kernelsmith main is byte-identical to the branch build (same /nix/store path): 18 ok, 1 declared SKIP.
The nix path builds every cell, boots every cell, and cuts the releases.
Nothing reads the Docker path any more, so it goes:
Dockerfile, .Dockerignore the embedded-toolchains image
build.sh, _in_container_build.sh the in-container build
.github/workflows/build.yml "Compile and release"
.github/workflows/clear_cache.yml its Harbor kernel-cache janitor
.gitmodules, linux/{4.10,6.13} the two fork branches patches/ replaced
The submodules were already dead and already wrong: .gitmodules declared
main_6.7 for linux/6.13, and the pin was a tree 89,775 commits from the
branch it named (see nix/source.nix, patches/README.md).
ONE capability came with build.sh rather than with Docker, so it is
ported rather than dropped: `--config-only`, which cpp-assembles a config,
runs savedefconfig, and diffs. nix/lint.nix does the same and is exposed
as `.#config-lint-<version>-<target>` and `.#config-lint`. It stays
ADVISORY -- the Docker version discarded its own exit status, because
savedefconfig prunes defaults and de-duplicates, so a readable #include-
structured config never equals its savedefconfig form. Asserting on it
would mean giving up the #include structure to satisfy the check.
Comments across nix/ still name _in_container_build.sh where they are a
faithful port of a specific step in it. Those are deliberate provenance
-- they record WHY a piece of the build looks the way it does -- and the
README now says so, and says where the file went.
The README described the submodule workflow and pointed at
panda-re/linux_builder for releases. Rewritten around the flake: quoting
cell names, where source comes from, the three checks (and why boot-check
is the one that matters), config linting, cutting a nixdev_* release, and
who consumes this repo. It also records the two things the Docker path did
silently: unversioned musl.cc toolchain downloads, so no kernel this repo
ever released has a recorded compiler identity; and the hand-built
x86_64-legacy toolchain pinned to binutils 2.30 whose unwritten reason was
the R_X86_64_PLT32 boundary.
Verified: `nix flake check` passes, and kernel-6.13-armel and boot-check
resolve to the same store paths as before this commit.
NOTE FOR SEQUENCING -- this removes the workflow that cuts non-prerelease
v* releases, which two Docker-path consumers still pull:
* penguin main pins linux_builder v3.6.5/kernels-latest.tar.gz
* igloo_driver's Docker CI pulls releases/latest/kernel-devel-all.tar.gz
Both keep working (existing releases stay downloadable) but neither can
receive a NEW kernel release after this lands. penguin#932 and
igloo_driver#100 are the nix-path replacements for those two consumers and
should merge first.
configs/<version>/<target> is a cpp fragment that #includes shared
pieces, so the file you edit is almost never the file an option comes
from -- and the config the kernel is BUILT with is a third thing again,
because olddefconfig runs afterwards and drops anything whose
dependencies are unmet. Three questions follow, one tool each.
.#config-required -- THE GATE. Asserts every cell ENDED UP with the
options IGLOO needs, reading the post-olddefconfig .config out of the
kernel's dev output rather than the fragments. That distinction is the
whole point: a fragment saying CONFIG_MODVERSIONS=y proves nothing about
what shipped. Seven options, each recorded with what breaks without it.
MODVERSIONS is the sharpest -- without it a mismatched igloo.ko loads
SILENTLY instead of being rejected, which is strictly worse than the
mismatch we spent this branch chasing. Reports every violation across
every cell before failing, rather than stopping at the first.
.#config-explain -- provenance. Resolves the #include chain and shows
where an option is set, which assignment wins, and via what path. It
exists because grep does not answer the question: CONFIG_IGLOO is set in
all-common.inc and no target sets it directly, so grepping a target
config finds nothing at all. Builds no kernel, so it stays instant.
.#config-redundant -- separates two things that look alike and are not:
an option assigned twice (always a bug, only the last has effect) and an
option that did not survive olddefconfig (usually an unnoticed
dependency). Advisory.
What they found on this tree:
* 595 duplicated assignments across the matrix, including
CONFIG_MODULES twice in 6.13/all-common.inc, 3 lines apart
* conflicting values where 6.13/powerpc re-declares =m over
all-common.inc's =y
* 2-68 options per cell that do not survive olddefconfig
All 19 cells DO satisfy the contract today; the gate is green.
Wired into CI's full-matrix job: config-required as a gate next to
boot-check (both exist for kernels that build and boot and then quietly
do not do their job), the redundancy report as continue-on-error so the
output lands in the log without failing the build.
The gate deliberately does not pipe python into tee: a pipeline's status
is its last command, so a failing check would be masked by a succeeding
tee unless pipefail is set. stdenv does set it -- but a gate that
silently passes when a shell option changes is the exact failure class
this file exists to catch.
Verified by negative test: adding a bogus required option fails all 19
cells with the per-cell reason, and removing it returns to green.
Removing build.yml took the release mechanism with it. nix.yml only ever
cut PRERELEASES from nixdev_* tags -- deliberately, so it could not race
the Docker pipeline for a version number -- so after that removal this
repo could not cut a vX.Y.Z release at all. Nothing downstream noticed
because existing releases stay downloadable, which is exactly why it
would have been found late.
Ported from build.yml into the full-matrix job:
* `full` now also runs on a push to main. With the Docker pipeline gone
it is the only thing that produces a release, so a merge has to build
the whole matrix, not the 2-cell gate.
* `reecetech/version-increment@2023.10.1` with `use_api: true`, then a
real (non-pre) release with both tarballs.
Prereleases stay as they were and stay separate: a nixdev_* tag IS its
own version, so a downstream repo can be handed a fixed, immutable kernel
set to test against without consuming a version number.
The nixdev_* prefix now outlives its original reason -- it existed to
avoid colliding with build.yml's dev_* trigger. Kept anyway: nixdev_0.1.0
and nixdev_0.1.1 are published and pinned by downstream lockfiles.
Comments that described build.yml as still existing are corrected rather
than deleted, since they explain why the prefix is what it is.
README documents both paths, and how to move the version LINE: push a
bare marker tag at main's tip and let the patch increment continue from
it. Explicitly NOT by setting `increment: minor`, which is not
self-clearing and silently bumps the release after it too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the two long-lived fork branches of
rehosting/linuxwith pristineupstream release tarballs + an IGLOO patch series, and replaces the Docker
build with a Nix one driven by kernelsmith.
Nothing downstream changes shape:
kernels-latest.tar.gzandkernel-devel-all.tar.gzcome out the same, so penguin needs no edits. Movingpenguin to consume this as a flake input is a separate, later change.
Why the fork branches had to go
The old
linux/6.13submodule pin was not a release. It was a tree based onv6.13~5with three upstream commits cherry-picked forward -- a base nobodywould choose deliberately, and one you cannot describe except by pointing at the
SHA. Those three commits are simply present in the
6.13tarball this branchbuilds from.
The IGLOO delta is now
patches/{4.10,6.13}/with quilt-styleseriesfiles (38and 33 patches, plus 5 shared
core/patches), andpatches/base.jsonnamingthe upstream tag and its hash.
scripts/verify-series.shproves a seriesreproduces its fork branch exactly, so the migration is checkable rather than
asserted.
Why the toolchains had to go
The Docker build got its cross compilers from unversioned
wget https://musl.cc/*-cross.tgzdownloads. Every kernel we have ever shippedtherefore has no recorded compiler identity -- there is no way to rebuild an
old release with the compiler that built it. kernelsmith resolves (kernel
version, arch) -> a pinned toolchain, so that identity is now in the lockfile.
What's built
19 cells: 7 targets on 4.10, 12 on 6.13. Each produces a kernel,
perf,igloo.ko, and osi/cosi analysis output, all individually addressable(
nix build .#packages.x86_64-linux."perf-6.13-mipsel") so a broken perf nolonger requires rebuilding the entire release tarball to see one compiler error.
igloo.kois built per cell via kernelsmith's newbuildModule(rehosting/kernelsmith#3). This is an acceptance test for the kernel
devoutput, not a claim about which repo should own the driver build: a build tree
that cannot compile the one out-of-tree module we care about is broken, and this
is where that should be caught.
Two silent-failure bugs found and fixed
Both had shipped. Both were invisible to every existing check because the build
succeeded and the artifact existed.
perf shipped one binary as four. It inherited kernel.nix's powerpc family
collapse and emitted byte-identical big-endian 64-bit binaries for
powerpc,powerpcle,powerpc64andpowerpc64le. Three of the four cannot run ontheir guest.
powerpclewas a phantom target. Its config asks forCONFIG_CPU_LITTLE_ENDIAN=y, whicharch/powerpc/platforms/Kconfig.cputypemakes
depends on PPC_BOOK3S_64.olddefconfigdrops it without a word andthe kernel comes out big-endian -- byte-identical to
powerpc's. A 32-bitlittle-endian powerpc kernel is not expressible in mainline Linux, so the
target is retired, with
configs/6.13/powerpcle.unusedrecording why andpreserving the original two lines.
Both are the same failure shape -- an ELF whose class or endianness disagrees
with its name -- so
nix/shape.nixnow asserts that for every artifact from thetarget name alone (deliberately not from the toolchain or the config; those are
what's being checked). It costs nothing and would have caught both.
A third:
0019-add-syscalls.h.patchwas corrupt, not merely churny2729 lines -> 163 (109 insertions, 10 deletions real). Every context line in
that patch carried a leading space that was never stripped -- it was generated
by treating a unified diff's body as file content. That is in the shipped tree,
not just the patch:
#directives at column 0#directives indented one spacestruct x;forward decls indentedIndented preprocessor directives are legal C, which is why this shipped in Aug
2025 without a compiler ever complaining. Repaired by reconstructing the header
against pristine upstream; the remaining 18 patches of the series rebased clean.
Verified:
diff -wis empty andsyscalls.his the only file that differstree-wide; the series re-imports from scratch (33 commits, clean); and a rebuilt
kernel-6.13-armelhas byte-identical.text,.rodata,.data,.init.textand__ksymtab. The only differences anywhere are.debug_line(-12 bytes),
.debug_infoand the 20-byte GNU build-id -- DWARF records columnnumbers, so de-indenting moves them. The raw boot image differs in exactly those
20 contiguous bytes out of 26.7 MB.
For scale: 4.10's equivalent patch was already clean at 141 lines, and a scan of
all 71 patches finds no other instance. Before the repair the two read as 141 vs
2729, which looked like "6.13 needs far more syscall machinery" and was purely
an artifact.
Verified
perf, 19/19igloo.ko,shape-checkgreen.nix build --rebuild, as is aspot-checked kernel cell.
verify-series.shreproduces both fork branches from their series.CI
Green on
rehosting-arc:series appliesfor both versions, andbuild {4.10,6.13}-armelcovering kernel + perf +igloo.koper cell.Nix install and Cachix wiring come from
rehosting/ci/actions/nix-setup@v1,the org's shared action, rather than anything defined here. Getting there
surfaced three things worth recording:
rehosting/nix-setup@v1, which does not exist.The real action lives at
rehosting/ci/actions/nix-setup@v1; nine workflowsacross penguin, fw2tar, qemu and vpnguin already use it.
cachix usefails with$USER must be set. TheUSER/LOGNAME/TMPDIR/XDG_CACHE_HOMEblock isa precondition for Nix on this pool, not decoration.
nix-setup'sskip-pushdefault, PR runs pull fromrehosting-toolsbut do not push; pushes tomainpopulate it. Thismatches the rest of the org. The expensive closures (the cross toolchains,
~250 MiB) are already cached, so PRs substitute rather than rebuild them.
One unrelated fix rides along:
ci: pin buildkit v0.30.0. The Dockerbuild-imagejob started failing withcan't mask dir "/proc/acpi"becausesetup-buildx-action's default buildkit drifted onto a version carrying that
regression.
mainlast ran 2026-07-28 and will hit the same thing on its nextrun.
Known gaps, stated plainly
kernelToolchainForinnix/kernel.nixstill carries aTODO(kernelsmith)--it belongs upstream in kernelsmith, not here.
Arc pool shares one
/nixper node, so paths built by an earlier job arealready in the local store. That will only be visible on a cold node.
here:
rehosting/ci'snix-release.yml(which is what the "Compile andrelease" job does by hand) and
update-flake-lock.yml(newly relevant --this branch gives the repo its first lockfile).