Repository navigation
Retire the Xen leftovers: kernel parameters, pillar paths, runx-initrd and docs - #6826
Conversation
|
@eriknordmark @rene @rucoder please let me know what you think. Is renaming the GRUB vars that contain |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6826 +/- ##
==========================================
+ Coverage 28.33% 28.64% +0.31%
==========================================
Files 537 548 +11
Lines 98110 99857 +1747
==========================================
+ Hits 27802 28608 +806
- Misses 68054 68800 +746
- Partials 2254 2449 +195 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@europaul i would keep dom0_ prefix, it stil makes sense. besides renaming user facing variables may break existent workflows |
eriknordmark
left a comment
There was a problem hiding this comment.
dom0_platform_tweaks is also documented and used.
How do you ensure that the dom0 grub settings which are documented and used in /config/grub.cfg will still work?
Does that handle using 'set' in addition to 'set_global'?
The safer thing would be to keep all of the names which are/might be used in /config/grub.cfg.
|
The repo's own README documents one, for the FR201 without eMMC ( With the PR, the board default Fielded devices use it too: in the command lines devices report in Could you add Eden still writes Smaller: the post-override check resolves a mismatch in favor of the old name, so a new override written with a bare |
|
Apart from the comments from @eriknordmark and @rucoder (which I agree), I think the documentation is missing at least a paragraph to explain the Xen was supported in the past and it was decommissioned in favor of kvm. If there is any action that should be taken from devices upgrading from older to newer versions, it must have a migration guide. Keep |
|
Claude finished @rene's task in 4m 10s —— View job Review completeI focused on correctness of the Xen→kernel/sys rename, since the compatibility-shimming in Two correctness findings, both posted inline on
One cross-cutting note on the On the design question raised in the thread (keeping the |
fc48449 to
7968dad
Compare
|
Thanks for the review guys, much appreciated! As per your comments I kept the |
|
/rerun red |
|
Could somebody kick off Eden please |
|
@europaul, you have a conflict to fix |
| set_global hv_console "console=com1" | ||
| set_global hv_platform_tweaks "$hv_platform_tweaks clocksource=pit" | ||
| set_global dom0_platform_tweaks "clocksource=tsc" | ||
| set_global dom0_console "console=ttyS0 console=hvc0" |
There was a problem hiding this comment.
I'm not sure if you need to remove hvc0, what about if EVE boot as a VM? It should be able to get console on the virtual console...
There was a problem hiding this comment.
ttyS0 should be enough for that. see #6826 (comment)
|
Claude finished @rene's task in 4m 20s —— View job Review completeI reviewed the current state of the branch (the One correctness finding (inline):
Answer to @rene's Everything else (docs,
|
| NODE_EXPORTER_TAG=$(linuxkit_tag pkg/node-exporter) | ||
| VECTOR_TAG=$(linuxkit_tag pkg/vector) | ||
| XENTOOLS_TAG=$(linuxkit_tag pkg/xen-tools) | ||
| QEMU_TAG=$(linuxkit_tag pkg/qemu) |
There was a problem hiding this comment.
Deleting pkg/xen-tools breaks tools/get-deps/main_test.go (TestNoneLinuxKitImage), which is not in this PR's diff but depends on the directory existing. The test feeds a synthetic Dockerfile containing FROM lfedge/eve-xen-tools:$XENTOOLS and asserts the filtered result is ["pkg/alpine", "pkg/xen-tools"]. But filterPkg() does os.Stat("pkg/xen-tools") and drops any package whose directory is absent (tools/get-deps/main.go:213). With pkg/xen-tools gone, the result is only ["pkg/alpine"], so the length check t.Fatalfs.
(Note: this contradicts the earlier bot review, which said these fixtures are "self-contained … and don't break" — they are not, because filterPkg stats the real filesystem.)
Fix: update the fixture in tools/get-deps/main_test.go to reference a package that still exists, e.g. swap the eve-xen-tools line for lfedge/eve-qemu:$QEMU and change expected to ["pkg/alpine", "pkg/qemu"].
make -C ./tools/get-deps (run in CI by check-docker-hashes-consistency.yml) only builds and won't catch this, but any go test ./... in that module will.
| set_global hv_platform_tweaks "$hv_platform_tweaks clocksource=pit" | ||
| set_global dom0_platform_tweaks "clocksource=tsc" | ||
| set_global dom0_console "console=ttyS0 console=hvc0" | ||
| set_global dom0_console "console=ttyS0" |
There was a problem hiding this comment.
Re @rene's question about EVE booting as a VM: removing console=hvc0 here is safe. This set_x86_64_qemu path (and the set_arm64 one) keeps console=ttyS0/console=tty0, which is the console EVE's own guest/VM setup actually provides — make run boots QEMU with -serial mon:stdio (Makefile:312), i.e. ttyS0/com1, and the cloud images (GCP etc.) use ttyS0 as well. hvc0 is a virtio-console / Xen-PV console device that EVE's QEMU invocation never wires up, so on KVM it was a dead console= entry (the kernel silently ignores a console that doesn't exist).
The only way this regresses is a deployment that runs EVE as a guest exposing only a virtio-console (virtio-serial+virtconsole) and no serial port — not a configuration EVE sets up or documents. hvc0 is correctly preserved on riscv64 (rootfs.cfg:374), where it is the SBI console.
7968dad to
f2d2c0c
Compare
f2d2c0c to
136982a
Compare
| but also applies for KVM, which means the "host" operating system) the following | ||
| arguments should be used: | ||
| As an example of using serial port and VGA as a console for the host operating | ||
| system (`dom0` is its Xen-era name, see [HYPERVISORS.md](HYPERVISORS.md#xen-historically)) |
There was a problem hiding this comment.
I don't think xen-historically covers the fact that we retain the xen'ish names for the grub options. We should have explicit text about that were we specify the grub option names and have this section refer to that.
There was a problem hiding this comment.
I moved the comment now to the section of BOOTING.md where we explain the GRUB params.
Or did you mean I should move it to pkg/grub/rootfs.cfg?
|
@europaul This PR conflicts with master and needs a rebase. Its last Go Tests and Fuzz Tests runs were cancelled at the 6h job limit while building |
dom0_vcpus_pin pinned the dom0 vCPUs under Xen; Linux ignores it. console=hvc0 was the Xen paravirtual console, so on every x86 and arm64 KVM boot it has only ever added a console that does not exist (the QEMU and iPXE entries, and the two i.MX8 boards that listed it). It stays on riscv64, where hvc0 is the SBI console and nothing to do with Xen. dom0_mem and dom0_max_vcpus are kept here: despite the name they size the eve cgroup on every hypervisor. Signed-off-by: Paul Gaiduk <paulg@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The last Xen left in the tree is vocabulary: domainmgr wrote every hypervisor's domain config under /run/domainmgr/xen as xen<N>.cfg, comments and docs still talked about xl, pciback and dom0, and the amd64 squashfs block-size rule and kernel-version.mk still listed an HV=xen flavor that the top-level Makefile no longer accepts. The config directory becomes /run/domainmgr/domains/dom<N>.cfg. It is tmpfs, recreated on every boot and read only by domainmgr, so nothing on a device sees the move. Everything else is comments, test strings and the domainmgr doc, whose PCI passthrough and domain config sections described the Xen implementation. Signed-off-by: Paul Gaiduk <paulg@zededa.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
runx-initrd is the initrd of the VMs that container apps run in. Its init script is one half of a protocol whose other half is pillar's containerd/oci.go: pillar serialises the app's OCI process spec into the cmdline, environment, ug and mountPoints files of the directory it shares into the VM at /mnt, and the initrd reads them back, mounts the volumes and execs the app. Nothing checks that the two sides agree; they work because they are changed together. Until now the initrd was built in pkg/xen-tools, which after the move of QEMU to its own package holds nothing else. Keeping it there means the format of those files can change in pillar while the initrd is versioned, built and reviewed elsewhere, under a name that says nothing about what it does. So the initrd sources move next to the code they talk to, and pillar's Dockerfile builds them as one more stage, installed in pillar's own rootfs. QEMU runs in a domain whose spec binds / to /hostfs, so the default ramdisk path becomes /hostfs/containers/services/pillar/rootfs/boot/runx-initrd, the same way the kernel is already taken from /hostfs/boot/kernel. The EVE-K boot image copies the file from the pillar image instead. pkg/xen-tools goes away entirely, the last package whose name carries Xen. The published lfedge/eve-xen-tools images stay: every stable branch still builds from them. The get-deps test fixture used lfedge/eve-xen-tools as its example of an in-tree package, and the test checks the package directory on disk. It now uses pkg/qemu. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Paul Gaiduk <paulg@zededa.com>
Xen support was removed in 17.6.0, but the documentation still described EVE as a two-hypervisor system: HYPERVISORS.md compared Xen and KVM device models, toolstacks and IOMMU handling, and linked a Xen spoofing patch that no longer exists; BOOTING.md, BUILD.md and several others called the host kernel "dom0" or named Xen as a reason for design choices; domainmgr.md kept a debugging tip about xencommons. The Xen-specific material goes. BOOTING.md, where the GRUB options are listed, now says which names carry the dom0 prefix and why they stay: they are user overrides in /config/grub.cfg, which a rootfs update does not replace, and Eden, evetest and support runbooks write them. HYPERVISORS.md gains a section that records when Xen was supported and which other vocabulary stays and refers to BOOTING.md for the GRUB names; pkg/dom0-ztools and the storage.dom0.* properties are kept for the same reason. The section also states that a device updating from a Xen-based release needs no manual step, which the legacy boot-path mapping in zedmanager and the unchanged override names guarantee. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Paul Gaiduk <paulg@zededa.com>
136982a to
f316c64
Compare
|
Please ignore evetest for now, I'm still fixing the workflows (e.g. #6970) |
|
Evetest selective skip for this PR (plumbing/Xen-cleanup only: dropping Xen-only kernel params, renaming internal domainmgr paths/comments, and relocating the runx-initrd build from Skipped as unrelated to the changed area:
Skipped as redundant duplicates:
Kept (directly or closely related): An engineer can comment |
Description
Follow-up to #6743, which moved QEMU into its own
pkg/qemuservice. Thisseries removes the Xen leftovers that outlived the Xen support itself: the
kernel parameters only Xen consumed, pillar's internal Xen paths and comments,
the last package whose name carries Xen (
pkg/xen-tools, which after #6743only built
runx-initrd), and the documentation that still described EVE asa two-hypervisor system.
Following the review, the
dom0_*GRUB variables and kernel parameters(
dom0_mem,dom0_max_vcpus,dom0_console,dom0_extra_args,dom0_platform_tweaks) keep their names. They are user overrides in/config/grub.cfg, which a rootfs update does not replace, and Eden, evetestand support runbooks write them. The docs now say why they stay.
One commit per logical change:
dom0_vcpus_pinpinned dom0 vCPUs under Xen and Linux ignores it.
console=hvc0was the Xenparavirtual console, so on every x86 and arm64 KVM boot it added a console
that does not exist. It stays on riscv64, where
hvc0is the SBI console.flags.
/run/domainmgr/xen/xen<N>.cfgbecomes/run/domainmgr/domains/dom<N>.cfg(tmpfs, read only by domainmgr); the restis comments, test strings, the domainmgr doc, and the stale
HV=xenentriesin
kernel-version.mkand the squashfs block-size rule.initrd's
init-initrdis one half of a protocol whose other half ispkg/pillar/containerd/oci.go(thecmdline,environment,ugandmountPointsfiles shared into the VM at/mnt). The sources move topkg/pillar/initrd/, pillar's Dockerfile builds them as one more stage, andthe file ships at
/boot/runx-initrdinside pillar's rootfs. QEMU runs in adomain whose spec binds
/to/hostfs, so the default ramdisk becomes/hostfs/containers/services/pillar/rootfs/boot/runx-initrd, the same waythe kernel already comes from
/hostfs/boot/kernel. The legacy/usr/lib/xen/boot/runx-initrda controller or checkpoint may still send ismapped to it by the table Upgrade QEMU to 11.1.2 and run it from its own pkg/qemu service #6743 added. The EVE-K
external-boot-imagecopiesthe initrd from the pillar image.
pkg/xen-toolsis deleted. Thetools/get-depstest fixture, which usedeve-xen-toolsas its in-treepackage example and checks the package directory on disk, now uses
pkg/qemu; that test is not run by CI, so the breakage only showed up inreview.
HYPERVISORS.mdloses the Xen device-model, toolstack and IOMMU material andthe dangling link to the Xen spoofing patch, and gains a "Xen, historically"
section: Xen was available up to 17.5.0 and removed in 17.6.0, which names
keep the
dom0prefix and why, and that a device updating from a Xen-basedrelease needs no manual step. The other docs stop calling the host kernel
"dom0" or naming Xen as a reason for design choices.
No new Docker Hub repository is needed. The published
lfedge/eve-xen-toolsimages stay, since every stable branch still builds from them.
Not in this PR, noted for follow-ups: the
dom0_mem/eve_mem/ctrd_memhard-limit regexes in
010-eve-cgroupcan match across arguments;pkg/dom0-ztoolskeeps its name;init-initrd'ssetsid agettyfor the shimconsole on
hvc1has never foundagetty(the kernel gives/initnoPATH, and musl's default path has no/sbin), which predates this series.PR dependencies
None. Builds on #6743, already merged.
How to test and validate this PR
Build and boot the image (
make pkgs live && make run-live), then on thedevice:
cat /proc/cmdline: containsdom0_mem=...anddom0_max_vcpus=...asbefore, and no
dom0_vcpus_pin,hvc0orxentoken (riscv64 keepsconsole=hvc0).set_global hv_dom0_mem_settings "dom0_mem=640M,max:800M dom0_max_vcpus=1"in
/config/grub.cfg, reboot, and check that the command line carries theoverride and the
evecgroup limits follow it.ls /containers/services/pillar/rootfs/boot/runx-initrdexists;/boot/runx-initrdand/containers/services/xen-toolsdo not.TEST_SMOKE/TEST_UAPP) and checkit reaches RUNNING. The domain's QEMU command line (
pson the device)shows
-initrd /hostfs/containers/services/pillar/rootfs/boot/runx-initrd.ls /run/domainmgr/domains/holds thedom<N>.cfgfiles of the runningdomains.
make HV=k pkgsbuildspkg/external-boot-image, and the imagecarries
/runx-initrd.Done on amd64 in QEMU: steps 1, 3 and 5, plus a hand-started shim VM (host
kernel, the initrd from pillar's rootfs, a 9p share with a busybox rootfs and
the
cmdline/environment/ugfiles pillar writes), which mounted theshare, ran the command through
chroot2and powered off cleanly. The pillarunit tests cover the legacy ramdisk path mapping (
TestSetBootDefaults),and
go test ./...intools/get-depspasses.Upgrade from an older release: the evetest
TestEVEUpgradeKVM case wasrun locally against this build. A device installed from 16.0.0-lts with a
container app running was upgraded to the PR image, rebooted once, came up on
the new version, and the app was RUNNING and reachable afterwards. The only
Xen-era value that can cross an update, an explicit
/usr/lib/xen/boot/...BootLoaderorRamdiskpath from the controller or the checkpoint, goesthrough the mapping #6743 added, whose
runx-initrdentry now points at thepillar path.
Changelog notes
The Xen-only
dom0_vcpus_pinandconsole=hvc0are no longer on the kernelcommand line of x86 and arm64 devices. No other user-facing change: the
dom0_*GRUB settings in/config/grub.cfgkeep their names.PR Backports
Checklist
And the last but not least:
check them.
🤖 Generated with Claude Code