zram: retry the device reset instead of dying silently at boot

[kyleinoregon]

What I found

My Centauri Carbon was running with no swap at all, although cosmos sets up a 200 % zram swap at boot. free showed Swap: 0, and the kernel log had zram: Added device: zram0 followed by nothing: no capacity change, no "Adding … swap". Two reboots later it came up fine, a third time it failed again, so it is a race.

I added a trace to /etc/init.d/zram and caught the failing boot:

=== boot trace: uptime 18s args=start ===
ls: /dev/zram0: No such file or directory
ls: /sys/block/zram0: No such file or directory
+ set -e
+ start
+ FACTOR=200
+ '[' -f /etc/default/zram ]
+ true
+ factor=200
+ grep MemTotal /proc/meminfo
+ awk ' { print $2 } '
+ memtotal=117232
+ disksize=240091136
+ modprobe zram 'num_devices=1'
+ echo 'zram devices probed successfully'
zram devices probed successfully
+ echo 1
sh: write error: Device or resource busy

Right after modprobe zram, udev opens the new block device to probe it. While it is open, the write to /sys/block/zram0/reset fails with EBUSY, and because the script runs with set -e it just stops there, before mkswap and swapon. Nothing reports it.

Running the same script by hand a minute later works every time, which is why this is easy to miss.

Why it matters on this board

With 114 MB of RAM the machine idles at about 16 MB free. Without swap, an ordinary event like uploading a gcode file from the slicer while a print is starting (Moonraker spawns its metadata scanner) took free memory from 20 MB to 8 MB in a few seconds, Klipper stalled, and the toolhead MCU shut down with "Timer too close". With the zram swap active there is over 200 MB of headroom for exactly that. I suspect a fair number of "Timer too close" reports come from machines that lost this race at boot.

Change

• udevadm settle --timeout=5 after the modprobe when udevadm exists (it does on the image).
• Retry the reset for up to five seconds (20 tries, 250 ms apart) instead of failing on the first EBUSY.
• If it still cannot reset, say so on the console and return non-zero, rather than dying silently.

Everything after the reset is unchanged: same algorithm, size, tuning and priority.

Tested

On a Centauri Carbon running 26.08.0. Before the change, with the traced script, the failure reproduced on one boot in four (trace above). After the change, three reboots in a row came up with the 234 MB zram swap active; the trace shows udevadm settle returning in about 60 ms and the reset then succeeding on the first try, so on this machine the settle alone closes the window and the retry loop is the safety net for boards without udevadm or with slower udev.

[Nikhil-Gohil]

I can confirm I am facing a similar issue. Was facing a lot of timer too close issues, looked like a swap issue. Fixed and retried the print, so far so good.

[jamesturton]

As a bit of background, this script comes from the openembedded layer meta-openembedded/plain/meta-oe/recipes-extended/zram/zram/init, but it's included as an override in our layer so we can tune the zram based off benchmarking I did for best performance on our SoC.

I have seen this error before first hand, but it happens so infrequently it's been very hard for me to track down what the issue might be. I'm curious why your board seems to present this issue much more frequently. How did you identify that it was the reset that was causing the issue here?

[jamesturton]

As this script is being run during start up, it's not impossible that udev is having a lot of things to do. Is waiting here for everything to be ready the best option?

[jamesturton]

Why do we not know if udevadm is installed or not? We choose what goes in our image. If it's a runtime requirement it should be added to the recipe as such.

[kyleinoregon]

How it was found: I added set -x tracing to the init script with output to a file on /data and watched boots. On the failing ones the trace ends at + echo 1 with sh: write error: Device or resource busy, and dmesg for those boots has "zram: Added device" but never "detected capacity change" or "Adding swap". With set -e the script dies there silently. Before the change it failed on 2 of 4 traced boots on this machine; since the change, 0 of 15.

Why this board hits it more often is a guess: it has a CANVAS, so udev is enumerating an extra USB MCU and running the toolhead and canvas firmware services right around S05, which would widen the window in which zram0 is still open for probing. I have no way to prove that beyond the timing.

Both inline points taken: the settle is gone, and with it the udevadm dependency. The reset is simply retried every 250 ms for up to 5 s and the script gives up with a message after that, which is bounded and only waits on this one device.

[kyleinoregon]

Apologies, the rework commit here was made from a damaged checkout and ended up deleting most of the tree alongside the zram change. Rebuilt the branch from current main as a single commit touching only the init script; the change itself is unchanged from what you reviewed.

[jamesturton]

LGTM