klipper: keep klippy resident in RAM - #320
Conversation
The board has 117 MB of RAM and the zram init sets swappiness to 150, so the kernel prefers to page program memory out over dropping file cache. Measured on a Centauri Carbon: klippy sat with 35 MB of its 48 MB in swap while idle, taking major page faults every second, and a G-code upload pushed it further out (about 150 faults in one second, free memory down to 7 MB). Starting a print right after the upload, as Orca does, then hit "Unable to obtain 'trsync_state' response" on the first homing move and "Timer too close" on the main MCU. Add a small Kalico module, [mlock], that raises RLIMIT_MEMLOCK and calls mlockall(MCL_CURRENT|MCL_FUTURE) at startup, and enable it in machine.cfg. With it klippy stays at 46 MB resident and took zero page faults through the same upload; Moonraker takes the paging instead, which it does not mind. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Correction to the write-up above, in fairness to anyone reading it: the memory lock did not fix the print-start shutdown on this printer. The same file failed the same way again with klippy fully locked and zero page faults, so I have taken the lock back out of the printer for now. What the shutdown dump showed instead: the main MCU was mid-homing, being fed step batches every 50 ms, and the host stopped sending for about 220 ms starting at the exact moment AFC's print-start timer ran. That timer fires 5 s into every print and fetched the file's metadata from Moonraker with a blocking What stands from this PR is the measurement: klippy on this board sits 35 of 48 MB in swap and page-faults during uploads, and the lock removes that. It is latency hardening, not the fix for this shutdown, and with only about 10 MB left free after locking it may cost as much as it buys. I'll leave it to you whether it is worth keeping open in that light; happy to reword the description or close it. |
If this fix didn't help then I see no reason to keep the PR open |
3 participants
This is the "Timer too close" that several of us have hit at print start (see the comments on #312). I chased one on a Centauri Carbon with CANVAS today and it is not CPU load, it is swap latency.
What happened. Orca uploaded a 7.5 MB G-code file and started the print three seconds later. Moonraker was still parsing the file for metadata. Five seconds in, the first
G28 Yfailed withUnable to obtain 'trsync_state' responseand the main MCU shut down withTimer too close. Host load was 0.2, nothing was printing yet.Why. The board has 117 MB of RAM and the zram init sets
vm.swappinessto 150, so the kernel prefers to page program memory out over dropping file cache. Sampling/proconce a second on an idle printer: klippy had 35 MB of its 48 MB in swap and was taking a handful of major page faults every second. During an upload it took about 150 faults in one second, its resident size fell to 10 MB and free memory to 7 MB. That is the state klippy was in when the MCU needed its trsync reply within a few milliseconds.All processes together use only 40 MB resident, so this is not a shortage of RAM, it is the swap policy applied to the one process with hard deadlines.
Fix. A small Kalico module,
[mlock], that at startup raisesRLIMIT_MEMLOCK(the default is 8 MB) and callsmlockall(MCL_CURRENT | MCL_FUTURE | MCL_ONFAULT), falling back to a plainmlockallon kernels withoutMCL_ONFAULT. It logs what it locked once klippy is ready. Shipped as one more patch in the Kalico recipe, enabled with[mlock]inmachine.cfgnext to[danger_options].Measured with it. klippy 46 MB resident, 6 MB of never-touched pages left in swap. Through the same upload and metadata parse: zero page faults, resident size unchanged. Moonraker took the paging instead, which it does not mind. Upload-and-print from Orca then homed and started normally, and the printer went on through a three-hour, 85-change CANVAS print with a pause, a file swap and a cancel in the middle, no Timer too close, no page faults on klippy.
Trade-off: about 48 MB of the 117 MB is now pinned. Free memory sits around 8 to 13 MB with the camera streamer running, and Moonraker lives a bit more in swap. Nothing else changed:
swappinessstays at 150, since that was benchmarked for a reason, and this only takes klippy out of that decision.swappinessalone would only shift the odds; locking the process makes the timing independent of what Moonraker, the camera or a file upload are doing.