Repository navigation
Memory stats and freezer management with cgroupv2 #10251
Description
Nomad cgroup-v2 integration as it has some cgroupv1-isms. Cgroups-v2 changed the filesystem representation and changed the memory metrics that Nomad has relied on, so Nomad reports 0 memory summary metric across ~all drivers.
First, Nomad memory reporting relies on cgroup-v1 metrics. Nomad defaults to using RSS as the top line memory summary value to report, and reports Kernel Max Usage, Kernel Usage, Max Usage, RSS, none of which are reported in cgroupv2. You can view the libcontainer reporting difference by comparing cgroup v1 memory stats with cgroup v2. This is pretty confusing.
Also, the executor DestroyCgroup method uses libcontainer cgroup v1 . This needs to be updated to account for v2 and ideally select the relevant cgroup backend.
It's not clear what the state of cgroup-v2 adoption is. Seems like Fedora and ArchLinux. Other distros, like RHEL and Ubuntu, provide it as an option but the default one.
Sample metrics of cgroup v2
Running on Fedora 33, I see the following stats info:
= 1e2bdcc2-983d-1e0c-d226-95577bffc188
Eval ID = dae9b0ab-d31a-446b-a9df-5f2cbf37dc53
Name = memory.cache[0]
Node ID = f7bf24d9-d3c0-c34e-0b80-1c6a5de7eddf
Node Name = ip-172-31-74-56.ec2.internal
Job ID = memory
Job Version = 1
Client Status = running
Client Description = Tasks are running
Desired Status = run
Desired Description = <none>
Created = 2021-03-28T17:52:15-04:00
Modified = 2021-03-28T17:52:33-04:00
Deployment ID = ff079acc-8f67-41bb-dc67-e5c506e9a795
Deployment Health = healthy
Evaluated Nodes = 1
Filtered Nodes = 0
Exhausted Nodes = 0
Allocation Time = 88.646µs
Failures = 0
Task "redis" is "running"
Task Resources
CPU Memory Disk Addresses
2465/500 MHz 0 B/1000 MiB 300 MiB
Memory Stats
Cache Kernel Max Usage Kernel Usage Max Usage RSS Swap Usage
0 B 0 B 0 B 0 B 0 B 0 B 261 MiB
CPU Stats
Percent System Mode Throttled Periods Throttled Time User Mode
98.64% 0.00% 0 0 98.64%
Task Events:
Started At = 2021-03-28T21:52:22Z
Finished At = N/A
Total Restarts = 0
Last Restart = N/A
Recent Events:
Time Type Description
2021-03-28T17:52:22-04:00 Started Task started by client
2021-03-28T17:52:20-04:00 Task Setup Building Task Directory
2021-03-28T17:52:15-04:00 Received Task received by client
Placement Metrics
Node binpack job-anti-affinity node-affinity node-reschedule-penalty final score
f7bf24d9-d3c0-c34e-0b80-1c6a5de7eddf 0.635 0 0 0 0.635
Also, here is docker memory stats for cgroup v1 and v2
Cgroup v2
{
"usage": 2744320,
"stats": {
"active_anon": 1757184,
"active_file": 0,
"anon": 1622016,
"anon_thp": 0,
"file": 0,
"file_dirty": 0,
"file_mapped": 0,
"file_writeback": 0,
"inactive_anon": 0,
"inactive_file": 0,
"kernel_stack": 73728,
"pgactivate": 0,
"pgdeactivate": 0,
"pgfault": 3531,
"pglazyfree": 0,
"pglazyfreed": 0,
"pgmajfault": 0,
"pgrefill": 0,
"pgscan": 0,
"pgsteal": 0,
"shmem": 0,
"slab": 573440,
"slab_reclaimable": 0,
"slab_unreclaimable": 573440,
"sock": 0,
"thp_collapse_alloc": 0,
"thp_fault_alloc": 0,
"unevictable": 0,
"workingset_activate": 0,
"workingset_nodereclaim": 0,
"workingset_refault": 0
},
"limit": 2036068352
}Cgroup v1
{
"usage": 6778880,
"max_usage": 9478144,
"stats": {
"active_anon": 1622016,
"active_file": 2297856,
"cache": 4055040,
"dirty": 0,
"hierarchical_memory_limit": 9223372036854772000,
"hierarchical_memsw_limit": 0,
"inactive_anon": 0,
"inactive_file": 1757184,
"mapped_file": 2027520,
"pgfault": 5049,
"pgmajfault": 33,
"pgpgin": 5016,
"pgpgout": 3591,
"rss": 1626112,
"rss_huge": 0,
"total_active_anon": 1622016,
"total_active_file": 2297856,
"total_cache": 4055040,
"total_dirty": 0,
"total_inactive_anon": 0,
"total_inactive_file": 1757184,
"total_mapped_file": 2027520,
"total_pgfault": 5049,
"total_pgmajfault": 33,
"total_pgpgin": 5016,
"total_pgpgout": 3591,
"total_rss": 1626112,
"total_rss_huge": 0,
"total_unevictable": 0,
"total_writeback": 0,
"unevictable": 0,
"writeback": 0
},
"limit": 1026154496
}
Activity
mircea-c commentedon Aug 20, 2021
Just upgraded to Debian “bullseye” and they are using
cgroup-v2as the default which is now causing this issue. Running nomad version 1.1.3.MorphBonehunter commentedon Aug 23, 2021
Also Ubuntu will change to
cgroup-v2with the upcoming 21.10 release.Himura2la commentedon Sep 28, 2021
As a workaround, you can switch the kernel to "hybrid" cgroup hierarchy. It fixes the issue. For debian-based distros, use the following script:
Details:
keslerm commentedon Dec 1, 2021
The workaround works for this, but I noticed that the memory usage reporting in the nomad dashboard only represents the RSS memory, where as docker considers both the RSS and the Cache. This results in things that might look okay memory wise in nomad being OOM killed by the kernel and it not being obvious without looking at dmesg.
Running
docker statsshows the combined memory usagemircea-c commentedon Jan 17, 2022
Any update on this issue? There's a PR open for this (timdaman/check_docker#82).
aep commentedon Jan 20, 2022
is there a workaround to disable cgroups for everything except the pid list? the raw executor is currently nonfunctional due to #10551 unless you disable cgroups entirely. unfortunately it doesn't use sessions so you leak processes without cgroups
m1kc commentedon Mar 31, 2022
Any updates on this one?
tgross commentedon Mar 31, 2022
Hi @m1kc improved cgroups v2 support is planned to ship in Nomad 1.3.0. Much of the work that @shoenig has done has already landed in
main.shoenig commentedon Apr 5, 2022
Between #11289 and #12419 (shipping in Nomad 1.3) I think we're reporting what's available in cgroups v2
exec:
raw_exec:
github-actions commentedon Oct 9, 2022
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.