Repository navigation
cpu: per-CPU topology facts for workload placement - #9
Conversation
|
@europaul @christoph-zededa any comments? |
|
@rucoder since you're already contributing directly to upstream jaypipes#479, maybe it makes sense to upstream the current PR first as well? (and then pull it into our fork) |
|
@rucoder are you planning to also report this new info as part of the hardware model to the controller or is it just for internal EVE use? |
europaul
left a comment
There was a problem hiding this comment.
Claude review (AI-assisted review, posted by @europaul). Findings from running the branch tests, a dry-run git am onto upstream jaypipes/ghw main, and a few probe tests against the new sysfsfixture. One real inconsistency, the rest are nits. Details inline.
Upstreamability: with the module path rewritten, commits 1–6 and 8 apply cleanly on upstream main and their tests pass there. Only commit 7 conflicts, in pkg/pci/pci_linux_test.go, because the fork's copy of that file dropped the jaypipes/pcidb import; the new test function itself is additive and its helpers exist upstream.
@europaul no, I do not have time to wait. i need these changes in EVE ASAP. the upstreaming should be easy since we now have a branch based on upstream release but hte change is big and probably I'll upstream it in pieces |
3cbd54f to
0394930
Compare
|
@christoph-zededa @europaul when #11 is merged i'll rebase this PR |
0394930 to
e5eaa04
Compare
e5eaa04 to
6060034
Compare
6060034 to
bc62bdb
Compare
europaul
left a comment
There was a problem hiding this comment.
Claude review, round 2 (AI-assisted, posted by @europaul). All five earlier comments are addressed in bc62bdb; I re-ran my probes and the suite on the new head and resolved those threads.
Two further topology bugs, found by a second-model (codex) pass and reproduced with sysfsfixture probes on bc62bdb. Details inline.
| // physical_package_id -1 from every CPU is the kernel saying it has no | ||
| // package information: one package. Next to CPUs with a real package it | ||
| // is unknown. | ||
| if allMinusOne { |
There was a problem hiding this comment.
[Claude review] The -1 → 0 package rewrite defeats cluster detection.
The sibling-set conflicts above are recorded under key{-1, core}, so clustered[-1] gets set. Then every procIDs[lpID] becomes 0, and the loop below looks up clustered[0], which is never true. Commit 2 and commit 5 cancel each other out on exactly the device-tree boards commit 2 targets.
Probe: 8 CPUs, every physical_package_id=-1, core_id=id%4, singleton thread_siblings_list:
pkg 0 core 0 lps [0 4]
pkg 0 core 1 lps [1 5]
pkg 0 core 2 lps [2 6]
pkg 0 core 3 lps [3 7]
TotalCores=4 TotalThreads=8
Expected 8 single-thread cores. Fix: normalize the package IDs before the sibling-set keys are built (or key the sets on the normalized ID). TestPackageIDMinusOne has no repeated core_id, so it does not catch this; a fixture combining the two would.
| if first, ok := firstSibling[lpID]; ok { | ||
| coreIDs[lpID] = first | ||
| } else { | ||
| coreIDs[lpID] = -1 |
There was a problem hiding this comment.
[Claude review] Several unknown cores merge into one fake SMT core.
In a clustered package every CPU whose thread_siblings_list cannot be read gets core -1 here, and processorsGet then calls CoreByID(-1), which groups them all into one multi-thread ProcessorCore. CoresForNode keys on {proc, -1} and does the same. The LogicalProcessor doc says two -1 core IDs are not known siblings, but Processors reports them as siblings anyway.
Probe: package 0, CPUs 0 and 1 with core_id=0 and siblings 0 / 1 (triggers the cluster rule), CPUs 2 and 3 with core_id 1 and 2 and no sibling list:
pkg 0 core 0 lps [0]
pkg 0 core 1 lps [1]
pkg 0 core -1 lps [2 3]
TotalCores=3 TotalThreads=4
The base branch kept CPUs 2 and 3 apart as cores 1 and 2. Suggestion: never use -1 as a grouping key; give each unresolved CPU its own core (e.g. keep the kernel core_id when it is unique among the unresolved ones, or number by the CPU itself). TestClusteredPackageCPUWithoutSiblings covers only one unreadable CPU, so a second one is needed to pin this.
bc62bdb to
62969c4
Compare
| } | ||
| } | ||
| // Clusters are found per package, so only once the package IDs are final. | ||
| siblingSets := map[key]map[int]bool{} |
There was a problem hiding this comment.
| siblingSets := map[key]map[int]bool{} | |
| siblingSets := map[key]map[int]struct{} |
There was a problem hiding this comment.
It should be map[key]map[int]struct{}{}. Without the trailing {} it's a type, not a value, so the build broke. The two lines that fill the set needed changing too: map[int]struct{}{} and struct{}{}. Done in the commit that adds this code, and force-pushed as 5734fab.
f026545 to
5734fab
Compare
|
@europaul @christoph-zededa I'm adding snapshots for other types of nodes: with P/E cores and ARM. Will updated the PR soon |
02a7875 to
048ed15
Compare
sysfs lists sets of logical processors as "0-3,7": the online CPUs, the CPUs of a NUMA node, the CPUs sharing a cache, the SMT siblings of a core, the CPUs local to a PCI device. ParseCPUList reads that format once for all of them. An empty list, or "(null)" for a mask the kernel never set, is nil; anything malformed is an error rather than a guess. IDs of 65536 or more are an error too, so that a corrupt list, from a snapshot say, cannot make it allocate gigabytes. ReadCPUList reads such a file, and ReadInt a file holding one integer. Unlike SafeIntFromFile, ReadInt does not warn and tells a missing or malformed file from a value of -1, which the kernel itself uses for "unknown" in some topology files. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Cores were grouped by the kernel's core_id within a package. That ID is unique within a package on x86, but an ARM device tree numbers cores per cluster, so core_id repeats: on a part with two 4-core clusters, CPUs 0 and 4 were reported as two threads of one core, and the package as four dual-threaded cores instead of eight single-threaded ones. Where one core_id covers more than one set of SMT siblings in a package, as thread_siblings_list reports them, every core of that package is now numbered by the lowest logical processor among its siblings; a CPU of such a package whose siblings cannot be read gets core ID -1 rather than a core_id another core now uses. A core ID of -1 is unknown, so CPUs with it are never grouped into one core: each is a core of its own, which also stops CPUs whose core_id and siblings both cannot be read from being reported as one multi-threaded core. Where core_id is unique, it is kept, so x86 reports the same IDs as before. CoresForNode identifies cores the same way, and also by package: a NUMA node holding two packages no longer merges their cores that share a core_id. It decides whether a package is clustered from all its online CPUs, as Info does, not from the node's alone: where clusters are separate NUMA nodes, a node's core_ids need not repeat, yet its cores must carry the same IDs as in Info. The tests build their sysfs trees with a new internal sysfsfixture package, which later tests reuse. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A cache's identity was used to merge the per-CPU views of it, but never reported, so a consumer could not tell which L2 or L3 a CPU uses, or name a cache the way the kernel and resctrl do. Cache gains ID: the kernel's identifier of the instance, unique within its level and type, or -1 where the kernel does not number it. CachesForNode fills it in. Caches lists every cache instance of the online CPUs, read from /sys/devices/system/cpu rather than per NUMA node, with all the CPUs sharing each one. Instances without a kernel id are told apart by the CPUs sharing them, so two such caches are never merged, and one with an id is never merged with one without. A CPU whose online file says 0 is left out even if its cache directory is still there, and a cache index whose level does not fit Cache.Level is skipped with a warning rather than wrapped around. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
CachesForNode read a cache's size from the cache index numbered by the cache's level rather than from the cache itself: index 1, the L1 instruction cache, for every level-1 cache. On an i7-1270P the P-cores' 48K L1d was reported as 32K and the E-cores' 32K L1d as 64K. It now reads each cache index with readCacheIndex and tells instances apart with cacheKey, as Caches does, so the two cannot drift apart: an instance without a kernel id is keyed by its shared_cpu_list rather than its shared_cpu_map, the same CPUs either way. Its own level, ID, type, size and CPU map readers are gone. That also fixes a size the kernel does not report, which came out as a wrapped-around SizeBytes and is now 0, and an index whose level cannot be read, which is now skipped with a warning rather than reported as a level-255 cache. It also keyed caches with a kernel id and caches without one in the same namespace, so a cache with id 3 and one whose CPU map reads "3" were merged into one. The two keys now differ. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…quencies Info listed packages and the cores in them, but nothing about a logical processor on its own: not its NUMA node, its frequencies or the caches it uses, and nothing about CPUs that are present but offline. A consumer placing work on CPUs had to read sysfs again for all of that. Info gains LogicalProcessors, one entry per online logical processor with its package, core (the IDs of its Processor and ProcessorCore), NUMA node and base and maximum frequencies; OfflineLogicalProcessors; and Caches, every cache instance of the online CPUs (memory.Caches). An ID that cannot be read is -1, never a plausible 0: a CPU no NUMA node lists is on node -1, while on a kernel without NUMA every CPU is on node 0. A frequency the kernel does not report is 0. physical_package_id -1 from every CPU is the kernel saying it has no package information, which means one package: it is now reported as 0, for Processor.ID too, and clusters are looked for in that package, as on boards that report one. Next to CPUs with a real package, and for a package ID that cannot be read, it stays -1. Processors and their cores are otherwise unchanged, in content and order. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com> # Conflicts: # pkg/cpu/cpu_linux.go
Heterogeneous parts mix core classes: Intel P-cores and E-cores, ARM big.LITTLE. LogicalProcessor gains CoreType: performance, efficiency, uniform for a homogeneous part, or unknown. Intel hybrid parts list their cores under the cpu_core PMU (performance) and the cpu_atom and cpu_lowpower PMUs (efficiency). Elsewhere each CPU's cpu_capacity ranks it: the highest capacity is a performance core, any lower an efficiency core, and one capacity for all CPUs a homogeneous part. Before Linux 6.16 a homogeneous x86 part reports no cpu_capacity; its single core PMU, /sys/devices/cpu, tells it instead. When none of these tells, or only some CPUs report a capacity, the type is unknown rather than guessed. The ranking sees only online CPUs and ranks by capacity alone: on a part with three classes of core the middle one reads as efficiency. The doc comment says so. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A device's DMA and interrupts are local to one NUMA node, and placing the work that drives it on that node's CPUs avoids crossing the interconnect. Device gains LocalCPUs, read from the device's local_cpulist: the CPUs of its NUMA node, or every CPU where the platform gives it no NUMA affinity. Unlike Node it does not depend on ghw having found a NUMA topology. Nil when the kernel does not report it; a list that cannot be read or parsed is also nil, with a warning. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A snapshot lacked files the CPU discovery now reads, so a snapshot taken on a hybrid or big.LITTLE host reproduced neither its offline CPUs, its frequencies nor its core types. Capture each CPU's online state, its cpufreq base and maximum frequencies and its cpu_capacity, the cpus files of the cpu_core, cpu_atom and cpu_lowpower PMUs, and the core PMU /sys/devices/cpu. The NUMA nodes' cpulist and the PCI devices' local_cpulist were already captured. The test snapshots three fixture hosts (Intel hybrid with an offline CPU, big.LITTLE, homogeneous x86) and checks that ghw reads the same CPU topology from each snapshot as from the host. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
No snapshot showed the files the CPU topology reads on a hybrid part: the i7-1270P one was taken before snapshots captured the cpu_core and cpu_atom PMUs and the cpufreq files, so core types and frequencies were tested on fixture trees only. linux-amd64-intel-i5-1335UE is a 13th-gen Core i5-1335UE: two P-cores with two threads each (CPUs 0-3, cpu_core) and eight E-cores (CPUs 4-11, cpu_atom), each module of four E-cores sharing an L2. It is a full capture, taken with this branch's ghwc from / in EVE's debug container (which shares the host's /sys, /dev and PID and network namespaces; only /proc/self/mounts is the container's), so other packages can test against it too, including its two NICs. Snapshots hold no DMI files or MAC addresses; the only identifying values left, the NVMe namespace's eui and wwid and an app instance UUID in /proc/self/mounts, are replaced by fake ones. ghwc reads the same cpu, memory, topology, block, pci and net information from it as from the unscrubbed capture, and the same CPU topology as from the machine. Turbo is off on that machine, so each CPU's maximum frequency equals its base frequency. TestHybridPMUSnapshot checks the cores, core types, frequencies and the L1d and L2 caches on it; the golden test of Processors covers it too. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
No snapshot had several cache instances at one level without kernel IDs, or a cpu_capacity file: the s390x one has caches without IDs, but its only shared level spans every CPU, so merging distinct instances would go unnoticed, and the x86 ones come from kernels that report no capacity there. linux-arm64-nvidia-jetson-agx-orin is a Jetson AGX Orin on NVIDIA's 5.15 kernel. It reports each cluster of four Cortex-A78AE cores as a package, so core IDs repeat across packages but not within one and are kept; every CPU has capacity 1024, a homogeneous part; and no cache has a kernel ID, so the three L3s, one per cluster, are told apart by the CPUs sharing them. It is a full capture, taken from / in EVE's debug container with this branch's ghwc plus the snapshot fixes of jaypipes#480 and jaypipes#481, so it holds the board's network interfaces and the DeviceTree properties ghw reads (the CPU model comes from compatible). The NVMe namespace's IDs and two app UUIDs in /proc/self/mounts are replaced by fake ones; ghwc reads the same information from it as from the unscrubbed capture, and the same CPU information as from the board. TestARMClusterPackagesSnapshot checks the packages, CPU model, cores, core type, frequency and caches on it. Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
9dc16de to
58599f7
Compare
Description
EVE places workloads on CPUs (core pinning, SMT- and NUMA-aware placement) and needs the host's CPU topology for it: which logical processors share a core, a cache, a NUMA node or a device's locality, which are offline, and which are performance or efficiency cores. ghw reported packages and cores, but not per-CPU facts, and a few of the facts it did report were wrong on clustered (ARM) and hybrid (Intel P/E) parts. This series adds the missing facts and fixes those, so EVE reads CPU topology from ghw instead of its own sysfs parser.
Commits, each building and passing the tests on its own:
util.ParseCPUList,util.ReadCPUListandutil.ReadInt, shared by the rest of the series.core_idrepeats per cluster, so cores of different clusters were merged into one. Cores in such a package are now numbered by their lowest sibling; x86 core IDs are unchanged.CoresForNodegets the same fix.memory.Caches(ctx)lists every cache instance of the online CPUs with the CPUs sharing it;Cache.IDis the kernel's id or -1.CachesForNoderead every level-1 cache's size from index 1 (the L1i), so an i7-1270P's 48K P-core L1d showed as 32K. It now shares the reader withCaches.Info.LogicalProcessors,Info.OfflineLogicalProcessorsandInfo.Caches. An ID that cannot be read is -1, never a plausible 0.Processorsand their cores are unchanged in content and order; a golden test checks that on every x86 snapshot.LogicalProcessor.CoreType: performance, efficiency, uniform or unknown. It is read from thecpu_core,cpu_atomandcpu_lowpowerPMUs on Intel hybrid parts, and otherwise fromcpu_capacityor the single core PMU. It is never guessed.Device.LocalCPUsfromlocal_cpulist.Known limits, written in the doc comments: core types are ranked by capacity among online CPUs only, so on a part with three core classes the middle one reads as efficiency.
How to test and validate this PR
go test -skip TestGPU ./...(TestGPUneeds a DRM card on the host and fails without one on the base branch too).LocalCPUsper socket).GOOS=linux GOARCH={arm64,riscv64,s390x},GOOS=windowsandGOOS=darwinbuilds.