Skip to content

cpu: per-CPU topology facts for workload placement - #9

Merged
rucoder merged 10 commits into
eve/v0.26.0from
rucoder/cpu-topology
Oct 27, 2026
Merged

rucoder merged 10 commits into
eve/v0.26.0from
rucoder/cpu-topology

Conversation

@rucoder

@rucoder rucoder commented Oct 21, 2026

Copy link
Copy Markdown

Description

EVE places workloads on CPUs (core pinning, SMT- and NUMA-aware placement) and needs the host's CPU topology for it: which logical processors share a core, a cache, a NUMA node or a device's locality, which are offline, and which are performance or efficiency cores. ghw reported packages and cores, but not per-CPU facts, and a few of the facts it did report were wrong on clustered (ARM) and hybrid (Intel P/E) parts. This series adds the missing facts and fixes those, so EVE reads CPU topology from ghw instead of its own sysfs parser.

Commits, each building and passing the tests on its own:

  1. util: read the kernel's CPU-list format. util.ParseCPUList, util.ReadCPUList and util.ReadInt, shared by the rest of the series.
  2. cpu: tell apart cores that share a core_id across clusters. On clustered ARM packages core_id repeats per cluster, so cores of different clusters were merged into one. Cores in such a package are now numbered by their lowest sibling; x86 core IDs are unchanged. CoresForNode gets the same fix.
  3. memory: report every cache instance with its id. memory.Caches(ctx) lists every cache instance of the online CPUs with the CPUs sharing it; Cache.ID is the kernel's id or -1.
  4. memory: read CachesForNode's caches the way Caches does. CachesForNode read every level-1 cache's size from index 1 (the L1i), so an i7-1270P's 48K P-core L1d showed as 32K. It now shares the reader with Caches.
  5. cpu: report each logical processor's package, core, NUMA node and frequencies. Info.LogicalProcessors, Info.OfflineLogicalProcessors and Info.Caches. An ID that cannot be read is -1, never a plausible 0. Processors and their cores are unchanged in content and order; a golden test checks that on every x86 snapshot.
  6. cpu: report each logical processor's core type. LogicalProcessor.CoreType: performance, efficiency, uniform or unknown. It is read from the cpu_core, cpu_atom and cpu_lowpower PMUs on Intel hybrid parts, and otherwise from cpu_capacity or the single core PMU. It is never guessed.
  7. pci: report the CPUs local to each device. Device.LocalCPUs from local_cpulist.
  8. snapshot: capture the files CPU topology reads. Each CPU's online state, frequencies and capacity, and the core PMUs. A round-trip test checks that a snapshot reproduces the CPU topology of its host.

Known limits, written in the doc comments: core types are ranked by capacity among online CPUs only, so on a part with three core classes the middle one reads as efficiency.

How to test and validate this PR

  • go test -skip TestGPU ./... (TestGPU needs a DRM card on the host and fails without one on the base branch too).
  • New fixture-based tests cover clustered packages, hybrid PMUs, big.LITTLE capacities, offline CPUs, missing or mixed IDs and caches without ids. Snapshot tests use the i7-1270P (hybrid) and the dual-socket Xeon L5640 (LocalCPUs per socket).
  • GOOS=linux GOARCH={arm64,riscv64,s390x}, GOOS=windows and GOOS=darwin builds.

@rucoder

rucoder commented Oct 22, 2026

Copy link
Copy Markdown
Author

@europaul @christoph-zededa any comments?

@christoph-zededa

Copy link
Copy Markdown

@claude

@europaul

europaul commented Oct 23, 2026 •

Copy link
Copy Markdown

@rucoder since you're already contributing directly to upstream jaypipes#479, maybe it makes sense to upstream the current PR first as well? (and then pull it into our fork)
Claude tells me the upstream should be trivial.

@europaul

Copy link
Copy Markdown

@rucoder are you planning to also report this new info as part of the hardware model to the controller or is it just for internal EVE use?

@europaul europaul left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review (AI-assisted review, posted by @europaul). Findings from running the branch tests, a dry-run git am onto upstream jaypipes/ghw main, and a few probe tests against the new sysfsfixture. One real inconsistency, the rest are nits. Details inline.

Upstreamability: with the module path rewritten, commits 1–6 and 8 apply cleanly on upstream main and their tests pass there. Only commit 7 conflicts, in pkg/pci/pci_linux_test.go, because the fork's copy of that file dropped the jaypipes/pcidb import; the new test function itself is additive and its helpers exist upstream.

Comment thread pkg/cpu/cpu_linux.go Resolved
Comment thread pkg/cpu/cpu.go Resolved
Comment thread pkg/cpu/cpu.go Resolved
Comment thread pkg/util/util.go Resolved
Comment thread pkg/memory/memory_cache_linux.go Resolved
@rucoder

rucoder commented Oct 23, 2026 •

Copy link
Copy Markdown
Author

@rucoder since you're already contributing directly to upstream jaypipes#479, maybe it makes sense to upstream the current PR first as well? (and then pull it into our fork) Claude tells me the upstream should be trivial.

@europaul no, I do not have time to wait. i need these changes in EVE ASAP. the upstreaming should be easy since we now have a branch based on upstream release but hte change is big and probably I'll upstream it in pieces

@rucoder

rucoder commented Oct 23, 2026 •

Copy link
Copy Markdown
Author

@rucoder are you planning to also report this new info as part of the hardware model to the controller or is it just for internal EVE use?

@europaul yes, this is for HW info and for core pinning feature, so we use it in both places

@rucoder

rucoder commented Oct 23, 2026

Copy link
Copy Markdown
Author

@christoph-zededa @europaul when #11 is merged i'll rebase this PR

@rucoder
rucoder force-pushed the rucoder/cpu-topology branch from 0394930 to e5eaa04 Compare October 23, 2026 16:05
Comment thread pkg/memory/memory_cache_linux.go Fixed
Comment thread pkg/memory/memory_cache_linux.go Fixed
@rucoder
rucoder force-pushed the rucoder/cpu-topology branch from e5eaa04 to 6060034 Compare October 23, 2026 16:09
Comment thread pkg/memory/memory_cache_linux.go Fixed
@rucoder
rucoder force-pushed the rucoder/cpu-topology branch from 6060034 to bc62bdb Compare October 23, 2026 16:13
@rucoder
rucoder requested a review from europaul October 23, 2026 16:16

@europaul europaul left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude review, round 2 (AI-assisted, posted by @europaul). All five earlier comments are addressed in bc62bdb; I re-ran my probes and the suite on the new head and resolved those threads.

Two further topology bugs, found by a second-model (codex) pass and reproduced with sysfsfixture probes on bc62bdb. Details inline.

Comment thread pkg/cpu/cpu_linux.go
// physical_package_id -1 from every CPU is the kernel saying it has no
// package information: one package. Next to CPUs with a real package it
// is unknown.
if allMinusOne {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Claude review] The -1 → 0 package rewrite defeats cluster detection.

The sibling-set conflicts above are recorded under key{-1, core}, so clustered[-1] gets set. Then every procIDs[lpID] becomes 0, and the loop below looks up clustered[0], which is never true. Commit 2 and commit 5 cancel each other out on exactly the device-tree boards commit 2 targets.

Probe: 8 CPUs, every physical_package_id=-1, core_id=id%4, singleton thread_siblings_list:

pkg 0 core 0 lps [0 4]
pkg 0 core 1 lps [1 5]
pkg 0 core 2 lps [2 6]
pkg 0 core 3 lps [3 7]
TotalCores=4 TotalThreads=8

Expected 8 single-thread cores. Fix: normalize the package IDs before the sibling-set keys are built (or key the sets on the normalized ID). TestPackageIDMinusOne has no repeated core_id, so it does not catch this; a fixture combining the two would.

Comment thread pkg/cpu/cpu_linux.go
if first, ok := firstSibling[lpID]; ok {
coreIDs[lpID] = first
} else {
coreIDs[lpID] = -1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Claude review] Several unknown cores merge into one fake SMT core.

In a clustered package every CPU whose thread_siblings_list cannot be read gets core -1 here, and processorsGet then calls CoreByID(-1), which groups them all into one multi-thread ProcessorCore. CoresForNode keys on {proc, -1} and does the same. The LogicalProcessor doc says two -1 core IDs are not known siblings, but Processors reports them as siblings anyway.

Probe: package 0, CPUs 0 and 1 with core_id=0 and siblings 0 / 1 (triggers the cluster rule), CPUs 2 and 3 with core_id 1 and 2 and no sibling list:

pkg 0 core 0 lps [0]
pkg 0 core 1 lps [1]
pkg 0 core -1 lps [2 3]
TotalCores=3 TotalThreads=4

The base branch kept CPUs 2 and 3 apart as cores 1 and 2. Suggestion: never use -1 as a grouping key; give each unresolved CPU its own core (e.g. keep the kernel core_id when it is unique among the unresolved ones, or number by the CPU itself). TestClusteredPackageCPUWithoutSiblings covers only one unreadable CPU, so a second one is needed to pin this.

@rucoder
rucoder force-pushed the rucoder/cpu-topology branch from bc62bdb to 62969c4 Compare October 23, 2026 22:39
Comment thread internal/sysfsfixture/sysfsfixture.go Resolved
Comment thread pkg/cpu/cpu_linux.go Outdated
}
}
// Clusters are found per package, so only once the package IDs are final.
siblingSets := map[key]map[int]bool{}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
siblingSets := map[key]map[int]bool{}
siblingSets := map[key]map[int]struct{}

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be map[key]map[int]struct{}{}. Without the trailing {} it's a type, not a value, so the build broke. The two lines that fill the set needed changing too: map[int]struct{}{} and struct{}{}. Done in the commit that adds this code, and force-pushed as 5734fab.

@rucoder
rucoder force-pushed the rucoder/cpu-topology branch from f026545 to 5734fab Compare October 27, 2026 08:16
@rucoder

rucoder commented Oct 27, 2026

Copy link
Copy Markdown
Author

@europaul @christoph-zededa I'm adding snapshots for other types of nodes: with P/E cores and ARM. Will updated the PR soon

@rucoder
rucoder force-pushed the rucoder/cpu-topology branch 2 times, most recently from 02a7875 to 048ed15 Compare October 27, 2026 08:43
sysfs lists sets of logical processors as "0-3,7": the online CPUs, the
CPUs of a NUMA node, the CPUs sharing a cache, the SMT siblings of a
core, the CPUs local to a PCI device. ParseCPUList reads that format
once for all of them. An empty list, or "(null)" for a mask the kernel
never set, is nil; anything malformed is an error rather than a guess.
IDs of 65536 or more are an error too, so that a corrupt list, from a
snapshot say, cannot make it allocate gigabytes.

ReadCPUList reads such a file, and ReadInt a file holding one integer.
Unlike SafeIntFromFile, ReadInt does not warn and tells a missing or
malformed file from a value of -1, which the kernel itself uses for
"unknown" in some topology files.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
Cores were grouped by the kernel's core_id within a package. That ID is
unique within a package on x86, but an ARM device tree numbers cores per
cluster, so core_id repeats: on a part with two 4-core clusters, CPUs 0
and 4 were reported as two threads of one core, and the package as four
dual-threaded cores instead of eight single-threaded ones.

Where one core_id covers more than one set of SMT siblings in a package,
as thread_siblings_list reports them, every core of that package is now
numbered by the lowest logical processor among its siblings; a CPU of
such a package whose siblings cannot be read gets core ID -1 rather than
a core_id another core now uses. A core ID of -1 is unknown, so CPUs
with it are never grouped into one core: each is a core of its own,
which also stops CPUs whose core_id and siblings both cannot be read
from being reported as one multi-threaded core. Where core_id is unique,
it is kept, so x86 reports the same IDs as before.

CoresForNode identifies cores the same way, and also by package: a
NUMA node holding two packages no longer merges their cores that share
a core_id. It decides whether a package is clustered from all its
online CPUs, as Info does, not from the node's alone: where clusters
are separate NUMA nodes, a node's core_ids need not repeat, yet its
cores must carry the same IDs as in Info.

The tests build their sysfs trees with a new internal sysfsfixture
package, which later tests reuse.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A cache's identity was used to merge the per-CPU views of it, but never
reported, so a consumer could not tell which L2 or L3 a CPU uses, or name
a cache the way the kernel and resctrl do. Cache gains ID: the kernel's
identifier of the instance, unique within its level and type, or -1
where the kernel does not number it. CachesForNode fills it in.

Caches lists every cache instance of the online CPUs, read from
/sys/devices/system/cpu rather than per NUMA node, with all the CPUs
sharing each one. Instances without a kernel id are told apart by the
CPUs sharing them, so two such caches are never merged, and one with an
id is never merged with one without. A CPU whose online file says 0 is
left out even if its cache directory is still there, and a cache index
whose level does not fit Cache.Level is skipped with a warning rather
than wrapped around.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
CachesForNode read a cache's size from the cache index numbered by the
cache's level rather than from the cache itself: index 1, the L1
instruction cache, for every level-1 cache. On an i7-1270P the P-cores'
48K L1d was reported as 32K and the E-cores' 32K L1d as 64K.

It now reads each cache index with readCacheIndex and tells instances
apart with cacheKey, as Caches does, so the two cannot drift apart: an
instance without a kernel id is keyed by its shared_cpu_list rather than
its shared_cpu_map, the same CPUs either way. Its own level, ID, type,
size and CPU map readers are gone. That also fixes a size the kernel
does not report, which came out as a wrapped-around SizeBytes and is now
0, and an index whose level cannot be read, which is now skipped with a
warning rather than reported as a level-255 cache.

It also keyed caches with a kernel id and caches without one in the same
namespace, so a cache with id 3 and one whose CPU map reads "3" were
merged into one. The two keys now differ.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
…quencies

Info listed packages and the cores in them, but nothing about a logical
processor on its own: not its NUMA node, its frequencies or the caches it
uses, and nothing about CPUs that are present but offline. A consumer
placing work on CPUs had to read sysfs again for all of that.

Info gains LogicalProcessors, one entry per online logical processor with
its package, core (the IDs of its Processor and ProcessorCore), NUMA node
and base and maximum frequencies; OfflineLogicalProcessors; and Caches,
every cache instance of the online CPUs (memory.Caches). An ID that cannot
be read is -1, never a plausible 0: a CPU no NUMA node lists is on node -1,
while on a kernel without NUMA every CPU is on node 0. A frequency the
kernel does not report is 0.

physical_package_id -1 from every CPU is the kernel saying it has no
package information, which means one package: it is now reported as 0,
for Processor.ID too, and clusters are looked for in that package, as on
boards that report one. Next to CPUs with a real package, and for a
package ID that cannot be read, it stays -1. Processors and their cores
are otherwise unchanged, in content and order.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>

# Conflicts:
#	pkg/cpu/cpu_linux.go
Heterogeneous parts mix core classes: Intel P-cores and E-cores, ARM
big.LITTLE. LogicalProcessor gains CoreType: performance, efficiency,
uniform for a homogeneous part, or unknown.

Intel hybrid parts list their cores under the cpu_core PMU (performance)
and the cpu_atom and cpu_lowpower PMUs (efficiency). Elsewhere each
CPU's cpu_capacity ranks it: the highest capacity is a performance core,
any lower an efficiency core, and one capacity for all CPUs a
homogeneous part. Before Linux 6.16 a homogeneous x86 part reports no
cpu_capacity; its single core PMU, /sys/devices/cpu, tells it instead.
When none of these tells, or only some CPUs report a capacity, the type
is unknown rather than guessed.

The ranking sees only online CPUs and ranks by capacity alone: on a part
with three classes of core the middle one reads as efficiency. The doc
comment says so.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A device's DMA and interrupts are local to one NUMA node, and placing the
work that drives it on that node's CPUs avoids crossing the interconnect.
Device gains LocalCPUs, read from the device's local_cpulist: the CPUs of
its NUMA node, or every CPU where the platform gives it no NUMA affinity.
Unlike Node it does not depend on ghw having found a NUMA topology. Nil
when the kernel does not report it; a list that cannot be read or parsed
is also nil, with a warning.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
A snapshot lacked files the CPU discovery now reads, so a snapshot taken
on a hybrid or big.LITTLE host reproduced neither its offline CPUs, its
frequencies nor its core types. Capture each CPU's online state, its
cpufreq base and maximum frequencies and its cpu_capacity, the cpus
files of the cpu_core, cpu_atom and cpu_lowpower PMUs, and the core PMU
/sys/devices/cpu. The NUMA nodes' cpulist and the PCI devices'
local_cpulist were already captured.

The test snapshots three fixture hosts (Intel hybrid with an offline
CPU, big.LITTLE, homogeneous x86) and checks that ghw reads the same
CPU topology from each snapshot as from the host.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
No snapshot showed the files the CPU topology reads on a hybrid part:
the i7-1270P one was taken before snapshots captured the cpu_core and
cpu_atom PMUs and the cpufreq files, so core types and frequencies were
tested on fixture trees only.

linux-amd64-intel-i5-1335UE is a 13th-gen Core i5-1335UE: two P-cores
with two threads each (CPUs 0-3, cpu_core) and eight E-cores (CPUs 4-11,
cpu_atom), each module of four E-cores sharing an L2. It is a full
capture, taken with this branch's ghwc from / in EVE's debug container
(which shares the host's /sys, /dev and PID and network namespaces; only
/proc/self/mounts is the container's), so other packages can test
against it too, including its two NICs. Snapshots hold no DMI files or
MAC addresses; the only identifying values left, the NVMe namespace's
eui and wwid and an app instance UUID in /proc/self/mounts, are replaced
by fake ones. ghwc reads the same cpu, memory, topology, block, pci and
net information from it as from the unscrubbed capture, and the same CPU
topology as from the machine. Turbo is off on that machine, so each
CPU's maximum frequency equals its base frequency.

TestHybridPMUSnapshot checks the cores, core types, frequencies and the
L1d and L2 caches on it; the golden test of Processors covers it too.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
No snapshot had several cache instances at one level without kernel
IDs, or a cpu_capacity file: the s390x one has caches without IDs, but
its only shared level spans every CPU, so merging distinct instances
would go unnoticed, and the x86 ones come from kernels that report no
capacity there.

linux-arm64-nvidia-jetson-agx-orin is a Jetson AGX Orin on NVIDIA's 5.15
kernel. It reports each cluster of four Cortex-A78AE cores as a package,
so core IDs repeat across packages but not within one and are kept;
every CPU has capacity 1024, a homogeneous part; and no cache has a
kernel ID, so the three L3s, one per cluster, are told apart by the CPUs
sharing them. It is a full capture, taken from / in EVE's debug
container with this branch's ghwc plus the snapshot fixes of
jaypipes#480 and jaypipes#481, so it holds the board's network interfaces
and the DeviceTree properties ghw reads (the CPU model comes from
compatible). The NVMe namespace's IDs and two app UUIDs in
/proc/self/mounts are replaced by fake ones; ghwc reads the same
information from it as from the unscrubbed capture, and the same CPU
information as from the board.

TestARMClusterPackagesSnapshot checks the packages, CPU model, cores,
core type, frequency and caches on it.

Signed-off-by: Mikhail Malyshev <mike.malyshev@gmail.com>
@rucoder
rucoder force-pushed the rucoder/cpu-topology branch from 9dc16de to 58599f7 Compare October 27, 2026 12:04
@rucoder
rucoder merged commit e655b8f into eve/v0.26.0 Oct 27, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants