Linux native-peer experiment — interest in a ggml/AVX CPU backend?

[Pointhairedboss]

Hi Nehanth — alongside the #22 work, I’ve been experimenting privately with Linux workers to understand the questions around #11.

I have a Linux WebGPU worker and a custom ggml CPU adapter that presents a WebGPU-compatible interface. The attached screenshot shows both participating alongside my PC’s GPU, using a Xeon machine with 256 GB of RAM. Worker startup is still scripted and rough.

This is an exploratory experiment, not a proposed implementation or acceptance test for #11. It demonstrates mixed workers generating together, but I haven’t established numerical equivalence or useful performance.

My longer-term interest is using the 512 GB of RAM across two Xeon boxes for larger models in another research project. I want to experiment with layer allocation and batching to see whether useful batch throughput is achievable.

Would a ggml/AVX CPU backend be of interest to Pooled, or would you prefer #11 to stay focused on headless peers using the existing WebGPU engine? If CPU support is interesting, should it be discussed separately, with an explicit engine interface rather than the WebGPU compatibility approach I used here?

I’m keeping the experimental code separate from the #22 extraction. For now, I’d appreciate your thoughts on the direction before proposing any contribution.

[Nehanth]

Hey, really cool experiment, sorry for the slow reply. Right now we're focused on GPUs, with system RAM coming in through MoE expert offload (on main, not released yet), so a CPU backend isn't on our plan yet.

Before we decide, could you share some numbers from your setup?

• tok/s for the room with and without the ggml CPU worker
• time per layer on the Xeon vs on your PC's GPU
• whether the output matches the GPU-only run (same tokens on a few greedy prompts)

If the numbers look good, we can do it properly: open an issue and we'll design a real engine interface for it instead of the WebGPU-compatible shim. Thanks for keeping it separate from #22.