PocketLLM

English | 中文

Run a large language model on one accelerator — a single GPU, an edge board, the phone in your pocket. One process owns one device. If the checkpoint does not fit, it is quantized further — never offloaded, never split across cards.

What it runs

Qwen3, on four accelerator classes, through one engine. The C core (src/, libpocketllm.so) is where a model actually runs: it reads a GGUF, tokenizes with the checkpoint's own BPE, walks the graph and decodes — on f16 weights and on packed quantized ones, the width ladder's K-quant rung (q2_k through q6_k) and its IQ2 rung (iq2_xxs, iq2_xs), decoded inside the kernel and never widened.

Device	Path	Qwen3 sizes	Format	Accuracy vs the reference
CPU (AVX2)	C core	0.6B, 1.7B	f16, q4_k_m, q2_k/q3_k/q5_k, iq2_xxs/iq2_xs	token-identical to llama.cpp
1× NVIDIA (sm_75+)	C core, eager kernels	0.6B, 1.7B	f16, q4_k_m, q2_k/q3_k/q5_k, iq2_xxs/iq2_xs	token-identical to llama.cpp
RDK S600 (Horizon Nash BPU)	prebuilt .hbm via the xlm delegate	0.6B → 8B	leap_llm int4 (w4/w8)	token agreement (prefix); at the quantizer's floor
Orange Pi AIpro (Ascend 310B1)	C core, CANN backend	0.6B → 8B	q4_k_m	token agreement; at the quantizer's floor at every size

Two honest limits run through the table. The C-core rows are checked token-for-token against llama.cpp on both widths; the two boards expose no logits surface, so "aligned to 1e-3" is expressible on them only as token agreement — and their weight schemes differ from the reference's, so a cross-board agreement number is a quantizer-provenance comparison, not a hardware ranking. The S600 is not a C-core backend: it runs a prebuilt graph the host compiles ahead of time, so it is listed by the artifact it executes, not by an op it implements.

The C-core rows' format list is where the width ladder is real: q2_k, q3_k, q5_k, iq2_xxs and iq2_xs join q4_k and q6_k as formats the engine decodes, and each new one is checked twice — the decoder block for block against llama.cpp's own decode of the same bytes, and the engine's greedy walk against a fresh llama.cpp prefill at every step on a file of that format. The block-for-block comparison is what caught this tree's Python q3_k decoder reading ggml's six-bit scales as four, and it is what pins the two codebook formats, whose decode is a table lookup: a wrong stride there produces weights that look like weights. The five are correctness-complete rather than fast: q4_k's packed AVX2 and int8 activation kernels are written for its own group arithmetic, so the others are decoded one weight at a time for now — 5.3 s against 0.7 s for 26 tokens at 0.6B on this machine. Packing those kernels is the follow-up, and the ladder's IQ3, IQ4 and IQ1 rungs behind them.

Beyond Qwen3, the same C-core rows run a dense llama-architecture checkpoint: MiniCPM5-1B (1B, Q4_K_M, a 2026 model) is checked token for token against llama.cpp on cpu and on a card, twelve steps each. It differs from qwen3 in the metadata prefix, an absent per-head QK norm, an untied head, and its rotary embedding — the interleaved convention, which is why there are two rope kernels.

They also run qwen3vl, end to end: Qwen3-VL-2B's text half is checked token-for-token the same way (8/8 on cpu and on a card), and the whole image path composes — the tower, an image of any size (the reference's own size rule and its two resamplers: align-corners truncating bilinear for the pixels, and the antialiased half-pixel one it uses for the learned position table), an image span's per-section rotary positions and its causal attention, and the text after it. The composition is one call, not three: runtime/vlm_session.h owns the order so the tool and the ABI cannot disagree about it. Measured against llama.cpp on the same image:

Image	Target	Span tokens	Agreement
768×768	768×768	576	corr 0.9926 on cpu, 0.9942 on a card
512×512	512×512	256	corr 0.9915
800×600	800×608	475	corr 0.9883, same argmax
1024×1024	1024×1024	1024	corr 0.9940, same argmax

They also run qwen3a — Qwen3-ASR-1.7B, speech in and a transcript out. A log-mel front end (the reference's own: a periodic Hann window, a Slaney mel bank with no step factor, log10 against a 2^-24 floor, and Whisper's normalisation) feeds a convolutional tower whose three convolutions, per-chunk positional table and 24 pre-norm blocks are checked against libmtmd's. The tower reaches correlation 0.99983003 against the reference's own output, and four seconds of speech transcribe to language English<asr_text>I have a dream that one day — the reference's answer for the same clip. pocketllm_asr_generate is one call for a host: a recording in, a transcript out, decoding greedily from the cache the composition filled.

What the two multimodal paths are not is fast. Everything above is a correctness claim, and both were brought up on this machine at f16/q8_0 without a benchmark of their own: the only number taken is a 4-second transcription in 12.8 s on one RTX 2080 Ti -- which is since down to about a second, after the two CUDA kernels #747 and #748 rewrote (the tower reached 2.2 s for an image, and the audio tower went from 719 ms to 251 ms for the same reason). The width ladder does not reach them yet either — the vision tower's loader widens to f32, so the q2_k…iq2 rungs the text rows are checked on have no multimodal counterpart. Both paths also have an ABI entry point now (pocketllm_vlm_* and pocketllm_asr_*, ABI 1.3), which is how a host reaches them without C++.

Reference performance — Qwen3-1.7B, f16, on one RTX 2080 Ti

	prefill (pp512)	decode (tg128)
CUDA, before the f32 GEMM was blocked	37.9 t/s	10.8 t/s
CUDA, after (#747, #748)	433 t/s	10.6 t/s

Eleven times the prefill, from two kernels rewritten to use a tile's worth of reuse: a blocked f32 GEMM (#747) and a streaming-softmax attention (#748). The same two moved a 768x768 image through the vision tower from 31.6 s to 2.2 s.

What this table does not say, and used not to be honest about. It carried a q4_k_m-against-llama.cpp ratio for both cpu and cuda, and neither is measurable here any more: the llama.cpp build on this host reports (none) for --list-devices, so there is no CUDA reference to divide by, and the CPU numbers move by several percent with the machine's load (load average 21 while this was written -- 218.9 and 228.3 t/s on two runs of the same binary). A ratio taken under that is a fact about the load and not about the engine. What is left in the table is the one comparison that is stable and does not need llama.cpp: f16 on the card, before against after, on one machine at one time. The q4_k_m arm of the ladder is unaffected by either change -- it runs gemm_quant, not gemm -- and its ratio is owed a quiet machine and a CUDA llama.cpp.

The f16 numbers above also explain something the old table obscured: the engine's f32 GEMM was the slow term for every path, not just the multimodal ones. That is why a commit aimed at a vision tower's tower moved the text prefill an order of magnitude. Full per-device numbers, methods and provenance: the fleet support matrix.

The rule

If it does not fit, it is quantized — not offloaded, and not split across cards.

The width ladder is Q4 → Q2 → IQ2 → IQ1 → ternary, in that order, stopping at the lowest format the model still answers correctly in. Host offload and multi-card parallelism are out of scope by construction: the ABI has no rank, no collective, and no second device, and EngineArgs has no tensor_parallel_size to set.

Current status

What works, what is a stub, and what has not been written. This table is the honest one.

Piece	State
src/ — the C++ engine (libpocketllm.so)	Runs Qwen3-0.6B and 1.7B in f16 and the seven packed formats q2_k through q6_k plus iq2_xxs/iq2_xs on cpu and one cuda card — GGUF read, BPE tokenize, forward, greedy decode, and temperature/top-k/top-p/min-p/seed sampling; every one of them decodes inside the kernel. Greedy is checked token-for-token against llama.cpp on both widths, and each packed format against llama.cpp's own decode of the same file block for block. The sampler is checked against the numpy reference. The CUDA path is correctness-complete and, since #747, no longer slow for the reason this row used to give: its f32 GEMM and its non-causal attention were both rewritten to use a tile's worth of reuse, which took f16 prefill from 37.9 to 433 t/s on a 2080 Ti. The dp4a activation lever on the packed path was measured and closed separately. The same library runs the two multimodal paths — see the next row. Details: the C engine page
src/kernel/ascend/ — Ascend 310B backend	Runs Qwen3-0.6B … 8B on an Orange Pi AIpro, at the quantizer's own floor.
src/model/ — the multimodal paths	Runs qwen3vl and qwen3a end to end, both compared against libmtmd on the same inputs: a vision tower with any-size preprocessing in, an image span and the text around it out; and a log-mel front end, an audio tower, and a transcript. Correctness is measured (correlation 0.9926-0.9942 for images, 0.99983003 for the audio tower, and llama-cli's own transcript for speech). Neither is benchmarked and neither has the width ladder: the vision tower widens its weights to f32, so q2_k…iq2 have no multimodal counterpart. runtime/vlm_session.h and runtime/asr_session.h own the two compositions, so a tool and the ABI cannot disagree about the order. Details: the C engine page
pocketllm.cli	Done for all six commands — devices, backends, architectures, ops, run, serve
pocketllm.protocol / pocketllm.server — OpenAI-compatible HTTP	Done. Driven over the C core by server/native_backend.py; one request at a time, no batch, no cancellation
pocketllm.kernels — the kernel ABI: descriptors, op schemas, dispatch, graph IR	Done. 19 ops declared; stdlib-only, no numpy
pocketllm.backends.reference — numpy oracle, every op, host memory	Done. The normative implementation
pocketllm.quant — GGML block decoders, vendored tables	Done. IQ4_NL, IQ4_XS, IQ1_M, IQ2/IQ3, q2_k–q6_k, q8_0
pocketllm.loader.gguf — GGUF reader	Done. numpy in, descriptors out
pocketllm.engine — executor, planner, memory, session lifecycle	Done, on the reference backend
pocketllm.backends.cpu — host CPU	Stub. Selection and declaration only
pocketllm.backends.{cuda,mps,qnn,horizon,ascend}	Stubs. Each names the runtime it waits for
pocketllm.architectures — model IR and builders	Scaffold. toy only
pocketllm.tokenizer — GGUF-vocabulary BPE	Skeleton. Whitespace works; BPE raises

The Python package does not run a model. It is the host side — the kernel ABI as spec, the numpy oracle, the GGUF loader, the quantization decoders, the execution layer and the OpenAI-compatible HTTP surface — and no Python backend implements a kernel: every backend except reference is a declaration whose session raises BackendNotImplementedError. The two halves meet at exactly one place: python/pocketllm/native.py, the ctypes bridge, which both pocketllm run and pocketllm serve drive — each reaches the C core. Neither is a Python backend, which is why the pocketllm.backends.* rows above stay stubs.

Installation

The base install is one dependency. The ABI is stdlib-only, the reference backend is numpy, and a device runtime is an optional extra — which is what lets the same wheel install on a CUDA box and on a phone.

pip install pocketllm

# With a device runtime
pip install "pocketllm[cuda]"      # NVIDIA, via torch
pip install "pocketllm[mps]"       # Apple Silicon, via torch

Requirements: Python >= 3.10. Nothing else is required for the base install — no compiler, no CUDA toolkit. There is no ext_modules in this tree and no build step: the C core is a separate native library (src/, CMake) loaded at runtime through ctypes, not a Python extension.

For a working copy:

git clone https://github.com/lvyufeng/PocketLLM.git
cd PocketLLM
pip install -e ".[dev]"

Try it

The read-only commands work on any host, including one with no accelerator and no torch:

pocketllm devices        # what this host can open, and what the rest are missing
pocketllm backends       # what each backend declares, available or not
pocketllm architectures  # the model structures this tree can build
pocketllm ops --op gemm_quant --device cuda

devices is the command to run when something is wrong, so it is the one command that must not depend on anything being installed. Every check behind it is a filesystem probe, and it imports no runtime. Note that available and implemented are different questions — and that this command answers both about the Python backends. An available row still has no Python kernels, and nothing here describes src/: pocketllm devices does not know the C core exists.

run reaches the C engine, so it generates — greedily by default, and only for an architecture the C core implements (Qwen3). It needs libpocketllm.so built; without it the command says so rather than pretending:

cmake -B build -S src && cmake --build build -j8
pocketllm run --model /path/to/qwen3-0.6b-f16.gguf --prompt "The capital of France is" --max-tokens 8

The capital of France is Paris, and the capital of Italy is Rome

run is greedy by default and samples when --temperature is above zero, with --top-k, --top-p, --min-p and --seed beside it. The draw comes from the host — random.Random(seed) — because the engine takes a uniform variate and holds no RNG, so the same seed reproduces the same text. src/tools/run.cpp is the same thing a level lower: it links the engine's objects directly and also takes --tokens, --device, and --print-top.

serve puts the same engine behind OpenAI-compatible HTTP, over the same checkpoint:

pocketllm serve --model ckpt.gguf --host 0.0.0.0 --port 8000
curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user", "content": "What is the capital of France?"}]}'

The prompt goes through the checkpoint's own chat template, read out of the GGUF's tokenizer.chat_template and rendered with Jinja — which needs jinja2, deliberately not a dependency of this package, so a chat request without it falls back to a plain rendering the model was not trained on. Sampling is per request ("temperature", "top_p", "top_k", "min_p", "seed"), as is "stop" and "n".

It serves one request at a time. A C Session holds one position and one KV cache with no locking, while the server is a thread-per-request ThreadingHTTPServer, so the adapter serializes behind a lock and declares supports_batch = False rather than promising overlap the engine cannot provide. Measured with six concurrent requests: 6/6 correct with the lock, 0/6 without it — and no error either way, which is why the lock is not an optimisation to revisit. Cancellation is refused for the same kind of reason. Fields with no op behind them — logprobs, the penalties, logit_bias, response_format, echo, suffix, best_of — are refused with a 400 naming the parameter rather than silently ignored.

Library

import pocketllm

print(pocketllm.__version__)   # 0.2.0.dev0

import pocketllm pulls in neither numpy nor torch — it declares the kernel vocabulary and nothing else, and LLM / AsyncLLM resolve lazily. A test enforces that on a fresh interpreter (tests/test_package_boundaries.py).

Why another inference engine

Because the constraint is different. Most runtimes scale a checkpoint across cards; PocketLLM's job is the opposite one — make one card, or one phone, sufficient. That single constraint changes the design rather than tuning it:

• A portable kernel ABI, not a framework binding. pocketllm.kernels is descriptors and declarations with no dependency at all — no numpy, no torch. A backend implements it; the engine drives it. That is what makes a Qualcomm NPU and a CUDA card the same kind of object, and what lets the same C core sit behind a CUDA card, an Ascend NPU and a CPU unchanged.
• The design is a ladder, not a flag. The width ladder is the fit strategy: a checkpoint that does not fit is quantized until it does. There is no offload path to take and no second device to reach for.
• Torch is optional, and it is a backend's business. The loader, the decoders and the reference backend are numpy, so a phone install reads a .gguf without the training stack.
• Native backends are declared, discovered and explained. A backend can arrive from a third-party package through the pocketllm.backends entry-point group, and pocketllm ops will tell you exactly why it was or was not chosen.

Documentation

Published at https://lvyufeng.github.io/PocketLLM/.

Section	What it holds
Getting started	Install, verify, and what to expect from a device with no accelerator
The fleet support matrix	What each accelerator class runs, its accuracy, and prefill/decode throughput
Architecture	The kernel ABI, the backend model, execution, and the device targets
Models	The support matrix and what an architecture is in this tree
Guides	Procedures, release flow
Reports	Long-form rendered reports

Roadmap

• The C core runs Qwen3-0.6B and 1.7B. GGUF read, BPE tokenize, dense forward and greedy decode, on cpu and on one cuda card, checked against llama.cpp.
• The K-quant and IQ2 rungs of the ladder in the C core. q2_k, q3_k, q4_k, q5_k, q6_k, iq2_xxs and iq2_xs decode inside the kernel, packed, never widened.
• The ladder's remaining rungs in the C core. iq3_xxs, iq4_xs, iq4_nl and iq1_m decode in Python and are refused by name at load; the ternary packs are addressable and not decoded.
• pocketllm run and pocketllm serve. Both reach the C core through native.py.
• The vision-language path. qwen3vl runs end to end — the tower, any-size preprocessing, the image span and the text around it — and agrees with libmtmd at correlation 0.99 on images from 512² to 1024². Reachable from C through pocketllm_vlm_*.
• The speech path. qwen3a runs end to end — log-mel, the tower, the transcript — with the tower at 0.99983003 against the reference and speech transcribing to the reference's own answer. Reachable from C through pocketllm_asr_*.
• Performance for the two multimodal paths. Both were brought up for correctness and neither has a benchmark: the vision tower reached 2.2 s for a 768x768 image and the audio tower 251 ms for a window after #747/#748, and what is left is the ladder rather than the kernels.
• The width ladder for the two multimodal paths. The vision tower's loader widens to f32, so none of the q2_k…iq2 rungs reach multimodal weights, and the multimodal towers are the largest thing this engine asks a phone to hold.
• The CPU path is ahead of llama.cpp on prefill and decode at a matched KV depth.
• The Python backends. cuda first; each is a declaration today.
• python/pocketllm/tokenizer/ — the BPE is a stub. The C core tokenizes correctly; the Python skeleton is behind it.
• architectures/xing4_0/, rebuilt against the ABI, and a golden fixture.
• A QNN backend and an Android delivery path — the target the whole repository is named for.

License

PocketLLM is released under the Apache License 2.0.

Model weights, tokenizer files, CUDA, PyTorch, GGUF assets and other third-party components are governed by their respective licenses. PocketLLM's code license grants no additional rights to third-party model assets. The vendored GGML header is MIT, from llama.cpp — see its provenance note.