NanoGEMM ⚡

🤝 Community, Issues & Support

• 🐛 Found a bug, numeric issue, or SIMD edge case? Please Open an Issue — reports are tracked and resolved quickly!
• 💬 Questions, performance ideas, or new kernels? Join our GitHub Discussions.
• ⭐ Find NanoGEMM useful? Give it a star on GitHub — it helps more engineers and researchers discover it!
• ☕ Support development (USDT TRC-20): TDVbEdnpgNgoAhNcn1EwXxWHiR3RKLit5P (Verify on TronScan)

🏆 Featured in The Daily Diff: Selected as a top AI & HPC story (Interest: 9/10, Depth: 9/10, Utility: 9/10). Ranked in GitHub Trending under Deep Learning topics.

📦 Multi-Platform Installation

Platform / Manager	Installation Command
PyPI (pip)	pip install nanogemm
PyPI (uv)	uv add nanogemm
Conda (conda-forge)	conda install -c conda-forge nanogemm
Conda (Anaconda Cloud)	conda install -c m_n_nik nanogemm
Ubuntu / Debian (APT PPA)	curl -sS https://eminsk.github.io/ppa/setup.sh | sudo bash sudo apt install python3-nanogemm
Ubuntu / Debian (.deb)	sudo dpkg -i python3-nanogemm_0.3.12-1_amd64.deb

pip install nanogemm
# or with uv
uv add nanogemm

NanoGEMM is a minimalist, bare-metal General Matrix Multiplication (GEMM) engine designed for sub-microsecond CPU inference and high-performance computing in Python.

Built with direct AVX2 / FMA (256-bit SIMD) and ARM NEON (128-bit SIMD) assembly-level register tiling and cache blocking, NanoGEMM eliminates the heavy function-call dispatch, thread-pool barriers, and memory-packing overhead of heavyweight BLAS libraries (OpenBLAS, MKL) for small-to-medium tensors.

---

🧩 Universal Compatibility Matrix

Runtime / Implementation	Supported Versions	Execution Mode	Status
CPython (Standard)	3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14, 3.15, 3.16 (Alpha)	Native C AVX2/FMA + Buffer Protocol	✅ Fully Supported
CPython (Free-Threaded)	3.13t, 3.14t, 3.15t	Multi-core No-GIL (Py_BEGIN_ALLOW_THREADS)	✅ Fully Supported
PyPy (JIT Accelerated)	3.8, 3.9, 3.10, 3.11, 3.12	High-speed JIT tracing / Pure ctypes FASM fallback	✅ Fully Supported
Operating Systems	Windows (7, 8, 10, 11), Linux, macOS (Intel & Apple Silicon)	x86_64, ARM64	✅ Fully Supported

---

🚀 Performance Benchmarks

Measured on Intel/AMD x86-64 CPU (AVX2 + FMA) against NumPy 2.2.3 (single-precision float32):

Matrix Dimension	NumPy Latency	NanoGEMM Latency	Speedup Factor	NanoGEMM Throughput
16 x 16	4.05 µs	1.71 µs (C: 0.68 µs)	🚀 2.36x – 5.50x FASTER	4.79 – 12.14 GFLOPS
24 x 24	19.39 µs	2.67 µs (C: 1.27 µs)	🚀 7.26x FASTER	10.36 – 21.73 GFLOPS
32 x 32	7.14 µs	4.21 µs (C: 2.18 µs)	🚀 1.70x – 2.95x FASTER	15.58 – 30.06 GFLOPS
48 x 48	12.65 µs	11.15 µs (C: 6.27 µs)	🚀 1.13x – 1.73x FASTER	19.83 – 35.29 GFLOPS
64 x 64	19.69 µs	13.89 µs	🚀 1.42x FASTER	37.74 GFLOPS
128 x 128	89.44 µs	115.90 µs	0.77x	36.19 GFLOPS

⚡ Quantized INT8 SIMD GEMM Performance (matmul_int8)

Measured live on Google Colab CPU (AVX2) against NumPy (int8 x int8 -> int32):

Matrix Dimension	NumPy Latency	NanoGEMM INT8	Speedup Factor	NanoGEMM Throughput
16 x 16	6.71 µs	4.57 µs	🚀 1.47x FASTER	1.79 GOP/s
32 x 32	40.02 µs	4.99 µs	🚀 8.03x FASTER	13.14 GOP/s
64 x 64	303.97 µs	28.88 µs	🚀 10.53x FASTER	18.16 GOP/s
128 x 128	2524.86 µs	221.79 µs	🚀 11.38x FASTER	18.91 GOP/s

🧠 Batched Matrix Multiplication (BMM) for Transformer Attention (bmm)

Measured live on Google Colab CPU (Multi-Head Attention $Q @ K^T$ shapes):

Attention Shape (Heads, Seq, Dim)	NumPy Latency	NanoGEMM BMM	Speedup Factor
Heads=8, Seq=32, Dim=32	25.48 µs	14.96 µs	🚀 1.70x FASTER
Heads=16, Seq=32, Dim=64	74.84 µs	50.56 µs	🚀 1.48x FASTER
Heads=32, Seq=64, Dim=64	450.56 µs	388.69 µs	🚀 1.16x FASTER
Heads=64, Seq=32, Dim=32	148.62 µs	106.96 µs	🚀 1.39x FASTER

💡 Why is NanoGEMM faster on small/medium matrices?
Traditional BLAS engines incur 3–10 µs of fixed overhead per invocation due to dynamic runtime dispatch, argument sanitization, thread synchronization, and packing buffers. NanoGEMM utilizes a zero-allocation, direct register-tiled microkernel that executes in sub-microsecond time immediately upon invocation.

⚙️ Benchmark Environment & Test Configuration

Parameter	Specification
CPU Architecture	x86-64 with AVX2 (256-bit SIMD) + FMA3 support
Execution Model	Single-Core / Single-Threaded (1 Thread) for NanoGEMM (zero thread-pool overhead)
NumPy Baseline	NumPy 2.2+ linked against OpenBLAS (standard runtime)
C Compiler	GCC (-O3 -mavx2 -mfma) / MSVC (/O2 /arch:AVX2)
Methodology	Median latency across 10,000 iterations per size with cache pre-warming
Data Type	IEEE-754 Single-Precision (float32), contiguous C-order layout

📌 Single-Thread Design Note: NanoGEMM runs strictly on a single CPU core without thread pools, pthreads, or mutex barriers. For small matrices ($16 \times 16$ to $64 \times 64$), thread synchronization in OpenMP costs more cycles than the matrix multiply itself. NumPy/OpenBLAS incurs dispatch latency, whereas NanoGEMM enters CPU registers directly.

---

⚖️ Architectural Comparison: NanoGEMM vs OpenBLAS vs LIBXSMM

When evaluating CPU matrix multiplication engines, understanding the design trade-offs between compilation models, binary footprint, dispatch latency, and target matrix shapes is critical:

Feature / Metric	NanoGEMM	LIBXSMM	OpenBLAS / MKL
Core Paradigm	Static AOT Register Tiling ($6\times 16$, $4\times 16$, $2\times 16$)	Runtime JIT Machine-Code Generation	Multi-threaded Dynamic BLAS with Buffer Packing
Binary Footprint	⚡ ~100 KB (ultra-lightweight single binary)	~15–30 MB	~30–50 MB
Code Generation	Ahead-of-Time (Zero-JIT)	Runtime JIT code emission into executable pages	Ahead-of-Time
Security / Hardened OS (W^X)	🛡️ 100% Compliant (No executable heap/stack required)	Requires writable & executable memory (PROT_EXEC)	🛡️ 100% Compliant
External Dependencies	None (pure C & FASM assembly, zero runtime deps)	C++ runtime, pthreads	OpenMP / pthreads, Fortran runtime
Python Invocation Latency	Sub-microsecond (< 1.5 µs) via Buffer Protocol & GIL release	Requires custom wrappers or ctypes bindings	3–5 µs (NumPy ufunc dispatch, C-API type checking)
Optimal Sweet Spot	Small tensors ($16 \times 16$ – $64 \times 64$) in Python / Edge AI	Small-to-medium matrices in pure C/C++	Large matrices ($512 \times 512$ – $4096 \times 4096+$)
Multi-Threading Model	Single-threaded per call (use ThreadPoolExecutor without lock contention)	Single/Multi-threaded	Multi-threaded OpenMP thread pool

💡 Key Design Rationale:

1. NanoGEMM vs OpenBLAS / MKL on Small Tensors:
   Traditional BLAS libraries are optimized for massive matrices ($1000 \times 1000+$). On small matrices ($16 \times 16$ to $64 \times 64$), the fixed overhead of thread synchronization, buffer packing, and NumPy ufunc argument sanitization takes 3–5 µs before arithmetic even begins. NanoGEMM eliminates this glue overhead completely, achieving sub-microsecond latency in Python.
2. NanoGEMM vs LIBXSMM:
   LIBXSMM (Intel Labs) is the gold standard for small-matrix GEMM in pure C/C++ when runtime JIT code emission is permissible. NanoGEMM adopts a different engineering philosophy: Zero-JIT, zero dynamic allocations, zero external dependencies, and a ~100 KB footprint. NanoGEMM deploys seamlessly in hardened environments (where W^X / DEP security policies block runtime JIT emission) and installs instantly via standard Python wheels.
3. v0.3.4 Boundary Microkernel Acceleration & Colab Validation:
   In NanoGEMM v0.3.4, dedicated $4 \times 16$ and $2 \times 16$ boundary register tiles (plus $8$-wide column tiles) are introduced. Standard square power-of-two matrices ($8 \times 8$, $16 \times 16$, $32 \times 32$, $64 \times 64$) now compute 100% inside AVX2+FMA vector registers with zero scalar tail fallback, verified live in Google Colab.

---

🛠 Architectural Design

1. Register Tiling ($6 \times 16$ Microkernel)

• Register allocation: Utilizes 12 ymm registers (ymm0 – ymm11) as 256-bit floating-point accumulators storing a $6 \times 16$ tile of matrix $C$.
• Vector broadcast & FMA: Two ymm registers load vectors from $B$, while individual scalar elements of $A$ are broadcast across ymm using _mm256_set1_ps and accumulated via fused multiply-add (_mm256_fmadd_ps).
• Zero Spilling: Fits completely inside the 16 available x86-64 YMM registers without stack eviction.

2. Multi-Level Cache Blocking

• $L_1$ / $L_2$ Cache Tiling: Matrices are processed in cache blocks ($M_c = 64, N_c = 128, K_c = 128$) to maintain maximum L1/L2 data cache hit ratios and eliminate memory bus thrashing.
• Vectorized Edge Handling: Arbitrary matrix dimensions (non-multiples of 6 or 16) are processed using boundary SIMD edge loops without padding or buffer allocations.

       Matrix A (M x K)              Matrix B (K x N)
     [ . . . . . . . . ]           [ . . . ymm0 . . . ]
     [ . . . . . . . . ]           [ . . . ymm1 . . . ]
     [ a0 a1 a2 a3 . . ]     x     [ . . . . .  . . . ]
     [ . . . . . . . . ]           [ . . . . .  . . . ]
     [ . . . . . . . . ]
             │                             │
             └──────────────┬──────────────┘
                            ▼
                Matrix C (6 x 16 Tile)
             [ ymm0  ymm1  ] -> Row 0
             [ ymm2  ymm3  ] -> Row 1
             [ ymm4  ymm5  ] -> Row 2
             [ ymm6  ymm7  ] -> Row 3
             [ ymm8  ymm9  ] -> Row 4
             [ ymm10 ymm11 ] -> Row 5

3. ARM NEON ($4 \times 8$ Microkernel)

• Target Architecture: Apple Silicon (M1/M2/M3/M4) and Linux AArch64 (AWS Graviton, Ampere Altra, Neoverse).
• Register Tiling: Operates on 8 128-bit float32x4_t accumulator registers (c00–c31) computing a $4 \times 8$ tile of matrix $C$.
• Vector FMA: Inner loop uses vld1q_f32 vector loads and vmlaq_n_f32 fused multiply-accumulates. Standard power-of-two matrix sizes ($16\times 16$, $32\times 32$, $64\times 64$) tile evenly with zero scalar tail fallback.

---

📦 Installation & Quickstart

Installation via PyPI (Recommended)

pip install nanogemm
# or with uv
uv add nanogemm

Python Compatibility: Fully tested and verified across Python 3.9 through 3.15 (including 3.15.0rc2).

⚡ No-GIL & Free-Threaded Ready (PEP 703):

• Free-Threaded CPython: Explicitly declares Py_MOD_GIL_NOT_USED to run safely without re-enabling the GIL under python3.13t, 3.14t, and 3.15t.
• GIL-Releasing Operations: Automatically releases the GIL (Py_BEGIN_ALLOW_THREADS) during hardware SIMD execution, allowing Python threads (threading.Thread, ThreadPoolExecutor) to run matrix multiplications concurrently across multiple CPU cores without lock contention.

Build from Source

git clone https://github.com/eminsk/nanogemm.git
cd nanogemm
pip install -e .

Python Usage

import nanogemm as ng
import numpy as np

# Verify SIMD hardware acceleration
print("Active ISA:", ng.get_simd_isa())
# Output: Active ISA: AVX2+FMA (256-bit SIMD, 6x16 register tiling)

# Allocate input matrices
A = np.random.randn(32, 64).astype(np.float32)
B = np.random.randn(64, 128).astype(np.float32)

# Direct hardware-accelerated MatMul: C = A @ B (supports 2D, 3D, and 4D tensors)
C = ng.matmul(A, B)

# Pre-allocated zero-copy output buffer for maximum throughput:
out = np.empty((32, 128), dtype=np.float32)
ng.matmul(A, B, out=out)

# Batched Multi-Head Attention GEMM for Transformers (B x H x S x D):
Q = np.random.randn(8, 16, 32, 64).astype(np.float32)
K_t = np.random.randn(8, 16, 64, 32).astype(np.float32)
scores = ng.bmm(Q, K_t)  # Output: (8, 16, 32, 32) with zero Python loop overhead

# Quantized INT8 GEMM (int8 x int8 -> int32, up to 5.0x faster than NumPy):
A_i8 = np.random.randint(-128, 128, size=(32, 64), dtype=np.int8)
B_i8 = np.random.randint(-128, 128, size=(64, 128), dtype=np.int8)
C_i32 = ng.matmul_int8(A_i8, B_i8)

# End-to-end Quantized Linear Layer forward pass:
Y_float = ng.quantized_matmul(A_i8, B_i8, scale_a=0.05, scale_b=0.02)

# Standard BLAS SGEMM interface: C = alpha * (A @ B) + beta * C
res = ng.sgemm(A, B, alpha=2.0, beta=0.5, c=out)

---

🤖 Native Model Context Protocol (MCP) Server

NanoGEMM includes a native, zero-dependency Model Context Protocol (MCP) server (nanogemm-mcp), allowing AI agents (Claude Desktop, Cursor, Antigravity, OpenManus) to query host SIMD instruction sets, execute ultra-low latency GEMM arithmetic, and run live CPU microbenchmarks.

🛠 Available MCP Tools

Tool	Parameters	Description
gemm_benchmark	m, k, n, iterations, dtype	Microbenchmark NanoGEMM vs NumPy on host CPU with exact GFLOPS & speedup factor
gemm_multiply	matrix_a, matrix_b, dtype	Execute direct SIMD matrix multiplication on nested numeric arrays (FP32/INT8)
gemm_hardware_info	(none)	Detect CPU architecture, SIMD extensions (AVX2, FMA, NEON, SSE2), and active kernel

🚀 Running the Server

Run directly via the CLI:

# Direct command
nanogemm mcp

# Or via dedicated script
nanogemm-mcp

⚙️ Client Integration Configurations

Cursor (.cursor/mcp.json)

{
  "mcpServers": {
    "nanogemm": {
      "command": "nanogemm-mcp"
    }
  }
}

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "nanogemm": {
      "command": "python",
      "args": ["-m", "nanogemm.mcp_server"]
    }
  }
}

---

🚀 Interactive Google Colab Demo

Run NanoGEMM benchmarks interactively in your browser with zero local setup:

The Interactive Benchmark Notebook demonstrates:

• Zero-Setup Installation & Hardware ISA Detection: Compiles native C/AVX2 on Google Colab CPU in seconds.
• 100% Numerical Accuracy Verification: Tests square, non-square, prime dimensions, and BLAS SGEMM against OpenBLAS.
• Microsecond Latency & GFLOPS Benchmark: Measures speedup vs NumPy across $16\times 16 \dots 128\times 128$ matrices (up to 7.26x faster on $24\times 24$).
• Interactive Performance Plotting: Side-by-side latency & speedup bar/line charts using Matplotlib with Edge AI sweet spot highlights.
• Real-Time Edge AI Loop: Demonstrates 100,000 iterations of zero-allocation in-place multiplication (out=C) for $24\times 24$ and $32\times 32$.
• Batched Matrix Multiplication (BMM): Multi-Head Attention benchmarks ($Q @ K^T$) across 8, 16, 32, and 64 heads.
• Quantized INT8 SIMD GEMM: Measures speedup vs NumPy integer multiplication across $16\times 16 \dots 128\times 128$ (up to 5.0x faster).

---

🧪 Testing & Verification

Run the full pytest suite (covering IEEE-754 correctness, batched Transformer BMM, INT8 SIMD, and FASM assembly):

uv run --extra dev pytest -v
# or with standard pytest
pytest -v

Or run individual test scripts directly:

# Correctness vs reference NumPy OpenBLAS:
python tests/test_correctness.py

# Batched BMM & INT8 SIMD GEMM verification:
python tests/test_bmm_int8.py

# Standalone FASM assembly DLL bridge verification:
python tests/test_fasm.py

# Official performance benchmark vs NumPy BLAS:
python benchmarks/bench_vs_numpy.py

---

⚡ Standalone Flat Assembler (FASM) 32-bit & 64-bit Engines

NanoGEMM includes native standalone assembly implementations written in Flat Assembler (FASM) for both 64-bit and 32-bit architectures in the asm/ directory:

• x86-64 Engine (asm/nanogemm64.dll, asm/test_nanogemm64.exe):
  • Microkernel with $4 \times 16$ and $4 \times 8$ register-tiled AVX2+FMA instructions (vfmadd231ps, vbroadcastss, vmovups).
  • Complies strictly with the Microsoft x64 ABI (preserving non-volatile registers RBX, RSI, RDI, R12–R15, XMM6–XMM15).
  • Peak throughput exceeding 38–40 GFLOPS on a single CPU core.
• x86 32-bit Engine (asm/nanogemm32.dll, asm/test_nanogemm32.exe):
  • Vectorized $4 \times 4$ SSE2 microkernel (movups, shufps, mulps, addps) using cdecl calling convention.
  • Compatible with all 32-bit x86 environments and 64-bit Windows via WoW64 with zero external dependencies.
  • Delivers 13–15 GFLOPS in pure 32-bit mode.

Building & Running FASM Tests

:: Build all 4 binaries and run native executable suites
cd asm
build.bat

:: Run Python verification and NumPy comparison suite
python tests/test_fasm.py

---

🌐 High-Performance Systems Ecosystem

NanoGEMM is developed by @eminsk as part of an engineering ecosystem focused on low-level hardware performance, assembly programming, and AI agent infrastructure:

• 🧠 AgentJIT — Just-In-Time Compiler for AI Agent Trajectories with speculative de-optimization guards and Native MCP Server (pip install agentjit).
• ⚡ NanoVector — Bare-metal C99/AVX2 vector search & episodic memory engine (~120KB) with Native MCP Server (pip install nanovector).
• 🖥️ NanoRecall — 100% Private, offline desktop memory & semantic screen search engine with Native MCP Server (pip install nanorecall).
• 🛒 avito-sdk — Headless Avito scraping & data extraction SDK with price drop tracking, Playwright cookies, Telegram/VK bots, and Native MCP Server (pip install avito-sdk).
• 📊 xlsx_vievers — Headless Excel formula engine (129+ functions), desktop spreadsheet viewer, SIMD SSE2 math, and Native MCP Server (pip install xlsx-viewer-pro).
• 📈 yfinance-ta-patterns — Candlestick & chart pattern scanner with AI Confluence Scoring, Backtesting, and Native MCP Server (pip install yfinance-ta-patterns).
• 🎥 screenvideo — Lightweight desktop screen recorder featuring WASAPI loopback audio and a standalone pure x64 Flat Assembler (FASM) native edition.

---

☕ Support, Community & Donations

If you find this project valuable and would like to support ongoing development:

• ⭐ Star the Repository: If NanoGEMM accelerates your matrix multiplications, give us a star on GitHub — it helps more developers discover high-performance SIMD!
• 💬 Join Discussions: Have ideas, use cases, or hardware benchmarks? Start or join a thread in GitHub Discussions!
• ☕ Donate (USDT TRC-20):
  TDVbEdnpgNgoAhNcn1EwXxWHiR3RKLit5P
  (Network: TRON / TRC-20 | Verify on TronScan)

---

📄 License

MIT License — Copyright (c) 2026 eminsk.