🦾 - IRON: Unlocking the Full Potential of NPUs - 🦾

IRON is an open-source & close-to-metal Python API enabling fast and efficient execution on AMD Ryzen™ AI NPUs. It relies on language bindings around the MLIR-AIE dialect.

Key Features:

• Close-to-metal NPU programming via MLIR-AIE Python bindings
• Pre-built operator library (GEMM, MHA, RMSNorm, RoPE, activations, etc.)
• Operator fusion for optimal performance
• Extensible architecture for custom operators
• End-to-end LLM inference (Llama 3.2 1B example included)

The IRON Python API for Ryzen™ AI NPUs is described in the following paper:

E. Hunhoff, J. Melber, K. Denolf, A. Bisca, S. Bayliss, S. Neuendorffer, J. Fifield, J. Lo, P. Vasireddy, P. James-Roxby, E. Keller. "Efficiency, Expressivity, and Extensibility in a Close-to-Metal NPU Programming Interface". In 33rd IEEE International Symposium On Field-Programmable Custom Computing Machines, May 2025.

🎯 Operator Dashboard

Section	Description	Datatype	AIE2	AIE2P	Status	Design Example
Element-wise Add	Element-wise addition kernel	bfloat16	✓	✓	🟢	iron/operators/elementwise_add/
Element-wise Mul	Element-wise multiplication kernel	bfloat16	✓	✓	🟢	iron/operators/elementwise_mul/
GEMM	General Matrix Multiplication kernel	bfloat16	✓	✓	🟢	iron/operators/gemm/
GEMM INT8	INT8 General Matrix Multiplication (i8→i8/i16/i32)	int8	✓	✓	🟢	iron/operators/gemm/
GEMV	General Matrix-Vector Multiplication kernel	bfloat16	✓	✓	🟢	iron/operators/gemv/
GQA	Grouped Query Attention kernel (Single pipeline)	bfloat16		✓	🟢	iron/operators/mha/
MHA	Multi-Head Attention kernel & Grouped Query Attention	bfloat16		✓	🟢	iron/operators/mha/
RMSNorm	RMSNorm kernel	bfloat16	✓	✓	🟢	iron/operators/rms_norm/
RoPE	Rotary Positional Embedding kernel	bfloat16	✓	✓	🟢	iron/operators/rope/
SiLU	Sigmoid Linear Unit activation kernel	bfloat16	✓	✓	🟢	iron/operators/silu/
Softmax	Softmax kernel	bfloat16	✓	✓	🟢	iron/operators/softmax/
Weighted RMSNorm	Weighted RMSNorm kernel	bfloat16	✓	✓	🟢	iron/operators/rms_norm/
Copy	Copy	bfloat16	✓	✓	🟢	iron/operators/mem_copy/
Transpose	Transpose	bfloat16	✓	✓	🟢	iron/operators/transpose/
AXPY	AXPY	bfloat16	✓	✓	🟢	iron/operators/axpy/
Reduction	Reduction	bfloat16			🟡
Dequant	Dequant Q4NX from AWQ to bfloat16	bfloat16	✓	✓	🟢	iron/operators/dequant/
RELU	RELU	bfloat16	✓	✓	🟢	iron/operators/relu/
Leaky RELU (WIP)	Leaky RELU kernel	bfloat16		✓	⚪	iron/operators/leaky_relu/
GELU	GELU	bfloat16	✓	✓	🟢	iron/operators/gelu/
LayerNorm	LayerNorm	bfloat16	✓	✓	🟢	iron/operators/layer_norm/
Convolution	Convolution	bfloat16			🟡
MaxPool	MaxPool	bfloat16			⚪
AveragePool	AveragePool	bfloat16			⚪
Tanh	Tanh kernel	bfloat16	✓	✓	🟢	iron/operators/tanh/
Sigmoid	Sigmoid kernel	bfloat16	✓	✓	🟢	iron/operators/sigmoid/

Use this dashboard to quickly check the status of each kernel and locate relevant setup, build, and usage information.

📌 Legend

Status	Meaning
🟢	Done
🟡	In Development
⚪	Not Assigned

Installation (Linux)

These instructions will guide you through everything required for building and executing a program on the Ryzen™ AI NPU, starting from a fresh bare-bones Ubuntu 24.04 or Ubuntu 24.10 install.

Initial Setup

Important: Ensure your system has the latest BIOS version that enables NPU support. Check your laptop/mini-PC manufacturer's support website for BIOS updates.

If starting from Ubuntu 24.04 you may need to update the Linux kernel to 6.11+ by installing the Hardware Enablement (HWE) stack:

sudo apt update
sudo apt install --install-recommends linux-generic-hwe-24.04
sudo reboot

1. Install XDNA™ Driver and XRT:

   Instructions from mlir-aie repository

2. Install the packages needed for IRON and MLIR-AIE:

   # Python versions 3.10, 3.12 and 3.13 are currently supported by our wheels
   sudo apt install \
   build-essential clang clang-14 lld lld-14 python3-venv python3-pip

3. Setup a virtual environment and activate it:

   python3 -m venv ironenv
   source ironenv/bin/activate
   python3 -m pip install --upgrade pip

4. Source XRT (installed in step 1):

   source /opt/xilinx/xrt/setup.sh

5. Install required Python packages (from requirements.txt):

   pip install -r requirements.txt

6. To test your installation, you can try to build and run the example below:

   pytest ./iron/operators/axpy/

Installation (Windows)

⚠️ Important: Toolchain Availability

The Python code of IRON is fully cross-platform and runs on Windows without modification. However, the upstream mlir_aie package (which provides aiecc.py, MLIR Python bindings, and XRT runtime integration) currently ships Linux-only wheels. The llvm-aie package (Peano compiler: clang++, llvm-ar, llvm-objcopy) does provide Windows wheels.

This means:

Component	Native Windows	Via WSL2
Python IRON code (operators, tests, API)	✅	✅
Peano compiler (clang++, llvm-ar)	✅	✅
aiecc.py (xclbin compilation)	❌	✅
MLIR Python bindings (aie.*)	❌	✅
XRT runtime (pyxrt)	✅ (separate SDK)	✅

Recommended approach: use WSL2 (Windows Subsystem for Linux) — all tools work identically to native Linux. Follow the Linux instructions above inside WSL2.

For native Windows development (editing, testing CPU-side code, running the Python API), the setup below works as-is.

Prerequisites (Native Windows)

Important: Ensure your system has the latest BIOS version that enables NPU support. Check your laptop/mini-PC manufacturer's support website for BIOS updates.

1. Install Python 3.10, 3.12, or 3.13 from python.org or the Microsoft Store. During installation, check "Add Python to PATH".

2. Install Visual Studio Build Tools (needed for compilation):

   • Download from Visual Studio Build Tools
   • Or install the full Visual Studio Community with the "Desktop development with C++" workload.

3. Install Git for Windows from git-scm.com.

4. Install the AMD XDNA™ Driver and XRT for Windows:

   Follow the AMD Ryzen AI SDK installation guide for Windows.

   XRT is typically installed to C:\Xilinx\XRT or C:\Program Files\AMD\XRT.

Setup (Native Windows)

1. Open PowerShell or Command Prompt and clone the repository:

   git clone https://github.com/amd/IRON.git
   cd IRON

2. Create and activate a virtual environment:

   python -m venv ironenv
   ironenv\Scripts\activate
   python -m pip install --upgrade pip

3. Set up XRT environment (adjust path to your XRT installation):

   call "C:\Xilinx\XRT\setup.bat"

4. Install required Python packages:

   pip install -r requirements.txt

5. Test your installation:

   pytest .\iron\operators\axpy\

Setup (WSL2 — Recommended)

1. Install WSL2 with Ubuntu 24.04:

   wsl --install -d Ubuntu-24.04

2. Inside WSL2, follow the Linux installation instructions above. All tools (mlir_aie, llvm-aie, XRT) work natively in WSL2.

3. Access your Windows files from WSL2 at /mnt/c/ if needed.

Windows Notes

• The LLVM/Clang tools from the llvm-aie pip package are used automatically (including clang++, llvm-objcopy, llvm-nm, llvm-ar). On Windows these ship as .exe files and are discovered by the build system.
• File copy operations during compilation use Python's shutil.copy2 instead of the Unix cp command.
• Symbol map generation (used by some operators) is done in pure Python on Windows, replacing the sh | awk pipeline used on Linux.
• The aiecc compiler tool is located via the MLIR-AIE package and handles both aiecc and aiecc.py entry points on Windows.
• When mlir_aie gains official Windows wheels, full native compilation will work without any code changes — the IRON build system is already prepared.

Building/Using & Testing Operators

All available operators can be found in iron/operators. These each contain:

• op.py: The Python operator interface -- an easy access point to integrate operators into your project that prescribes how to compile the operator (build artifacts) and how to call it at runtime (buffer sizes, etc.)
• design.py: The implementation of the operator's NPU code. Often references a kernel in aie_kernels for the compute core code and describes the data movement using ObjectFIFOs.
• reference.py: A reference CPU implementation to validate the correctness of the NPU implementation.
• test.py: An end-to-end test that instantiates and builds the operator, runs it and verifies its outputs against the reference.

NOTE: Be sure the XRT setup script has been sourced and the Python environment is activated: Linux: source /opt/xilinx/xrt/setup.sh + source /path/to/ironenv/bin/activate Windows: call "C:\Xilinx\XRT\setup.bat" + ironenv\Scripts\activate

To build and test all the operators:

pytest iron/operators/ -m "not extensive"

To run the extensive test suite:

pytest iron/operators/

To run a specific operator's tests:

pytest iron/operators/axpy/

Git Hooks (Optional but Recommended)

To ensure your code passes CI linting checks before pushing, install the pre-push hook:

Linux / macOS:

cp scripts/hooks/pre-push .git/hooks/pre-push
chmod +x .git/hooks/pre-push

Windows (PowerShell):

Copy-Item scripts\hooks\pre-push .git\hooks\pre-push

The hook will run the same linting checks as CI:

• License checks (reuse)
• Python formatting (black)
• C++ formatting (clang-format)

To bypass the hook if needed: git push --no-verify

Applications

Llama 3.2 1B Inference

IRON includes a complete LLM inference example demonstrating NPU acceleration:

• Location: iron/applications/llama_3.2_1b/
• Model: Meta Llama 3.2 1B
• Features: Multi-head attention, fused operators, bfloat16 quantization

See iron/applications/llama_3.2_1b/README.md for setup and usage instructions.

Architecture

IRON uses a three-layer architecture:

1. Operators (iron/operators/): High-level Python API for NPU operations

   • Each operator has: op.py (interface), design.py (MLIR-AIE implementation), reference.py (CPU reference), test.py (validation)

2. AIE Kernels (aie_kernels/): Low-level C++ compute kernels

   • Organized by architecture: generic/, aie2/, aie2p/
   • Vectorized using AIE API for optimal performance

3. Common Infrastructure (iron/common/): Compilation, device management, and utilities

   • MLIR-AIE compilation pipeline
   • XRT runtime integration
   • Operator fusion framework

Performance

IRON operators are designed for maximum NPU utilization:

• Parallel execution across multiple AIE columns
• Optimized data movement via ObjectFIFOs
• Fused operations to minimize host-NPU transfers
• Vectorized kernels using AIE intrinsics

Run benchmarks:

# Run all operators with performance metrics stored in tests_latest.csv
pytest iron/operators/ -m "not extensive" -v

ggml-xdna Backend (Windows)

This fork includes a ggml-xdna backend for llama.cpp — enabling LLM inference on AMD XDNA NPUs via the ggml ecosystem.

Based on albiol2004/llama.cpp ggml-xdna branch. See ggml-org/llama.cpp#21725 for the upstream feature request.

Supported operations (bf16)

• ✅ GEMM/GEMV (prefill & decode)
• ✅ SwiGLU (fused decode, prefill)
• ✅ QKV projection (fused)
• ✅ RMSNorm, RoPE, Softmax
• ✅ Attention (FlowKV decode)

INT8 support (in progress)

• ✅ INT8 GEMM operator (i8×i8→i8/i16/i32)
• 🔜 INT8 GEMV (decode path)
• 🔜 INT8 SwiGLU (W8A16 decode, W8A8 prefill)

Files

ggml-xdna backend перенесён в llama.cpp-xdna (ggml/src/ggml-xdna/).

Community and Support

• 💬 Discord: Join our Discord server for discussions and support
• 🐛 Issues: Report bugs and request features via GitHub Issues
• 📖 Contributing: See CONTRIBUTING.md for development guidelines
• 📚 Documentation: Operator examples in iron/operators/, kernel docs in aie_kernels/README.md

License

IRON is licensed under the Apache License 2.0. See LICENSE for details.

---

Copyright© 2025-2026 Advanced Micro Devices, Inc