Skip to content

quic/efficient-transformers

BranchesTags

Repository files navigation

Qualcomm Dragonfly logo

Qualcomm AI200 rack

Efficient Transformers Library

Latest news 🔥

  • [07/2026] Added dynamo flag to QEFFAutoModelForCausalLM.export() to support torch.onnx.export dynamo-based ONNX export for CausalLM models
  • [06/2026] Added support for Gemma4 models, google/gemma-4-E2B-it, google/gemma-4-26B-A4B-it
  • [06/2026] Added support for Qwen3.6 model Qwen/Qwen3.6-35B-A3B
  • [06/2026] Added support for Qwen3.5 modelQwen/Qwen3.5-0.8B
  • [06/2026] Added support for Qwen3 VL and Qwen3 MoE VL models, Qwen/Qwen3-VL-30B-A3B-Instruct, Qwen/Qwen3-VL-32B-Instruct
  • [06/2026] Added support for GLM-4.5 MoE model and enabled its disaggregated mode zai-org/GLM-4.5
  • [06/2026] Enabled disaggregated support for Qwen3 MoE model
  • [04/2026] Added WAN non-unified execution support in QEffWanPipeline with separate transformer_high and transformer_low modules
  • [04/2026] Added first-block-cache support for WAN non-unified mode and FLUX (QEffWanPipeline, QEffFluxPipeline)
More

Overview

Train anywhere, Infer on Qualcomm Cloud AI with a Developer-centric Toolchain

This library provides reimplemented blocks of LLMs which are used to make the models functional and highly performant on Qualcomm Cloud AIxxx (AI100, AI200 and so on). There are several models which can be directly transformed from a pre-trained original form to a deployment ready optimized form. For other models, there is comprehensive documentation to inspire upon the changes needed and How-To(s).

Typically for LLMs, the library provides:

  1. Reimplemented blocks from Transformers which enable efficient on-device retention of intermediate states.
  2. Graph transformations to enable execution of key operations in lower precision
  3. Graph transformations to replace some operations to other mathematically equivalent operations
  4. Handling for under-flows and overflows in lower precision
  5. Patcher modules to map weights of original model's operations to updated model's operations
  6. Exporter module to export the model source into a ONNX Graph.
  7. Sample example applications and demo notebooks
  8. Unit test templates.

It is mandatory for each Pull Request to include tests such as:

  1. If the PR is for adding support for a model, the tests should include successful execution of the model post changes (the changes included as part of PR) on Pytorch and ONNXRT. Successful exit criteria is MSE between output of original model and updated model.
  2. If the PR modifies any common utilities, tests need to be included to execute tests of all models included in the library.

Quick Installation

# Create Python virtual env and activate it. (Recommended Python 3.12)
sudo apt install python3.12-venv
python3.12 -m venv qeff_env
source qeff_env/bin/activate
pip install -U pip

# Clone and Install the QEfficient repository from the mainline branch
pip install git+https://github.com/quic/efficient-transformers

# Clone and Install the QEfficient repository from a specific branch, tag or commit by appending @ref
# Release branch (e.g., release/v1.20.0):
pip install "git+https://github.com/quic/efficient-transformers@release/v1.20.0"

# Or build wheel package using the below command.
pip install build wheel
python -m build --wheel --outdir dist
pip install dist/qefficient-0.0.1.dev0-py3-none-any.whl

For more details about using QEfficient via Cloud AIxxx (AI100, AI200 and so on) Apps SDK, visit Linux Installation Guide

Documentation

Note: More details are here: https://quic.github.io/cloud-ai-sdk-pages/latest/Getting-Started/Model-Architecture-Support/Large-Language-Models/llm/

Acknowledgements

Thanks to:

  • HuggingFace transformers for work in LLM GenAI modeling implementation
  • ONNX, Pytorch, ONNXruntime community.

Support

If you run into any problems with the code, please file Github issues directly to this repo.

Contributing

This project welcomes contributions and suggestions. Please check the License. Integration with a CLA Bot is underway.

About

This library empowers users to seamlessly port pretrained models and checkpoints on the HuggingFace (HF) hub (developed using HF transformers library) into inference-ready formats that run efficiently on Qualcomm Cloud AIxx (AI100, AI200 and so on) accelerators.

quic.github.io/efficient-transformers/

Topics

acceleratoraicloudllmqualcomm

Resources

Readme
License

Code of conduct

Code of conduct

Contributing

Contributing
Activity
Custom properties

Stars

90 stars

Watchers

5 watching

Forks

92 forks
Report repository

Releases

Packages

Used by

Contributors

Languages