TensorRT-LLM Throughput Benchmark

Setup

(official documentation)

Install System Dependencies

sudo apt-get -y install libopenmpi-dev

Install Python Dependencies

Using pip

Create and activate a virtual environment:

python -m venv .venv
source .venv/bin/activate

Install the dependencies from pyproject.toml using:

pip install -e .

Or if you want to install without editable mode:

pip install .

Using uv (faster alternative)

Install dependencies:

uv pip install -e .

Or for development dependencies:

uv pip install -e ".[dev]"

Usage

Virtual Environment

These scripts should be executed in the virtual environment so that the CLI commands are recognized.

source .venv/bin/activate

Supported models

Check if the model architecture has already been tested with TensorRT-LLM (here).

If so, go ahead and add your preferred HF name of the model to HF_MODELS in benchmark.py. Use the key associated with the HF name in BenchmarkConfig.model.