[Bug]: KV cache manager v2 startup fails with HostOOMError: mmap failed when vm.overcommit_memory=2 #20967

System Info

x86_64, 1 × H200, kernel 5.14 (EL9), vm.overcommit_memory=2, overcommit_ratio=80, no swap; MemTotal 2,112,922,436 kB, CommitLimit 1,681,084,264 kB. TensorRT-LLM 1.4.0rc1.dev202610201007, PyTorch backend. The code involved is the same in v1.4.0rc1 and on main.

Who can help?

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

On a host with vm.overcommit_memory=2 and physical RAM larger than CommitLimit, starting a server with the default KV cache settings fails during KV cache setup. The host tier reserves address space equal to physical RAM, which is above CommitLimit, so the reservation is refused regardless of how much memory is free. Hosts in mode 0 or 1 are not affected. To check a host: sysctl vm.overcommit_memory and grep -E 'MemTotal|CommitLimit' /proc/meminfo.

trtllm-serve serve meta-llama/Llama-3.1-8B-Instruct --backend pytorch --tp_size 1

Expected behavior

The server starts, with the host KV cache tier sized automatically.

actual behavior

Startup fails with:

tensorrt_llm.bindings.internal.batch_manager.kv_cache_manager_v2.HostOOMError: mmap failed: Cannot allocate memory

Workaround: kv_cache_config: {host_cache_size: 0}.

additional notes

• Since [None][perf] Fill the KVCM2 host tier progressively #19650 the host tier reserves address space equal to physical RAM by default (core.cpp:765).
• hostMmap maps it PROT_READ|PROT_WRITE with MAP_NORESERVE (hostMem.cpp:58).
• In mode 2 the kernel ignores MAP_NORESERVE and charges a private writable mapping in full (overcommit-accounting).

An LD_PRELOAD wrapper around mmap logged the failing call: 2,163,656,884,224 bytes (1.00 × physical RAM), called from libtensorrt_llm.so, errno 12 (ENOMEM).

reserve() already keeps the range PROT_NONE until commit() opens it (hostMemBacking.h:90-93). If the range is mapped PROT_NONE from the start, mode 2 does not charge the reservation, only the ranges that commit() later opens. A PR with that change will follow.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.