Repository navigation
Conversation
Lifecycle fixes needed by an engine that takes a minute or more to start and forks helper processes, such as vLLM. None of them is engine-specific. - engine:status and engine:get-installed no longer wait for a running lifecycle operation, so a toggle during a slow start does not time out. - Shutdown caps the stop grace at 3s so the force kill lands inside the parent's window, and a SIGTERM no longer blocks on an idle stdin. - A stop also kills helper processes left in the engine's process group, so a crash mid-load does not orphan a process holding GPU memory. - An orphaned engine run by an interpreter (a Python package's script) is recognised by its command line, so the next launch can stop it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…ache Add an hf_cache manifest action that lists the models in a Hugging Face hub cache directory. It reads the disk, so it works while the engine is stopped, which lets an engine that serves one model per process (vLLM) show every model it could be switched to. A repository is listed only when its snapshot is a complete, text-model-shaped download: vision-only and GGUF-only repositories, interrupted downloads and links whose blob is gone are left out. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a launch_model manifest action for an engine that has no load call because
the model is a launch argument (vLLM). It takes {"model": "<id>"}, writes the
model into the saved launch settings under the declared flag, and restarts a
running engine on it. If the new model does not come up, the previous one is
restored and the error says so. A stopped engine only has its saved launch
changed, and asking for the model already in the launch arguments restarts
nothing.
The action requires editable launch settings and a readiness probe on every
platform, and refuses model ids that could be read as a flag, a path or a
shell fragment.
Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Register vllm alongside Ollama, LM Studio and llama.cpp: facade port 8000, engine port 8001, discovery key "vl". The proxy and broker treat it like any other managed engine, and the terminal interface gets its model wire. The engine-manager manifest (Linux amd64/arm64) is detect-first: an existing vllm is adopted, otherwise a sha256-pinned uv installs vllm 0.32.0 into a private venv after checking for an NVIDIA driver with CUDA 13+ and enough disk. vLLM serves one model per process, so its list_models is an hf_cache action, load_model is a launch_model action, and a new manifest field, advertised_models: loaded, advertises only the served model to peers and the proxy, since a request for any other cached model would fail at the engine. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Check the bundled vLLM manifest's install, launch and model actions, that the install refuses drivers without CUDA 13, and, across real binaries, that the broker starts a vLLM proxy and the facade serves the model list and routes exact model ids. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Show vLLM next to the other engines: type, display name, links, welcome-screen default (Linux only), proxy identity and port state, icon and downloaded-model matcher. The icon is an original lettermark, not the vLLM project logo. vLLM serves one model per process and has no pull, eject or delete call, so a new hasPullModel capability hides Add model for it. Its model list is the Hugging Face cache and Load switches the served model by restarting vLLM; that load is observed so a failure is attributed to the model row, and the row stays locked for up to 20 minutes while the engine restarts. Document the engine across the desktop and top-level docs, tick the roadmap item, and add the vLLM model store to the data-reset notes. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Models with a 256K default context need more KV cache than the default 0.5 GPU memory share provides and fail to start. Pass --max-model-len 32768 by default and document it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…ts it vLLM serves one model per process, so loading another restarts the engine. Show 'starting' instead of 'stopped', keep the model list, and await the load call so the switch does not look like the engine was disabled. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…lling per model vLLM refuses a context above the model's own limit, reserves 92% of the GPU by default, and rejects tool_choice auto without a tool-call parser. Agent clients such as Hermes need a 64K window and tool calling. Add a manifest auto_launch (hf_context) step that reads the served model's config.json and the free GPU memory at every start and appends --max-model-len (64K target, shrunk to fit), YaRN --hf-overrides when the model's window is shorter, --gpu-memory-utilization sized to the weights and KV cache, and --enable-auto-tool-choice with the hermes parser for Qwen families. Arguments the user saved always win, nothing is written back to saved settings, and tensor/pipeline-parallel launches are left alone. Drop the fixed 0.5 GPU share and 32K cap from the default launch. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…for vLLM vLLM has no pull API, so a model not already cached could only be used by naming it in the launch arguments. Add an hf_pull action kind for pull_model: it reads the repo's file list and sizes from Hugging Face, selects the files vLLM reads, and runs the hf tool from the engine's virtualenv, reporting progress from the bytes landing in the hub cache. It needs no running engine and stops when cancelled or stalled. Add a vLLM engine:catalog source: open, text-generating safetensors repos of a few publishers (text-generation and image-text-to-text), with parameter count and size, plus a searchable query like llama.cpp's. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Enable hasPullModel for vLLM, give it an engine hub, let the model hub search the upstream catalog for any engine that reports it (a new hubSearch capability, set for llama.cpp and vLLM), and warm the vLLM catalog with the others. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM had no delete, so a cached model could only be removed by hand. Add an hf_cache delete variant for delete_model: it removes a model's folder and download locks from the hub cache, needs no running engine, refuses a path outside the cache, and refuses the model the engine is set to serve (running or not), failing closed when that cannot be determined. Enable hasDeleteModel for vLLM in the desktop, with a new deleteBlockedWhileLoaded capability that disables Delete on the loaded model. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM cannot drop its model while running, so Eject (unload_model) is a new stop_engine action that stops the engine and frees all of its GPU memory. Load (launch_model) on a stopped engine now saves the choice and starts it, restoring the previous choice if it does not start. Enable hasEject for vLLM and keep its model list, Add model and Load available while stopped, since the list is read from disk. Delete removed only a model's folder, but recent Hugging Face tooling keeps large files in a shared hub/blobs store linked from each repo, so the weights stayed on disk. Delete now also removes shared files no other model uses, with their sidecars, and reports the bytes freed. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add an optional runtime front: PAIR listens on the engine's public port and runs the engine itself on a loopback port at a fixed offset, forwarding requests unchanged, streaming responses, cancelling the engine's request when the client leaves, and answering 503 while the engine is not up. Only the launch that is spawned is repointed; the launch users see and validate keeps the public port. A stale engine on the internal port from a crashed run is reclaimed at the next start. This is the base for idle unload and wake on request. Only vllm.json declares a front, and the hooks in shared lifecycle code do nothing without one; a test checks that every other engine's launch is unchanged. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
An engine behind a gateway now tracks inference activity at the gateway (not GET/HEAD/OPTIONS, so health checks and the model list do not count, and a request in flight keeps the engine loaded). After front.idle_timeout_s (600 s for vLLM, 0 never) with none, the manager stops the engine's process, freeing the GPU, while the gateway and the ON state stay. The status reports idle, /health still answers, inference is refused with 503, and the model it is set to serve is still advertised with nothing marked loaded. Eject on such an engine now unloads and keeps it ON like the other engines, and Load of the model it is set to serve loads it again; a failed load leaves the engine idle. Waking on an incoming request follows. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
A request that needs the model and arrives while the engine is unloaded now loads it behind the gateway and is answered once it is ready, as Ollama and llama.cpp do, instead of getting a 503. Requests that arrive together share one start, a client that leaves does not abandon the load, and Eject followed by a request reloads it. A failed load answers 503 with the reason and leaves the engine idle so the next request retries. Health checks, the model list and other reads never load the model. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…e proxy The proxy routes by each node's advertised models and builds its own model list from the engine's /v1/models. While the engine was unloading or loading, the inventory briefly lost the model, and the idle gateway refused the model list, so a request arriving then was answered 'no node advertises the requested model' and never reached the wake. Record the served model when the engine goes idle, mark the state idle before the gateway so no reader sees an engine that is neither loaded nor idle, keep the engine marked running and idle while it loads (no stopped flicker), and answer GET /v1/models from the gateway while unloaded or loading. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…or loading The health check kept probing /health while the engine loaded again after an idle unload. The gateway forwards that probe to the engine for the start's readiness check, so it failed for the whole load and raised 'failed its health probe', which cleared itself once the load finished. Skip the probe while the engine is idle or loading, and test that no warning is raised while one stands. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
…out settable Start, restart and the restore after PAIR restarts now bring an engine with a front up idle, with nothing loaded, as Ollama and llama.cpp start a server: about a second, the engine ON and healthy, the model loaded by the first request or a Load. The launch is still validated and the ports checked, so a launch that cannot work is refused at Start. Applying launch settings and switching or loading the model still load it at once, so a bad setting is found and rolled back. Add get_idle_timeout and set_idle_timeout (front_idle) actions: the limit is read and changed in seconds, 0 for never, applies at once and is saved in the engine's override file. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
The status of an engine that is on with nothing loaded now carries idle, and the card reads 'Idle · loads on next request'. For vLLM the model row's expiry control becomes the idle limit (1 minute to 1 hour, or stays loaded), read from and sent to the engine manager as seconds through get_idle_timeout and set_idle_timeout; a limit that is not offered, such as one set by editing the override file, shows as the nearest one. A new engine-level list marks which engines use the expiry this way, and a test keeps it and the capability in step. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Unload an idle vLLM after five minutes, not ten, and show five minutes as the limit until the engine manager reports the real one. Update the docs. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a darwin/arm64 platform to the vLLM manifest. The install fetches the sha256-pinned uv, creates a managed Python 3.12 venv and installs the pinned vLLM 0.32.0 core and vLLM-Metal wheels, each checked against a digest. It refuses early on non-arm64, macOS older than 15, or under 10 GB free. Allow the install on macOS in the desktop engine capabilities, update the tests that assumed Linux only, and document the new platform. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
vLLM serves one model per process, so a request naming another downloaded model used to be refused: only the loaded model was advertised. The gateway in front of vLLM now reads the request's JSON "model" field and, for a different downloaded model, lets the requests already running finish, unloads the model, saves --model, loads the new one behind the same gateway, and answers the held request. Requests that arrive together share one switch. A model that is not downloaded gets a 404 and nothing changes. A switch that fails is not rolled back: the request gets a 503 naming the model and the reason, and vLLM is left unloaded and ON, so the next request loads again. vLLM now advertises every cached model, including while idle. The gateway's /v1/models names them all, and loaded_models asks for /v1/models?resident=1 so only the loaded one is reported as resident. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
- Re-check for a switch under the gateway lock before admitting a request, so a request cannot start on a model that is being unloaded or load the old one while the switch runs. - Verify the current model was really unloaded before saving the new one, and wait out a load that was only just starting. - Leave a name that is not a downloaded model (such as a --served-model-name alias) to the engine instead of answering 404. - Read the downloaded-model list once per interval for all requests, keep the last good list when a read fails, and bound the model-list body separately from the request peek. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
95d8ec6 to
7c0f178
Compare
Several docs described vLLM as it was before later commits in this series: - vLLM also runs on Apple Silicon macOS, not only Linux. - Every cached model is advertised, and a request for one that is not loaded switches vLLM to it; the docs still said only the served model was advertised and other requests were refused. - The PAIR gateway listens on 8001 with vllm serve behind it on loopback 18001; the docs still put vllm serve on 8001 and said vLLM had no load, unload, pull or delete actions. Also document GET /v1/models?resident=1, and say "vLLM engine" rather than "vLLM backend" in the services parity table. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Add a probe that asks the installed vLLM which of its tool-call parsers reads a model's tool-call format. It runs with vLLM's own Python, renders the model's chat template with one example tool call, and offers that text to every registered parser; a parser matches when it extracts exactly that call. A tokenizer that ships response_template metadata gets vLLM's hf parser. Nothing is downloaded and no code from the model repository runs. Among parsers that read a format equally, the general-purpose ones are preferred. Results are kept per vLLM binary and snapshot for the life of the service; a failed probe is tried again on the next start. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Tool calling was enabled only for the Qwen families, with the hermes parser, so every other model answered HTTP 400 to agent clients, which send tools with tool_choice "auto" on every request. Replace the family table with the probe: a vLLM launch now gets --enable-auto-tool-choice and the parser that reads the model's own tool-call format, or no tool flags when none does. A parser named in the launch arguments is still used as given. Checked against vLLM 0.32.0: Qwen2.5 and Qwen3 get hermes as before; Qwen3.5 and Granite 4.2 now get qwen3_coder; Nemotron Nano v2 and Qwen2.5-VL get none. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
… parser A model that no vLLM parser reads (Nemotron Nano v2, or a template with no tool support) still failed every chat from an agent client: vLLM rejects tool_choice "auto" without a parser, and such clients offer their tools on every request. The gateway now knows whether the running launch parses tool calls. When it does not, it removes tools, tool_choice and parallel_tool_calls from chat requests that leave tool_choice unset or "auto", so the model answers in plain text, and marks the response with X-PAIR-Tool-Calls. A request with tool_choice "required" or a named function is forwarded unchanged and gets vLLM's own error. Bodies are still never logged. Update the engine-manager README and the engine settings guide for the parser choice and this fallback. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
kjlubick
left a comment
kjlubick
left a comment
There was a problem hiding this comment.
waitsForEngineReadiness in nvpair-engine-manager/remoteclient.go has a branch for ollama that might also be needed for vllm with regards to remote vLLM loads being canceled too early
| settings "nvpair-shared/enginesettings" | ||
| ) | ||
|
|
||
| // Front puts a PAIR-owned HTTP gateway on an engine's public port and runs the |
There was a problem hiding this comment.
Could you explain the architectural decision to add the inference gateway in nvpair-engine-manager/front.go? From my understanding, features like inference proxying should be handled by nvpair-proxy, or at least that's what the engine-manager spec says.
There was a problem hiding this comment.
I want to +1 this question. I don't understand the need for another proxy.
There was a problem hiding this comment.
Good question, and you're both right: this conflicts with the spec. nvpair-engine-manager/spec.md puts inference traffic out of scope ("this service never proxies /api/chat etc."), and the gateway does exactly that. I updated the README but not the spec, and I should have raised the design before building on it.
Why it's there: vLLM serves one model per process, loads it at startup, has no load/unload API, and holds most of the GPU until the process exits. The other engines can load a model on the first request, free memory when idle, and serve any model by name. To give vLLM the same behaviour, something has to see each request and start, stop or relaunch the process. I put that next to the lifecycle code in the engine manager, but in hindsight it's the wrong layer. It also has a practical gap: a request that triggers a large model load can outlast the proxy's 120 s response timeout.
Proposal: take the gateway out of this PR.
- This PR lands vLLM as an engine the user manages directly. Load restarts vLLM with the new model, and Eject stops it and frees the GPU. Hugging Face pull/list/delete, desktop support and tool-call parser selection stay; tool-call parsers are chosen at launch, with no request rewriting.
- Removed from this PR:
front.go, idle unload, wake on request, switch on request, removingtoolsfrom requests, and the idle-limit setting. Only the loaded model is advertised for routing. - On-demand loading gets a separate design discussion. The likely shape: the engine manager detects idleness from vLLM's own
/metrics, with no traffic through it, and the vLLM facade innvpair-proxy, which already sees every request, holds a request and asks the engine manager through the broker to load the model. vLLM's sleep mode is worth evaluating there too.
Does that direction work for you, or would you prefer something else? I'll hold off on the rework until we agree.
There was a problem hiding this comment.
@Noah-Tervalon-Nvidia, the reply above covers your +1 too. In short: the gateway was added so vLLM could load and unload models on demand like the other engines, but it doesn't belong in the engine manager, so I've proposed removing it from this PR and leaving Load and Eject to restart and stop vLLM directly. Are you OK with that direction? I'll hold off on the rework until you and @kjlubick agree.
There was a problem hiding this comment.
I like the direction of scoping this down. I also don't know that we should even be doing the gateway at all (even in another PR). It seems like by doing that we start to lose the core idea that we are just a passthrough. That gateway doing work of managing what's utilized etc. is basically a wrapper around vllm managing it, which seems outside of our scope.
There was a problem hiding this comment.
Thanks, agreed. I'll remove the gateway from this PR, so vLLM is handled like a plain server: Load sets the model and restarts vLLM, Eject stops it, and nvpair-proxy talks to vLLM directly.
One question for later, not for this PR. Without the gateway, vLLM keeps the GPU until someone ejects it, unlike llama.cpp, where we pass the engine's own --sleep-idle-seconds 300. vLLM has no supported equivalent yet; its sleep mode needs VLLM_SERVER_DEV_MODE=1. Would a small idle stop in the engine manager fit the scope? It would read vLLM's /metrics and stop vLLM after N idle minutes, without proxying any traffic. If not, we'll leave it manual and wait for vLLM to support this itself. I'll open an issue only if you think it's worth discussing.
I'll push the rework here and re-request review from you and @kjlubick.
engine:get-installed listed every manifest, and reported an engine with no block for the host's OS and CPU, such as vLLM on Windows, as an empty not-installed entry. The desktop and TUI then offered an install that could only fail. Leave such an engine out of the listing, and make engine:status return an error for it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
A node's engine manager now leaves out an engine the node cannot run. Once a node has listed its engines, mark the missing ones unsupported and give them no row in the engine list, tray, endpoints or first-run setup. This hides vLLM on Windows, while a Linux node viewed from Windows still shows it. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
11269f5 to
a0eff8f
Compare
Noah-Tervalon-Nvidia
left a comment
Noah-Tervalon-Nvidia
left a comment
There was a problem hiding this comment.
I reviewed everything up through 4e8d2f4 (where you add the front.go). Since I don't really understand the rationale for adding that I skimmed the rest and they all rely on that so going to hold off any further comments.
|
|
||
| // hfDroppedSuffixes are file types a vLLM-served model never reads. Skipping | ||
| // them keeps a repo that ships several formats from downloading each of them. | ||
| var hfDroppedSuffixes = []string{".gguf", ".onnx", ".h5", ".msgpack", ".ot", ".tflite"} |
There was a problem hiding this comment.
Just asking the question: is this something we want to do? If a user pulls a model they probably expect it to be all there in case they want to go use something else with that model that they assume is all on disk. If they try to run it and parts seem to be missing that might be misleading.
There was a problem hiding this comment.
Yes, this is deliberate. A pull downloads what the engine will load, not the whole repository, which is how every engine PAIR manages already behaves: llama.cpp fetches a single GGUF quantization (repo:Q4_K_M), Ollama pulls a single tag, and lms get downloads one variant. vLLM has no download command of its own, so PAIR makes that choice in hfpull.go, and it is the same choice vLLM makes when it downloads a model on first start: the weight format it loads (safetensors when present, otherwise .bin) plus config and tokenizer, skipping original/ checkpoints and other formats. Fetching everything would often multiply the size; Llama 3.3 70B Instruct, for example, carries about 141 GB of safetensors plus about 140 GB of original/ checkpoints.
On using the model with something else: the files sit in the standard Hugging Face cache, which is per file rather than a copy of the repository, so transformers, hf download or another Hugging Face–based tool reuses what is there and fetches only what it is missing. The catalog shows the size of the weights being downloaded, not of the whole repository. I agree it should not be implicit, though, so the action's description and the README now state that it downloads the files vLLM loads and that other formats in the repository are not fetched.
There was a problem hiding this comment.
awesome, thanks for the explanation
There was a problem hiding this comment.
Don't we skip consolidated weights but vLLM includes/prefers them? So it's not quite the same. in some circumstances.
| if err := os.RemoveAll(repo); err != nil { | ||
| return nil, fmt.Errorf("delete %s: %w", id, err) | ||
| } | ||
| _ = os.RemoveAll(filepath.Join(hubDir, ".locks", name)) |
There was a problem hiding this comment.
is there a reason not to just delete the directory and everything below it (not following sym links)?
There was a problem hiding this comment.
That is what happens for the repository itself: its folder is removed with os.RemoveAll, which deletes symlinks without following them. The additional steps cover files that Hugging Face keeps outside that folder. Download locks live in hub/.locks/<repo>, so they are removed separately. Recent Hugging Face tooling also stores the weight files in a store shared by all repositories (hub/blobs/<xx>/<hash>) and leaves only symlinks in the repository's own blobs directory. Deleting the folder alone would therefore leave the weights on disk, so the delete also removes any shared files that no remaining repository links to. Files that another model still uses are kept.
There was a problem hiding this comment.
gotcha, makes sense.
There was a problem hiding this comment.
Doesn't the implementation use a glob, ignoring Dataset and Space references? How do we prevent deleting one model from removing a shared file some other process needs?
There was a problem hiding this comment.
Could we delegate deletion and shared-blob cleanup to the installed Hugging Face cache API, keeping PAIR’s running-model checks and selection handling? That would preserve Delete’s intended behavior while avoiding our own implementation of cache references and locking.
| settings "nvpair-shared/enginesettings" | ||
| ) | ||
|
|
||
| // Front puts a PAIR-owned HTTP gateway on an engine's public port and runs the |
There was a problem hiding this comment.
I want to +1 this question. I don't understand the need for another proxy.
…tore the previous model A remote load used the ordinary 30 s response-header budget for every engine except Ollama. Loading a vLLM model restarts vLLM on it, which can take minutes, so the caller gave up, the peer's handler was cancelled mid start, and the rollback to the previous model ran under the same cancelled context and failed at once, leaving vLLM down with the new model saved. Remote vLLM loads now use the readiness-sized client, and the previous model is restored under a context the caller cannot cancel. A test cancels a slow switch and checks that the previous model is back and running. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Replying to this review point. @kjlubick Thanks, good catch. Confirmed and fixed in 5c27d65. For vLLM, The fix:
Verified across two clustered machines with a real vLLM (RTX 6000 Ada → RTX A6000 over LAN mTLS):
|
167fedb to
13666dc
Compare
(review) Thank you for the detailed review. I've replied to each of your comments in its own thread, and the resulting changes are in the last commit. On |
|
Follow-ups from the review, kept out of this PR To keep this PR focused, these improvements came out of the review but are deferred. I'll open an issue for each once the PR merges.
Pending in this PR: removing the vLLM gateway ( To verify before merge: the Qwen3.5-4B default on Apple Silicon through vLLM-Metal, and launch sizing on a DGX Spark. Let me know you thoughts / comments on this |
- Match Hugging Face shard file names without a leading ^.+ in hfShardRe. - Drop the redundant shard-count comparison; the loop over shard numbers is the real completeness check, and a test now covers shards numbered from zero. - Allow deleting the model a stopped vLLM is set to serve. The delete takes --model out of the saved launch arguments, and an engine with a launch_model action refuses to start while its launch names no model, rather than letting vLLM download a default of its own. Loading a model selects one again. The model a vLLM process serves is still refused; an engine ejected behind its gateway is stopped first, so Eject then Delete works. - Say in hfpull.go that the download is done by Hugging Face's hf CLI, and state in the pull_model description and README that it fetches the files vLLM loads rather than the whole repository. - Fold setting and removing a launch flag into one helper, dropLaunchFlag. - Make Qwen/Qwen3.5-4B the default vLLM model in place of Qwen2.5-0.5B, state that it needs a GPU with at least 16 GB, and raise the install's free-disk check to cover the engine and that model (25 GB on Linux, 20 GB on macOS). - Size vLLM launches on unified-memory machines such as DGX Spark, where nvidia-smi reports [N/A] for GPU memory, from /proc/meminfo, as nvpair-node-info already does. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
13666dc to
74f8e14
Compare
Resolve two conflicts with develop: - services/nvpair-engine-manager/README.md: keep develop's shutdown-sweep paragraphs and this branch's signal and engine:status paragraphs. - services/nvpair-ui-broker/enginesettings_test.go: use develop's reserveSettingsPort for the harness ports, keeping the fifth port this branch added for the vLLM proxy. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
Inference traffic belongs to nvpair-proxy, and the engine manager never proxies it (spec.md). The gateway in front of vLLM broke that rule and had PAIR manage vLLM's model use itself, so it is removed, as agreed in review. vLLM is now handled like a plain server the engine manager starts and stops: Load saves --model and restarts vLLM, restoring the previous model if the new one fails, and Eject stops it. Start loads the model straight away, and only the model being served is advertised, read from vLLM's own /v1/models. Removed with it: the idle unload and its limit actions, loading on the first request, switching models on request, and the fallback that took tools out of requests to a model with no tool-call parser; vLLM now answers those requests with its own error. The tool-call parser is still chosen per model at start. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
The engine manager no longer unloads vLLM when it sits idle, so the card's "Idle · loads on next request" label and the model row's idle-limit control go, along with the idle field on the engine status. Eject stops vLLM and Load starts it on the chosen model. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
The spec merged from develop names the Ollama, LM Studio and llama.cpp facades only. vLLM's facade is handled like llama.cpp's: a vllm-proxy: namespace relayed to nvpair-proxy, the default engine forwarder, and a facade on :8000 or a safe fallback with no compatibility-port gate. Signed-off-by: Tushar Khinvasara <tkhinvasara@nvidia.com>
| // Not downloaded yet, so its size and limits are unknown. Only the memory | ||
| // share can be bounded sensibly. | ||
| if haveGPU && !userUtil && !userKV { | ||
| util := math.Floor(math.Min(0.5, float64(free-freeMargin)/float64(total))*100) / 100 |
There was a problem hiding this comment.
I looked at Qwen/Qwen3.5-4B (the model mentioned in the README). "The model is about 9.3 GB of weights, so the default needs a GPU with at least 16 GB of memory; on a smaller GPU." We set the memory to be half of what we have, so even if the user does have 16 GB of memory, we cap it at 8 and I bet that leads to failures or an unnecessary retry because the model is bigger than that. Could we obtain the model metadata before calculating the first launch’s memory budget? Could we ensure the first launch gets an appropriate budget and cover this empty-cache case with a regression test?
|
|
||
| // hfDroppedSuffixes are file types a vLLM-served model never reads. Skipping | ||
| // them keeps a repo that ships several formats from downloading each of them. | ||
| var hfDroppedSuffixes = []string{".gguf", ".onnx", ".h5", ".msgpack", ".ot", ".tflite"} |
There was a problem hiding this comment.
Don't we skip consolidated weights but vLLM includes/prefers them? So it's not quite the same. in some circumstances.
| @@ -0,0 +1,349 @@ | |||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
launchFlagValue seems to return the first thing it finds, not both/all
| if err := os.RemoveAll(repo); err != nil { | ||
| return nil, fmt.Errorf("delete %s: %w", id, err) | ||
| } | ||
| _ = os.RemoveAll(filepath.Join(hubDir, ".locks", name)) |
There was a problem hiding this comment.
Doesn't the implementation use a glob, ignoring Dataset and Space references? How do we prevent deleting one model from removing a shared file some other process needs?
| // is not what a peer or the proxy may route to, so it is not queried here. | ||
| if mf.AdvertisedModels == advertisedModelsLoaded && loadedSpec != nil { | ||
| advertiseLoaded[i] = true | ||
| listSpec = nil |
There was a problem hiding this comment.
Could we also have remote desktop and TUI model management read the cached inventory? These views currently use the advertised list of loaded models, so ejecting the model empties the management list even though the files remain cached. Users then lose the rows needed to Load or Delete those models.
| if err != nil || json.Unmarshal(raw, &cfg) != nil { | ||
| continue | ||
| } | ||
| return dir, cfg, bytes, true |
There was a problem hiding this comment.
Could we select the snapshot matching the revision vLLM will load? For example, the cache could contain:
refs/main → bbb222
snapshots/aaa111/config.json # older revision: 8K context
snapshots/bbb222/config.json # current revision: 128K context
os.ReadDir returns entries in filename order, so this selects aaa111 even though cached main points to bbb222. An explicit --revision bbb222 is also ignored here. Automatic configuration can therefore derive context, memory, and tool-parser settings from a different revision than vLLM loads. Could we resolve the intended revision and skip automatic configuration when its snapshot isn’t cached?
| if err != nil { | ||
| return toolParserProbe{}, err | ||
| } | ||
| ctx, cancel := context.WithTimeout(context.Background(), toolParserProbeTimeout) |
There was a problem hiding this comment.
Instead of using context.Background() here, can you pass in a parent context? We want to be able to interrupt this via a higher level cancel or StopAll.
bringUpProcess(ctx, ...)
→ applyAutoLaunch(ctx, ...)
→ planToolCalling(ctx, ...)
→ detectToolParser(ctx, ...)
→ runToolParserProbe(ctx, ...)
| return true | ||
| } | ||
| if len(weights) == 1 { | ||
| return true |
There was a problem hiding this comment.
Could we restrict the single-file shortcut to unsharded weights? A lone model-00001-of-00004.safetensors currently passes the completeness check even when the other three shards are missing, so an incomplete download can appear ready to load.
Could we also add a test where only model-00001-of-00004.safetensors is present and verify that the model is considered incomplete?
| if err := os.RemoveAll(repo); err != nil { | ||
| return nil, fmt.Errorf("delete %s: %w", id, err) | ||
| } | ||
| _ = os.RemoveAll(filepath.Join(hubDir, ".locks", name)) |
There was a problem hiding this comment.
Could we delegate deletion and shared-blob cleanup to the installed Hugging Face cache API, keeping PAIR’s running-model checks and selection handling? That would preserve Delete’s intended behavior while avoiding our own implementation of cache references and locking.
| // not fit in GPU memory, or the id does not exist) the previous model is | ||
| // restored, so a failed switch does not leave the engine down. A stopped engine | ||
| // only has its saved launch text changed. | ||
| func (e *Executor) switchLaunchModel(ctx context.Context, engine string, spec *ActionLaunchModel, params json.RawMessage) (json.RawMessage, error) { |
There was a problem hiding this comment.
Could we extract the shared launch-change machinery behind ConfigureLaunch and reuse it for model switching and selection clearing? This could remove repeated snapshot/configure/start/restore sequences and hold one engine lock across the operation, while preserving each operation’s existing failure policy.
|
this looks good to me once we address Kaylee's comments |
Description
Adds vLLM as a supported engine alongside Ollama, LM Studio, and llama.cpp. PAIR installs and manages vLLM and passes requests straight to it; it does not load or unload models on its own.
hf, and Delete removes a cached model. vLLM serves one model per process, so Load saves the model into the launch arguments and restarts vLLM on it, restoring the previous model if the new one does not start, and Eject stops vLLM, which frees its GPU memory. Peers and the proxy are told only about the model vLLM is serving.toolswithtool_choice: "auto"on every request, which vLLM rejects with HTTP 400 unless it was started with a tool-call parser. PAIR now picks the parser from the model itself: it renders the model's chat template with one example tool call and keeps the installed vLLM's parser that reads it back exactly (hfwhen the tokenizer shipsresponse_template). No list of model families is kept, nothing is downloaded, and no code from the model repository runs. When no parser fits, tool calling stays off and vLLM answers a request that asks for tool calls with its own error.Other engines keep their current model and request behavior. Shared code changes in these ways:
engine:statusandengine:get-installedno longer wait for a running lifecycle operation.hf_cache,hf_pull,launch_model, andstop_engine, which only the vLLM manifest uses.engine:get-installedleaves out an engine whose manifest has no block for the host's OS and CPU, where it used to report an empty not-installed entry, andengine:statusreturns an error for it. The desktop marks an engine missing from a node's listing asunsupportedand gives it no row. Today this affects only vLLM.Release intent
Changelog title
vLLM engine support
Changelog body
Bumps
Scope
Included: the vLLM engine (manifest, installer, auto-sizing, tool-call parser choice), model download, load, eject and delete, desktop UI, proxy/broker/TUI wiring for the new engine, docs.
Excluded: loading or unloading vLLM models automatically (on a request, or after an idle time), which an earlier revision did through a gateway in the engine manager; it was removed after review because inference traffic belongs to
nvpair-proxyand PAIR passes requests through. Also excluded: loaded-node preference innvpair-proxy(cross-engine, separate change); fixing vLLM memory sizing for some multimodal/quantized models.This is a large PR, so it is organised as focused commits that can be reviewed one at a time. The groups below are in commit order. The gateway commits in groups 3–6 are undone by the gateway-removal commits in group 10, so a reviewer can skip them and read the final state of
services/nvpair-engine-managerinstead:hf_cachemanifest action, which lists Hugging Face cache models.launch_modelmanifest action, which switches an engine's model through its launch flag.front.goand its tests, the idle unload, loading on the first request, switching models on request, the tool-removal fallback and the idle-limit actions are removed; vLLM advertises only the model it serves again. The desktop drops the idle state and the idle-limit control. The broker spec, merged fromdevelopmeanwhile, now lists the vLLM facade.Validation
go test ./...inservices/nvpair-engine-manager,services/nvpair-proxyand the other touched modules, all passing after the gateway removal.npm run lint,typecheck,test:unit,dead-code:check,service-contracts:checkindesktop/.go test ./...inservices/tests(cross-process, real binaries), run with no other PAIR instance on the machine.node scripts/spdx-headers.mjs.qwen3_coderand Qwen3-8B withhermes, and each returns a parsedget_weathercall, plain and streamed; Nemotron Nano 9B v2 gets no parser, answers plain chat, and a tool request gets vLLM's own 400.engine:restore-enabledbrings vLLM back on the model it was serving, and after a Stop it stays off.--modelcleared, a start is refused with no download, and loading another model starts vLLM and answers a chat request.vllm serveitself on the engine port, and it answers; Load switches to Qwen2.5-0.5B, which answers, including a tool request withtool_choice: "auto"; only the served model is advertised, and nothing while stopped; a request for another cached model gets vLLM's 404 and changes nothing; Eject stops vLLM and frees the GPU (7.9 GB → 18 MB); Load starts it again; Delete after Eject clears--model, Start is then refused with nothing downloaded, and Load recovers.NVPAIR_LIVE_VLLM_BIN=… NVPAIR_LIVE_VLLM_TOOL_MODELS=… go test -run TestToolParserProbe_FindsTheParserWithTheInstalledVLLM):hermesfor Qwen2.5-0.5B, Qwen3-8B and Qwen3-32B,qwen3_coderfor Qwen3.5-2B, Qwen3.5-4B and Granite 4.2 8B, none for Nemotron Nano 9B v2 and Qwen2.5-VL-7B NVFP4.Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.jsonis written by automation — do not edit it by hand.