Add opt-in prefill progress streaming (prompt_progress SSE chunks)

[eliaskg]

Is your pull request related to a problem? Please describe.

During a long prefill (e.g. 64k+ context), the SSE stream is silent until the first generated token arrives. External clients see no data for minutes and cannot tell a slow prefill from a hang. There is no way to opt into progress updates.

ExLlamaV3 already computes per-chunk prefill progress. Its Job.prefill() method appends {"stage": "prefill", "curr_progress": N, "max_progress": M} to its results after every chunk_size batch. generate_gen() receives these events and feeds them to the server console status line, but never forwards them to the SSE stream. The data is already there. It just is not exposed to the client.

Why should this feature be added?

• Parity with llama.cpp, which already offers this via its return_progress flag.
• No new computation. The change only forwards data ExLlamaV3 already produces.
• Opt-in and backward compatible. return_progress defaults to false. Existing clients see no change.
• The prompt_progress field is a vendor extension. Clients that ignore unknown fields are unaffected.

Examples

When the client sets return_progress: true, the server emits one prompt_progress chunk per chunk_size batch during prefill, matching llama.cpp's format:

{
  "id": "chatcmpl-...",
  "object": "chat.completion.chunk",
  "choices": [{"index": 0, "delta": {"role": "assistant", "content": null}, "finish_reason": null}],
  "prompt_progress": {"total": 29000, "cache": 12400, "processed": 14448, "time_ms": 200}
}

Field	Source
total	max_progress from ExLlamaV3
cache	cached_tokens from the started event
processed	curr_progress from ExLlamaV3
time_ms	elapsed since the started event (server-side)

Test it:

curl -s http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "messages": [{"role": "user", "content": "Say OK"}],
    "max_tokens": 10,
    "stream": true,
    "return_progress": true,
    "stream_options": {"include_usage": true}
  }'

A small prompt emits one prompt_progress chunk (prefill fits in one batch). A large prompt emits several chunks with increasing processed values, then content chunks, then the final usage chunk. Without the flag, the response is identical to before.

Additional context

Files changed (44 lines added, 0 removed):

File	Change
endpoints/OAI/types/common.py	Add return_progress: bool = False to CommonCompletionRequest
backends/exllamav3/model.py	Track prefill start time and cached tokens. Yield a _prefill_progress event when the flag is set
endpoints/OAI/utils/chat_completion.py	Forward _prefill_progress events through the stream collector. Serialize them as SSE chunks with the prompt_progress field
docker/Dockerfile.prefill	Thin overlay on the official image for easy testing (copies the 3 modified files)

Scope notes:

• Non-streaming requests are unaffected (progress events are skipped in the non-streaming path).
• Only the chat/completions endpoint is modified. The completion endpoint is unchanged.
• This is ExLlamaV3 only. ExLlamaV2 does not emit per-chunk prefill progress, so it is out of scope.

[eliaskg]

One practical application would be to access the prefill progress from a harness like pi:

This is especially useful on smaller machines / bigger models