Add mlx-lm as a third server, and settle the quant and peg-native anomalies

[devYRPauli]

M6. Adds a third server, retracts one published reading, and closes the two
outstanding anomalies at the replication bar. 230 measured runs behind it.

Retraction first

M5 measured nine quantization arms once each and read Qwen2.5-7B's 47/45/48 as
the lowest quant beating the highest, against folklore. That does not hold.

Re-running every arm five times reproduced the M5 numbers exactly, which looked
like confirmation and was not: the runner hardcoded temperature 0.0 and seed 42,
so decoding was greedy and each repeat was the same computation. All 45 runs
matched down to the identical set of failing scenario ids. Under five distinct
seeds at temperature 0.7 the three 7B quants land at 46.8, 45.4 and 46.4 with
fully overlapping ranges.

What survives is narrower: quantization level from Q8_0 to Q3_K_M does not
predict tool-calling ability on this corpus, and the ordering is not monotonic
(Qwen2.5-1.5B puts Q4_K_M below the more aggressively quantized Q3_K_M).

--seed and --temperature are now flags, defaults unchanged so existing
results stay reproducible. Amendment 7 records that greedy repetition shows
reproducibility, and only a seed-varied arm supports a claim that two arms
differ.

Decode-constraint disclosure

Required for launch and previously missing. llama.cpp compiles the supplied tool
definitions into a GBNF grammar and constrains decoding, so a call naming a
function that was never supplied cannot be sampled. Ollama and mlx-lm generate
unconstrained and parse afterwards. That favours llama.cpp in every row, so a
llama.cpp-vs-Ollama delta is a stack property, not a server defect or a model
difference. Now stated in the README and on the matrix page, and recorded per
result in server.quirk_flags. Verified for mlx-lm 0.31.3 by source inspection:
tools reach the chat template and nothing else, and parse failures are swallowed.

peg-native anomaly: replicated, excluded, not version-bound

Llama-3.1-8B loses 7-9 of 50 scenarios per run to llama.cpp returning HTTP 500,
"does not match the expected peg-native format". The model is fine; it emits
correct calls as bare JSON, often separated by "; ". Its lazy GBNF trigger never
fires, so generation is unconstrained, and the post-hoc PEG re-parse then
rejects the text and escalates to a 500 with no fallback.

Homebrew is pinned at b10050 so brew upgrade was a no-op; the arm was re-run
against the upstream b10075 binary instead. Identical on every arm and run, with
a clean Qwen control. Not version-bound. The three Llama-3.1 llama.cpp arms are
excluded with reason from the quantization conclusion and shown as such on the
site. Recorded as a candidate upstream issue, not filed.

mlx-lm, and the one real quantization effect

Six seeded rows, chosen to overlap the existing model set. Qwen2.5-7B 4bit
scores 46/50, Qwen2.5-1.5B 41/50.

Llama-3.1-8B, Mistral-7B and Phi-4-mini all score exactly 7/50, passing only the
negative traps. mlx-lm logs "model does not support tool calling" on all 50
scenarios for each, so the tools never reach the model. A gap in mlx-lm's
parser coverage, which is what a whole-stack cell should expose.

Qwen2.5-7B scores 46/50 at 4bit and 7/50 at 8bit, five runs each, identical
every run. The 8bit conversion emits doubled braces:

8bit:  {{"name": "get_weather", "arguments": {"city": "Boston"}}}
4bit:  {"name": "get_weather", "arguments": {"city": "Boston"}}

The bytes were recovered by generating through mlx_lm directly with the 4bit
conversion as control, before writing any claim. That is what turned this from
"the server dropped a call" into "the model emitted malformed JSON" - the same
correction this project had to make once before. mlx-lm's parser is correct.
The server defect that remains is the reporting: HTTP 200, finish_reason
tool_calls, and a message carrying neither content nor tool_calls, so a client
cannot tell this from silence. Same class as the Ollama silent-discard issue.
Both recorded as candidates, neither filed.

Also

• New mlx_lm preset on a dedicated port 8081, since mlx-lm's own default of
  8080 collides with the llama.cpp convention; 8081 joins the contention
  preflight.
• README documents two ways to get an mlx-lm row wrong: MLX rows are converted
  weights, so the one-blob-two-servers control does not cross that boundary,
  and mlx-lm's /v1/models lists the entire local cache rather than the loaded
  model. A first pass of the seeding script used it for discovery and filed
  three rows under the wrong model name; those runs were discarded and re-run
  behind an identity assertion.
• Site gains the MLX LM column, the decode-constraint paragraph, replication
  totals, and an excluded-rows block.

Verification

fmt, clippy and 97 tests green. All 32 result files pass the CI schema job
locally, and evidence hashes verify. Every model run was CERES health-gated on
macstudio and the guard never tripped.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 13ce3bdb2b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Align quirk metadata before claiming every result records it

I checked the published results/*.json files, and the pre-existing Ollama and llama.cpp rows still have empty metadata.server.quirk_flags. With this new wording, those rows are documented as if an empty flag set means "unverified", so readers and downstream tooling will misclassify the existing Ollama/llama.cpp measurements unless the result files are backfilled or this statement is scoped to newly generated results.

Useful? React with 👍 / 👎.