Eredu for Mesh

Run a local MLX model on your Apple Silicon Mac with Eredu, served through Mesh's normal OpenAI-compatible API.

Early access: Apple Silicon (macOS arm64) only. Text chat, streaming and checkpoint-native function calls through OpenAI JSON/SSE. Tools require an Eredu registered native-tool template; unsupported templates fail closed.

Tool requests support tools, tool_choice (auto, none, required), parallel_tool_calls (default false), and thinking control through enable_thinking, reasoning_effort (none disables thinking) or chat_template_kwargs.enable_thinking — the form Mesh forwards, so these knobs work through Mesh too. Other chat_template_kwargs reach the checkpoint template unchanged. Named-function tool_choice objects are not supported and are rejected, not silently ignored. The plugin returns calls for your client to execute; it never executes tools. Replay the assistant message and a role: "tool" message with its tool_call_id to supply results. Function arguments on the wire are JSON strings.

Validated locally with an existing mlx-community/Qwen3-1.7B-4bit checkpoint: required tool call, result replay with tool_choice: "none" and a nonempty follow-up, plus streaming tool-call output (enable_thinking: false). This is single-node adapter evidence, not qualification of every model, parallel tool execution, agent harness, or remote Mesh routing. Readiness does not advertise blanket model tool capability.

1. Install

mesh-llm plugins install Mesh-LLM/eredu-plugin

This downloads the latest release built from main. It is a single binary (ad-hoc signed, not notarized). It never downloads model weights.

2. Pick a model

Point it at a local MLX safetensors model directory (with config.json, tokenizer.json and *.safetensors). For example, if you already have mlx-community/Qwen3-1.7B-4bit in your Hugging Face cache:

ls -d ~/.cache/huggingface/hub/models--mlx-community--Qwen3-1.7B-4bit/snapshots/*

3. Add it to Mesh

Add to ~/.mesh-llm/config.toml (edit an existing [runtime] section rather than adding a second one):

[runtime]
mode = "on_demand"

[[plugin]]
name = "eredu-plugin"
args = ["--model", "/absolute/path/to/model-dir", "--model-id", "qwen3-1.7b-eredu"]

Then:

mesh-llm serve
curl http://127.0.0.1:9337/v1/models
curl http://127.0.0.1:9337/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-1.7b-eredu","messages":[{"role":"user","content":"Hello"}]}'

The model loads when Mesh starts the plugin and appears in the model list once loaded. Stopping Mesh stops the plugin.

Try it without Mesh

~/.mesh-llm/plugins/installed/eredu-plugin/eredu-plugin --model /path/to/model-dir --port 8080
# then talk to http://127.0.0.1:8080/v1

Development

just verify (mock tests, fmt, Clippy), just build (native MLX build; needs Xcode with the Metal toolchain: xcodebuild -downloadComponent MetalToolchain), just package v0.1.0 (release archive in dist/), just clean.

Every push to main publishes a release (v0.1.<run>), so mesh-llm plugins install Mesh-LLM/eredu-plugin always gets the latest.

Eredu is MIT OR Apache-2.0; this plugin is Apache-2.0. Model weights have their own licences.