Skip to content

Local AI: Effort Off doesn't stop llama-server thinking, and newer gateways return HTTP 500 for unsupported levels #1976

Open

Description

@joelagnel

Summary

Reasoning control is broken for Local AI (the llama-server the app launches and registers as the llamacpp provider):

  1. Effort Off / /think off does not stop thinking. The model keeps reasoning; the gateway only hides the output, so Off still costs the full thinking time.
  2. Newer gateways can fail outright. Gateway 2026.10.5 beta sends reasoning_effort to local OpenAI-compatible providers by default. With the app's current provider entry, /think minimal on Qwen3.8 returns HTTP 500 from llama-server.

Root cause

  • LlamaServerRouterConfiguration.cs:295 writes reasoning = on into the llama-server preset. With --reasoning on, llama.cpp ignores reasoning_effort: "none", so the per-request off switch has no effect. This is an upstream llama.cpp bug (still on master); under the default --reasoning auto, "none" turns thinking off for every template in the Local AI catalog.
  • LocalAiGatewayProviderDefinition.BuildProviderJson declares reasoning = true but no compat.supportsReasoningEffort / supportedReasoningEfforts. The gateway therefore either sends nothing the server honors (older gateways) or sends levels the model template does not support, such as minimal (2026.10.5 beta).
  • GatewayActionTransport.cs:122 sends thinking: "low" with every canvas (A2UI) action. The gateway persists that level on the chat session, so one canvas action silently switches the user's Effort to Low and turns Off back into thinking.

Proposed fix (standard reasoning_effort, not model-specific)

Use OpenAI's reasoning_effort field, which llama-server implements and the gateway already supports per model, with Off sent as "none":

  1. Drop reasoning = on from the llama-server preset (fall back to the default auto).
  2. Declare per-model effort support in the provider entry, sourced from the model catalog for every entry:
    • Qwen3.8: supportedReasoningEfforts: [none, low, medium, high], hide Minimal (its template has low / medium / xhigh, and accepts high as xhigh).
    • Qwen3.6 and Qwen3.5: [none, medium], hide Minimal and High (on/off only).
    • "none" must be listed, or the gateway drops Off.
    • Do not hide Low: the gateway rejects an explicitly requested hidden level rather than clamping it, and paired phones request Low.
  3. Stop canvas actions from sending thinking: "low".

Resulting pickers: Off / Low / Medium / High / Ultra for Qwen3.8, Off / Low / Medium / Ultra for Qwen3.6 (Ultra is a gateway mode and cannot be hidden).

Both app config write paths (config set --batch-file and config.patch) preserve the hidden-level settings and pass the exact readback check. Existing installs need a provider-entry migration so an upgrade does not hit the "changed outside the companion" repair state.

Trade-offs to decide

  • Default effort drops for Qwen3.8 from xhigh to medium (the gateway default). In return, switching between Medium and Off reuses the prompt cache instead of re-prefilling the agent prompt (about 21 s on an RTX Spark), because the Qwen3.8 template writes the effort level at the top of the system prompt.
  • Gateway floor rises to v2026.6.5, since older gateways reject the level-hiding setting. Only native-gateway installs are affected; WSL setup installs the latest gateway.
  • Phones can still switch Effort to Low through voice transcripts and the iOS share extension. That needs an upstream gateway change.

Validation so far

Proven end to end through real gateways 2026.9.8, 2026.10.2 and 2026.10.5 beta on Linux with an RTX 4090, using a small stand-in model loaded with the real Qwen chat templates.

Not yet tested: the real catalog models, Windows, the app's own Effort UI and canvas path, and llama-server router mode under auto.

Activity

  1. added
    bugSomething isn't working
    local-modelsRelated to management and serving of local model inference
    on Nov 2, 2026
  2. added
    clawsweeper:intake:bugPreliminary submission classification; full source review pending.
    clawsweeper:intake:needs-reproPreliminary submission classification; full source review pending.
    P2Normal priority bug or improvement with limited blast radius.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    impact:auth-providerThis issue is about auth, provider routing, model choice, or SecretRef resolution.
    impact:session-stateThis issue is about session, memory, transcript, context, or agent state drift.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    and removed
    clawsweeper:intake:bugPreliminary submission classification; full source review pending.
    clawsweeper:intake:needs-reproPreliminary submission classification; full source review pending.
    on Nov 2, 2026
  3. clawsweeper commented on Nov 2, 2026

    @clawsweeper

    Codex review: this still needs some work.

    Summary
    Keep this issue open: current main retains the reported configuration defects, the partial fix remains unmerged, and this collaborator-authored report requires explicit maintainer judgment.

    Reproducibility: Current source establishes forced reasoning, absent effort metadata and the canvas Low override; the report and linked candidate supply runtime observations, but this read-only review ran no tests or live reproduction.

    Root-cause cluster
    Relationship: canonical
    Canonical: #1976
    Summary: This issue retains the broader Local AI effort-control problem; the existing candidate implements a qualified-Qwen and canvas subset.

    Members:

    Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

    Maintainer decision needed

    • Question: Should qualified-Qwen support ship first with an explicitly accepted default-effort policy while this issue retains non-Qwen coverage?
    • Recommendation: Accept a staged Qwen repair: Confirm the intended default behavior, resolve the remaining proof gaps in the existing candidate, and retain broader model coverage here.
    • Why: The candidate deliberately limits model coverage, and the inspected discussion records no owner acceptance of the staged scope or the report's default-effort tradeoff.

    Next step
    Review #2012 (fix: honor Effort Off for qualified Qwen local AI models) for the remaining migration and pinned-runtime proof, then confirm its default-effort policy and deferred model scope.

    Review details

    Best possible solution:

    Honor validated per-model effort capabilities while preserving session choices, credentials, external edits and recoverable managed upgrades.

    Do we have a high-confidence way to reproduce the issue?

    Current source establishes forced reasoning, absent effort metadata and the canvas Low override; the report and linked candidate supply runtime observations, but this read-only review ran no tests or live reproduction.

    Is this the best way to solve the issue?

    Catalog-backed capabilities and session inheritance fit the existing owners, but the original fixed Gateway floor should not be assumed necessary because the candidate now proposes guarded degradation on older versions.

    AGENTS.md: found and applied where relevant.

    Remaining risk / open question:

    • The candidate's native provider migration and WSL in-place migration with persisted restart recovery remain unverified through their production owners.
    • The app comparison uses llama.cpp b12372 with reasoning already unforced in both runs, leaving the router change on pinned b11320 incompletely demonstrated.
    • Acceptance of the intended default-effort behavior and deferred non-Qwen coverage remains unresolved.

    Codex review notes: model internal, reasoning medium; reviewed against 212fac114f6f.

    Label changes

    Label changes:

    No label changes.

    Label justifications:

    • P2: Optional Local AI effort control and selected inference requests are affected, without an established core-runtime outage or blocked onboarding flow.
    • impact:auth-provider: Generated provider capabilities do not constrain reasoning requests to the local model's supported vocabulary.
    • impact:session-state: Canvas requests explicitly override effort with Low, and the contributor reports that value replacing the session's selected Off state.
    Evidence reviewed

    What I checked:

    • Current default branch still forces reasoning: REST main matches checkout 212fac1; BuildPreset writes reasoning = on, and LlamaServerRuntimeService writes that preset before launching the process. (src/OpenClaw.Connection/LocalAi/LlamaServerRouterConfiguration.cs:297, 212fac114f6f)
    • Provider metadata lacks effort capabilities: BuildProviderJson declares reasoning=true, but compat contains only supportsTools and supportsUsageInStreaming, with no supported effort vocabulary or Off mapping. (src/OpenClaw.Connection/LocalAi/LocalAiGatewayProviderDefinition.cs:124, 212fac114f6f)
    • Canvas sends an explicit effort override: The production agent.request payload includes thinking=low; GatewayActionTransportTests currently require that override rather than session inheritance. (src/OpenClaw.Tray.WinUI/A2UI/Actions/GatewayActionTransport.cs:122, 212fac114f6f)
    • Provider migration must preserve ownership checks: Provider comparison requires exact object properties, and QuiesceAsync refuses a mismatched provider as externally modified; adding metadata therefore needs explicit recognition of known legacy managed definitions. (src/OpenClaw.Tray.WinUI/Services/LocalAiGatewayProviderCoordinator.cs:70, 212fac114f6f)
    • Existing candidate owns a partial repair: fix: honor Effort Off for qualified Qwen local AI models #2012 (fix: honor Effort Off for qualified Qwen local AI models) remains open at ed6d88f. Its updated body reports a v2026.11.3 managed WSL upgrade with credential retention and restart recovery, but explicitly excludes native migration, the WSL in-place migration owner, and a complete pinned-runtime comparison; Bonsai, Nemotron Bolt and custom-model effort metadata remain deferred. The proposal now has no hard Gateway version floor. (ed6d88f90217)
    • Prepared visual evidence inspected: All four prepared screenshots were inspected locally: the candidate Qwen3.8 menu omits Minimal, the comparison menu includes it, and transcripts show completed replies and selected effort states. Screenshots do not independently establish suppressed reasoning; that result comes from turutupa's reported request observations in the linked PR.

    Likely related people:

    • Chris Anderson: Raw commit 9fa43f3 adds src/OpenClaw.Tray.WinUI/A2UI/Actions/GatewayActionTransport.cs:122 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 9fa43f347703; files: src/OpenClaw.Tray.WinUI/A2UI/Actions/GatewayActionTransport.cs)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge; those execution lanes are retired.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

    Reviewed November 17, 2026, 10:25 PM ET / November 18, 2026, 03:25 UTC.

  4. github-actions commented on Nov 17, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale due to inactivity.
    Please add updated OpenClaw Windows details or it will be closed.

  5. added
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    and removed
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal priority bug or improvement with limited blast radius.bugSomething isn't workingclawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerThis issue is about auth, provider routing, model choice, or SecretRef resolution.impact:session-stateThis issue is about session, memory, transcript, context, or agent state drift.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.local-modelsRelated to management and serving of local model inference

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions