Skip to content

Compare local AI models for the explanation layer #408

Open

Description

@patelankeet2

As the AI-layer owner, I want a measured comparison of candidate local Ollama models for the explanation layer, so that "Warden" is a data-backed choice rather than an unexamined default.

Acceptance criteria

  • At least two other locally-servable Ollama models are run against the same held-out set tools/explanation_quality.py already scores Warden against.
  • Latency and approximate memory/resource footprint are measured for each candidate on real hardware, not estimated.
  • A written comparison names which model is recommended and why, or confirms Warden stays, with the measurement as the reason rather than a preference.

Shape

AI layer, tooling and measurement only. No change to ai/explain.py's contract or to the F-1 finding shape.

Why this is worth doing

ai/Modelfile currently commits to one model with no comparison on record. #385 and #407 already built the held-out scoring infrastructure this would reuse directly, so the marginal cost of a real comparison is low.


Epic: Candidate idea (AI layer) - not yet agreed as a specialist pick
Size: not yet estimated
Assigned: @patelankeet2

Proposed 27 October 2026 as a candidate feature idea. Not yet discussed or agreed by the team; open for refinement or rejection at the next planning session, the same status every other idea in this batch carries.

Activity

  1. self-assigned this
    on Oct 26, 2026
  2. added
    enhancementNew feature or request
    layer: aiLayer 2 — local LLM explanation and Q&A
    phase: 3Market readiness -- what stands between us and a product someone could buy.
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestlayer: aiLayer 2 — local LLM explanation and Q&Aphase: 3Market readiness -- what stands between us and a product someone could buy.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions