mesen (目線) — UI Evaluation Harness

Capture UI evidence, check measurable defects, and compare model judgments with the evidence. The bundled local judge is a ViT-B/16 ONNX classifier; Qwen and specialist UI/UX teacher paths are research code, not the default CLI inference path.

---

1. What does mesen run today?

The local mesen judge command loads models/onnx/mesen_jev_vlm.onnx. Its neural input is one screenshot resized to 224×224. A separate evidence evaluator consumes validated witness state and can reject a measured defect even when image logits disagree. Document-level horizontal overflow is a hard responsive defect; child containers that scroll horizontally remain informational. One screenshot cannot establish semantic consistency across breakpoints; without a measured document defect, the responsive answer abstains.

The harness contract is: capture screenshots and browser facts, validate the witness, apply measurable checks, then record the model answer and any disagreement. Its usefulness depends on real-browser holdouts and human review of false negatives and confidence, not on the backbone name.

---

2. Core Architecture

[ Browser screenshots + measured viewport/DOM facts ]
                         │
                         ▼
             [ Validated witness state ]
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
      [ ViT-B/16 ONNX ]      [ Evidence evaluator ]
      one 224px image       geometry and rule registry
              └──────────┬──────────┘
                         ▼
          [ Typed answers + consultation ]

---

3. Decision Contract Schema

The harness returns these typed answers. Measured evidence can override or abstain from an image-only prediction:

Dimension	Type	Description
primary_action_reachable	choice (yes/no/unknown)	Are next primary actions and controls visible and reachable at all viewports?
visual_integrity	choice (yes/no/unknown)	Are controls free of clipping, overlapping, and obstruction?
responsive_consistency	choice (yes/no/unknown)	Does workflow meaning and layout remain consistent across breakpoints?
evidence_consistency	choice (yes/no/unknown)	Do screenshots agree with deterministic contract, accessibility & geometry facts?
operator_clarity	choice (yes/no/unknown)	Is the next operator action and its consequences clear from the UI?
overall_quality	score (0..3)	0 = Blocked/Unsafe, 1 = Confusing/Workaround, 2 = Usable/Clear, 3 = Excellent

---

4. Research model paths

The repository contains Qwen3.5-2B training/export code and names afx-team/UI-UX and inclusionAI/UI-Venus-2-9B as candidate teachers. Those names do not establish which checkpoint produced a local judgment. Record the exact checkpoint, training manifest, validation receipt, and inference path before claiming teacher distillation or model performance.

---

5. Deployment Options

Option A: Experimental GPU server

The server code exists, but mesen judge --remote is not implemented. Use a separately validated server client before treating this as a release gate:

mesen serve --host 0.0.0.0 --port 8088 --checkpoint models/onnx/mesen_jev_vlm.onnx

--checkpoint is a path to an ONNX graph, not a safetensors file; the judge and the two PP-OCR models are read from that graph's own directory. The local CLI rejects --remote explicitly. mesen serve runs the judge on CPU — measured 0.3–1s per 224px screenshot on a 2-thread session, not sub-millisecond. mesen export is not implemented and writes nothing; use scripts/export_system1_onnx.py.

Option B: Local ONNX Runtime (CPU)

Use the bundled ONNX checkpoint with a captured witness and screenshot:

mesen judge --state state.json --images 375.png

--state is required; --images is repeatable and takes the first readable file.

---

Setup

The ONNX artifacts are gitignored, so a fresh clone has no judge. Any mesen command that loads the model will fetch and verify them on first use. To stage them ahead of time:

uv sync --extra dev
mesen fetch-models          # or: uv run python -c "from mesen.hub import ensure_models; ensure_models()"

src/mesen/hub.py is the only place that knows which artifacts exist and what they must hash to. It resumes an interrupted transfer, skips a file that already matches its pin, and raises rather than proceeding with a file whose SHA-256 does not match. Artifacts are served from the models-v1 release; override the destination with MESEN_MODELS_DIR. Without an override they land in models/onnx inside a checkout, and ~/.cache/mesen/models/onnx otherwise.

CI stages through the same hub, so a runner and a developer resolve artifacts identically. If the artifact is missing, the judge-backed tests fail rather than skip.

---

Development / QC

uv sync --extra dev   # install ruff + pytest
bash scripts/qc.sh    # ruff check + format check + pytest (CI runs the same)

System 1 Consultant Head Training

python scripts/train_system1_consultant.py

The training gate enforces monotonic improvement against the held-out validation baseline (n_val=113, floor val_mean_f1 >= 0.6018, overall_quality_acc >= 0.7699).

• If validation metrics beat the baseline, the active checkpoint and receipt are updated.
• If the validation dataset size changes (n_val != 113), the gate fails closed and writes candidate artifacts (candidate_consultant_heads.pt and candidate_consultant_receipt.json) without touching active artifacts.
• To ratify a new baseline after modifying the validation set, inspect the candidate metrics and promote them:

  cp models/onnx/heads_v1/candidate_consultant_heads.pt models/onnx/heads_v1/consultant_heads.pt
  cp models/onnx/heads_v1/candidate_consultant_receipt.json models/onnx/heads_v1/consultant_receipt.json

Ruff config lives in pyproject.toml ([tool.ruff]); pytest in [tool.pytest.ini_options]. Pydantic models stay snake_case — camelCase wire names survive only as field aliases.

---

6. License

MIT License.