Skip to content

fix(goal): judge on the session's selected model (+ goal_wait gating, compact resume, model picker) - #1022

Merged
Pizzaface merged 3 commits into
mainfrom
fix/goal-session-model
Nov 5, 2026
Merged

Pizzaface merged 3 commits into
mainfrom
fix/goal-session-model

Conversation

@Pizzaface

Copy link
Copy Markdown
Owner

Summary

/goal judged most checks on the cheapest authenticated model, not the session's model. It reused the session context on the session model only until the first evaluation with cacheRead = 0, then switched to the cheapest non-reasoning text model for the rest of the session. On openai-codex that miss was close to guaranteed: the judge call never passed sessionId, which is what pi turns into prompt_cache_key.

Changes

Evaluator model (goal/index.ts, evaluator.ts, session-context.ts)

  • The judge uses the session's selected model (ctx.model) unless goal.evaluatorModel pins one.
    • With a captured session prefix, it re-sends that prefix and passes sessionId, so prompt caching can apply.
    • Without one, it sends a standalone transcript, still on the session model.
  • If the pinned model is unusable, it falls back to the session model.
  • If there is no session model or no auth, the check returns uncertain and logs a warning.
  • Removed the cache-miss downgrade (recordCacheOutcome, isCacheReuseViable) and the cheapest-model lookup (findSmallFastModels).

Other goal fixes included in this PR (also not yet in main)

  • goal_wait is only an active tool while a goal is active.
  • A manual /compact resumes an active goal once the session is idle with nothing queued. Before, the aborted run left the goal loop stalled.
  • Runner settings: new evaluator model picker. The default is "Session model"; choosing it clears any pinned model.

Docs: removed the "cheapest authenticated model" wording from features/goals.mdx and customization/configuration.mdx.

Trade-off

If the cache misses, each check pays full input price on the session model. It no longer switches to a cheap model.

Tests

  • bun test packages/cli/src/extensions/goal/: 124 pass. New tests:
    • sessionId is forwarded to the provider call.
    • With no captured prefix and an unusable pinned model, the session model is used.
    • After a cache miss, checks stay on the session model.
    • goal_wait gating.
    • Resume after /compact.
  • GoalEvaluatorSettings.test.tsx: 4 pass.

@Pizzaface

Copy link
Copy Markdown
Owner Author

!pizza - fix the checks

@Pizzaface

Pizzaface commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner Author

✅ Fixed and rebased onto origin/main (97fddbc). Pushed head: 0b6db24.

  • Retained the Ubuntu 24.04 browser-test runner pin now also on main.
  • Removed unused goal-test imports that failed lint.
  • Replaced a flaky append timing ratio with deterministic cache-reuse and output checks; verified appended tools appear. Mutation probes reject both disabled cache reuse and dropped appends.

Verification:

  • All 14 PR checks are green, including Unit Tests, all five binary targets, Android, both E2E install flows, runner image, and CodeQL.
  • Rebased local lint, typecheck, build, full test orchestration with disposable Redis, workflow structure validation, and diff checks passed.
  • Independent review: LGTM.

PR is mergeable. Not merged or deployed.

…ompact resume, evaluator model picker

- The /goal LLM evaluator always runs on the session's currently selected
  model unless goal.evaluatorModel pins one. Removed the cache-miss downgrade
  to the cheapest authenticated model; an unusable pinned model now falls
  back to the session model.
- Session-context reuse passes the pi sessionId so providers keyed on it
  (openai-codex prompt_cache_key) can hit the session's prompt cache.
- goal_wait is only an active tool while a goal is active.
- Manual /compact resumes an active goal once the session is idle.
- Runner settings: evaluator model picker (default: Session model).
@Pizzaface
Pizzaface force-pushed the fix/goal-session-model branch from c40c41e to 0b6db24 Compare November 5, 2026 18:38
@Pizzaface
Pizzaface merged commit ac3e88b into main Nov 5, 2026
14 checks passed
@Pizzaface
Pizzaface deleted the fix/goal-session-model branch November 5, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant