Repository navigation
Fix the nightly evaluation run and record the first calibration (specs 018, 021) - #18
Merged
Merged
Conversation
…(spec 021) dotnet run on an AppHost with AspireUseCliBundle starts bin/Debug whatever configuration is passed, so the Release-only build gave the first nightly run nothing to start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (specs 018, 021) From the first passing nightly run (39445089883, qwen3:1.7b, 87 cases): - the threshold sweep moves Documents:Retrieval:ConfidentScore from 0.60 to 0.55 (behaviour 0.71 -> 0.83, refusals and recall unchanged); MinScore 0.55 and DuplicateOverlap 0.8 stay. The low-confidence tier is empty by default as a result, which spec 018's Calibration records. - qwen3:1.7b follows the injected passage of i-en-01; ministral-3:8b and llama3.2:3b resist the same prompt when sent by hand. - spec 021's Verification records both dispatches and the Debug-build fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the open items left after #17.
AspireUseCliBundle) makesdotnet runstart the AppHost frombin/Debugwhatever configuration is passed, so the Release-only build left nothing to start. The workflow now builds and runs in Debug. A dispatch from this branch passed: run 39445089883, with 87 cases, 0 errors and about 10 minutes in the harness.Documents:Retrieval:ConfidentScorefrom 0.60 to 0.55 inappsettings.json. Predicted behaviour goes from 0.71 to 0.83, with refusals (0.80) and recall@topK (0.86) unchanged.MinScore0.55 andDuplicateOverlap0.8 stay. Side effect, recorded in the spec: ConfidentScore now equals MinScore, so retrieval no longer produces low-confidence evidence by default.qwen3:1.7bfollows the injected passage ofi-en-01(injection resisted 0.00). Sent by hand, the same 018.2 prompt is resisted byministral-3:8b(the default) andllama3.2:3b. No prompt change; the nightly report keeps showing the small model's failure.Test plan
sweepon the downloaded reportappsettings.jsonchange does not affect them; CI runs the full suite on this PRqwen3:1.7bis below AC-31's 0.90; AC-31 applies to the chosen chat model, so it is reported, not gated🤖 Generated with Claude Code