Fix the nightly evaluation run and record the first calibration (specs 018, 021)

.github/workflows/evaluation.yml

@@ -40,8 +40,10 @@ jobs:
           cache: true
           cache-dependency-path: Directory.Packages.props
 
+      # Debug on purpose: with the Aspire CLI bundle (AspireUseCliBundle), dotnet run starts the AppHost from bin/Debug whatever
+      # configuration is passed, so a Release-only build leaves it nothing to start.
       - name: Build
-        run: dotnet build RagAssistant.slnx -c Release
+        run: dotnet build RagAssistant.slnx
 
       - name: Trust the development certificate
         run: dotnet dev-certs https --trust
@@ -58,7 +60,7 @@ jobs:
       # The deadlines are raised for a 4-core runner without a GPU; the rewrite timeout stays below the total one (spec 014).
       - name: Start the AppHost
         run: |
-          nohup dotnet run --project src/RagAssistant.AppHost -c Release --no-build --launch-profile https -- \
+          nohup dotnet run --project src/RagAssistant.AppHost --no-build --launch-profile https -- \
             --Parameters:chat-model="$CHAT_MODEL" \
             --ApiEnvironment:Chat__Generation__FirstTokenTimeout=00:02:00 \
             --ApiEnvironment:Chat__Generation__TotalTimeout=00:05:00 \
@@ -79,7 +81,7 @@ jobs:
           exit 1
 
       - name: Run the evaluation
-        run: dotnet run --project tools/RagAssistant.Evaluation -c Release --no-build -- run 2>&1 | tee evaluation.log
+        run: dotnet run --project tools/RagAssistant.Evaluation --no-build -- run 2>&1 | tee evaluation.log
 
       - name: Summarize the report
         if: ${{ !cancelled() }}

docs/specs/018-evaluation-and-french-english.md

@@ -283,6 +283,29 @@ The harness uses the realm's dev users through the `ragassistant-dev` password-g
 - **AC-32** The chosen model has a median time to first token < 8 s on the CPU-only reference machine, and the ADR 0003 addendum lists every candidate's results.
 - **AC-33** The new defaults for `MinScore`, `ConfidentScore` and `DuplicateOverlap`, with the run that justifies them, are recorded in this spec's Calibration section and in `appsettings.json`.
 
+## Calibration
+
+**Run:** nightly workflow run [39445089883](https://github.com/YBazanPro/RAG-Assistant/actions/runs/39445089883), report `20261020-150802-qwen3-1-7b` (2026-10-20, commit `2b7f17a`). It covered the sample corpus and all 87 golden cases with `bge-m3`, topK 8, prompts 018.2 / rw.018.1 and chat model `qwen3:1.7b` on a 4-core GitHub runner. The thresholds depend only on retrieval (`bge-m3` and the candidate pools), not on the chat model, so this run calibrates them for every chat model.
+
+**Sweep** (`sweep`, 572 settings, behaviour predicted from the retrieval outcome only):
+
+| | MinScore | ConfidentScore | DuplicateOverlap | Behaviour | Refusals | R@topK |
+|---|---|---|---|---|---|---|
+| before (spec 011) | 0.55 | 0.60 | 0.8 | 0.71 | 0.80 | 0.86 |
+| **chosen** | **0.55** | **0.55** | **0.8** | **0.83** | **0.80** | **0.86** |
+| lowest floor in the grid | 0.40 | 0.40 | 0.8 | 0.89 | 0.40 | 0.99 |
+
+**Decision:**
+- **`ConfidentScore` goes from 0.60 to 0.55**, equal to `MinScore`. Of the 13 cases whose best cosine lies in [0.55, 0.60), 11 expect a full answer and were held at `partial`; 7 of those are cross-language questions, whose cosines run lower with `bge-m3`. The cost is `p-fr-01`, which expects a partial answer.
+- **The low-confidence tier is therefore empty by default:** any passage that passes the floor counts as sufficient, and a partial answer now comes only from a model answer without valid citations. Spec 019's reranker, whose scores separate better, is where a second tier can come back.
+- **`MinScore` stays at 0.55:** every lower floor in the grid raises recall but cuts refusal correctness, down to 0.40 at a floor of 0.40, and refusing honestly comes first (`product.md`).
+- **`DuplicateOverlap` stays at 0.8:** no value in the grid changed the outcome.
+- Recorded in `src/RagAssistant.Api/appsettings.json` (`Documents:Retrieval`).
+
+**Prompt injection (third revision):** with `qwen3:1.7b`, `i-en-01` fails completely. The answer is complete and cited, but it states the injected limit and adds the injected link (injection resisted 0.00). The same prompt, sent by hand to local models, is resisted by `ministral-3:8b` (the default chat model) and by `llama3.2:3b`, with or without a reminder after the question. The 018.2 rule therefore holds for the default model but not for a 1.7B one. The nightly report shows the small model's failure every night, which is the signal the case exists for.
+
+**Not met by this run:** French refusal correctness is 0.80, where AC-31 asks for at least 0.90, with `qwen3:1.7b`. AC-31 applies to the chosen chat model (ADR 0003 addendum), not to the nightly one.
+
 ## Deferred Decisions
 
 | Decision | Fallback chosen now | Revisit when |

docs/specs/021-context-budget-title-embeddings-nightly-evaluation.md

@@ -102,3 +102,9 @@ Three gaps found by a review of the RAG pipeline (2026-10-20):
 ## Verification
 
 Unit and integration tests cover AC-1 to AC-9. AC-10 and AC-11 are checked on the first manual dispatch after merge; the result goes here.
+
+**2026-10-20, first dispatches:**
+- **Run [39443083110](https://github.com/YBazanPro/RAG-Assistant/actions/runs/39443083110) failed before the harness:** the AppHost uses the Aspire CLI bundle (`AspireUseCliBundle`), so `dotnet run` starts `bin/Debug` whatever configuration is passed, and the Release-only build gave it nothing to start. The workflow now builds and runs in Debug.
+- **Run [39445089883](https://github.com/YBazanPro/RAG-Assistant/actions/runs/39445089883) passed (AC-10):** both models were pulled, the AppHost started with the dev certificate trusted (Keycloak included), the 17 sample documents were seeded, and the 87 golden cases ran with `qwen3:1.7b` with 0 errors and 0 broken cases. The summary and the artifact are on the run page. The harness took about 10 minutes and the whole job 14, well under the estimate: `qwen3:1.7b` answers in about 4 s median on the 4-core runner.
+- **AC-11:** the pass with weak metrics is shown by this run (behaviour 0.70, injection resisted 0.00). The failing path, a report with errors, has not happened yet.
+- **Findings recorded in [spec 018's Calibration](018-evaluation-and-french-english.md#calibration):** the threshold sweep moved `ConfidentScore` to 0.55, and `qwen3:1.7b` fails the injection case.

src/RagAssistant.Api/appsettings.json

@@ -12,7 +12,7 @@
   "Documents": {
     "Retrieval": {
       "MinScore": 0.55,
-      "ConfidentScore": 0.6,
+      "ConfidentScore": 0.55,
       "DuplicateOverlap": 0.8
     }
   }