US-39 — Show me progress during a long scan #331

Activity

1. ARSH871-bot

   added this to the Phase 3 - market readiness milestone on Sep 10, 2026
2. ARSH871-bot

   assigned

   SamikaPerera

   on Sep 10, 2026
3. ARSH871-bot

   added

   size: SDays. One person, one PR.Days. One person, one PR.

   phase: 3Market readiness -- what stands between us and a product someone could buy.Market readiness -- what stands between us and a product someone could buy.

   on Sep 10, 2026
4. SamikaPerera commented on Oct 12, 2026

   

   SamikaPerera

   on Oct 12, 2026

   Collaborator

   More actions

   Measured the full GET /api/findings wait, cold, three runs per setup, on a CPU-only laptop (integrated AMD graphics, Ollama at 100% CPU, machine at 80% memory, so a loaded machine rather than a quiet bench). rtr-us5-insecure with Ollama running: median 68.0s total = 7.4s analyse() + 60.6s explanations (5 found findings, about 12s each warm). Same config with Ollama unreachable (deterministic fallback): 7.4s. rtr-us5-messy (6 found), model not resident: 163.9s including the 2.6 GB model load; warm 73.4s (two runs only). Largest fixture multi-device-10: 7.6s, but it produces no found findings so no model calls; the wait scales with found findings, not file size. _attach_remediation: 0.00s in every run. So analyse() is a steady 6-8s and the explanation loop is 89-96% of the wait whenever the model runs, and the hardcoded up-to-15-seconds text in runScan() is false in that configuration by 4.5x to 11x. Caveats: one machine, three runs per setup; Arsh has a dedicated GPU and measured about 1s per finding on #303, so the wait there is short, but CPU-only is a realistic client case. Worth re-measuring on another machine before sizing. This changes the scope: per-stage progress covers about 8% of the wait; the explanation stage is where n-of-m is countable and an honest estimate is hardest. A different option is to stop explanations blocking the findings: return findings after analyse() and fill explanations in through a separate GET (the per-concern shape from #365). That touches the findings contract, so I am raising it for a decision, not building it.

   ---

   The raw runs, so nobody has to take the medians

   setup                                    total    analyse   explanations   remediation  explained
   A  rtr-us5-insecure, Ollama up           73.59s     7.85s        65.73s        0.00s    5 model
                                            68.01s     7.43s        60.56s        0.00s    5 model
                                            60.57s     7.12s        53.44s        0.00s    5 model
   B  rtr-us5-insecure, Ollama unreachable   7.39s     6.98s         0.40s        0.00s    5 fallback
                                             7.39s     6.96s         0.41s        0.00s    5 fallback
                                             7.62s     7.20s         0.40s        0.00s    5 fallback
   C  multi-device-10 (largest), Ollama up    7.82s     7.78s         0.00s        0.00s    0
                                              7.59s     7.56s         0.00s        0.00s    0
                                              7.57s     7.53s         0.00s        0.00s    0
   D  rtr-us5-messy, 6 found, model cold    163.88s     6.32s       157.54s        0.00s    6 model
      rtr-us5-messy, 6 found, model warm     73.35s     6.07s        67.27s        0.00s    6 model
   

   Method: one fresh Python process per run, so _analysis_cache and
   _explanation_cache start empty by construction rather than by being cleared
   — the harness asserts both are empty before issuing the request. Timed by
   wrapping the three module globals the endpoint resolves at call time, so the
   real endpoint runs the real path. Setup B used OLLAMA_HOST=127.0.0.1:1
   rather than stopping the service; explanation_sources: {"fallback": 5}
   confirms it took the fallback branch.

   Re-checked on today's main (9c75940), one cold run

   web/main.py has changed since I measured (#382, four lines, unrelated to
   this path), so I re-ran rather than assuming:

   83.24s total  =  6.67s analyse()  +  76.55s explanations     5 found, 5 model
   

   The absolute total moves with machine load — 83.2s today against a 68.0s
   median a week ago — and the split does not. analyse() is 6.67s, inside
   the 6–8s band every run has produced. Explanations are 92% of the wait. So
   treat the medians above as the shape rather than as a benchmark: the useful
   claim is the ratio and the order of magnitude, not the second decimal.

   Two things I would ask the team to notice

   multi-device-10 being the fastest setup is the counter-intuitive result
   and it is the one that matters for scoping. It is the largest fixture by
   some margin — ten files, 230 lines — and it finishes in 7.6s because it
   produces no found findings at all. Nothing about file size predicts the
   wait. The number of problems does.

   That means a clean network is always fast and a badly misconfigured one is
   always slow, which is exactly backwards from the user's point of view: the
   person who most needs the answer waits longest for it.

   _attach_remediation is 0.00s in all eleven runs. It is deterministic
   text with no model call. It does not belong in any progress indicator, and I
   would rather say so now than have it listed as a stage because it sits next
   to one in the code.