Repository navigation
Scientific hardening: 4-state decision engine (P0 direction fix), Bonferroni-coherent Newcombe CI, honest edge contracts - #2
Merged
Conversation
…ges (P0) Direction (P0): the engine decided with abs(diff) > MPE, labeling a significantly worse B as 'Vencedor' (reproduced: A=60% vs B=58%, n=10k/group, p=0.004, diff=-2pp). Now: Melhora (p<alpha_adj and diff>+mpe), Regressão (p<alpha_adj and diff<-mpe), Efeito Fraco (p<alpha_adj and |diff|<=mpe), Inconclusivo (p>=alpha_adj or undefined test). Adds direction field and public 'regressao' scenario. CI (P1): Newcombe interval via confint_proportions_2indep(compare=diff, method=newcomb, alpha=alpha_adj) for pB-pA; response exposes ci_level and ci_method; UI can show the real level (e.g. 98.33%). Decision and interval are now family-wise coherent (before: decision at alpha_adj but Wald CI at nominal alpha labeled 95%). Pooled score test + unpooled CI documented as a method choice in docs/STATISTICAL_METHOD.md. Edge contracts: structured 400 for non-object JSON bodies (was 500); strict integer validation rejecting bool/fractional/NaN/Inf/negative counts (was silent int() truncation); degenerate cases (0% vs 0%, 100% vs 100%) return test_defined=false, p_value=null and warnings (was NaN masked as p=1.0) with the Newcombe CI still reported; uplift=null when pA=0 (was fake 0.0); sample-size guards (positive MDE, finite inputs, <=100M per group); normal-approximation warning when min(successes, failures) < 5 per arm; GET /api/assumptions and assumptions in /api/demo.
G1 equality, G2 melhora, G3 regression guard (historical 60% vs 58% never Melhora), G4 weak effect, G5 Bonferroni flip, G6 CI vs statsmodels reference in 5 families (normal, Bonferroni, unbalanced, near-0, near-1) + ci_level, G7 non-object bodies 400, G8 float/NaN counts 400, G9 bool 400, G10/G11 degenerate without fake p, G12 baseline-zero uplift null, G13 MDE guards. Plus closed-form Cohen's h sample-size, power monotonicity, alpha sensitivity, scenario coverage and legacy validations.
…arnings Status colors for Melhora/Regressão/Efeito Fraco/Inconclusivo; direction chip (favor/against B); p-value shown as 'indefinido' when test_defined is false; uplift shows 'Não definido (baseline zero)' on null; CI displays the actual ci_level (e.g. 98.33%) and method; warnings block; assumptions details fetched from /api/assumptions; alpha input on Plan tab; scenario chips updated to the five public scenarios (melhora, regressao, efeito_fraco, inconclusivo, zero_conversoes).
New docs/STATISTICAL_METHOD.md (four states, pooled score test vs Newcombe CI rationale, Bonferroni-coherent intervals, degenerate contract, uplift baseline-zero, absolute MDE semantics, assumptions, language policy). api-contract.md rewritten for the new response. architecture-guardrails.md rewritten to the current dual-project Flask architecture (was stale FastAPI single-project with English labels). README switched to 4-state wording and canonical statlab-ab.vercel.app URL. TESTING.md documents the 47-test golden suite. DEMO_SCRIPT.md gains the Regressão step. TECHNICAL_DECISIONS.md adds ADR-006/007. CHANGELOG entry. Historical snapshots (AUDIT_REPORT, HANDOFF, PORTFOLIO_HANDOFF, PLAN, SUMMARY, portfolio-*) now carry an explicit banner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P0 — Direção do efeito (CONFIRMADO e corrigido)
O motor decidia com
abs(diff) > MPE: B significativamente pior era rotulado "Vencedor". Reproduzido no baseline85ed425(local e produção): A=60% vs B=58%, n=10k/grupo → p=0.0040, diff=−2pp → "Vencedor".Novo motor (4 estados, direção assinada
diff = pB − pA):Golden test G3 + cenário público
regressaotravam o bug.P1 — IC coerente com Bonferroni
Antes: decisão em
alpha_adjmas IC Wald com alpha nominal e UI dizendo 95%. Agora: IC de Newcombe (confint_proportions_2indep,method="newcomb") para pB−pA comalpha=alpha_adj; resposta expõeci_level/ci_methode a UI mostra o nível real (ex.: IC 98,33% com 3 comparações). Score test pooled + IC unpooled documentados como escolha de método (docs/STATISTICAL_METHOD.md).Edge cases
test_defined=false,p_value=null+ warnings (era NaN→1.0); IC segue reportadopA=0:uplift=null(era 0.0 falso)GET /api/assumptions+ premissas na UITestes
47 pytest (golden G1–G13, CI contra referência statsmodels em 5 famílias de casos) — verde local; frontend lint/typecheck/build verdes.
Docs
STATISTICAL_METHOD.mdnovo; api-contract, guardrails (estava descrevendo arquitetura FastAPI antiga), README (4 estados, URL canônica), TESTING, DEMO_SCRIPT, TECHNICAL_DECISIONS (ADR-006/007) atualizados; snapshots históricos com banner.