sota-verify

A 6-test verification suite for scoring formulas and evaluation metrics. Given a formula that maps empirical signals to a score, sota-verify checks whether that formula is fit for purpose — robust, well-behaved, and better than trivial baselines.

Based on the k-server-bench evaluation methodology. This fork corrects bugs in the original baseline, robustness, gradient, and axiom tests; the original suite remains available as run_all_original().

The 6 Tests

#	Test	What it checks	Pass criterion
1	Combined Score	Product ranking across metrics and mean per-metric correlation both align with ground truth	Combined Spearman ρ > 0.5 and mean per-metric ρ > 0.5
2	Robustness	Gaussian noise at 4 levels keeps the formula meaningfully correlated with quality	Mean perturbed ρ > 0.3 and min perturbed ρ > 0.2 at every level; formula must start positively correlated
3	Gradient / Monotonicity	Active partial derivatives have the expected sign; hard gates fire or the formula is monotone	E is positively active; every other active variable has the expected sign; gate check uses tolerance
4	Baseline Comparison	Formula outperforms random and inverted-prediction baselines	ρ(B, E) > 0.3, ρ(B, E) > ρ(random, E) + 0.05, and ρ(B, E) > ρ(-B, E) + 0.05 on every metric
5	Axiom Checks	Structural properties hold: near-zero below gate, monotonic in E, maximum at ideal point	All checks pass within tolerance; hard gate waived for fully monotone formulas
6	Cross-Instance Transfer	Formula's transfer correlation exceeds raw quality's transfer correlation	Wins > 50% of pairs and mean ρ_formula > mean ρ_E

Verdict Scale

Tests Passed	Verdict
6/6	SOTA — formula passes all verification
5/6	Near-SOTA — minor weakness
4/6	Promising — needs work
3/6	Framework — correct structure, poor performance
< 3/6	Not viable — fundamental issues

Quick Start

1. Install from source:

   pip install -e .

2. Run the bundled example to see the suite in action:

   sota-verify examples/demo_formula.py

3. Define your own formula in a Python file:

   # my_formula.py
   import numpy as np
   
   EMPIRICAL_QUALITY_THRESHOLD = 0.5
   
   
   def formula(E, S, U, C, R, M):
       """Your scoring function. E=quality, S=surprise, U=coverage, C=challenge, R=risk, M=complexity."""
       if E < EMPIRICAL_QUALITY_THRESHOLD:
           return 0.0
       sigma = C / (1 + C)
       return (E * S * U * sigma) / (R * (1 + M))
   
   
   def load_data():
       """Minimal runnable example data."""
       rng = np.random.default_rng(0)
       names = [f"c{i}" for i in range(10)]
       rows = []
       for n in names:
           E = rng.uniform(0.5, 1.0)
           S = rng.uniform(0.1, 1.0)
           U = rng.uniform(0.5, 2.0)
           C = rng.uniform(0.1, 0.9)
           R = max(rng.uniform(0.05, 0.4), 1e-6)
           M = int(rng.integers(0, 4))
           B = formula(E, S, U, C, R, M)
           rows.append({"name": n, "E": E, "S": S, "U": U, "C": C, "R": R, "M": M, "B_formula": B})
       return {
           "per_metric": {"metric_a": rows, "metric_b": rows},
           "cross_instance": [{"rho_E": 0.6, "rho_formula": 0.65}],
       }

4. Run the suite:

   sota-verify my_formula.py

   run_all() returns one result dict per test:

   [
       {"test": "combined_score", "passed": True, "rho": 0.92},
       {"test": "robustness", "passed": True},
       {"test": "gradients", "passed": True},
       {"test": "baseline_comparison", "passed": True},
       {"test": "axioms", "passed": True},
       {"test": "cross_instance", "passed": True},
   ]

5. Or use it programmatically, including the original (uncorrected) suite:

   from sota_verify import run_all, run_all_original
   import my_formula
   
   results = run_all(my_formula.load_data(), my_formula.formula)
   original = run_all_original(my_formula.load_data(), my_formula.formula)

Data Format

Your load_data() function must return:

{
    "per_metric": {
        "metric_name_1": [
            {
                "name": "candidate_a",
                "E": 0.82,       # empirical quality [0, 1]
                "S": 0.64,       # surprise [0, 1]
                "U": 1.3,        # hard-case coverage [0, inf)
                "C": 0.45,       # challenge magnitude [0, 1]
                "R": 0.12,       # reproducibility risk [0, 1]
                "M": 2,          # complexity (int, >= 0)
                "B_formula": 0.0  # pre-computed formula output
            },
            # ... more candidates
        ],
        "metric_name_2": [ ... ],
    },
    "cross_instance": [
        {"rho_E": 0.45, "rho_formula": 0.62},   # correlation on a transfer pair
        # ... more pairs
    ]
}

What Changed

This fork fixes the following issues in the original suite:

1. Baseline comparison no longer compares against -ground_truth (which always yields ρ = -1.0 and is impossible to beat). It compares signed correlations against random scores and an inverted-prediction baseline.
2. Robustness now requires the formula to start positively correlated with quality and to stay meaningfully correlated after noise.
3. Gradient checks skip variables the formula does not use, so simple-but-valid formulas like B = E are not rejected.
4. Axiom checks use tolerance and waive the hard-gate requirement for fully monotone formulas.
5. Cross-instance transfer now requires a majority of wins and a higher mean transfer correlation.

API Reference

run_all(data, formula_fn, hard_gate_threshold=0.5)

Run the corrected 6-test suite. Returns a list of result dicts, each with "test", "passed", and test-specific fields.

run_all_original(data, formula_fn, hard_gate_threshold=0.5)

Run the original (uncorrected) 6-test suite for comparison.

Individual Tests

Each corrected test can be called independently:

• test_combined_score(data, formula_key="B_formula")
• test_robustness(data, formula_fn, formula_key="B_formula", n_perturbations=50)
• test_gradients(formula_fn, hard_gate_threshold=0.5)
• test_baseline_comparison(data, formula_key="B_formula")
• test_axioms(formula_fn, hard_gate_threshold=0.5)
• test_cross_instance(data, formula_key="rho_formula", e_key="rho_E")

CLI

usage: sota-verify [-h] [--json] [--gate GATE] formula_script

positional arguments:
  formula_script    Path to Python file with formula() and load_data()

options:
  --json            Output results as JSON
  --gate GATE       Hard gate threshold for E (default: 0.5)

License

MIT