Skip to content

Hide rankings with failed secret runs - #235

Merged
msaroufim merged 1 commit into
mainfrom
fix-secret-failed-ranking-filter
Jun 17, 2026
Merged

msaroufim merged 1 commit into
mainfrom
fix-secret-failed-ranking-filter

Conversation

@msaroufim

Copy link
Copy Markdown
Member

Summary

  • hide public scored runs from kernelboard rankings when the same submission has any failed secret run on the same runner
  • apply the filter to leaderboard API results, trend endpoints, summary cards, legacy pages, and rank notification query
  • add a regression test for the QR failure pattern: public leaderboard run passes while secret benchmark run fails

Why

Kernelbot now filters these cases from its API, but kernelboard also queries the leaderboard DB directly. Without the same predicate, submissions that fail secret validation can still appear in kernelboard-rendered rankings or summaries.

The filter intentionally matches on submission_id and runner, not mode, because the observed QR issue has a visible public leaderboard score and a failed secret benchmark run.

Validation

  • uv run python -m py_compile kernelboard/api/leaderboard.py kernelboard/api/leaderboard_summaries.py kernelboard/index.py kernelboard/leaderboard.py ranking_worker.py tests/api/test_leaderboard_api.py
  • git diff --check
  • uv run ruff check kernelboard/api/leaderboard.py kernelboard/api/leaderboard_summaries.py kernelboard/index.py kernelboard/leaderboard.py tests/api/test_leaderboard_api.py

Attempted REDIS_URL=redis://localhost:6380/0 uv run pytest -q tests/api/test_leaderboard_api.py, but the local fixture invokes bare psql with an environment that drops Homebrew's /opt/homebrew/bin from PATH, so it fails before tests run with FileNotFoundError: psql on this machine.

@msaroufim
msaroufim merged commit 969de4e into main Jun 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant