Repository navigation
Name valid values when a benchmark filter matches nothing (#1041) - #1056
Merged
yakew7 merged 1 commit intoOct 18, 2026
Merged
Conversation
get_benchmark_results returned total_matches: 0 with no explanation when a filter value matched no rows. Because audit names are lowercase, a call like audit="COMPAS" came back empty, which an agent can misread as "no data." _get_benchmark_results_impl now adds a 'notes' entry naming that column's valid values when a filter value matches none of them, with a hint when the mismatch is only in case. The empty result is kept, and a value that does match is unaffected. Updates the tool docstring and the tests.
Contributor
|
@sammist2 is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1041.
get_benchmark_resultsreturnedtotal_matches: 0with no explanation when a filter value matched no rows. Because the audit names are lowercase, a call likeaudit="COMPAS"came back empty, which an agent can misread as "no data" rather than a case mismatch._get_benchmark_results_implnow adds anotesentry naming that column's valid values whenever a filter value matches none of them, with a hint when the mismatch is only in case (e.g.audit 'COMPAS' not found; available: ... (values are case-sensitive; did you mean 'compas'?)). The empty result is kept, and a value that does match is unaffected. The tool docstring now documents this.Updated the existing no-match test and added a case-mismatch test in
tests/test_mcp_server.py.