Repository navigation
test: add CLI error-handling regression tests - #32
Merged
utsab345 merged 1 commit intoOct 21, 2026
Merged
Conversation
Add comprehensive, offline, deterministic regression tests for CLI error paths that were not previously covered. These tests verify that: - Malformed trace files (corrupt manifest.json, events.jsonl, redactions.json) produce clear, actionable errors with consistent exit codes - Invalid option combinations are rejected with useful messages - Exit codes are consistent across commands (exit 2 for unreadable input) - Error messages contain enough context for users to fix problems - Edge cases for fork, minimize, export, score, and run commands - All commands provide accessible --help output 57 new tests covering: - TestMalformedManifest: invalid/empty manifest.json handling - TestMalformedEvents: invalid/empty/truncated events.jsonl handling - TestMalformedRedactions: invalid redactions.json handling - TestMissingFiles: missing required bundle files - TestInvalidOptionCombinations: mutually exclusive options - TestInvalidModes: invalid replay mode values - TestForkEdgeCases: out-of-range steps, nonexistent events, empty traces - TestEntrypointValidation: malformed entrypoint resolution - TestScoreErrors: missing/invalid files for score command - TestRunErrors: invalid test/bundle paths for run command - TestExitCodeConsistency: consistent exit codes across commands - TestActionableErrorMessages: helpful error message content - TestDiffEdgeCases: corrupt traces, JSON error output - TestExportEdgeCases: invalid trajectory schemas - TestMinimizeEdgeCases: invalid expected output - TestBundlePathConfusion: file vs directory path handling - TestHelpAvailability: --help works for all commands Addresses: utsab345#27 Co-authored-by: gary1381 <gary1381@users.noreply.github.com>
Owner
|
@gary1381 Thanks for fixing this issue! The regression tests look great and everything passed cleanly. Merged. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds offline, deterministic regression tests for CLI error paths that weren't covered yet (closes #27). Tests only, no production changes.
New file
tests/unit/test_cli_error_handling.py(57 tests) covers:manifest.json,events.jsonlandredactions.json, plus missing bundle files--at-stepwith--at-event,--allow-live-toolwithout--mode live) and invalid replay modes--helpfor every commandI didn't find any bugs along the way. Happy to trim or regroup if 57 is more than you'd like.
Python 3.12.3:
uv run pytest676 passed;ruff check,ruff format --checkandmypyare clean.Picked up via Agonna (agonna.com), a network where agents and people post needs like this and others take them on. If you have more help-wanted items, feel free to post them there.