Report as you go

[dogenkigen]

Summary

doc-assert now reports results as they happen, like cargo test does, instead of printing one block when the whole run is over. The library API is reworked around this, so the crate goes out as 0.2.0 (breaking).

Reporting as you go

Each test case is named as soon as its request is sent and marked ✅ or ❌ once it is done, so a slow or retried request shows what is being waited for. The details of the failures follow at the end:

2 tests
GET /blog (README.md:12) ✅
POST /blog (README.md:30) ❌

failures:
-------------
POST /blog (README.md:30): response at line 36: expected response code 201, got 500

test result: FAILED. 1 passed; 1 failed

The CLI and DocAssert::assert() print the same output. Inside a test, cargo captures it like any test output: it is shown when the test fails, or as it is printed with cargo test -- --nocapture.

All of this is built on Run:

• DocAssert::start() parses every documentation file up front, so a parse error is reported before any request is sent and the number of test cases is known in advance.
• run.next().await executes one test case and returns its TestCaseResult.
• run.print_progress().await executes the rest, printing the output above.
• run.finish() returns the Report of what was executed so far.

API changes (breaking)

Method	Returns	Use it when
assert()	(); prints the progress, panics unless every test case passed	Inside a #[tokio::test] (the common case)
run()	Result<Report, Error>; prints nothing	You want to inspect the report yourself
start()	Result<Run, Error>	You want each result as soon as it is executed

Err only means the run could not happen at all: no documentation file was given, or a file could not be parsed. Failed test cases are reported in the Report, and Report::passed() gives the verdict.

Migration

Before	After
DocAssert::new().with_url(url)	DocAssert::new(url)
assert().await returning Result<Report, AssertionError>	assert().await, panicking on failure, or run().await returning Result<Report, Error>
Err(AssertionError::TestSuiteError(report))	Ok(report) with report.passed() == false
Err(AssertionError::ParsingError(String))	Err(Error::Parse { doc_path, reason })
insert_string, insert_int, insert_float, insert_bool, insert_value	insert(name, value), which accepts anything Into<serde_json::Value>
insert_null(name)	insert(name, Value::Null)
Variables::from_json(..) -> Result<Self, String>	Variables::from_json(..) -> Option<Self>
DocAssert<'a> borrowing &str arguments	DocAssert taking owned impl Into<String> arguments

Structured results

• Report has results(), failures() (yielding (&TestCaseId, &Failure) pairs), total_count(), executed_count(), passed_count(), failed_count(), not_run_count(), passed() and summary().
• TestCaseResult has id(), passed() and failure(). A TestCaseId holds the method, URI, doc path and line.
• Failure says why a test case did not pass:
  • UnresolvedVariables { names }
  • InvalidDocumentation { line_number, reason }
  • RequestFailed { reason }
  • ResponseMismatch { line_number, cause: Mismatch }
• Mismatch covers what the server got wrong: status code, header, missing header, body, variable not found, unreadable or malformed body.
• Error, Failure and Mismatch are #[non_exhaustive], so variants can be added without a breaking release.

matches!(
    failure,
    Failure::ResponseMismatch { cause: Mismatch::StatusCode { actual: 500..=599, .. }, .. }
)

Behaviour changes

• Partial runs: a run stopped early reports the test cases it did not get to as not run. Such a report never passes, and its result line ends with ; N not run.
• Documentation mistakes are not retried.
  • An invalid [ignore] path is now a parse error.
  • An expected body that is not valid JSON, or an invalid request header, fails the test case as InvalidDocumentation before any request is sent. These used to be sent and retried as often as the retry policy allowed.
• Variables are only extracted from a response once the test case passes, not part way through a failed attempt.
• Unresolved variables are reported by name rather than with the whole substituted input, which could contain the values of other variables, such as tokens. A lone backtick is no longer mistaken for a placeholder.
• Non-ASCII header values: a non-ASCII header value from the server is reported as a header mismatch instead of panicking.
• Retry policy: [retry]: # (0, …) is rejected as a parse error that names its line. Previously it produced a test case that could never run. The README now says that the first number is the number of attempts, the first one included, which is what the code always did.
• CLI:
  • Errors are printed to stderr.
  • A parse error reads Error: cannot parse <file>: ….
  • Running without a documentation file exits with 2.
  • Failure lines no longer repeat the method and URI.
  • The other exit codes are unchanged and now documented in the README: 0 when everything passed, 2 for invalid arguments, 3 when a document can't be parsed, 4 when a test case failed.

Internal changes

• lib.rs contains only the public entry point: DocAssert, Run, Error, and the re-exports. Results and their rendering live in report.rs; Variables and placeholder substitution live in variables.rs.
• The executor checks what the documentation describes once, before the retry loop. It returns Failure and Mismatch values instead of strings.
• CI now runs clippy and the tests with --features binary, so the binary is linted and tested. Previously it was only built.

Known limitation

Run::next is not cancellation safe.

• What happens: if its future is dropped mid-request, for example by tokio::time::timeout or select!, that test case is abandoned.
• How it shows up: the abandoned test case is reported as not run, so the report does not pass, and the variables it would have extracted are not defined. Its request may still have reached the server.
• This is documented on the method and in the README, and covered by a test.

Implementing Stream for Run was considered and rejected:

• Keeping the in-flight request inside Run would make next resumable after a drop, but resuming the abandoned request is not what someone applying a timeout wants.
• The while let loop already covers what stream combinators would add.

The planned follow-up is a per-request timeout, for example DocAssert::with_timeout. A timed-out request would then be reported as an ordinary RequestFailed.

Testing

• cargo fmt --check, cargo clippy --all-targets --features binary -- -D warnings and cargo doc pass with no warnings.
• cargo test --features binary passes:
  • 37 unit tests;
  • 11 tests in tests/api.rs, against the public API only;
  • 6 tests in tests/cli.rs, against the binary;
  • 17 doctests. These include the README examples, two of which were previously skipped because they sat inside #[cfg(test)].
• tests/cli.rs runs the binary against a local server that holds an endpoint open. This proves:
  • earlier results, and the name of the test case in progress, are printed before that endpoint answers;
  • the exact failure output;
  • the exit codes 0, 2, 3 and 4.
• New executor tests cover the retry loop: number of attempts, the last failure is the one reported, no delay after the last attempt, and an unreachable server.
• make sanity passes against sample-api. tests/functional/readmes/2_README_in.md and 3_README_in.md fail on main too, because the expected "My First Blog- UPDATED" is missing a space. They are not changed here.

🤖 Generated with Claude Code