Example: Benchmark BOOTH's Latency and Accuracy Overhead (100 Questions) #32

Anyone adopting BOOTH wants to understand the trade-off between answer reliability and computational cost.

We currently have manually measured numbers (e.g., 3 samples resulting in approximately 3× the model calls in #29), but no controlled, reproducible comparison.

This issue aims to create a small benchmark that runs the same questions through a plain model call and BOOTH, comparing accuracy, latency, and model-call overhead.

What to build

Create a self-contained example:

examples/{github_username}_latency_accuracy/

├── run_example.py

├── RESULTS.md

└── .gitignore

1. Question dataset

Use 100 questions by default, with configurable sample sizes of 50, 100, or 200.

Suggested distribution:

Also include:

• Per-repetition results.

• Grading and normalization methodology.

• Dataset IDs or a reference to the committed question manifest.

• Brief observations, failures, and limitations.

Important: Commit actual benchmark output, not fabricated or illustrative measurements. Report negative or unexpected results honestly.

8. Implementation constraints

• No changes to src/booth/.

• No new BOOTH dependencies.

• No committed API keys or secrets.

• Keep the example self-contained and configurable.

• Use .gitignore for temporary files and local configuration.

Acceptance criteria

• Add the three requested files under examples/{github_username}_latency_accuracy/.

• Support 50, 100, and 200 questions (100 by default).

• Sample from all three suggested datasets using a fixed seed.

• Preserve question IDs and reference answers.

• Implement deterministic grading.

• Implement all three execution arms.

• Run at least 3 repetitions with randomized or alternating order.

• Report accuracy, latency, runtime, model calls, and all outcome categories.

• Include actual results and reproducibility details in RESULTS.md.

• Verify the example works with a local model without an API key.

• Do not modify BOOTH source code or add dependencies.

Out of scope

• Changes to the BOOTH library.

• Adding timing to BoothResult.

• Calibration analysis (tracked in Measure whether BOOTH's self-reported confidence predicts correct answers (calibration example) #24).

• Building a general-purpose benchmark framework.

Contribution claim

To claim this issue, comment with:

1. Proposed provider.

2. Model and version.

3. Local or hosted execution (local preferred).

Claim window: 7 days.

The goal is an honest, reproducible demonstration of BOOTH's accuracy, latency, and model-call trade-offs.

Results from 100 questions are illustrative and should not be treated as general performance claims.

Background

Anyone adopting BOOTH wants to understand the trade-off between answer reliability and computational cost.

We currently have manually measured numbers (e.g., 3 samples resulting in approximately 3× the model calls in #29), but no controlled, reproducible comparison.

This issue aims to create a small benchmark that runs the same questions through a plain model call and BOOTH, comparing accuracy, latency, and model-call overhead.

What to build

Create a self-contained example:

examples/{github_username}_latency_accuracy/
├── run_example.py
├── RESULTS.md
└── .gitignore

1. Question dataset

Use 100 questions by default, with configurable sample sizes of 50, 100, or 200.

Suggested distribution:

Requirements:

• Use a fixed sampling seed.
• Preserve dataset IDs and reference answers.
• Check dataset licences and attribution requirements.
• Use deterministic grading; no LLM judge.
• Document answer normalization and numerical comparison rules.

2. Compare three execution arms

Run the same questions using the same model:

Optionally include a fourth arm using check_with_evidence() where suitable evidence is available.

3. Metrics

Report the following for every arm:

Clearly explain the accuracy denominator. Count unsuccessful responses separately and do not silently exclude them from grading.

4. Fair comparison

• Run at least 3 repetitions by default, configurable by the user.
• Randomize or alternate execution order.
• Use identical model and generation settings across all arms.
• Apply consistent timeout and error-handling policies.
• Report individual repetitions and aggregated results.
• Distinguish setup/warm-up time from benchmark execution time.

5. Model and provider

Support at least one local model so the benchmark can run without an API key.

Suggested configuration:

• Provider: Ollama
• Model: qwen2.5:7b-instruct or another suitable local model.
• Temperature: 0.
• Fixed seed where supported.

Keep provider integration simple: isolate model invocation in one function so contributors can switch providers easily.

Hosted providers are welcome, but API keys must come from environment variables.

Use Python's standard library for HTTP requests where possible.

6. Reproducibility

Document and pin the following:

• Latest published boothpy version at implementation time (record the exact installed version).
• Model tag and digest/revision.
• Provider and generation settings.
• Python version and operating system.
• Hardware (CPU, GPU, and RAM).
• Dataset versions and question-sampling seed.
• Number of questions and repetitions.

7. RESULTS.md — Expected format

Use a consistent reporting format:

Experiment configuration

Benchmark comparison

Also include:

• Per-repetition results.
• Grading and normalization methodology.
• Dataset IDs or a reference to the committed question manifest.
• Brief observations, failures, and limitations.

Important: Commit actual benchmark output, not fabricated or illustrative measurements. Report negative or unexpected results honestly.

8. Implementation constraints

• No changes to src/booth/.
• No new BOOTH dependencies.
• No committed API keys or secrets.
• Keep the example self-contained and configurable.
• Use .gitignore for temporary files and local configuration.

Acceptance criteria

• Add the three requested files under examples/{github_username}_latency_accuracy/.
• Support 50, 100, and 200 questions (100 by default).
• Sample from all three suggested datasets using a fixed seed.
• Preserve question IDs and reference answers.
• Implement deterministic grading.
• Implement all three execution arms.
• Run at least 3 repetitions with randomized or alternating order.
• Report accuracy, latency, runtime, model calls, and all outcome categories.
• Include actual results and reproducibility details in RESULTS.md.
• Verify the example works with a local model without an API key.
• Do not modify BOOTH source code or add dependencies.

Out of scope

• Changes to the BOOTH library.
• Adding timing to BoothResult.
• Calibration analysis (tracked in Measure whether BOOTH's self-reported confidence predicts correct answers (calibration example) #24).
• Building a general-purpose benchmark framework.

Contribution claim

To claim this issue, comment with:

1. Proposed provider.
2. Model and version.
3. Local or hosted execution (local preferred).

Claim window: 7 days.

The goal is an honest, reproducible demonstration of BOOTH's accuracy, latency, and model-call trade-offs.

Results from 100 questions are illustrative and should not be treated as general performance claims.