feat(tasks): add declarative task slices and rollout validation

[yuecideng]

Description

Stack

• Layer: 8/14
• Base: feat/mllm-expert-program-frontend
• Depends on: feat(agents): add strict expert program frontend #498
• Followed by: refactor(atomic-actions): add typed tracking contracts #501

Consolidation

Consolidates #499 into this layer. Scope: declarative task vertical slices and deterministic rollout validation.
Add deterministic Expert Program rollout measurement: fixed-seed execution, raw trial JSON, failure-preserving aggregation, Markdown reports, static migration-size checks, and supported-simulation gates.

Benchmark reporting distinguishes framework coverage, checked-in integrations, and physical acceptance instead of treating compilation success as task success.

Refs #471
Refs #474

Type of change

• Enhancement (non-breaking change which improves existing functionality)

Screenshots

Not applicable.

Validation

• Focused coverage: tests/benchmark/expert_program/test_demo_success.py and tests/scripts/tools/test_expert_program_rollout_report.py
• Final affected-suite regression on the stack tip: 1215 passed, 2 skipped, 8 deselected
• Changed Python files pass Black 26.3.1; the Sphinx build and rollout-report drift check pass at the stack tip

Checklist

• Changed Python files pass Black 26.3.1.
• Corresponding public/design documentation is included in this stack.
• Tests cover the affected behavior.
• No dependency update is required.

[greptile-apps]

Greptile Summary

Adds fixed-seed Expert Program rollout measurement and deterministic static rollout reporting.

• Introduces raw per-trial JSON, failure-preserving aggregation, Markdown benchmark reports, and live Gym execution.
• Adds a generated capability and migration-size report with a drift-check command.
• Adds pure-Python benchmark coverage and a conditional Open Drawer simulation gate.

Confidence Score: 5/5

The PR appears safe to merge with no concrete blocking or independently actionable non-blocking issues identified.

The new benchmark validates artifact inputs, preserves executor failures, discards episode data, closes simulation resources, and distinguishes static framework coverage from physical acceptance.

Important Files Changed

Filename	Overview
scripts/benchmark/expert_program/demo_success.py	Adds validated trial models, no-retry collection, failure-preserving aggregation, artifact generation, and live/offline CLI paths.
scripts/tools/expert_program_rollout_report.py	Generates the static capability and migration-size report and supports checked-in drift validation.
tests/benchmark/expert_program/test_demo_success.py	Covers model validation, execution lifecycle, aggregation, serialization, CLI validation, and cleanup behavior.
tests/benchmark/expert_program/test_demo_success_open_drawer_sim.py	Adds a slow supported-simulation gate validating successful Open Drawer artifacts through the public CLI.
docs/design/expert_program_rollout_report.md	Records framework coverage, checked-in integrations, migration-size metrics, and benchmark limitations.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[CLI invocation] --> B{Run simulation?}
    B -- Yes --> C[Build Gym environment]
    C --> D[Reset once per fixed seed]
    D --> E[Execute demo episode]
    E --> F[Normalize rows and preserve failures]
    F --> G[Discard episode buffer]
    G --> H[Write raw trial JSON]
    B -- No --> I[Load existing raw JSON]
    I --> J[Validate and aggregate trials]
    H --> J
    J --> K[Write three-table Markdown report]
    L[Static rollout-report tool] --> M[Count declared task sources]
    M --> N[Render capability and migration snapshot]
    N --> O{Check mode?}
    O -- Yes --> P[Reject report drift]
    O -- No --> Q[Write design report]

Loading

_{Reviews (1): Last reviewed commit: "feat(benchmark): add expert program roll..." | Re-trigger Greptile}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.