Add data-query category: AL query-generation benchmark

[Onat Buyukakkus (onbuyuka)]

What

Adds a new data-query evaluation category: an offline benchmark for AL query generation. Given a natural-language data question, the agent authors a single AL query object (query.al) and it's scored deterministically — compile + run the generated query and a gold reference query against the container's Contoso demo data, pass if the result sets match. No MCP server, no LLM judge.

This complements the AI Test Toolkit evals in the platform repo: those test the MCP server end-to-end; this benchmarks models/agents on query generation.

How it works

• Dataset (dataset/dataquery.jsonl) — 11 entries, each with nl_prompt + gold_query (the reference AL query whose result set defines "correct") + environment_setup_version + ordered.

• Pipeline (evaluate/dataquery.py) — the agent writes query.al; the harness compiles + runs the generated and gold queries and compares result sets.

  • build = generated query compiled and ran; resolved (ResolutionRate) = result set matches gold.

  • Rows compared by value; numeric columns are normalized scale-insensitively (500 == 500.0) while string Code/No. columns are compared verbatim ("001" != "1"). Column names/order ignored; order-insensitive unless the entry marks the question ordered.

• Run mechanism (operations/bc_operations.py) — wrap_query_as_api turns each query into an API query; execute_al_query publishes a throwaway app to the container and reads its OData endpoint (following @odata.nextLink so large result sets aren't truncated).

• Execution-based category (requires_container = True, runner GitHub-BCBench) — a stock BC sandbox artifact (Cronus/Contoso data) suffices; no special build needed. Setup-ContainerAndRepository.ps1 skips the git clone for data-query (there's no repo) and just provisions the container.

• Skill — an agent-facing al-query-authoring SKILL.md (instructions/dataquery-bc/skills/) documenting AL query authoring, including the canonical column(Name) { Method = Count; } idiom (a field-less Count; a source field trips AL0353).

Scoring integrity & robustness

• Fail loud on broken golds — evaluation runs the gold query first and does not swallow its errors: a gold that doesn't compile/run raises and fails the run, so it gets fixed rather than silently scored. An empty agent output is recorded as a tracked build failure (No query.al produced).
• Deterministic comparison — numeric-only decimal canonicalization (Code/No. strings preserved verbatim, so "001" != "1"; amounts scale-insensitive, 500 == 500.0), OData @odata.nextLink paging, timeout → BuildTimeoutExpired, malformed agent output → BuildError, case-insensitive query/QueryType wrapping, OrderBy treated as a property, and throwaway apps uninstalled between runs to avoid object-ID conflicts.

How to run

Actions → Evaluation with GitHub Copilot (or Claude Code) → Run workflow → category = data-query. The self-hosted runner provisions the container; no local setup.

New --skills dispatch input (mirrors --al-lsp / --al-mcp) toggles the agent skill per run — default off. Its exact shape is being reconciled with #761's skill handling on rebase.

Tested

• ✅ Unit tests: result_sets_match (incl. digit-only Code non-collapsing), wrap_query_as_api, run-template wiring, and --skills override (tests/test_dataquery_evaluation.py, tests/test_agent_skills.py); full suite 695 pass; ruff + ty clean.
• ✅ End-to-end on the self-hosted runner: the execute_al_query container round-trip (compile → publish → OData → result-set compare) is validated — generated and gold queries build, publish, and compare, producing real build/resolved scores.

Results

Run 30405864654 (claude-sonnet-4.6, skills on + AL LSP): 9/11 resolved, 11/11 build. Every generated query compiles and runs, and the two misses (avg-invoice-amount-by-country, total-purchase-amount-by-vendor) both build but diverge from gold on metric-definition ambiguity (which amount / net vs incl-VAT, per-invoice vs per-line) — left as documented misses for a future gold/prompt-reconciliation pass rather than harness defects.

Known non-blocking CI caveat: on a data-query run the matrix + summarize + bceval-upload jobs succeed, but the summarize job's Update leaderboard step fails because it checks out main, whose EvaluationCategory enum has no data-query yet. This self-resolves once this PR merges.

Depends on #761: generic non-repo-category support (skip-repo as a category property, optional repo/base_commit) is landed in #761; this PR merges after it and takes up those abstractions on rebase (dropping the data-query clone-skip special-case).

Review feedback has been addressed in follow-up commits and every review thread is resolved (bar the --skills thread, kept open pending the #761 rebase) — see the review-response comments.

[Magnus Hartvig Grønbech (Groenbech96)]

Looks good. Good stuff.

[Copilot]

Pull request overview

Adds the data-query benchmark for deterministic AL query generation and execution against Business Central demo data.

Changes:

• Adds 11 query-generation dataset entries and agent guidance.
• Implements query wrapping, execution, comparison, and result reporting.
• Integrates container setup, workflows, tests, and documentation.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
dataset/dataquery.jsonl	Adds benchmark entries and gold queries.
src/bcbench/evaluate/dataquery.py	Implements evaluation and result comparison.
src/bcbench/operations/bc_operations.py	Adds query wrapping and OData execution.
src/bcbench/dataset/dataset_entry.py	Defines data-query entries.
src/bcbench/dataset/__init__.py	Exports the new entry type.
src/bcbench/types.py	Registers category runtime behavior.
src/bcbench/results/base.py	Adds a general execution-result factory.
src/bcbench/evaluate/__init__.py	Exports the pipeline.
src/bcbench/operations/__init__.py	Exports query operations.
src/bcbench/commands/evaluate.py	Supports mock data-query evaluation.
src/bcbench/agent/shared/config.yaml	Adds the agent prompt template.
src/bcbench/agent/shared/instructions/dataquery-bc/skills/al-query-authoring/SKILL.md	Adds AL query authoring guidance.
scripts/Setup-ContainerAndRepository.ps1	Creates clone-free workspaces.
scripts/BCBenchUtils.psm1	Resolves the new dataset category.
.github/workflows/copilot-evaluation.yml	Enables Copilot runs.
.github/workflows/claude-evaluation.yml	Enables Claude runs.
tests/test_dataquery_evaluation.py	Tests comparison and wrapping logic.
tests/conftest.py	Adds data-query fixtures.
tests/test_type_exhaustiveness.py	Covers category type dispatch.
docs/data-query.md	Documents the benchmark.
docs/index.md	Links the new category.

Comments suppressed due to low confidence (1)

src/bcbench/evaluate/dataquery.py:115

• execute_al_query also raises BuildTimeoutExpired on a gold-query timeout, and it is not a BuildError. This exception escapes instead of taking the intended harness/container failure path.

        except BuildError as e:
            logger.exception(f"Gold query failed to compile/run for {context.entry.instance_id}")
            self.save_result(
                context,

[Onat Buyukakkus (onbuyuka)]

Review feedback addressed (commit 129c5ef)

Thanks for the thorough review. Summary of what changed:

Scoring integrity

• Numeric rounding false-positives — result_sets_match now canonicalizes numbers with Decimal.normalize() (full precision, scale-insensitive) instead of rounding through float to 4 decimals. 1.00001 ≠ 1.00002; 500 == 500.0.
• OData pagination — the fetch now follows @odata.nextLink until exhausted, so large result sets aren't silently truncated to the first page.
• Gold-failure biasing scores — a gold/harness failure is now recorded as unscorable (new scorable flag) and excluded from resolved/total/build/instance_results, so it no longer counts against the agent's ResolutionRate.
• Timeout escaping — BuildTimeoutExpired is now caught around both generated and gold execution (it isn't a BuildError), so a timeout is recorded instead of breaking summarization.
• ValueError on malformed output — wrap_query_as_api now raises BuildError (handled downstream) when there's no query declaration or no object body.

Robustness

• Case sensitivity — the query keyword reassignment and QueryType removal are now case-insensitive and don't require a leading newline, so Query 50123 / { QueryType = Normal; ... } no longer break ID reassignment or create a duplicate QueryType.
• Object-ID conflict on re-run — execute_al_query now uninstalls/unpublishes the throwaway app before and after each run, so re-running locally against the same container doesn't hit a 50100/50101 conflict.

Docs/cleanup

• SKILL.md: OrderBy corrected to a property (OrderBy = descending(Col);), not a block.
• types.py: stale MCP/seed comments removed (folded DATA_QUERY into the existing match arms).
• PR description: validation scope updated to the current 11 gold queries; the 5 new ones are in runner shakeout now.

Added unit tests for the precision fix, the case-insensitive/malformed wrap_query_as_api paths, and the paging/cleanup wiring. Full suite: 635 pass, ruff + ty clean.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/bcbench/operations/bc_operations.py:293

• AL escapes a quote inside a quoted identifier by doubling it ("A ""quoted"" query"), not with a backslash. This regex stops at the first doubled quote, leaves the rest of the original name behind, and turns an otherwise valid query into invalid AL. Match doubled quotes in the quoted-name branch.

    text, replaced = re.subn(
        r'(\bquery\s+)\d+\s+("(?:[^"\\]|\\.)*"|\w+)',
        rf"\g<1>{object_id} {safe_name}",

src/bcbench/evaluate/dataquery.py:31

• This converts every numeric-looking string to a number, so distinct AL text/code values such as "001" and "1" compare equal (and the earlier None conversion similarly equates null with ""). That can award resolution to a query returning the wrong identifier. Preserve string/null identity and normalize only values known to be numeric, or carry type information into comparison.

    try:
        # Canonical decimal form: scale/trailing-zero-insensitive (500 == 500.0) but full precision
        # preserved, so distinct values like 1.00001 and 1.00002 are NOT collapsed. No float rounding.
        return str(Decimal(text).normalize())
    except (InvalidOperation, ValueError):

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/results/base.py:116

• scorable=False is not propagated to the bc-eval records. category_metrics exports only resolved=False and build=True, and ResolutionRate/BuildRate score those values directly, so the externally reported headline metrics still count a gold-query failure as a resolution failure (and a build success), contrary to the new unscorable semantics. Export scorable and make the downstream evaluators skip these records, or omit unscorable records from the bc-eval export.

    def create_unscorable(cls, context: "EvaluationContext", output: str, error_message: str) -> Self:
        """A harness/dataset failure (not the agent's fault) that must not count toward the resolution rate."""
        return cls(**cls._base_fields(context), output=output, build=True, resolved=False, scorable=False, error_message=error_message)

src/bcbench/operations/bc_operations.py:426

• The OData JSON is parsed through Python float before _normalize_value sees it, so high-magnitude BC Decimal values can lose precision and distinct results can compare equal (for example, adjacent cent values near BC Decimal's upper range). Parse JSON decimal literals directly as Decimal to preserve the deterministic comparison promised by the matcher.

    rows = json.loads(result_file.read_text(encoding="utf-8-sig") or "[]")

src/bcbench/evaluate/dataquery.py:88

• The new pipeline's outcome logic is not covered by the added tests: there are no tests that mock execute_al_query and verify match, mismatch, generated build failure, and gold-query unscorable results. These branches define the benchmark's scores, and the current ordering/export issues are examples that helper-only tests do not catch. Add focused pipeline tests like those used for the existing evaluation pipelines.

    def evaluate(self, context: EvaluationContext[DataQueryEntry]) -> None:

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/bcbench/operations/bc_operations.py:305

• The transform is not comment-aware. In a valid query with a preceding comment such as // QueryType = Normal;, this substitution removes the comment occurrence because count=1, leaves the real property, and then injects a second QueryType, causing compilation to fail. Likewise, text.find("{") can select a brace in a leading comment. Locate the declaration/body with comment-aware parsing and remove the actual object-level property rather than the first textual match.

    text = re.sub(r"\bQueryType\s*=\s*\w+\s*;", "", text, count=1, flags=re.IGNORECASE)

    brace_index = text.find("{")
    if brace_index == -1:
        raise BuildError("query-wrap", f"Generated query has no object body ('{{' not found):\n{query_text}")

src/bcbench/agent/shared/instructions/dataquery-bc/skills/al-query-authoring/SKILL.md:27

• This example is effectively the gold solution for dataquery__outstanding-sales-value-by-customer-1: it uses the same Customer → Sales Line join, Order filter, and Outstanding Amount sum. Any run with this skill enabled receives the answer to a benchmark entry (including the first test-run entry), inflating that experiment's score. Replace it with a valid query pattern that is not represented in the dataset.

            dataitem(SalesLine; "Sales Line")
            {
                DataItemLink = "Sell-to Customer No." = Customer."No.";
                DataItemTableFilter = "Document Type" = const(Order);
                column(OutstandingAmount; "Outstanding Amount") { Method = Sum; }

src/bcbench/results/bceval_export.py:53

• This scoring-critical skip path has no regression coverage: tests/test_result_writer.py comprehensively exercises write_bceval_results, but no test creates an execution result with scorable=False. Add mixed and all-unscorable cases to verify these records never reach the bc-eval JSONL output.

            # Unscorable results (harness/dataset failures, e.g. a gold query that didn't compile) must
            # not reach the uploaded/core score, or they'd count against the agent's ResolutionRate.
            if isinstance(result, ExecutionBasedEvaluationResult) and not result.scorable:
                logger.info(f"Skipping unscorable result from bceval export: {result.instance_id}")
                continue

[Copilot]

🟡 Not ready to approve

Result normalization can produce false positives, and gold validation plus query wrapping still have correctness gaps.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (2)

src/bcbench/evaluate/dataquery.py:114

• This early return runs before the gold query, reintroducing the case where a broken gold is recorded as an ordinary agent build failure whenever query.al is missing. Move gold validation ahead of every generated-output failure so the documented “fail loud on broken golds” guarantee is independent of agent output.

        if not generated_query:
            logger.warning(f"Agent produced no {GENERATED_QUERY_FILE} for {context.entry.instance_id}")
            self.save_result(context, ExecutionBasedEvaluationResult.create_build_failure(context, output="", error_message=f"No {GENERATED_QUERY_FILE} produced"))
            return

src/bcbench/operations/bc_operations.py:306

• The opening brace is searched from the start of the file rather than from the matched query declaration. A valid query preceded by a comment containing { gets the API properties injected into the comment, and the otherwise valid agent output fails compilation. Scope both brace discovery and QueryType removal to the matched object body.

    text = re.sub(r"\bQueryType\s*=\s*\w+\s*;", "", text, count=1, flags=re.IGNORECASE)

    brace_index = text.find("{")

• Files reviewed: 26/26 changed files
• Comments generated: 1
• Review effort level: Medium

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

Type-erasing normalization can produce false passes, while comment-sensitive query wrapping can reject valid AL queries.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (2)

src/bcbench/evaluate/dataquery.py:25

• Normalization still erases the JSON type: a Code value "1" and numeric 1 both normalize to the same string (likewise "true"/true and ""/null). Because column names are intentionally ignored, a query returning a value from a wrong-typed column can therefore be marked resolved. Keep a type tag in the normalized value while applying decimal canonicalization only within the numeric tag, and cover the "1" versus 1 case.

def _normalize_value(value: object) -> str:
    if value is None:
        return ""
    if isinstance(value, bool):
        return str(value).lower()

src/bcbench/operations/bc_operations.py:304

• These regex/string searches are not scoped to AL syntax. A valid leading comment containing { makes this select the comment brace, and a comment containing QueryType = Normal; can consume the one removal and leave the real property duplicated with the injected API property. Such otherwise-valid agent queries are then scored as build failures. Locate the opening brace from the matched query declaration and remove the actual top-level QueryType property (ideally via token-aware parsing), rather than matching comments/strings.

    text = re.sub(r"\bQueryType\s*=\s*\w+\s*;", "", text, count=1, flags=re.IGNORECASE)

    brace_index = text.find("{")

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Medium

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

Gold-answer exposure and result-normalization collisions can invalidate benchmark scores.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Comments suppressed due to low confidence (4)

src/bcbench/evaluate/dataquery.py:75

• The agent runs with unrestricted filesystem tools while repo_path is the checkout's testbed child, so it can read ../dataset/dataquery.jsonl (or recover it from ../.git) and copy the exact gold_query. This makes resolved scores vulnerable to direct answer leakage. Withhold the gold and the Git object database during the agent phase, or run the agent in an isolated workspace that cannot access the checkout.

        with github_log_group(f"{context.agent_name} -- Entry: {context.entry.instance_id}"):
            context.metrics, context.experiment = agent_runner(context)

src/bcbench/evaluate/dataquery.py:25

• Normalization still loses the JSON type: a Code value "1" and numeric 1 both normalize to "1" (similarly "true"/true and ""/null). Because column names are ignored, a query returning the wrong typed field can be accepted. Keep a type tag in the canonical value while normalizing only the numeric payload.

def _normalize_value(value: object) -> str:
    if value is None:
        return ""
    if isinstance(value, bool):
        return str(value).lower()

src/bcbench/evaluate/dataquery.py:94

• The empty-output return happens before the gold query is validated. If an entry's gold is broken and this agent produces no file, the run records an agent build failure instead of failing loudly on the dataset defect, so gold validity still depends on agent output. Run the gold before this branch.

        if not generated_query:
            logger.warning(f"Agent produced no {GENERATED_QUERY_FILE} for {context.entry.instance_id}")
            self.save_result(context, ExecutionBasedEvaluationResult.create_build_failure(context, output="", error_message=f"No {GENERATED_QUERY_FILE} produced"))
            return

docs/data-query.md:8

• This says there is no live server in the loop, but the next paragraph and execution section require a running BC container and query its OData server. Clarify that there is no MCP or external production service; otherwise the category's runtime requirements are documented inaccurately.

This category benchmarks an agent's ability to **generate Business Central AL queries** from a natural-language data question — an offline query-generation benchmark. There is **no MCP server and no live server in the loop**: the agent writes an AL query, and the query is evaluated deterministically.

• Files reviewed: 28/28 changed files
• Comments generated: 0 new
• Review effort level: Medium

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Sun Haoran (haoranpb)]

[recommendation] I think this is a no-op here

[Sun Haoran (haoranpb)]

[recommendation] suffix should have a stronger type, so it's enforced to have value generated or gold. Alternatively, make it a boolean => serialize to $0$ and $1$

[Sun Haoran (haoranpb)]

[recommendation] consider extracting the normalization here into a dedicated function, it will be useful to future categories as well

[Sun Haoran (haoranpb)]

[critical] You might want to check if gold_rows is empty, if so, fail the evaluation => requires some design on the dataset.

If the gold_rows is empty, the generated query only need to return nothing.

[Sun Haoran (haoranpb)]

[recommendation] We are taking an inexplicit dependency on the index 0 company here, not sure how that is resvoled.

You will have an easier time to hardcode a company name that we know will exist.

---

We could also pin the company name into the dataset, so each entry has to specify the company name. Together with pinned version, this should provide a more deterministic runs.