Skip to content

feat(storage): surface the typed-table definition in table-detail (#621) - #630

Closed
padak wants to merge 1 commit into
mainfrom
claude/issue-621-table-detail-definition
Closed

feat(storage): surface the typed-table definition in table-detail (#621)#630
padak wants to merge 1 commit into
mainfrom
claude/issue-621-table-detail-definition

Conversation

@padak

@padak padak commented Aug 21, 2026

Copy link
Copy Markdown
Member

What

storage table-detail now returns the table's definition — the typed-table layout
carrying primaryKeysNames and, on BigQuery, timePartitioning, rangePartitioning
and clustering. null for an untyped table.

Why

kbagent could already write that layout (storage create-table --source-table-id --time-partitioning-field --clustering-field, promoted with storage swap-tables) but
had no way to read it back. StorageService.get_table_detail() built its response from
an explicit field allowlist that dropped definition, even though
GET /v2/storage/tables/{id} returns it for typed tables with no include= parameter.
The write half of the repartition flow was supported; the verify half was not.

That matters because the table ID is unchanged whether the swap happened or not — the
layout is the only field that distinguishes a completed repartition from a failed one.
On a Keboola-managed BigQuery project where the caller holds no bigquery.jobs.create,
the Storage definition is also the only reachable view of the registered layout.

Changes

  • service (services/storage_service.py::get_table_detail) — definition passes
    through verbatim; None on untyped tables, which needs no special-casing.
    Deliberately detail-only: the LIST endpoint genuinely does not return it, so
    storage tables would mean one detail request per table — a different change with a
    different cost profile (the issue calls this out as out of scope).

  • command — human mode gains Partitioning: / Clustering: rows between the
    primary key and last-import lines:

    Table: out.c-my-bucket.my-table
      Rows: 6,290,737
      Primary key: id
      Partitioning: DAY on created_at
      Clustering: tenant_id, country
      Last import: 2026-08-19T17:26:18+0200
    

    Rows are emitted only when a layout exists, so Snowflake and untyped-table output is
    byte-identical. Ingestion-time partitioning (no field) prints just the type.

  • RESTGET /storage/table-detail/... delegates to the service, so the route
    picks the field up with no router change. No new command, so no OPERATION_REGISTRY
    entry.

  • file-size budgetcommands/storage.py is past its grandfathered ceiling, so
    rather than golf the diff, the table-detail human renderer moved into a private
    commands/_storage_format.py (the same pattern as _storage_snapshots.py) together
    with the new layout formatter. Net effect on commands/storage.py is a 22-line
    shrink
    (2246 → 2224).

Robustness note

The formatter type-guards at every level (isinstance(..., dict) on the definition and
on each sub-object). A Storage API deploy once served definition as [] rather than an
object — the incident pinned by tests/test_storage_empty_definition.py, which broke the
legacy Go CLI's strict decoder. A test asserts the renderer returns nothing for that shape
so the crash class cannot come back through this path.

Version

Bumped to 0.87.1 with a changelog entry. Renumber freely if it should land under a
different version.

Testing

  • make check green: lint, format, typecheck, skill-check, version-check,
    command-sync-check, changelog-check, error-codes, sentinel-guards, loc-check, and
    5711 tests passed.
  • New tests/test_storage_table_definition.py: service passthrough (typed + untyped),
    seven formatter cases (no layout, non-dict definition, time partitioning with and
    without field, range partitioning with and without bounds, clustering), and three CLI
    cases (human output shows and orders the rows; human output unchanged without a layout;
    --json carries definition).
  • Not verified against a live BigQuery project — no live-project access in this run. The
    response shape follows the issue's reported v0.86.0 output and the existing
    create-table write-side field names.

Note for the merge queue

Touches the same get_table_detail function as #628 (issue #624). Whichever lands second
needs a trivial rebase; there is no semantic overlap — #628 changes how column
descriptions are read, this one adds a field to the return dict.

Fixes #621

`storage create-table` can apply BigQuery `timePartitioning`,
`rangePartitioning` and `clustering`, and `storage swap-tables` promotes the
result into place -- but nothing could read that layout back.
`StorageService.get_table_detail()` built its response from an explicit field
allowlist that dropped `definition`, even though the underlying
`GET /v2/storage/tables/{id}` returns it for typed tables with no `include=`
parameter. The write half of the repartition flow was supported and the verify
half was not.

That gap matters because the table ID is unchanged whether the swap happened or
not, so the layout is the only field that tells a completed repartition from a
failed one -- and on a Keboola-managed BigQuery project without
`bigquery.jobs.create`, the Storage `definition` is the only reachable view of
the registered layout.

- service: `definition` passes through verbatim, `None` for untyped tables.
  Deliberately detail-only -- the LIST endpoint does not return it.
- command: human mode gains `Partitioning:` / `Clustering:` rows between the
  primary key and last-import lines, emitted only when a layout exists, so
  Snowflake and untyped-table output stays byte-identical.
- `commands/storage.py` is past its grandfathered line budget, so the
  table-detail human renderer moved to a private `_storage_format` module
  alongside the new layout formatter. Net effect is a 22-line shrink.

The formatter type-guards every level: a Storage API deploy once served
`definition` as `[]` (see tests/test_storage_empty_definition.py) and that
crash class must not come back.
@padak

padak commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

Closing as a duplicate of #629, which was opened concurrently and is the stronger implementation: it also surfaces requirePartitionFilter and the partitions[] count, adds an E2E case, and updates keboola-expert.md / storage-types-workflow.md. Nothing here is worth porting over — the isinstance guarding against the "definition":[] wire shape (see tests/test_storage_empty_definition.py) is present in #629 too.

Opened by the daily issue-triage run before #629 existed; no reviewer time was spent on it.

@padak padak closed this Aug 21, 2026
@padak
padak deleted the claude/issue-621-table-detail-definition branch August 21, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage table-detail drops the table definition (BigQuery partitioning/clustering) from the API response

1 participant