Skip to content

feat(mdcode): deploy the deployments that --deployment names, after every check - #554

Merged
feixufeixu merged 2 commits into
mainfrom
feat/push/tpp-2-named-deployments
Oct 24, 2026
Merged

feixufeixu merged 2 commits into
mainfrom
feat/push/tpp-2-named-deployments

Conversation

@feixufeixu

@feixufeixu feixufeixu commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

kcmd push --deployment <names> publishes the model to Knowledge Catalog, then
deploys each named deployment in the order given, with the profile that
deployment names. Names are comma-separated, or the flag repeats. Every check of
every destination runs before the first write.

After the checks of the --no-deployment path, the first phase builds each
selected deployment's model in deploymentPlan:

  1. A Spanner deployment that names no profile, in a model with metrics, is
    refused, since a Spanner graph cannot hold metrics.
  2. The deployment's profile is applied with applyProfile. A profile file is
    read again first, with its actions.
  3. The inheritance and expression rules run before pruning, since pruning can
    remove the field they are about.
  4. deployments is narrowed to the deployment's own entry, which keeps each
    deployment to its own target. The model is then validated and its DDL
    generated offline.

After the file-copy type lookup and the catalog preflight, liveErrors runs
each deployment's checks that call a service. It reads the BigQuery dataset
through datasetLocation, which the graph deploy already uses, or the Spanner
database through the new SpannerClient.getDatabase. It probes each BigQuery
source, dry-runs the BigQuery DDL in the dataset's location and checks each
action's DML. When the dataset's location cannot be read, push prints that
warning if a check fails or under --validate-only. A push that deploys prints
it with the graph, as today. The second phase writes Knowledge Catalog, then
deploys each deployment, under its own header when there are several. An
AlloyDB deployment deploys no graph and prints a note.

Push refuses --deployment with --no-deployment in either order, a bare or
empty --deployment, a name the model lacks and --deployment in a scope with
no model. typedDeployment reads both flags from the command line as typed,
since cac turns a value that looks like a number into a number and drops
--deployment when --no-deployment follows it.

Today's path accepts and refuses what it did before. Its refusal of a graph push
of a named profile no longer says that a deployment cannot name a profile yet.
pruningNote, printGraph and generateOffline move out of it, and its
output does not change. Comments in runtime.ts, store.ts and skills.ts no
longer say that push refuses several deployments, since --deployment deploys
several.

namedProfileErrors now takes a callback that says what push's own checks cover for each profile. Today's path leaves the profile it pushes to its own checks, as before. A --deployment push checks every profile in full, and leaves to each selected deployment's checks only the warning about SQL in its profile that kcmd cannot read. It also takes the profile whose join push's own key
check reports, which the profile consistency check leaves out. The two-phase
path passes default, and today's path the profile it pushes.

This version differs from the finished stack in these ways:

  1. Push does not yet check that a deployment's tables are in the database it
    deploys to, or hold a deployment that names a profile to the metric rules
    of its engine. A Spanner deployment whose profile leaves a metric available
    deploys its graph without that metric. A BigQuery deployment whose profile
    leaves a metric that is not a single SUM, AVG, COUNT, MIN or MAX deploys
    its graph without that measure. The generator warns about each metric it
    leaves out. Push does not yet warn that default is unbound when every
    deployment names a profile. A later PR in this stack adds these checks.
  2. A push with --deployment does not yet apply catalog.yaml's
    default_profile the way the runtime does, or warn that it chooses no
    binding. A later PR in this stack adds both.
  3. A plain push still refuses a deployment that names a profile, and
    kcmd profiles still reports the errors of a plain push, until a later PR
    in this stack runs every model that lists deployments in two phases.
  4. The errors for a deployment that names a legacy profile and for a graph the
    generator cannot build are tested in a later PR in this stack, beside the
    report that prints them. That PR also tests that a deployment's error prints
    once.
  5. Until a later PR sends a plain push of a model that lists deployments
    through two phases, the test harness passes every deployment's name to
    --deployment when a case with a Google-flavor model gives none. So the
    two-phase tests already state what the final plain push does.

Point to check: Spanner cannot dry-run DDL, so a Spanner deployment whose DDL
Spanner rejects fails in the second phase, after Knowledge Catalog is written.
reference.md says so.

@feixufeixu
feixufeixu requested a review from mmotoyama October 23, 2026 16:00
@feixufeixu
feixufeixu force-pushed the feat/push/tpp-2-named-deployments branch 2 times, most recently from 7bf16f0 to d9e9f60 Compare October 23, 2026 22:02
@feixufeixu
feixufeixu force-pushed the feat/push/tpp-2-named-deployments branch from d9e9f60 to 2d4504b Compare October 23, 2026 22:24

@mmotoyama mmotoyama left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI only found one issue, so LGTM if the finding is valid and addressed!

}

return {kcModels, fileCopies, overviews};
const plans: DeploymentPlan[] = [];

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI found only one medium urgency issue:

When a profile is selected by --deployment, validateProfileCompleteness skips several checks on that profile—including physical column-name checks on keys and joins, field-expression reference checks, and profile-introduced hierarchy errors—so they are not reported twice once deploymentPlan validates the applied model.

However, checkBeforeWrites returns early whenever profileErrors or kcErrors is non-empty (see line 1872), before deploymentPlan runs. In addition, the deploymentPlan loop returns on the first failing deployment rather than collecting errors across all selected deployments. As a result, whenever a push has any error in profileErrors, kcErrors, or an earlier deployment plan, the checks that were deferred to deploymentPlan are silently dropped instead of being reported in the same run.

Here is a test case illustrating the problem:

  test(
      'reports deferred profile errors alongside other profile errors when a deployment selects that profile',
      async () => {
        // Setup: A model with a 'bq' deployment using profile 'bq'. In 'profiles/bq.yaml',
        // entity 'Customer' has a non-physical primary_key ('UPPER(c_key)') and entity
        // 'Order' is missing its 'source'.
        const model = `
semantic_model:
  - name: retail
    deployments:
      - name: bq
        target: bigquery/project/proj/dataset/ds
        profile: bq
    entities:
      - name: Customer
        primary_key: [c_key]
        fields:
          - name: c_key
            expression:
              dialects:
                - dialect: ANSI_SQL
                  expression: c_key
      - name: Order
        primary_key: [o_key]
        fields:
          - name: o_key
            expression:
              dialects:
                - dialect: ANSI_SQL
                  expression: o_key
`;
        const bqProfile = `
name: bq
entities:
  - name: Customer
    source: proj.ds.customers
    primary_key: [UPPER(c_key)]
  - name: Order
    primary_key: [o_key]
`;
        writeWorkspace(
            tempDir, {'models/retail.yaml': model},
            {'profiles/bq.yaml': bqProfile});

        // Current behavior on this PR: checkBeforeWrites passes applied=true for
        // profile 'bq' to namedProfileErrors (which skips the physical key column check
        // on Customer so deploymentPlan will report it), but then returns early on
        // profileErrors before deploymentPlan runs—dropping the Customer error.
        // Expected behavior: Both the missing source on 'Order' and the non-physical
        // key column 'UPPER(c_key)' on 'Customer' are reported in the same run, just as
        // they are when --no-deployment is used.
        const out = await runPush(['--deployment', 'bq']);

        expect(out.code).toBe(1);
        expect(out.errors).toEqual([
          expect.stringContaining(
              `[profiles/bq.yaml] profile 'bq': entity 'Order' has no source in this profile`),
          expect.stringContaining(
              `'UPPER(c_key)' is not a physical column name`),
        ]);
      });

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. With --deployment, the profile check skipped the key, join, field expression and hierarchy rules for each selected deployment's profile and left them to that deployment's own checks. Those checks run only after every other local check passes, so these errors showed up on the next run instead of the same one. Push wrote nothing in either case.

I changed the profile check to apply every rule to every profile, as --no-deployment already does. It now leaves only one thing to a selected deployment's checks: the warning about SQL kcmd cannot read, which those checks give themselves. With a valid graph target, your example now prints both errors in one run, and I added a case like it to the two-phase path tests.

The deployment loop still stops at the first deployment that fails, as the live checks after it do.

The warning about SQL kcmd cannot read still prints only when those checks run, so a push that fails earlier shows it on a later run.

…ther errors

A push with --deployment left the key, join, expression and hierarchy
checks of each selected deployment's profile to that deployment's own
checks. Those run only after every other local check passes, and stop at
the first deployment that fails. A push that failed an earlier check
reported these errors only on the next run.

The profile check now gives every error of every profile, as it does
under --no-deployment. It leaves to a selected deployment's checks only
the warning about SQL in its profile that kcmd cannot read.
@feixufeixu
feixufeixu merged commit ba9aa70 into main Oct 24, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants