Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions toolbox/mdcode/docs/semantic-model/profiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,9 +247,10 @@ catalog/EntryGroups/commerce_eg/
The `<name>` in the file name must match the `name:` inside the file, and
`default` is reserved for the inline bindings. A profile file carries `name`,
`entities`, `relationships`, `metrics_exclude` and `actions`, and names no
deployment target. For now `kcmd push` refuses a deployment that names a
profile, except under `--no-deployment`, and refuses to deploy a graph for a
profile file. It says so before it deploys anything.
deployment target. `kcmd push --deployment` deploys each deployment it names
with the profile that deployment names. A push without `--deployment` or
`--no-deployment` refuses a deployment that names a profile, and refuses to
deploy a graph for a profile file. It says so before it deploys anything.
`kcmd profiles --print-store` and `kcmd skills-generate`
take a profile file's store from the deployments that name the profile in
`profile`. When none does, they report that there is no database to run it
Expand Down
20 changes: 14 additions & 6 deletions toolbox/mdcode/docs/semantic-model/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,15 +40,21 @@ whether the catalog leg runs.
| `--all-profiles` | Deploy the graph for every defined binding profile (plus the inline bindings when the document itself declares a target), instead of a single one. The Knowledge Catalog leg still records one canonical view — the default binding. Mutually exclusive with `--profile` and `--no-profile`. |
| `--no-profile` | Deploy the graph for no binding profile: publish only the logical model to Knowledge Catalog, leaving any deployed graph untouched. |
| `--no-kc` | Skip the Knowledge Catalog metadata push and deploy only the graph the selected profile targets. Knowledge Catalog is pushed by default. A push that deploys no graph (a logical model, or `--no-profile`) can only reach Knowledge Catalog, so `--no-kc` on it is an error, as is `--no-profile --no-kc` (nothing left to deploy). |
| `--no-deployment` | Publish the model to Knowledge Catalog and deploy nothing, after every check of the two-phase path below. It publishes the model file as written, whereas `--no-profile` publishes the binding `default_profile` names. |
| `--deployment <names>` | Deploy only these entries of the model's `deployments:` list, comma-separated or repeated, in the order given. Push still publishes the model to Knowledge Catalog first. |
| `--no-deployment` | Publish the model to Knowledge Catalog and deploy nothing, after every check of the two-phase path below. It publishes the model file as written, whereas `--no-profile` publishes the binding `default_profile` names. Push refuses it with `--deployment`, in either order. |
| `--validate-only` | Run every validation check and report pass/fail, but write nothing. |
| `--print` | Print each destination's generated artifact (BigQuery or Spanner Graph SQL DDL, Knowledge Catalog entry plan). Combine with `--validate-only` to preview without deploying. |
| `--force-remove` | Delete models in the entry group that this push no longer includes (see [Updating and removing models](README.md#updating-and-removing-models)). |
| `--emit-expressions` | Also write the SQL-expression fields (per-field `schema.semantics` and `semantic-metric.expression`) to Knowledge Catalog. Off by default: the published system-type templates do not carry them yet. Knowledge Catalog push only. |

A push with `--no-deployment` runs in two phases. The first runs every check
for Knowledge Catalog and writes nothing. The second publishes the model to
Knowledge Catalog. On this path `--no-kc`, `--profile`,
A push with `--deployment` or `--no-deployment` runs in two phases. The first
runs every check, for Knowledge Catalog and for each deployment, and writes
nothing. The second publishes the model to Knowledge Catalog, then deploys each
deployment in order with the profile it names. An AlloyDB deployment deploys no
graph. The first phase dry-runs each BigQuery graph's DDL. Spanner cannot check
a graph's DDL without running it, so a Spanner deployment whose DDL Spanner
rejects, for example one that reads a missing column, fails in the second phase,
after Knowledge Catalog is written. On this path `--no-kc`, `--profile`,
`--all-profiles` and `--no-profile` are errors, and catalog.yaml's
`default_profile` chooses no binding: Knowledge Catalog receives the model file
as written.
Expand Down Expand Up @@ -407,9 +413,11 @@ differs from BigQuery Graph in five ways:
- **Bare table and graph names.** A Spanner property graph lives inside one
database and names tables in that same database, so the generator emits bare
names — no `project.dataset.` qualifier on either the tables or the graph.
- **No `MEASURE`.** Metrics are dropped (warned per metric), never errored. The
- **No `MEASURE`.** Metrics are dropped (warned per metric). The
BigQuery-only rule that a metric must resolve to a single entity therefore does
not apply to a Spanner target (see [Validation](#validation)).
not apply to a Spanner target (see [Validation](#validation)). Push refuses a
Spanner deployment that names no profile in a model with metrics; name a
profile with `metrics_exclude: "*"`. This holds for a push with `--deployment`.
- **No per-element `OPTIONS`.** `description` / `synonyms` are not emitted into
the Spanner DDL; they ride into Knowledge Catalog instead — mirroring how
BigQuery's graph-level `OPTIONS` is dropped. See
Expand Down
12 changes: 10 additions & 2 deletions toolbox/mdcode/src/libts/gcp/spanner.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
// API client for Cloud Spanner, covering two surfaces the semantic-model push
// leg needs: Database Admin (updateDatabaseDdl and getOperation) to deploy
// Spanner Graph DDL, and the data plane (sessions and executeSql) to PLAN an
// leg needs: Database Admin (updateDatabaseDdl, getDatabase and getOperation)
// to check the database and deploy Spanner Graph DDL, and the data plane
// (sessions and executeSql) to PLAN an
// action's DML against the live database before publishing it.

import * as api from './api';
Expand Down Expand Up @@ -62,6 +63,13 @@ export class SpannerClient extends api.ApiClient {
return await this._patch<Operation>(name, {statements});
}

// Fetches a database's metadata.
async getDatabase(project: string, instance: string, database: string):
Promise<api.ApiResult<{name?: string; [key: string]: any}>> {
return await this._get(
`projects/${project}/instances/${instance}/databases/${database}`);
}

// Fetches a long-running operation by its resource name (as returned in
// Operation.name, e.g.
// `projects/.../instances/.../databases/.../operations/...`).
Expand Down
2 changes: 1 addition & 1 deletion toolbox/mdcode/src/libts/semantic/deploy_bigquery.ts
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ async function awaitQueryDone(
// location, but record a warning so the degraded path is visible -- otherwise a
// silent fallback can route a non-US CREATE to the wrong region and fail later
// with an opaque location error.
async function datasetLocation(
export async function datasetLocation(
bigQuery: BigQueryClient, target: BigQueryGraphTarget,
warnings: string[]): Promise<string|undefined> {
const res = await bigQuery.getDataset(target.project, target.dataset);
Expand Down
4 changes: 2 additions & 2 deletions toolbox/mdcode/src/libts/semantic/deployment_target.ts
Original file line number Diff line number Diff line change
Expand Up @@ -307,10 +307,10 @@ export function targetError(at: string, target: string): string {
/**
* The deployment names a `--deployment` flag selects, or undefined when the
* flag is absent or negated. The flag is repeated or comma-separated. It
* arrives as a string, as an array when it is repeated, or as a boolean when
* arrives as a string, as an array of them, or as a boolean when
* it is absent or negated. Pieces are trimmed, empty ones dropped, and a
* repeated name kept once, at its first position. Throws when any occurrence
* of the flag names no deployment. A bare occurrence among repeated ones
* of the flag names no deployment. A bare occurrence in an array
* arrives as `true` and names none.
*/
export function parseDeploymentSelection(
Expand Down
8 changes: 7 additions & 1 deletion toolbox/mdcode/src/libts/semantic/resolve_profiles.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1993,6 +1993,10 @@ export interface CompletenessOptions extends CloudSqlLoad {
// applied, the caller stops reading what the SQL reads. This check then
// reads what the profile's field expressions read, and gives the warning.
applied?: boolean;
// The caller warns about the profile's SQL that the parser cannot read. This
// check then gives that warning only when the hierarchy does not resolve
// with the profile applied.
sqlWarned?: boolean;
// The caller warns about the model file's SQL that it reads, which it does
// only when the hierarchy of the model it checks resolves. It reads an
// ANSI_SQL text in `dialect`, or as generic SQL when that is unset, and
Expand Down Expand Up @@ -2392,7 +2396,9 @@ export function validateProfileCompleteness(
return what === fieldLabel(f.slice(0, dot), f.slice(dot + 1));
});
for (const what of unreadable) {
if (applied ? !profiledError : !own.has(what) && callerWarns(what)) {
if (applied ? !profiledError :
opts.sqlWarned && !profiledError ||
!own.has(what) && callerWarns(what)) {
continue;
}
warnings.push(
Expand Down
4 changes: 2 additions & 2 deletions toolbox/mdcode/src/libts/semantic/runtime/runtime.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ export interface SemanticRuntime {
*/
storeFailed?: boolean;
/**
* Set with `storeFailed` when push rejects the model's deployments too, so
* the model is wrong whatever it is used for. One such case is a target in
* Set with `storeFailed` when the model's deployments give no store
* whatever the model is used for. One such case is a target in
* no accepted form in any deployment. Another is a GOOGLE block that is not
* valid JSON. A third is several targets for the profile with no single
* store. A table bound outside the store and a failed AlloyDB lookup are
Expand Down
11 changes: 5 additions & 6 deletions toolbox/mdcode/src/libts/semantic/runtime/store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -186,12 +186,11 @@ function strayBindings(model: SemanticModel, store: ChosenStore): string[] {
*
* `named` is set when the model's deployments name a target and still no
* store results, so a caller that runs against the store fails. `rejected` is
* set with it when push rejects those deployments too, so the model is wrong
* whatever it is used for. Push rejects a target in no accepted form in any
* deployment, whichever profile the deployment is for, and a target listed
* twice. It also rejects a GOOGLE block whose data is not valid JSON, and
* several targets for the profile that leave the store ambiguous. `named`
* alone means the model binds a table outside its Spanner or AlloyDB store.
* set with it when the deployments give no store whatever the model is used
* for: a target in no accepted form in any deployment, a target listed twice,
* a GOOGLE block whose data is not valid JSON, or several targets for the
* profile. `named` alone means the model binds a table outside its Spanner or
* AlloyDB store.
*
* Neither is set when the profile names no store, unless a target is in no
* accepted form or listed twice, or a GOOGLE block is not valid JSON. Neither
Expand Down
4 changes: 2 additions & 2 deletions toolbox/mdcode/src/libts/semantic/skills.ts
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ export interface GenerateSkillOptions {
* The skill for one model, as files a caller writes out.
*
* Returns an error for a name the Agent Skills spec would reject. It also
* returns the store error when push rejects the model's deployments, and
* returns the store error when the model's deployments are rejected, and
* when the profile names a store that cannot be resolved for a model with
* actions, whose skill names the store. A model with no actions is not an
* error -- it is a skill that says the model declares nothing to act on, which
Expand All @@ -106,7 +106,7 @@ export function generateSkill(opts: GenerateSkillOptions): SkillPackage|{
const {actions, instruction} = modelTools({runtime});
// The skill names the store only in what it says about actions. So a store
// the profile names and that cannot be resolved stops only a skill with
// actions. Deployments push rejects stop any skill.
// actions. Deployments the runtime marks rejected stop any skill.
if (runtime.deploymentsRejected || (runtime.storeFailed && actions.length)) {
return {error: runtimeStoreError(runtime) ?? ''};
}
Expand Down
2 changes: 1 addition & 1 deletion toolbox/mdcode/src/libts/semantic/validate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1952,7 +1952,7 @@ function firstLine(message: string): string {
// as `Unexpected \"$1\"`, where the detail says `Unexpected "$1"`. BigQuery
// sends no such detail, so the fallback is the normal path there, not an
// unusual one.
function storeMessage(body: string|undefined, status: number): string {
export function storeMessage(body: string|undefined, status: number): string {
const raw = (body ?? '').trim();
if (!raw) return `HTTP ${status}.`;
try {
Expand Down
Loading
Loading