Skip to content

feat(semantic): write entities, fields and field expressions into the second-generation catalog types - #556

Merged
feixufeixu merged 2 commits into
mainfrom
feat/kc-v2/entities-fields-expressions
Oct 23, 2026
Merged

feixufeixu merged 2 commits into
mainfrom
feat/kc-v2/entities-fields-expressions

Conversation

@feixufeixu

Copy link
Copy Markdown
Collaborator

With KC_V2_ASPECTS on, every entity gets an entry with the second-generation aspects, abstract entities included.

The mapping rules here were settled by Fei Xu with Dmitry Lychagin, the author of the mapping.

  • semantic-entity holds the source as written in source.resource and the entity's own primaryKey and uniqueConstraints as written. It also holds the direct parents in extends, abstract, synonyms, and one record per named profile that lists the entity, with the source, excluded fields and keys that profile states. A unique key with no column is left out.
  • A built-in reference link runs from each entity's entry to the entry of each parent in extends. Its ID is semantic-extends- plus the first 32 hex characters of the SHA-256 of <model>/<child>/<parent>. Push owns a reference link whose ID starts with semantic-extends-, as it owns the semantic-relationship- ones.
  • schema lists only the fields the entity declares. A field it inherits and only rebinds is a record in sql-expressions, with nothing in schema. Each field has its description, logical data type and metadataType, semantic: DIMENSION for a dimension, its synonyms, and annotations for label and, when declared, is_time. A field a profile excludes is still listed. schema holds no keys and no semantics block.
  • A field's instructions and custom ai_context members, its third-party blocks and its examples go in guidelines, semantic-extensions and overview at @Schema.<field> on the entity that declares the field, as the entity's own go at entry level. A member whose value holds .inf, -.inf, .nan or a number past 2^53 is left out with the same warning as on the model, given once under the entity that declares the field. An entity's update lists semantic-extensions@* and overview@* beside guidelines@*, and leaves out the exact field keys those cover.
  • Push writes an entity's whole overview from the workspace, as it does for the model: examples from the model file, and content, contentType and links from the sidecar <model>/entities/<entity>.overview.md. A field's overview holds its examples alone. An entity's update lists overview, so an overview the workspace no longer gives is removed.
  • sql-expressions holds the expressions of the entity's own fields, then those of each named profile, with the profile's name in qualifier. An expression written as a plain string gives a record with no dialect.
  • push --no-deployment publishes the model as its files declare it, with a record for each profile. Every other push still gives the catalog the model with the selected profile applied. On a graph push that model is also pruned, as before. A Google-flavor model's profile files get no record on these pushes. Under --profile or default_profile, the entity's own source, keys and records hold that profile's bindings. A vanilla model's GOOGLE-block profile is then written both untagged and with its name in qualifier.
  • The missing-key warning skips abstract entities and counts unique keys and the keys named profiles state.
  • An action parameter that projects a field a profile excludes, and that the catalog still lists, no longer warns that the push leaves the field out. An action that names an abstract entity no longer warns that the push does not publish it.

New error, from push and kcmd profiles before any write or graph deploy: entry '<id>' would hold <n> <type> aspects, ..., where <type> is overview, guidelines or semantic-extensions.

Please look closely at:

  1. declaredFields and extendsLinks in knowledge_catalog.ts.
  2. fieldExpressionsData and the record rules for each way an expression is written.
  3. instanceCountError, which refuses an entity entry that would hold 100 instances of one aspect type.

… second-generation catalog types

With KC_V2_ASPECTS on, every entity gets an entry, abstract ones included.
Its semantic-entity aspect holds the source and keys as written, its direct
parents and one record per named profile. Its schema lists its own and
inherited fields with their logical types, and a field's instructions,
custom ai_context members, third-party blocks and examples go in aspects at
the field's path. Field expressions go in sql-expressions, with the profile
that supplied each one. An entity entry that would hold 100 instances of
overview, guidelines or semantic-extensions fails the push before any write.
With KC_V2_ASPECTS on, an entity's schema lists only the fields it
declares, and a field's aspects sit on the entity that declares it. A
reference link runs from each entity to each parent it extends, with an id
that starts with semantic-extends-. Push owns those links as it owns the
semantic-relationship- ones.

Push reads an entity's overview sidecar, <model>/entities/<entity>.overview.md.
An entity's update lists overview and overview@*, so push removes an
overview the workspace no longer gives. With the flag off, the schema
aspect is built as before.

Every rule here was settled by Fei Xu with Dmitry Lychagin, the author of the
mapping.
@feixufeixu
feixufeixu requested a review from merr1988 October 23, 2026 19:21
@feixufeixu
feixufeixu added this pull request to stack #558 October 23, 2026 21:39

@merr1988 merr1988 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Fei, this lines up with the 10/21 mapping, and it's nice to see the extends links and entity sidecars land here. tsc and every suite pass, also merged with #552. One question:

[Question] The 100-instance check is per aspect type (deploy_knowledge_catalog.ts L560)
instanceCountError counts overview, guidelines and semantic-extensions separately, but today's pull reads all of an entry's semantic aspect types in one lookupEntry (pull_kc.ts L107). An entity with, say, 40 fields with instructions, 40 with examples and 30 with extension blocks passes the check, yet holds ~110 field instances plus semantic-entity/schema/sql-expressions, so a combined read would stop at 100. Is the plan for the v2 pull to read one aspect type per lookup? If so, a one-line comment saying that would make the per-type limit self-explanatory.

Approving.

@feixufeixu

Copy link
Copy Markdown
Collaborator Author

Thanks Miguel. Yes: #557 makes pull read an entry again once per aspect type when one read returns 100 aspects, so the limit holds per type. I added that sentence to the comment on instanceCountError in #557, which brings the re-read.

@feixufeixu
feixufeixu merged commit 9a5e38b into main Oct 23, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants