Repository navigation
feat(semantic): write entities, fields and field expressions into the second-generation catalog types - #556
Conversation
… second-generation catalog types With KC_V2_ASPECTS on, every entity gets an entry, abstract ones included. Its semantic-entity aspect holds the source and keys as written, its direct parents and one record per named profile. Its schema lists its own and inherited fields with their logical types, and a field's instructions, custom ai_context members, third-party blocks and examples go in aspects at the field's path. Field expressions go in sql-expressions, with the profile that supplied each one. An entity entry that would hold 100 instances of overview, guidelines or semantic-extensions fails the push before any write.
With KC_V2_ASPECTS on, an entity's schema lists only the fields it declares, and a field's aspects sit on the entity that declares it. A reference link runs from each entity to each parent it extends, with an id that starts with semantic-extends-. Push owns those links as it owns the semantic-relationship- ones. Push reads an entity's overview sidecar, <model>/entities/<entity>.overview.md. An entity's update lists overview and overview@*, so push removes an overview the workspace no longer gives. With the flag off, the schema aspect is built as before. Every rule here was settled by Fei Xu with Dmitry Lychagin, the author of the mapping.
merr1988
left a comment
There was a problem hiding this comment.
Thanks Fei, this lines up with the 10/21 mapping, and it's nice to see the extends links and entity sidecars land here. tsc and every suite pass, also merged with #552. One question:
[Question] The 100-instance check is per aspect type (deploy_knowledge_catalog.ts L560)
instanceCountError counts overview, guidelines and semantic-extensions separately, but today's pull reads all of an entry's semantic aspect types in one lookupEntry (pull_kc.ts L107). An entity with, say, 40 fields with instructions, 40 with examples and 30 with extension blocks passes the check, yet holds ~110 field instances plus semantic-entity/schema/sql-expressions, so a combined read would stop at 100. Is the plan for the v2 pull to read one aspect type per lookup? If so, a one-line comment saying that would make the per-type limit self-explanatory.
Approving.
With
KC_V2_ASPECTSon, every entity gets an entry with the second-generation aspects, abstract entities included.The mapping rules here were settled by Fei Xu with Dmitry Lychagin, the author of the mapping.
semantic-entityholds the source as written insource.resourceand the entity's ownprimaryKeyanduniqueConstraintsas written. It also holds the direct parents inextends,abstract, synonyms, and one record per named profile that lists the entity, with the source, excluded fields and keys that profile states. A unique key with no column is left out.referencelink runs from each entity's entry to the entry of each parent inextends. Its ID issemantic-extends-plus the first 32 hex characters of the SHA-256 of<model>/<child>/<parent>. Push owns areferencelink whose ID starts withsemantic-extends-, as it owns thesemantic-relationship-ones.schemalists only the fields the entity declares. A field it inherits and only rebinds is a record insql-expressions, with nothing inschema. Each field has its description, logical data type and metadataType,semantic: DIMENSIONfor a dimension, its synonyms, and annotations forlabeland, when declared,is_time. A field a profile excludes is still listed.schemaholds no keys and nosemanticsblock.ai_contextmembers, its third-party blocks and its examples go inguidelines,semantic-extensionsandoverviewat@Schema.<field>on the entity that declares the field, as the entity's own go at entry level. A member whose value holds.inf,-.inf,.nanor a number past 2^53 is left out with the same warning as on the model, given once under the entity that declares the field. An entity's update listssemantic-extensions@*andoverview@*besideguidelines@*, and leaves out the exact field keys those cover.overviewfrom the workspace, as it does for the model:examplesfrom the model file, andcontent,contentTypeandlinksfrom the sidecar<model>/entities/<entity>.overview.md. A field's overview holds its examples alone. An entity's update listsoverview, so an overview the workspace no longer gives is removed.sql-expressionsholds the expressions of the entity's own fields, then those of each named profile, with the profile's name inqualifier. An expression written as a plain string gives a record with no dialect.push --no-deploymentpublishes the model as its files declare it, with a record for each profile. Every other push still gives the catalog the model with the selected profile applied. On a graph push that model is also pruned, as before. A Google-flavor model's profile files get no record on these pushes. Under--profileordefault_profile, the entity's own source, keys and records hold that profile's bindings. A vanilla model's GOOGLE-block profile is then written both untagged and with its name inqualifier.New error, from push and
kcmd profilesbefore any write or graph deploy:entry '<id>' would hold <n> <type> aspects, ..., where<type>isoverview,guidelinesorsemantic-extensions.Please look closely at:
declaredFieldsandextendsLinksin knowledge_catalog.ts.fieldExpressionsDataand the record rules for each way an expression is written.instanceCountError, which refuses an entity entry that would hold 100 instances of one aspect type.