Adding ontology converters

[vmihalovski]

OSI Ontology Converters — Python Project

This PR introduces converters/ontology as a standalone, installable Python project providing bidirectional conversion between ontology formats used across the OSI ecosystem. It adds a complete package: the in-memory OSI ontology model, spec (de)serialisation, a Palantir ingestion layer, three converters, a runnable CLI script, and a test suite.

34 files, ~8.2k lines added. Everything lands under converters/ontology/ — no existing files are modified.

Model layer

• src/osi/model.py — the in-memory OSI ontology model, organised in three layers:

  • ontology — Concept (entity / value / component types), Relationship, Role, RelationshipMultiplicity, Formula, OntologyComponent, OsiOntology
  • semantic — Dataset, DatasetField, Dimension, JoinPath, Metric, DialectExpression / DialectExpressionSet, SemanticModel
  • mapping — ObjectMapping, ReferentMapping, LinkMapping, ConceptMapping, OntologyMapping

  Also includes verbalization parsing (parse_verbalizations) that splits madlib-style relationship text into roles using the dash convention, and FormulaFactory / MappingFormulaFactory extension points so downstream consumers can plug in their own expression parsers for derived_by / requires and mapping expressions.

• src/osi/spec.py — pydantic DTOs mirroring the OSI YAML specification (OsiSpec and friends), used as the serialisation boundary.

• src/osi/__init__.py — a curated public API surface, so consumers import from osi rather than deep sub-paths.

Converters

Converter	Direction	Notes
palantir_to_osi	Palantir ontology → OSI model	maps object types, properties, and datasets; handles many-to-one, many-to-many, and intermediary relations; derives identifying fields and falls back gracefully when none are found; attaches link mappings to concept mappings
osi_to_spec	OSI model → OSI spec YAML	emits non-built-in concepts in dependency order, including ontology-level and concept-level requires
spec_to_osi	OSI spec YAML/JSON → OSI model	completes the round trip; topologically sorts the concept dependency graph before construction

OsiParser (src/osi/parser/) is the entrypoint for reading a spec file (YAML or JSON, UTF-8 pinned for reproducibility) and returning an OsiOntology.

Palantir ingestion

• src/osi/external/palantir/model.py — DTOs for a Palantir export: Ontology, ObjectType, Property, DataSet / DataSetColumn, and the relation hierarchy (ManyToOneRelation, ManyToManyRelation, IntermediaryRelation).
• src/osi/external/palantir/parser/ — PalantirParser accepts a .zip archive or an already-extracted folder, and tolerates both top-level and single-root-directory layouts. It parses the ontology JSON plus a data_sets/ folder of dataset specs, handles both old- and new-style relation encodings, and raises clear errors for ambiguous or missing inputs.

Common utilities

• common/graph.py — topological sort, cycle detection, and a cycle-breaking variant used to order concept and mapping dependencies.
• common/file_utils.py — zip/directory traversal helpers with input validation.
• common/utils.py — identifier and verbalization string normalisation.

Tests

A pytest suite (tests/, 7 files) driven off the flights example:

• test_osi_parser.py — spec → model conversion: document metadata, ontology- and concept-level requires, value-type inheritance, identifiers, relationship multiplicity, ontology mappings, YAML/JSON loading, error handling, formula-factory isolation between parsers, and round-trip structural invariants
• test_palantir_parser.py — every supported export layout (top-level zip, single-root zip, extracted dir, single-root dir, dir wrapping a zip) plus the unsupported/invalid cases (missing path, non-zip file, missing or empty data_sets, ambiguous top-level JSON)
• test_verbalization_parsing.py — prefix/postfix splitting, multi-word decoration, ternary relationships, built-in concept resolution, and error cases
• test_flights_snapshot.py — structure and round-trip YAML snapshots (pytest-snapshot), which also assert the output is deterministic for fixed input
• test_examples_in_sync.py — guards the vendored tests/fixtures/flights.yaml against drift from the canonical examples/flights.yaml. Test inputs are vendored so the suite runs from an sdist/wheel or a subset checkout; this test skips when examples/ isn't present.

Run with pytest from converters/ontology.

Project setup

• pyproject.toml — runtime deps (pydantic, pyyaml), dev extras (pytest, pytest-snapshot, pip-tools), src/ layout, pytest and pyright config; requires Python ≥ 3.11
• requirements.lock — fully reproducible installs, generated from pyproject.toml via pip-compile
• src/osi/ layout — importable as osi after installation
• .gitignore — Python artifacts, virtualenvs, pyenv, VS Code, JetBrains
• README.md — pyenv/virtualenv setup, lock-file regeneration, usage, and script docs

Script

scripts/palantir_to_osi.py — takes a Palantir export (.zip or extracted folder) and writes OSI YAML to stdout, warnings to stderr. Snowflake database and schema names used to qualify table references are configurable via SNOWFLAKE_DATABASE_NAME and SNOWFLAKE_SCHEMA_NAME (both default to PALANTIR).

Not included

No CI wiring — the suite is currently run locally.

[Copilot]

Pull request overview

Introduces a new standalone Python package under converters/ontology that implements OSI ontology converters and supporting parsing/model layers, enabling bidirectional conversion between Palantir exports, an OSI runtime model, and an OSI YAML spec format.

Changes:

• Added Pydantic DTO schema (OsiSpec) plus YAML load/dump helpers for OSI spec serialization.
• Added a runtime ontology + semantic model implementation and converters: Spec ↔ OSI model, Palantir → OSI model.
• Added Palantir ZIP parsing utilities/models and a CLI script + packaging/docs scaffolding for installable usage.

Reviewed changes

Copilot reviewed 16 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
converters/ontology/src/osi/spec.py	Defines the Pydantic spec DTOs and YAML load/dump helpers.
converters/ontology/src/osi/model.py	Adds the runtime ontology/semantic/mapping model used by converters.
converters/ontology/src/osi/parser/init.py	Adds an OsiParser entrypoint for reading YAML/JSON into the runtime model.
converters/ontology/src/osi/converter/spec_to_osi/converter.py	Implements Spec → runtime model conversion (incl. mapping-expression resolution).
converters/ontology/src/osi/converter/osi_to_spec/converter.py	Implements runtime model → Spec DTO conversion and mapping-expression rendering.
converters/ontology/src/osi/external/palantir/parser/init.py	Adds Palantir export ZIP parsing (ontology + dataset specs).
converters/ontology/src/osi/external/palantir/model.py	Adds Palantir ontology/data model types used by the parser/converter.
converters/ontology/src/osi/converter/palantir_to_osi/converter.py	Implements Palantir → runtime model conversion (concepts, relationships, mappings, datasets).
converters/ontology/src/osi/common/utils.py	Adds small shared string/name utilities used by converters/parsers.
converters/ontology/src/osi/common/graph.py	Adds topo-sort utilities used for dependency ordering (incl. cycle breaking).
converters/ontology/src/osi/common/file_utils.py	Adds ZIP helpers for reading top-level and dataset JSON files.
converters/ontology/src/osi/init.py	Defines the package’s intended public API surface (osi.*).
converters/ontology/scripts/palantir_to_osi.py	Adds a runnable CLI script for Palantir ZIP → OSI YAML conversion.
converters/ontology/pyproject.toml	Adds Python project packaging/configuration for the new package.
converters/ontology/requirements.lock	Adds a pinned dependency lock file for reproducible installs.
converters/ontology/README.md	Adds setup and usage documentation for the package/script.
converters/ontology/.gitignore	Adds Python-focused ignores for the new project directory.
converters/ontology/src/osi/converter/init.py	Package init for converter modules.
converters/ontology/src/osi/converter/spec_to_osi/init.py	Package init for Spec → OSI converter.
converters/ontology/src/osi/converter/osi_to_spec/init.py	Package init for OSI → Spec converter.
converters/ontology/src/osi/converter/palantir_to_osi/init.py	Package init for Palantir → OSI converter.
converters/ontology/src/osi/external/init.py	Package init for external integrations.
converters/ontology/src/osi/external/palantir/init.py	Package init for Palantir integration.
converters/ontology/src/osi/common/init.py	Package init for common utilities.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 24 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 24 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 30 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 30 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 31 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 33 changed files in this pull request and generated 3 comments.

[jbonofre]

License headers + CI workflow

Two blocking ASF-convention gaps from the review, with fixes attached.

1. License headers — 21 inline suggestions below add the standard Apache-2.0 header (one-click Apply). The 8 empty __init__.py files need it too but have no suggestable line; the generated requirements.lock, .gitignore, and the tests/snapshots/** files should stay header-less (a header in the round-trip snapshot would break that test — exclude them via RAT instead).

2. CI workflow — every other converter ships .github/workflows/converter-<name>-ci.yml; this PR has none, so the suite never runs on PRs. New file .github/workflows/converter-ontology-ci.yml (mirrors converter-omni-ci.yml, adapted to this converter's pip/requirements.lock setup and 3.11+ floor):

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

name: Converters Ontology CI

on:
  push:
    branches: [ "main" ]
    paths:
      - 'converters/ontology/**'
      - '.github/workflows/converter-ontology-ci.yml'
  pull_request:
    branches: [ "main" ]
    paths:
      - 'converters/ontology/**'
      - '.github/workflows/converter-ontology-ci.yml'

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12", "3.13", "3.14"]

    steps:
      - name: Checkout project
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        working-directory: converters/ontology
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.lock
          pip install -e ".[dev]"

      - name: Unit Tests
        working-directory: converters/ontology
        run: |
          pytest

I verified the header additions + these exact CI steps in a clean venv: 52 passed. (Couldn't push directly — Allow edits by maintainers is off on this PR.)

[jbonofre]

At least the ASF header is missing.

Also, are you sure about the name? Is it ontology converter or palantir converter?

Can you also rename from OSI to Ossie?

[vmihalovski]

@jbonofre thank you for the detailed review — all points addressed.

• Allow edits by maintainers — enabled.

• OSI → Ossie naming — renamed throughout. The package is now ossie_ontology (was osi) and the distribution apache-ossie-ontology (was ontology), with the module paths and public types following: converter/palantir_to_ossie/, converter/ossie_to_spec/, converter/spec_to_ossie/, and OssieOntology, OssieParser, OssieSpec, PalantirToOssieConverter, OssieToSpecConverter, SpecToOssieConverter. Done with git mv so file history is preserved. The on-disk YAML format contains no osi tokens, so it is unchanged and the fixtures did not move.

• License headers — now on every file in the package. Rather than excluding the snapshots via RAT, I made the header part of the value the snapshot tests assert: tests/conftest.py overrides the snapshot fixture to prepend it, so it applies both when comparing and when regenerating with --snapshot-update, and the tests themselves stay free of boilerplate. The header therefore cannot drift out of sync with the snapshots. # starts a comment in YAML, so flights_roundtrip.yaml is still parseable as an Ossie document.

The one remaining file without a header is the generated requirements.lock, since pip-compile rewrites it and would drop the header — consistent with how the other Python converters treat their uv.lock.

• CI — added .github/workflows/converter-ontology-ci.yml, mirroring converter-omni-ci.yml and adapted to this converter's pip/requirements.lock setup, over a 3.11–3.14 matrix. I verified the install steps and the suite locally: 52 tests pass.

• On the ontology package name — it's deliberately format-neutral rather than named after any one source. The package holds several converters already (Palantir → Ossie, Ossie ↔ Spec YAML) and we expect to add more, most likely OWL, so naming it after a single format would age badly. As apache-ossie-ontology under converters/ontology it also stays consistent with the other converter packages.

Could you take another look when you get a chance? Thanks again.