diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 94fdf54b..6fcc9656 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -85,8 +85,8 @@ jobs: suite: - name: pipelines args: tests/cdm_data_loaders/pipelines/ - - name: docker - args: --run-docker -m "docker and not requires_spark and not requires_ceph" tests/cdm_data_loaders/pipelines/ + # - name: docker + # args: --run-docker -m "docker and not requires_spark and not requires_ceph" tests/cdm_data_loaders/pipelines/ - name: integration args: tests/integration/ - name: everything else diff --git a/docs/xsd_to_pyarrow_specification.md b/docs/xsd_to_pyarrow_specification.md new file mode 100644 index 00000000..2744c0a9 --- /dev/null +++ b/docs/xsd_to_pyarrow_specification.md @@ -0,0 +1,1122 @@ +# XSD-to-PyArrow Specification + +This document specifies how an XML Schema (`.xsd`) and target `xml_tag` determine: +1. The **PyArrow `pa.Schema`** for the table yielded by the pipeline. +2. The **deterministic Python / JSON row dictionaries** produced by schema-driven XML element conversion (`convert_xml_element`) before constructing `pa.Table.from_pylist(rows, schema=pa_schema)`. +3. The **error conditions** raised when an XSD or `xml_tag` uses unsupported or ambiguous constructs. + +--- + +## 1. Summary of Conversion Rules + +### 1.1 Top-Level `xml_tag` Resolution & Unwrapping +- **Resolution:** Search the XSD for `xml_tag` starting at depth 0 (global root elements) and depth 1 (direct children of global root elements), deduplicating matches by `id(element.type)`. + - If **0** elements match (even after falling back to deeper elements), raise an error. + - If **>1** distinct `XsdType` definitions match at the shallowest matching depth, raise an ambiguity error. +- **Unwrapping:** Each streamed `<{xml_tag}>` element becomes one row in the output `pa.Table`. + - `xml_tag` must have a `complexType` that maps to a `pa.StructType` (at least one attribute, `#text`, or child element field). The outer `xml_tag` key is unwrapped so that the top-level columns of `pa.Schema` are the struct fields of `xml_tag`. + - If `xml_tag` maps to a scalar, list, or empty marker type with no attributes and no struct fields, schema generation raises a `ValueError` (see Error E3 in Section 3). + +### 1.2 Naming & Field Ordering +- **Attributes:** Prefixed with `_` (e.g., `id` $\rightarrow$ `_id`). Namespace declarations (`xmlns`, `xmlns:*`) and `xsi:*` attributes (`xsi:nil`, `xsi:schemaLocation`, `xsi:noNamespaceSchemaLocation`) are stripped. +- **Text content (`#text`):** When a complex type has both attributes and text (`simpleContent` with attributes, or `mixed="true"` with attributes), text is stored under `#text`. +- **Struct Field Order:** + 1. Declared attributes (`_attr`), in XSD declaration order (base type attributes before extension attributes). + 2. `#text` (if applicable). + 3. Child elements, in XSD content-model traversal order (deduplicated on first appearance if the same element name appears in multiple branches of a `` or ``). + +### 1.3 Element Classification (Shape) +| XSD Element Type | Cardinality (`overall_max_occurs`) | Python / JSON Value | PyArrow DataType | +| :--- | :--- | :--- | :--- | +| **`xs:simpleType`** (atomic) | `1` | Scalar (`str`, `int`, `float`, `bool`, `date`, `bytes`) or `None` | Scalar `pa.DataType` | +| **`xs:simpleType`** (atomic) | `> 1` or `unbounded` | `list[Scalar \| None]` or `None` (when absent) | `pa.list_(Scalar)` | +| **`xs:simpleType`** (`xs:list`) | `1` | `list[ItemScalar]` or `None` | `pa.list_(ItemScalar)` | +| **`xs:simpleType`** (`xs:list`) | `> 1` or `unbounded` | `list[list[ItemScalar] \| None]` or `None` | `pa.list_(pa.list_(ItemScalar))` | +| **`complexType` (`simpleContent`)** without attributes | `1` / `> 1` | Same as underlying `simpleType` | Same as underlying `simpleType` | +| **`complexType` (`simpleContent`)** with attributes | `1` | `{"_attr": ..., "#text": Scalar}` or `None` | `pa.struct([("_attr", ...), ("#text", Scalar)])` | +| **`complexType` (`mixed="true"`)** without attributes | `1` | Raw inner XML string `str` or `None` | `pa.string()` | +| **`complexType` (`mixed="true"`)** with attributes | `1` | `{"_attr": ..., "#text": str}` or `None` | `pa.struct([("_attr", ...), ("#text", pa.string())])` | +| **`complexType` (attribute-only)** | `1` | `{"_attr": ...}` or `None` | `pa.struct([("_attr", ...)])` | +| **`complexType` (element-only)** | `1` | `{"_attr": ..., "child": ...}` or `None` | `pa.struct([...])` | +| **`complexType` (empty: no attrs, no children, not mixed)** | `1` / `> 1` | `""` (when present) or `None` (when absent) / `["", ...]` | `pa.string()` / `pa.list_(pa.string())` | + +*(For any `complexType` row above with `overall_max_occurs > 1` or `unbounded`, the Python value is `list[...]` and the PyArrow type is `pa.list_(...)`. When structural recursion exceeds `max_recursion_depth`, a single recursive element becomes `pa.string()` and a repeated recursive element (`overall_max_occurs > 1`) becomes `pa.list_(pa.string())`, where each truncated item is serialized as a JSON string.)* + +### 1.4 Field Nullability (`nullable=False` vs `nullable=True`) & Empty Tag Coercion +- **Attributes (`_attr`):** + - `nullable=False` when `use="required"` (and the enclosing element is not a `nillable="true"` element that can be nilled without attributes). + - `nullable=True` when `use="optional"` (default). +- **Child Elements:** + - `nullable=False` when **all** of the following hold: + 1. The element particle has `minOccurs >= 1`. + 2. The element declaration has `nillable="false"` (default). + 3. Every enclosing compositor/group (``, ``, ``) up to the parent `complexType` has `minOccurs >= 1`, and the element is not conditional inside an `` branch (unless required in every branch of the choice). + 4. The parent element is not `nillable="true"` (since a parent with `xsi:nil="true"` and attributes, e.g. ``, produces a struct `{"_type": "function"}` whose child elements are `None`). + - `nullable=True` in all other cases. +- **`#text` Field Inside a Struct with Attributes:** + - `nullable=True` for `mixed="true"` types, `nillable="true"` elements, and string-based `simpleContent` types (where `` has no `#text` key). + - `nullable=False` for non-nillable `simpleContent` types with non-string atomic base types (e.g., `xs:int`, `xs:decimal`), where valid XML always provides lexical text. +- **Empty Standalone Tag Coercion (`` $\rightarrow$ `""`):** + - When a standalone element mapped to `pa.string()` (atomic string `simpleType`, string `simpleContent` without attributes, `mixed="true"` without attributes, or empty `complexType` marker) is **present** in the XML as `` or `` (without `xsi:nil="true"`), its value is coerced to `""` so that present empty elements satisfy `nullable=False` and are distinguishable from omitted optional elements (`None`). + +### 1.5 Built-In XSD Scalar Type Mapping & Field Metadata +| XSD Built-In Type(s) | Python Type (Coerced) | PyArrow `DataType` | +| :--- | :--- | :--- | +| `xs:string`, `xs:normalizedString`, `xs:token`, `xs:language`, `xs:Name`, `xs:NCName`, `xs:NMTOKEN`, `xs:ID`, `xs:IDREF`, `xs:ENTITY`, `xs:anyURI`, `xs:QName`, `xs:NOTATION` | `str` | `pa.string()` | +| `xs:boolean` (`"true"`/`"1"` $\rightarrow$ `True`, `"false"`/`"0"` $\rightarrow$ `False`) | `bool` | `pa.bool_()` | +| `xs:byte` | `int` | `pa.int8()` | +| `xs:short` | `int` | `pa.int16()` | +| `xs:int` | `int` | `pa.int32()` | +| `xs:long`, `xs:integer`, `xs:positiveInteger`, `xs:nonNegativeInteger`, `xs:negativeInteger`, `xs:nonPositiveInteger` | `int` | `pa.int64()` | +| `xs:unsignedByte` | `int` | `pa.uint8()` | +| `xs:unsignedShort` | `int` | `pa.uint16()` | +| `xs:unsignedInt` | `int` | `pa.uint32()` | +| `xs:unsignedLong` | `int` | `pa.uint64()` | +| `xs:float`, `xs:double`, `xs:decimal` | `float` | `pa.float64()` | +| `xs:date` | `datetime.date` | `pa.date32()` | +| `xs:dateTime`, `xs:time`, `xs:duration`, `xs:gYear`, `xs:gYearMonth`, `xs:gMonthDay`, `xs:gDay`, `xs:gMonth` | `str` | `pa.string()` | +| `xs:base64Binary` | `bytes` (`base64.b64decode`) | `pa.binary()` | +| `xs:hexBinary` | `bytes` (`bytes.fromhex`) | `pa.binary()` | +| `xs:NMTOKENS`, `xs:IDREFS`, `xs:ENTITIES` | `list[str]` (`.split()`) | `pa.list_(pa.string())` | + +- **`xs:restriction`:** Inherits the primitive/built-in base type mapping above. +- **`xs:list`:** Maps to `pa.list_(item_type)` and coerces by splitting whitespace (`value.split()`) and coercing each item. +- **`xs:union`:** If all `memberTypes` map to the same `pa.DataType`, uses that type; otherwise falls back to `pa.string()` (`str`). +- **`pa.Field.metadata`:** + - **`"xsd_type"`:** Recorded on every leaf scalar `pa.Field` (including `#text` and list `"item"` fields when list elements are scalars) with the prefixed built-in XSD type name (e.g., `"xs:string"`, `"xs:int"`, `"xs:float"`, `"xs:double"`, `"xs:decimal"`, or `"xs:date | xs:gYearMonth | xs:gYear"` for unions). Omitted on `pa.struct(...)`, outer `pa.list_(...)`, empty-marker `pa.string()`, and recursion-truncated JSON `pa.string()` fields. + - **`"comment"`:** Extracted from `` on the element or attribute declaration (falling back to its referenced `type`'s `` if the declaration itself has none), whitespace-normalized with `" ".join(doc.split())`. + +--- + +## 2. Specification Examples + +### Example 1: Scalar Types, Restrictions, Lists (`xs:list`), and Unions (`xs:union`) +Covers built-in numeric/boolean/date/binary scalars, `xs:restriction` enumerations, `xs:list` (`intListType` from [`uniprot.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/uniprot/uniprot_kb/uniprot.xsd#L1122-L1124)), and `xs:union` (`date | gYearMonth | gYear` from [`uniprot.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/uniprot/uniprot_kb/uniprot.xsd#L341-L345)). + +#### XSD +```xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +#### Sample XML (`xml_tag="record"`) +```xml + + + true + -5 + 1024 + 100000 + 999999999999 + 18446744073709551615 + 1.5 + 2.718281828 + 3.14 + 2025-06-15 + 2025-06 + 42 + 10 20 30 + 48656c6c6f + SGVsbG8= + + + 0 + + +``` + +#### Python / JSON Translation +```python +[ + { + "_id": "rec1", + "_status": "active", + "_evidence": [1, 4, 9], + "_version": 2, + "flag": True, + "smallInt": -5, + "medInt": 1024, + "stdInt": 100000, + "bigInt": 999999999999, + "ulong": 18446744073709551615, + "f32": 1.5, + "f64": 2.718281828, + "dec": 3.14, + "created": datetime.date(2025, 6, 15), + "published": "2025-06", + "code": "42", # int + short -> different Arrow types (int32 vs int16) -> falls back to str + "scores": [10, 20, 30], + "rawHex": b"Hello", + "rawB64": b"Hello", + }, + { + "_id": "rec2", + "_status": "archived", + "_version": 1, + "flag": False, + }, +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:ID"}), + pa.field("_status", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "_evidence", + pa.list_(pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"})), + nullable=True, + ), + pa.field("_version", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field("flag", pa.bool_(), nullable=False, metadata={"xsd_type": "xs:boolean"}), + pa.field("smallInt", pa.int8(), nullable=True, metadata={"xsd_type": "xs:byte"}), + pa.field("medInt", pa.int16(), nullable=True, metadata={"xsd_type": "xs:short"}), + pa.field("stdInt", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field("bigInt", pa.int64(), nullable=True, metadata={"xsd_type": "xs:integer"}), + pa.field("ulong", pa.uint64(), nullable=True, metadata={"xsd_type": "xs:unsignedLong"}), + pa.field("f32", pa.float64(), nullable=True, metadata={"xsd_type": "xs:float"}), + pa.field("f64", pa.float64(), nullable=True, metadata={"xsd_type": "xs:double"}), + pa.field("dec", pa.float64(), nullable=True, metadata={"xsd_type": "xs:decimal"}), + pa.field("created", pa.date32(), nullable=True, metadata={"xsd_type": "xs:date"}), + pa.field("published", pa.string(), nullable=True, metadata={"xsd_type": "xs:date | xs:gYearMonth | xs:gYear"}), + pa.field("code", pa.string(), nullable=True, metadata={"xsd_type": "xs:int | xs:short"}), + pa.field( + "scores", + pa.list_(pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"})), + nullable=True, + ), + pa.field("rawHex", pa.binary(), nullable=True, metadata={"xsd_type": "xs:hexBinary"}), + pa.field("rawB64", pa.binary(), nullable=True, metadata={"xsd_type": "xs:base64Binary"}), + ] +) +``` +*(In Examples 2–9 below, `metadata={"xsd_type": ...}` on scalar fields is omitted for brevity.)* + +--- + +### Example 2: Namespaces & `nillable="true"` (`xsi:nil="true"`) +Covers `targetNamespace`, `elementFormDefault="qualified"`, stripping `xmlns` / `xmlns:xsi`, and the 3 cases of `xsi:nil="true"`: +1. Nillable element with **no other attributes** $\rightarrow$ `None`. +2. Nillable complex element **with attributes** (like `` in [`uniprot.xsd` L47](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/uniprot/uniprot_kb/uniprot.xsd#L47)) $\rightarrow$ `{"_type": "function"}` (with `_xsi:nil` stripped and child fields `null`). +3. Nillable `simpleContent` element **with attributes** $\rightarrow$ `{"_lang": "en"}` (`#text` is `null`). + +#### XSD +```xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +#### Sample XML (`xml_tag="entry"`) +```xml + + + + <label lang="en" xsi:nil="true"/> + <comment type="function" xsi:nil="true"/> + <comment type="subunit"> + <text>Homodimer</text> + </comment> + </entry> + <entry id="e2"> + <title>Normal Title + + +``` + +#### Python / JSON Translation +```python +[ + { + "_id": "e1", + "title": None, + "label": {"_lang": "en"}, + "comment": [ + {"_type": "function"}, + {"_type": "subunit", "text": "Homodimer"}, + ], + }, + { + "_id": "e2", + "title": "Normal Title", + "label": None, + }, +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field("_id", pa.string(), nullable=False), + pa.field("title", pa.string(), nullable=True), + pa.field( + "label", + pa.struct( + [ + pa.field("_lang", pa.string(), nullable=True), + pa.field("#text", pa.string(), nullable=True), + ] + ), + nullable=True, + ), + pa.field( + "comment", + pa.list_( + pa.struct( + [ + pa.field("_type", pa.string(), nullable=False), + pa.field("text", pa.string(), nullable=True), + ] + ) + ), + nullable=True, + ), + ] +) +``` + +--- + +### Example 3: Mixed Content Kept as Raw Text +Covers `mixed="true"` elements with inline HTML/formatting tags (``, ``, ``, ``, ``), both with and without attributes on the mixed element itself (as in [`pubmed.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/pubmed/pubmed.xsd) `ArticleTitle`, `AbstractText`, and [`10_combinations/schema.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/schema.xsd) `ParaType`, `NoteType`, `RemarkType`). + +- Child elements declared inside a `mixed="true"` complexType are **not** emitted as struct fields; instead, the mixed element's entire inner XML is captured as a raw text string. +- Character entities in text nodes (`&`, `<`, `>`) remain XML-escaped in the captured raw text string so that literal `<` or `>` is distinguishable from inline HTML tags like ``. +- If the `mixed="true"` type has **no attributes**, its PyArrow type is `pa.string()` (and an empty tag `` without `xsi:nil="true"` coerces to `""`). +- If the `mixed="true"` type **has attributes**, its PyArrow type is `pa.struct([("_attr", ...), ("#text", pa.string())])`. + +#### XSD +```xml +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:group name="inlineFormatting"> + <xs:choice> + <xs:element name="b" type="formattedText"/> + <xs:element name="i" type="formattedText"/> + <xs:element name="sup" type="formattedText"/> + </xs:choice> + </xs:group> + + <xs:complexType name="formattedText" mixed="true"> + <xs:group ref="inlineFormatting" minOccurs="0" maxOccurs="unbounded"/> + </xs:complexType> + + <xs:complexType name="abstractTextType" mixed="true"> + <xs:group ref="inlineFormatting" minOccurs="0" maxOccurs="unbounded"/> + <xs:attribute name="Label" type="xs:string"/> + </xs:complexType> + + <xs:element name="articles"> + <xs:complexType> + <xs:sequence> + <xs:element name="article" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="title" type="formattedText"/> + <xs:element name="abstractText" type="abstractTextType" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> +``` + +#### Sample XML (`xml_tag="article"`) +```xml +<articles> + <article> + <title> Ca<sup>2+</sup> signaling in <i>E. <b>coli</b></i> cells + Studied bold & italic text. + Unlabelled paragraph with tags. + + + +
+ + </article> +</articles> +``` + +#### Python / JSON Translation +```python +[ + { + "title": "Ca<sup>2+</sup> signaling in <i>E. <b>coli</b></i> cells", + "abstractText": [ + {"_Label": "BACKGROUND", "#text": "Studied <b>bold</b> & <i>italic</i> text."}, + {"#text": "Unlabelled paragraph with <b>tags</b>."}, + {"_Label": "EMPTY"}, + None, + ], + }, + { + "title": "", + }, +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field("title", pa.string(), nullable=False), + pa.field( + "abstractText", + pa.list_( + pa.struct( + [ + pa.field("_Label", pa.string(), nullable=True), + pa.field("#text", pa.string(), nullable=True), + ] + ) + ), + nullable=True, + ), + ] +) +``` + +--- + +### Example 4: Structural Recursion with Configurable `max_recursion_depth` +Covers both: +1. **Type self-recursion** (`TreeType` $\rightarrow$ `branch: TreeType`) +2. **Element-ref self-recursion** (`Section` $\rightarrow$ `<xs:element ref="Section"/>` as in [`pubmed.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/pubmed/pubmed.xsd)) +3. **Mutual recursion** (`NodeA` $\rightarrow$ `NodeB` $\rightarrow$ `NodeA`) + +With `max_recursion_depth = 2`: +- Occurrence 1 (top-level `Section` or `tree`) = depth 1 (`pa.struct`) +- Occurrence 2 (child `Section` or `branch`) = depth 2 (`pa.struct`) +- Occurrence 3 (grandchild `Section` or `branch` beyond `max_recursion_depth=2`) = flattened to JSON string (`pa.string()` when `overall_max_occurs == 1`, or `pa.list_(pa.string())` when `overall_max_occurs > 1`). + +#### XSD +```xml +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:complexType name="TreeType"> + <xs:sequence> + <xs:element name="branch" type="TreeType" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="label" type="xs:string"/> + </xs:complexType> + + <xs:element name="doc"> + <xs:complexType> + <xs:sequence> + <xs:element name="record" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="tree" type="TreeType" minOccurs="0"/> + </xs:sequence> + <xs:attribute name="id" type="xs:string" use="required"/> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> +``` + +#### Sample XML (`xml_tag="record"`, `max_recursion_depth=2`) +```xml +<doc> + <record id="r1"> + <tree label="L1"> + <branch label="L2"> + <branch label="L3"> + <branch label="L4"/> + </branch> + <branch label="L3_sibling"/> + </branch> + </tree> + </record> +</doc> +``` + +#### Python / JSON Translation (`max_recursion_depth=2`) +```python +[ + { + "_id": "r1", + "tree": { + "_label": "L1", + "branch": [ + { + "_label": "L2", + "branch": [ + '{"_label": "L3", "branch": [{"_label": "L4"}]}', + '{"_label": "L3_sibling"}', + ], + } + ], + }, + } +] +``` + +#### PyArrow Schema (`max_recursion_depth=2`) +```python +pa.schema( + [ + pa.field("_id", pa.string(), nullable=False), + pa.field( + "tree", + pa.struct( + [ # depth 1 of TreeType + pa.field("_label", pa.string(), nullable=True), + pa.field( + "branch", + pa.list_( + pa.struct( + [ # depth 2 of TreeType + pa.field("_label", pa.string(), nullable=True), + pa.field( + "branch", pa.list_(pa.string()), nullable=True + ), # depth > 2 -> JSON string per item + ] + ) + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] +) +``` + +--- + +### Example 5: Model Groups (`xs:group`), Attribute Groups (`xs:attributeGroup`), `<xs:all>`, and Compositors with Duplicate Element Names +Covers: +- `<xs:group ref="..."/>` (including with `maxOccurs="unbounded"`) +- `<xs:attributeGroup ref="..."/>` +- Unrepeated `<xs:choice>` where the **same element name** appears in multiple branches (e.g., `(StartPage, EndPage?, MedlinePgn?) | MedlinePgn` from [`pubmed.xsd` L672–679](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/pubmed/pubmed.xsd#L672-L679)) $\rightarrow$ `MedlinePgn` has `overall_max_occurs == 1`, so it is a **single** `pa.string()` field emitted once. +- `<xs:sequence>` where the **same element name** appears twice (`<xs:element name="interactant"/><xs:element name="interactant"/>`) $\rightarrow$ `overall_max_occurs == 2`, so it is a **list** field emitted once. +- `<xs:all>` compositor. + +#### XSD +```xml +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:attributeGroup name="auditAttrs"> + <xs:attribute name="createdBy" type="xs:string"/> + <xs:attribute name="version" type="xs:int"/> + </xs:attributeGroup> + + <xs:group name="kvGroup"> + <xs:sequence> + <xs:element name="k" type="xs:string"/> + <xs:element name="v" type="xs:int"/> + </xs:sequence> + </xs:group> + + <xs:complexType name="paginationType"> + <xs:choice> + <xs:sequence> + <xs:element name="StartPage" type="xs:string"/> + <xs:element name="EndPage" type="xs:string" minOccurs="0"/> + <xs:element name="MedlinePgn" type="xs:string" minOccurs="0"/> + </xs:sequence> + <xs:element name="MedlinePgn" type="xs:string"/> + </xs:choice> + </xs:complexType> + + <xs:complexType name="metadataAllType"> + <xs:all> + <xs:element name="author" type="xs:string"/> + <xs:element name="year" type="xs:int" minOccurs="0"/> + </xs:all> + </xs:complexType> + + <xs:element name="dataset"> + <xs:complexType> + <xs:sequence> + <xs:element name="item" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="pagination" type="paginationType" minOccurs="0"/> + <xs:element name="meta" type="metadataAllType" minOccurs="0"/> + <xs:element name="pairs" minOccurs="0"> + <xs:complexType> + <xs:group ref="kvGroup" maxOccurs="unbounded"/> + </xs:complexType> + </xs:element> + </xs:sequence> + <xs:attributeGroup ref="auditAttrs"/> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> +``` + +#### Sample XML (`xml_tag="item"`) +```xml +<dataset> + <item createdBy="alice" version="1"> + <pagination> + <StartPage>10</StartPage> + <EndPage>20</EndPage> + <MedlinePgn>10-20</MedlinePgn> + </pagination> + <meta> + <year>2025</year> + <author>Bob</author> + </meta> + <pairs> + <k>a</k> + <v>1</v> + <k>b</k> + <v>2</v> + </pairs> + </item> + <item> + <pagination> + <MedlinePgn>e12345</MedlinePgn> + </pagination> + </item> +</dataset> +``` + +#### Python / JSON Translation +```python +[ + { + "_createdBy": "alice", + "_version": 1, + "pagination": { + "StartPage": "10", + "EndPage": "20", + "MedlinePgn": "10-20", + }, + "meta": { + "author": "Bob", + "year": 2025, + }, + "pairs": { + "k": ["a", "b"], + "v": [1, 2], + }, + }, + { + "pagination": { + "MedlinePgn": "e12345", + }, + }, +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field("_createdBy", pa.string(), nullable=True), + pa.field("_version", pa.int32(), nullable=True), + pa.field( + "pagination", + pa.struct( + [ + pa.field("StartPage", pa.string(), nullable=True), + pa.field("EndPage", pa.string(), nullable=True), + pa.field("MedlinePgn", pa.string(), nullable=True), + ] + ), + nullable=True, + ), + pa.field( + "meta", + pa.struct( + [ + pa.field("author", pa.string(), nullable=False), + pa.field("year", pa.int32(), nullable=True), + ] + ), + nullable=True, + ), + pa.field( + "pairs", + pa.struct( + [ + pa.field("k", pa.list_(pa.string()), nullable=False), + pa.field("v", pa.list_(pa.int32()), nullable=False), + ] + ), + nullable=True, + ), + ] +) +``` + +--- + +### Example 6: Context-Dependent Same `(parent, child)` Names with Different Types +In [`06_context_dependent_paths/schema.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/06_context_dependent_paths/schema.xsd) and [`10_combinations/schema.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/schema.xsd#L195-L226), `left/node/leaf` is a single `xs:string` while `right/node/leaf` is an unbounded `LabelType` (complex `simpleContent` with attribute `lang`). +Because the full ancestor path from `xml_tag` (`("root", "left", "node")` vs `("root", "right", "node")`) distinguishes them, each gets its exact PyArrow type instead of merging `("node", "leaf")` globally. + +#### XSD +```xml +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:element name="root"> + <xs:complexType> + <xs:sequence> + <xs:element name="left"> + <xs:complexType> + <xs:sequence> + <xs:element name="node"> + <xs:complexType> + <xs:sequence> + <xs:element name="leaf" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> + <xs:element name="right"> + <xs:complexType> + <xs:sequence> + <xs:element name="node"> + <xs:complexType> + <xs:sequence> + <xs:element name="leaf" maxOccurs="unbounded"> + <xs:complexType> + <xs:simpleContent> + <xs:extension base="xs:string"> + <xs:attribute name="lang" type="xs:string"/> + </xs:extension> + </xs:simpleContent> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> +``` + +#### Sample XML (`xml_tag="root"`) +```xml +<root> + <left> + <node> + <leaf>l1</leaf> + </node> + </left> + <right> + <node> + <leaf>r1</leaf> + <leaf lang="en">r2</leaf> + </node> + </right> +</root> +``` + +#### Python / JSON Translation +```python +[ + { + "left": { + "node": { + "leaf": "l1", + } + }, + "right": { + "node": { + "leaf": [ + {"#text": "r1"}, + {"_lang": "en", "#text": "r2"}, + ] + } + }, + } +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field( + "left", + pa.struct( + [ + pa.field( + "node", + pa.struct( + [ + pa.field("leaf", pa.string(), nullable=False), + ] + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "right", + pa.struct( + [ + pa.field( + "node", + pa.struct( + [ + pa.field( + "leaf", + pa.list_( + pa.struct( + [ + pa.field("_lang", pa.string(), nullable=True), + pa.field("#text", pa.string(), nullable=True), + ] + ) + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + ] +) +``` + +--- + +### Example 7: Multiple Root Elements Sharing the Same Child `xml_tag` (`uniref.xsd` Pattern) +In [`uniref.xsd`](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/uniprot/uniref/uniref.xsd), four root elements (`UniRef100`, `UniRef90`, `UniRef50`, `UniRef`) each declare a child `<xs:element name="entry" type="entryType"/>`. Because all four `entry` declarations share the same `entryType`, resolving `xml_tag="entry"` succeeds unambiguously. + +--- + +### Example 8: Field Nullability & Empty String Tag Coercion +Covers: +1. Required attributes (`use="required"` $\rightarrow$ `nullable=False`) vs optional attributes (`use="optional"` $\rightarrow$ `nullable=True`). +2. Required non-nillable elements (`minOccurs="1"`, `nillable="false"` $\rightarrow$ `nullable=False`), with present empty string tags `<summary/>` coerced to `""`. +3. Elements inside `<xs:choice>` branches (`nullable=True`). +4. Children of `nillable="true"` complex elements (`nullable=True` so `<comment type="function" xsi:nil="true"/>` can omit child `text`). + +#### XSD +```xml +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:complexType name="commentType"> + <xs:sequence> + <xs:element name="text" type="xs:string"/> + </xs:sequence> + <xs:attribute name="type" type="xs:string" use="required"/> + </xs:complexType> + + <xs:element name="root"> + <xs:complexType> + <xs:sequence> + <xs:element name="entry" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" type="xs:string"/> + <xs:element name="summary" type="xs:string"/> + <xs:element name="optionalNote" type="xs:string" minOccurs="0"/> + <xs:choice> + <xs:element name="geneId" type="xs:int"/> + <xs:element name="proteinId" type="xs:string"/> + </xs:choice> + <xs:element name="comment" type="commentType" nillable="true"/> + </xs:sequence> + <xs:attribute name="id" type="xs:string" use="required"/> + <xs:attribute name="source" type="xs:string" use="optional"/> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> +``` + +#### Sample XML (`xml_tag="entry"`) +```xml +<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <entry id="e1"> + <name>BRCA1</name> + <summary/> + <geneId>672</geneId> + <comment type="function" xsi:nil="true"/> + </entry> +</root> +``` + +#### Python / JSON Translation +```python +[ + { + "_id": "e1", + "name": "BRCA1", + "summary": "", # coerced from None -> "" to satisfy nullable=False + "geneId": 672, + "comment": {"_type": "function"}, + } +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field("_id", pa.string(), nullable=False), # use="required" + pa.field("_source", pa.string(), nullable=True), # use="optional" + pa.field("name", pa.string(), nullable=False), # minOccurs=1, non-nillable + pa.field("summary", pa.string(), nullable=False), # minOccurs=1, non-nillable + pa.field("optionalNote", pa.string(), nullable=True), # minOccurs=0 + pa.field("geneId", pa.int32(), nullable=True), # inside xs:choice -> nullable=True + pa.field("proteinId", pa.string(), nullable=True), # inside xs:choice -> nullable=True + pa.field( + "comment", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=False), # use="required" + pa.field("text", pa.string(), nullable=True), # parent is nillable="true" -> nullable=True + ] + ), + nullable=True, + ), # nillable="true" -> nullable=True + ] +) +``` + +--- + +### Example 9: Purely Empty `complexType` Marker Elements +Covers DTD-derived empty marker elements like [`pubmed.xsd` L1312](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/pubmed/pubmed.xsd#L1312) (`<BookInProcess/>`) and [`10_combinations/schema.xsd` L11–12](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/schema.xsd#L11-L12) (`EmptyType`): +- Mapped to `pa.string()` (or `pa.list_(pa.string())` when `overall_max_occurs > 1`). +- Coerced to `""` when present in the XML, and `None` when omitted. + +#### XSD +```xml +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:complexType name="EmptyType"/> + + <xs:element name="books"> + <xs:complexType> + <xs:sequence> + <xs:element name="book" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="title" type="xs:string"/> + <xs:element name="BookInProcess" type="EmptyType" minOccurs="0"/> + <xs:element name="flag" type="EmptyType" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> +``` + +#### Sample XML (`xml_tag="book"`) +```xml +<books> + <book> + <title>In-Process Book + + + + + + Completed Book + + +``` + +#### Python / JSON Translation +```python +[ + { + "title": "In-Process Book", + "BookInProcess": "", + "flag": ["", ""], + }, + { + "title": "Completed Book", + }, +] +``` + +#### PyArrow Schema +```python +pa.schema( + [ + pa.field("title", pa.string(), nullable=False), + pa.field("BookInProcess", pa.string(), nullable=True), + pa.field("flag", pa.list_(pa.string()), nullable=True), + ] +) +``` + +--- + +## 3. Error Specification (Unsupported / Invalid XSD Features) + +The converter raises a descriptive `ValueError` during schema generation when encountering any of the following: + +| # | Error Case | Example XSD Pattern | Reason | +| :--- | :--- | :--- | :--- | +| **E1** | **`xml_tag` Not Found** | `` or `xml_tag="nonexistent"` | Target element does not exist in the XSD. | +| **E2** | **Ambiguous `xml_tag` Type** | Root `A` has `` and root `B` has `` | Cannot determine a single `pa.Schema` for `xml_tag="entry"` when declarations at the same resolution depth have different types. | +| **E3** | **Non-Struct Top-Level `xml_tag`** | `` or empty `` with no attributes/children | A `pa.Table` schema requires top-level columns (at least one attribute, child element, or `#text` field). | +| **E4** | **Element Wildcard (``)** | `` | Arbitrary undeclared child elements cannot be represented as fixed fields in a `pa.struct`. | +| **E5** | **Attribute Wildcard (``)** | `` | Arbitrary undeclared attributes cannot be represented as fixed fields in a `pa.struct`. | +| **E6** | **Abstract Element or Abstract Type (`abstract="true"`)** | `` used as an element's type, or `` | Concrete runtime instances require `xsi:type` or `substitutionGroup` polymorphism with varying schemas. | +| **E7** | **Substitution Group with Heterogeneous Types** | `` | Substituted elements introduce different element names/types not declared in the parent's content model. | +| **E8** | **Prefixed Attribute / Child Name Collision** | Element has both `` (mapped to `_id`) and `` | Both map to field name `"_id"` in the same `pa.struct`. | +| **E9** | **Non-JSON-Serializable Scalar Type Under Recursion** | Recursive element subtree contains `xs:date`, `xs:dateTime`, `xs:hexBinary`, or `xs:base64Binary` | Elements truncated beyond `max_recursion_depth` are serialized to JSON (`json.dumps`), which does not natively serialize `datetime.date` or `bytes` without lossy/ambiguous string encoding. | + +--- + +## 4. Generated Specification Fixtures, Real-World Examples & Tests + +### 4.1 Specification Fixtures (`tests/data/xsd/spec/`) + +| Suite | `xml_tag` | XSD | Expected PyArrow Schema | Sample XML | Expected JSON | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **`01_scalars_and_simple_types`** | `record` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/01_scalars_and_simple_types/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/01_scalars_and_simple_types/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/01_scalars_and_simple_types/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/01_scalars_and_simple_types/expected.json) | +| **`02_namespaces_and_nillable`** | `entry` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/02_namespaces_and_nillable/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/02_namespaces_and_nillable/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/02_namespaces_and_nillable/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/02_namespaces_and_nillable/expected.json) | +| **`03_mixed_content_raw_text`** | `article` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/03_mixed_content_raw_text/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/03_mixed_content_raw_text/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/03_mixed_content_raw_text/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/03_mixed_content_raw_text/expected.json) | +| **`04_recursion_depth`** | `record` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/04_recursion_depth/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/04_recursion_depth/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/04_recursion_depth/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/04_recursion_depth/expected.json), [expected_depth_1.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/04_recursion_depth/expected_depth_1.json) | +| **`05_groups_and_compositors`** | `item` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/05_groups_and_compositors/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/05_groups_and_compositors/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/05_groups_and_compositors/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/05_groups_and_compositors/expected.json) | +| **`06_context_dependent_paths`** | `root` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/06_context_dependent_paths/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/06_context_dependent_paths/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/06_context_dependent_paths/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/06_context_dependent_paths/expected.json) | +| **`07_multi_root_shared_type`** | `entry` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/07_multi_root_shared_type/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/07_multi_root_shared_type/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/07_multi_root_shared_type/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/07_multi_root_shared_type/expected.json) | +| **`08_nullability_and_empty_tags`** | `entry` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/08_nullability_and_empty_tags/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/08_nullability_and_empty_tags/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/08_nullability_and_empty_tags/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/08_nullability_and_empty_tags/expected.json) | +| **`09_empty_marker_elements`** | `book` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/09_empty_marker_elements/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/09_empty_marker_elements/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/09_empty_marker_elements/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/09_empty_marker_elements/expected.json) | +| **`10_combinations`** | `record` | [schema.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/schema.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/expected_schema.py) | [data.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/data.xml) | [expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/10_combinations/expected.json) | + +### 4.2 Error Specification Fixtures (`tests/data/xsd/spec/errors/`) + +- [ambiguous_xml_tag.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/ambiguous_xml_tag.xsd) (E2) +- [scalar_top_level.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/scalar_top_level.xsd) (E3) +- [empty_marker_top_level.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/empty_marker_top_level.xsd) (E3) +- [wildcard_any_element.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/wildcard_any_element.xsd) (E4) +- [wildcard_any_attribute.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/wildcard_any_attribute.xsd) (E5) +- [abstract_element.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/abstract_element.xsd) (E6) +- [abstract_type.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/abstract_type.xsd) (E6) +- [substitution_group.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/substitution_group.xsd) (E7) +- [attr_child_name_collision.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/attr_child_name_collision.xsd) (E8) +- [nested_date_under_recursion.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/nested_date_under_recursion.xsd) (E9) +- [nested_binary_under_recursion.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/spec/errors/nested_binary_under_recursion.xsd) (E9) + +### 4.3 Real-World Minimal & Maximal Examples (`tests/data/xsd/examples/`) + +| Dataset | `xml_tag` | XSD | Expected PyArrow Schema | Minimal XML & Expected JSON | Maximal XML & Expected JSON | +| :--- | :--- | :--- | :--- | :--- | :--- | +| **UniRef** | `entry` | [uniref.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/uniprot/uniref/uniref.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniref/expected_schema.py) | [minimal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniref/minimal.xml), [minimal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniref/minimal_expected.json) | [maximal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniref/maximal.xml), [maximal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniref/maximal_expected.json) | +| **UniProtKB** | `entry` | [uniprot.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/uniprot/uniprot_kb/uniprot.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniprot/expected_schema.py) | [minimal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniprot/minimal.xml), [minimal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniprot/minimal_expected.json) | [maximal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniprot/maximal.xml), [maximal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/uniprot/maximal_expected.json) | +| **PubMed (`PubmedArticle`)** | `PubmedArticle` | [pubmed.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/pubmed/pubmed.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_article/expected_schema.py) | [minimal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_article/minimal.xml), [minimal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_article/minimal_expected.json) | [maximal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_article/maximal.xml), [maximal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_article/maximal_expected.json) | +| **PubMed (`PubmedBookArticle`)** | `PubmedBookArticle` | [pubmed.xsd](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/pubmed/pubmed.xsd) | [expected_schema.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_book_article/expected_schema.py) | [minimal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_book_article/minimal.xml), [minimal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_book_article/minimal_expected.json) | [maximal.xml](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_book_article/maximal.xml), [maximal_expected.json](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/data/xsd/examples/pubmed_book_article/maximal_expected.json) | + +### 4.4 Specification Test Suite + +- [test_xsd_spec.py](file:///Users/gwg/code/kbase/cdm-data-loaders-no-batcher/tests/cdm_data_loaders/readers/test_xsd_spec.py): Validates all fixtures against their XSDs, tests `generate_pyarrow_schema`, tests error cases E1–E9, tests `convert_xml_element` + PyArrow Table / Parquet round-trips, and tests `process_xml_file_to_dict` yielding `pa.Table` batches. + diff --git a/pyproject.toml b/pyproject.toml index 2efdf762..76631030 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -49,6 +49,7 @@ pdb_manifest = "cdm_data_loaders.pipelines.pdb_manifest:cli" uniprot_cdm = "cdm_data_loaders.pipelines.uniprot_kb_cdm:cli" uniref_cdm = "cdm_data_loaders.pipelines.uniref_cdm:cli" xmltodict = "cdm_data_loaders.pipelines.xmltodict.pipeline:cli" +xmltopyarrow = "cdm_data_loaders.pipelines.xmltopyarrow.pipeline:cli" xml2db = "cdm_data_loaders.pipelines.xml2db.pipeline:cli" xsv = "cdm_data_loaders.pipelines.xsv.pipeline:cli" hello_world = "cdm_data_loaders.utils.hello_world:cli" diff --git a/src/cdm_data_loaders/pipelines/ncbi_ftp_promote.py b/src/cdm_data_loaders/pipelines/ncbi_ftp_promote.py index 33f17387..8d111629 100644 --- a/src/cdm_data_loaders/pipelines/ncbi_ftp_promote.py +++ b/src/cdm_data_loaders/pipelines/ncbi_ftp_promote.py @@ -4,7 +4,6 @@ final location in the Lakehouse. """ -import logging from pathlib import Path, PurePosixPath from typing import Final @@ -22,8 +21,6 @@ DEFAULT_TRANSFER_MANIFEST_FILE: PurePosixPath = PurePosixPath("transfer_manifest.txt") ESTIMATED_FILES_PER_ASSEMBLY: int = 21 -logger = logging.getLogger("dlt") - class PromoteSettings(LoggerSettings): """Configuration for the NCBI FTP file promotion pipeline.""" diff --git a/src/cdm_data_loaders/pipelines/xmltodict/pipeline.py b/src/cdm_data_loaders/pipelines/xmltodict/pipeline.py index dfe57fa7..d7b4afc2 100644 --- a/src/cdm_data_loaders/pipelines/xmltodict/pipeline.py +++ b/src/cdm_data_loaders/pipelines/xmltodict/pipeline.py @@ -1,7 +1,6 @@ """XML to dictionary pipeline for the KBase CTS.""" from collections.abc import Generator, Iterator -from logging import Logger, getLogger from pathlib import Path from typing import TYPE_CHECKING, Any @@ -18,15 +17,15 @@ if TYPE_CHECKING: from dlt.extract import DltResource -logger: Logger = getLogger(__name__) - -def _read_items(items: Iterator[FileItemDict], settings: XmlToDictSettings) -> Generator[TDataItems, Any, Any]: +def _read_xmltodict_items( + items: Iterator[FileItemDict], settings: XmlToDictSettings +) -> Generator[TDataItems, Any, Any]: """Read each file in items. Yields one dict per `xml_tag` element. :param items: file items to read :type items: Iterator[FileItemDict] - :param settings: pipeline config, including xml_tag and xsd-derived paths + :param settings: pipeline config, including xml_tag :type settings: XmlToDictSettings :yield: one dict per matched element :rtype: Generator[TDataItems, Any, Any] @@ -41,8 +40,7 @@ def run_xmltodict_pipeline(settings: XmlToDictSettings) -> LoadInfo | None: :param settings: pipeline configuration :type settings: XmlToDictSettings """ - reader = dlt.transformer(_read_items, name="xmltodict_reader", parallelized=True) - + reader = dlt.transformer(_read_xmltodict_items, name="xmltodict_reader", parallelized=True) files = filesystem(bucket_url=str(settings.input_dir), file_glob=settings.file_glob) xmltodict_resource: DltResource = files | reader(settings) diff --git a/src/cdm_data_loaders/pipelines/xmltodict/settings.py b/src/cdm_data_loaders/pipelines/xmltodict/settings.py index 22eed4cb..43b44fdf 100644 --- a/src/cdm_data_loaders/pipelines/xmltodict/settings.py +++ b/src/cdm_data_loaders/pipelines/xmltodict/settings.py @@ -1,9 +1,8 @@ """Settings class for the XML to dictionary pipeline for the KBase CTS.""" -from functools import cached_property -from typing import Annotated, Any, Final, Self +from typing import Annotated, Final -from pydantic import Field, FilePath, PrivateAttr, computed_field, model_validator +from pydantic import Field from pydantic_settings import SettingsConfigDict from cdm_data_loaders.core.fields import ( @@ -17,7 +16,6 @@ FileFormatIngestSettings, default_settings_with_shortcuts, ) -from cdm_data_loaders.readers.xsd import find_list_and_single_child_paths, load_schema PIPELINE_NAME: Final[str] = "xmltodict" @@ -33,9 +31,6 @@ class XmlToDictSettings(FileFormatIngestSettings): }, ) - _paths: Annotated[dict[str, Any], Field(description="parent-child paths in the schema")] = PrivateAttr( - default_factory=dict - ) file_glob: Annotated[ FileGlob, Field( @@ -50,45 +45,3 @@ class XmlToDictSettings(FileFormatIngestSettings): description="XML tag to capture the contents of", ), ] - xsd_file: Annotated[ - FilePath | None, - Field( - None, - description="Path to the XSD file describing the XML documents to be imported. Including the schema will ensure that the appropriate relationships (1-to-1 vs 1-to-many) are created between parent and child elements.", - ), - ] - - @model_validator(mode="after") - def check_xsd_file(self) -> Self: - """Check whether the schema file is present and correct; if so, init self._paths.""" - if self.xsd_file: - try: - schema = load_schema(self.xsd_file) - list_paths, single_paths = find_list_and_single_child_paths(schema) - self._paths = {"list": list_paths, "single": single_paths} - except Exception as e: - err_msg = f"Could not generate parent-child relationships for schema: {e!s}" - raise ValueError(err_msg) from e - - return self - - @computed_field - @cached_property - def xmltodict_args(self) -> dict[str, Any]: - """Extra parser configuration arguments for running xmltodict over a dataset. - - :return: kwargs for configuring xmltodict.parse() - :rtype: dict[str, Any] - """ - if not self._paths: - return {} - - single_paths = self._paths["single"] - - def force_list(path: list[tuple[str, str | None]], key: str, _: Any) -> bool: - if not path: - return False - - return (path[-1][0], key) not in single_paths - - return {"force_list": force_list} diff --git a/src/cdm_data_loaders/pipelines/xmltopyarrow/__init__.py b/src/cdm_data_loaders/pipelines/xmltopyarrow/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/src/cdm_data_loaders/pipelines/xmltopyarrow/pipeline.py b/src/cdm_data_loaders/pipelines/xmltopyarrow/pipeline.py new file mode 100644 index 00000000..4dfabd81 --- /dev/null +++ b/src/cdm_data_loaders/pipelines/xmltopyarrow/pipeline.py @@ -0,0 +1,63 @@ +"""XML to pyarrow pipeline for the KBase CTS.""" + +from collections.abc import Generator, Iterator +from pathlib import Path +from typing import TYPE_CHECKING, Any + +import dlt +from dlt.common.pipeline import LoadInfo +from dlt.common.storages.fsspec_filesystem import FileItemDict +from dlt.common.typing import TDataItems +from dlt.sources.filesystem import filesystem + +from cdm_data_loaders.pipelines.core import PipelineCli, run_pipeline +from cdm_data_loaders.pipelines.xmltopyarrow.settings import PIPELINE_NAME, XmlToPyArrowSettings +from cdm_data_loaders.readers.xml import process_xml_file_to_pyarrow + +if TYPE_CHECKING: + from dlt.extract import DltResource + + +def _read_items(items: Iterator[FileItemDict], settings: XmlToPyArrowSettings) -> Generator[TDataItems, Any, Any]: + """Read each file in items. Yields PyArrow tables for matched `xml_tag` elements. + + :param items: file items to read + :type items: Iterator[FileItemDict] + :param settings: pipeline config, including xml_tag and XSD schema + :type settings: XmlToPyArrowSettings + :yield: PyArrow tables of matched elements + :rtype: Generator[TDataItems, Any, Any] + """ + for file_item in items: + yield from process_xml_file_to_pyarrow(settings, file_path=Path(file_item.local_file_path)) + + +def run_xmltopyarrow_pipeline(settings: XmlToPyArrowSettings) -> LoadInfo | None: + """Run the XML to PyArrow pipeline on matching files in settings.input_dir. + + :param settings: pipeline configuration + :type settings: XmlToPyArrowSettings + """ + reader = dlt.transformer(_read_items, name="xmltopyarrow_reader", parallelized=True) + files = filesystem(bucket_url=str(settings.input_dir), file_glob=settings.file_glob) + + xmltopyarrow_resource: DltResource = files | reader(settings) + + return run_pipeline( + settings=settings, + resource=xmltopyarrow_resource, + pipeline_kwargs={ + "pipeline_name": PIPELINE_NAME, + "dataset_name": settings.dataset_name, + }, + pipeline_run_kwargs={ + "loader_file_format": str(settings.loader_file_format), + }, + ) + + +cli = PipelineCli(XmlToPyArrowSettings, run_xmltopyarrow_pipeline) + + +if __name__ == "__main__": + cli() diff --git a/src/cdm_data_loaders/pipelines/xmltopyarrow/settings.py b/src/cdm_data_loaders/pipelines/xmltopyarrow/settings.py new file mode 100644 index 00000000..e5c1f2e4 --- /dev/null +++ b/src/cdm_data_loaders/pipelines/xmltopyarrow/settings.py @@ -0,0 +1,95 @@ +"""Settings class for the XML to PyArrow pipeline for the KBase CTS.""" + +from typing import Annotated, Final, Self + +import pyarrow as pa +import xmlschema +from pydantic import Field, FilePath, PositiveInt, PrivateAttr, model_validator +from pydantic_settings import SettingsConfigDict + +from cdm_data_loaders.core.fields import ( + DEFAULT_XML_FILE_GLOB, + FILE_GLOB, + FileGlob, + NonEmptyStr, + TableName, +) +from cdm_data_loaders.core.settings import ( + FileFormatIngestSettings, + default_settings_with_shortcuts, +) +from cdm_data_loaders.readers.xsd import ( + DEFAULT_MAX_RECURSION_DEPTH, + generate_pyarrow_schema, + load_schema, +) + +PIPELINE_NAME: Final[str] = "xmltopyarrow" + + +class XmlToPyArrowSettings(FileFormatIngestSettings): + """Settings for the XML to PyArrow ingestion pipeline.""" + + model_config: SettingsConfigDict = default_settings_with_shortcuts( + cli_prog_name=PIPELINE_NAME, + cli_shortcuts={ + **FileFormatIngestSettings.model_config["cli_shortcuts"], + FILE_GLOB: "g", + }, + ) + + _xsd_schema: xmlschema.XMLSchema = PrivateAttr() + _pyarrow_schema: pa.Schema = PrivateAttr() + file_glob: Annotated[ + FileGlob, + Field( + default=DEFAULT_XML_FILE_GLOB, + description="Glob pattern for XML files inside each entity's input subdirectory.", + ), + ] + table_name: TableName + xml_tag: Annotated[ + NonEmptyStr, + Field( + description="XML tag to capture the contents of", + ), + ] + xsd_file: Annotated[ + FilePath, + Field( + description="Path to the XSD file describing the XML documents to be imported. Including the schema will ensure that the appropriate relationships (1-to-1 vs 1-to-many) are created between parent and child elements, and that element text is stored consistently.", + ), + ] + max_recursion_depth: Annotated[ + PositiveInt, + Field( + default=DEFAULT_MAX_RECURSION_DEPTH, + description="Maximum recursion depth for self-referential XSD types before serializing deeper content as JSON.", + ), + ] + + @model_validator(mode="after") + def check_xsd_file(self) -> Self: + """Check whether the schema file is present and valid; load the XSD and generate the PyArrow schema.""" + try: + self._xsd_schema = load_schema(self.xsd_file) + self._pyarrow_schema = generate_pyarrow_schema( + self._xsd_schema, + self.xml_tag, + max_recursion_depth=self.max_recursion_depth, + ) + except Exception as e: + err_msg = f"Could not generate PyArrow schema from XSD: {e!s}" + raise ValueError(err_msg) from e + + return self + + @property + def xsd_schema(self) -> xmlschema.XMLSchema: + """Return the loaded XMLSchema instance.""" + return self._xsd_schema + + @property + def pyarrow_schema(self) -> pa.Schema: + """Return the generated PyArrow schema for xml_tag.""" + return self._pyarrow_schema diff --git a/src/cdm_data_loaders/readers/jsonschema_xsv/xsv_validator/helpers.py b/src/cdm_data_loaders/readers/jsonschema_xsv/xsv_validator/helpers.py index d8a7c54b..d839a49d 100644 --- a/src/cdm_data_loaders/readers/jsonschema_xsv/xsv_validator/helpers.py +++ b/src/cdm_data_loaders/readers/jsonschema_xsv/xsv_validator/helpers.py @@ -4,7 +4,6 @@ import subprocess from dataclasses import dataclass from functools import cached_property -from logging import Logger, getLogger from pathlib import Path from shutil import copy, move from typing import Annotated, Final, Self @@ -35,9 +34,6 @@ VALID_SUFFIX: Final[str] = "-valid" -logger: Logger = getLogger(__name__) - - class ErrorRecord(BaseModel): """A single error encountered while cleaning/validating a file. diff --git a/src/cdm_data_loaders/readers/xml.py b/src/cdm_data_loaders/readers/xml.py index bb40f84e..05683f62 100644 --- a/src/cdm_data_loaders/readers/xml.py +++ b/src/cdm_data_loaders/readers/xml.py @@ -16,8 +16,10 @@ from cdm_data_loaders.core.fields import DEFAULT_XML_FILE_GLOB from cdm_data_loaders.core.settings import CtsSettings from cdm_data_loaders.pipelines.xmltodict.settings import XmlToDictSettings +from cdm_data_loaders.pipelines.xmltopyarrow.settings import XmlToPyArrowSettings +from cdm_data_loaders.readers.xsd import convert_xml_element from cdm_data_loaders.utils.batcher import get_file_batches -from cdm_data_loaders.utils.buffer import DictBuffer, ListBuffer +from cdm_data_loaders.utils.buffer import ArrowListBuffer, DictBuffer, ListBuffer from cdm_data_loaders.utils.gz import open_maybe_gzip logger: Logger = getLogger(__name__) @@ -64,7 +66,9 @@ def parse_head_matter(file_path: str | Path) -> dict[str, str]: def stream_xml_file(file_path: str | Path, element_with_ns: str) -> Generator[Element, Any]: - """Stream XML elements from a file. + """Stream the outermost matching XML elements from a file. + + A matching element nested inside another matching element is yielded as part of the outer element. :param file_path: path to the XML file; file can be gzipped or not. :type file_path: str | Path @@ -76,10 +80,21 @@ def stream_xml_file(file_path: str | Path, element_with_ns: str) -> Generator[El logger.debug("Streaming XML from %s", file_path) with open_maybe_gzip(file_path) as f: - for _, elem in iterparse(f, tag=(element_with_ns), remove_blank_text=True): + # number of open matching elements; 0 means the parser is outside any match + depth = 0 + for event, elem in iterparse(f, events=("start", "end"), tag=element_with_ns, remove_blank_text=True): + if event == "start": + depth += 1 + continue + depth -= 1 + if depth: + # nested match: leave it in place as part of the enclosing match + continue logger.debug(elem) yield elem elem.clear() + while elem.getprevious() is not None: + del elem.getparent()[0] def _log_interval_progress(n_entries: int, log_interval: int) -> None: @@ -94,16 +109,59 @@ def _log_interval_final(n_entries: int, file_path: Path, log_interval: int) -> N logger.debug("Processed %d entries from %s", n_entries + 1, file_path.name) +def process_xml_file_to_pyarrow(settings: XmlToPyArrowSettings, file_path: Path) -> Generator[DataItemWithMeta]: + """Convert XML elements to PyArrow table batches using an XSD schema. + + Each matching element is converted using ``settings.xsd_schema`` and buffered into PyArrow + tables matching ``settings.pyarrow_schema``. + + Use process_xml_file for parsing functions that return data spread across several tables. + + :param settings: pipeline config, including xml_tag, table_name, xsd_schema, pyarrow_schema, and buffer_size + :type settings: XmlToPyArrowSettings + :param file_path: file path, as a Path + :type file_path: Path + :yield: pages of PyArrow tables + :rtype: Generator[DataItemWithMeta] + """ + logger.info("Reading from %s", str(file_path)) + n_entries = -1 + + stream_tag = settings.xml_tag + if not stream_tag.startswith("{"): + default_ns = parse_head_matter(file_path).get("") + if default_ns: + stream_tag = f"{{{default_ns}}}{stream_tag}" + + arrow_buffer = ArrowListBuffer( + table_to_schema_map={settings.table_name: settings.pyarrow_schema}, + max_items=settings.buffer_size, + ) + for n_entries, element in enumerate(stream_xml_file(file_path, stream_tag)): + parsed_row = convert_xml_element( + element, + settings.xsd_schema, + settings.xml_tag, + max_recursion_depth=settings.max_recursion_depth, + ) + yield from arrow_buffer.add_item(parsed_row) + _log_interval_progress(n_entries, settings.log_interval) + + _log_interval_final(n_entries, file_path, settings.log_interval) + yield from arrow_buffer.flush() + return + + def process_xml_file_to_dict(settings: XmlToDictSettings, file_path: Path) -> Generator[DataItemWithMeta]: - """Convert XML to dictionary form. + """Convert XML elements to dictionary form using ``xmltodict.parse``. - Note: each incoming element produces a single dictionary as output. + Each matching element is converted with ``xmltodict.parse`` and buffered as dicts. Use process_xml_file for parsing functions that return data spread across several tables. :param settings: pipeline config, including xml_tag, table_name, and buffer_size :type settings: XmlToDictSettings - :param file_path: file path, as a string + :param file_path: file path, as a Path :type file_path: Path :yield: pages of dictionary items :rtype: Generator[DataItemWithMeta] @@ -113,7 +171,7 @@ def process_xml_file_to_dict(settings: XmlToDictSettings, file_path: Path) -> Ge buffer = ListBuffer(table_name=settings.table_name, max_items=settings.buffer_size) for n_entries, element in enumerate(stream_xml_file(file_path, settings.xml_tag)): - parsed_element = xmltodict.parse(tostring(element), **DEFAULT_XMLTODICT_ARGS, **settings.xmltodict_args) + parsed_element = xmltodict.parse(tostring(element), **DEFAULT_XMLTODICT_ARGS) if parsed_element: for k, v in parsed_element.items(): # remove the xmlns declarations diff --git a/src/cdm_data_loaders/readers/xsd.py b/src/cdm_data_loaders/readers/xsd.py index 7e2cb24d..e766fe9e 100644 --- a/src/cdm_data_loaders/readers/xsd.py +++ b/src/cdm_data_loaders/readers/xsd.py @@ -1,76 +1,972 @@ -"""Functions for reading XSD files.""" +"""Functions for reading XSD files, generating PyArrow schemas, and converting XML elements.""" +import base64 +import datetime as dt +import json +from collections.abc import Mapping +from dataclasses import dataclass from pathlib import Path +from typing import Any, Final +import pyarrow as pa import xmlschema +from lxml.etree import Element +from xmlschema.validators import ( + XsdAnyAttribute, + XsdAnyElement, + XsdAttribute, + XsdComplexType, + XsdElement, + XsdGroup, + XsdSimpleType, + XsdType, +) + +DEFAULT_MAX_RECURSION_DEPTH: Final[int] = 2 + +XSD_NAMESPACE: Final[str] = "{http://www.w3.org/2001/XMLSchema}" +XSI_NAMESPACE: Final[str] = "{http://www.w3.org/2001/XMLSchema-instance}" +XSI_NIL: Final[str] = f"{XSI_NAMESPACE}nil" + +FORBIDDEN_RECURSIVE_XSD_TYPES: Final[frozenset[str]] = frozenset( + { + "xs:date", + "xs:dateTime", + "xs:hexBinary", + "xs:base64Binary", + } +) + +BUILTIN_LIST_ITEM_TYPES: Final[Mapping[str, str]] = { + "xs:NMTOKENS": "xs:NMTOKEN", + "xs:IDREFS": "xs:IDREF", + "xs:ENTITIES": "xs:ENTITY", +} + +SCALAR_ARROW_TYPES: Final[Mapping[str, pa.DataType]] = { + "xs:string": pa.string(), + "xs:normalizedString": pa.string(), + "xs:token": pa.string(), + "xs:language": pa.string(), + "xs:Name": pa.string(), + "xs:NCName": pa.string(), + "xs:NMTOKEN": pa.string(), + "xs:ID": pa.string(), + "xs:IDREF": pa.string(), + "xs:ENTITY": pa.string(), + "xs:anyURI": pa.string(), + "xs:QName": pa.string(), + "xs:NOTATION": pa.string(), + "xs:dateTime": pa.string(), + "xs:time": pa.string(), + "xs:duration": pa.string(), + "xs:gYear": pa.string(), + "xs:gYearMonth": pa.string(), + "xs:gMonthDay": pa.string(), + "xs:gDay": pa.string(), + "xs:gMonth": pa.string(), + "xs:boolean": pa.bool_(), + "xs:byte": pa.int8(), + "xs:short": pa.int16(), + "xs:int": pa.int32(), + "xs:long": pa.int64(), + "xs:integer": pa.int64(), + "xs:positiveInteger": pa.int64(), + "xs:nonNegativeInteger": pa.int64(), + "xs:negativeInteger": pa.int64(), + "xs:nonPositiveInteger": pa.int64(), + "xs:unsignedByte": pa.uint8(), + "xs:unsignedShort": pa.uint16(), + "xs:unsignedInt": pa.uint32(), + "xs:unsignedLong": pa.uint64(), + "xs:float": pa.float64(), + "xs:double": pa.float64(), + "xs:decimal": pa.float64(), + "xs:date": pa.date32(), + "xs:base64Binary": pa.binary(), + "xs:hexBinary": pa.binary(), +} def load_schema(schema_path: str | Path) -> xmlschema.XMLSchema: """Load an XML schema document into an XMLSchema object. :param schema_path: path to the schema file - :type schema_path: str | Path + :type schema_path: str | Path :return: loaded schema - :rtype: xmlschema.XMLSchema + :rtype : xmlschema.XMLSchema """ return xmlschema.XMLSchema(str(schema_path)) -def find_list_and_single_child_paths( - schema: xmlschema.XMLSchema, -) -> tuple[set[tuple[str, str]], set[tuple[str, str]]]: - """Traverse an xmlschema XMLSchema and classify parent/child element paths. - - A child is put in list_paths if it can occur zero, one, or more times - (maxOccurs is unbounded or greater than 1), meaning it should be - represented as a list. Otherwise it goes in single_paths (occurs at most - once, whether required or optional). - - :param schema: the schema to traverse - :type schema: xmlschema.XMLSchema - :return: tuple of (list_paths, single_paths) - :rtype: tuple[set[tuple[str, str]], set[tuple[str, str]]] +def _local_name(tag: str) -> str: + """Strip namespace URI from a Clark-notation tag or prefixed attribute name.""" + if "}" in tag: + return tag.rsplit("}", maxsplit=1)[-1] + return tag + + +def _extract_doc_comment(component: object, fallback_type: object = None) -> str | None: + """Extract whitespace-normalized ```` text from an XSD component or its type.""" + candidates = [component] + ref = getattr(component, "ref", None) + if ref is not None and ref is not component: + candidates.append(ref) + if fallback_type is not None: + candidates.append(fallback_type) + + for candidate in candidates: + annotation = getattr(candidate, "annotation", None) + docs = getattr(annotation, "documentation", None) + if not docs: + continue + parts = [" ".join(doc.text.split()) for doc in docs if getattr(doc, "text", None) and doc.text.strip()] + if parts: + return " ".join(parts) + return None + + +def _build_field_metadata( + *, + xsd_type: str | None = None, + comment: str | None = None, +) -> dict[str, str] | None: + """Construct the ``pa.Field`` metadata dictionary from optional comment and XSD type strings.""" + metadata: dict[str, str] = {} + if comment: + metadata["comment"] = comment + if xsd_type: + metadata["xsd_type"] = xsd_type + return metadata or None + + +def _builtin_xsd_name(simple_type: XsdType) -> str: + """Resolve an atomic simpleType or restriction to its built-in ``xs:`` identifier.""" + cur: Any = simple_type + while cur is not None: + name = getattr(cur, "name", None) + if isinstance(name, str) and name.startswith(XSD_NAMESPACE): + return f"xs:{cur.local_name}" + cur = getattr(cur, "base_type", None) + return "xs:string" + + +def _unwrap_member_types(simple_type: XsdType) -> list[XsdSimpleType]: + """Return the member types of an ``xs:union`` simpleType (unwrapping restrictions).""" + cur: Any = simple_type + while cur is not None: + members = getattr(cur, "member_types", None) + if members: + return list(members) + cur = getattr(cur, "base_type", None) + return [] + + +def _unwrap_item_type(simple_type: XsdType) -> XsdSimpleType | None: + """Return the item type of an ``xs:list`` simpleType (unwrapping restrictions).""" + cur: Any = simple_type + while cur is not None: + item_type = getattr(cur, "item_type", None) + if item_type is not None: + return item_type + cur = getattr(cur, "base_type", None) + return None + + +def _resolve_scalar_type(simple_type: XsdType) -> tuple[pa.DataType, str]: + """Map a non-list XSD simpleType (atomic or union) to ``(pa.DataType, xsd_type_str)``.""" + if simple_type.is_union(): + members = _unwrap_member_types(simple_type) + member_pairs = [_resolve_scalar_type(m) for m in members] + xsd_type_str = " | ".join(xsd_name for _, xsd_name in member_pairs) + unique_arrow_types = {arrow_t for arrow_t, _ in member_pairs} + arrow_type = next(iter(unique_arrow_types)) if len(unique_arrow_types) == 1 else pa.string() + return arrow_type, xsd_type_str + + xsd_name = _builtin_xsd_name(simple_type) + arrow_type = SCALAR_ARROW_TYPES.get(xsd_name, pa.string()) + return arrow_type, xsd_name + + +def _simple_type_to_arrow(simple_type: XsdType) -> tuple[pa.DataType, str | None]: + """Map any XSD simpleType (atomic, union, or list) to ``(pa.DataType, leaf_xsd_type_or_None)``. + + When ``simple_type`` is a list type (``xs:list``, ``xs:NMTOKENS``, ``xs:IDREFS``, ``xs:ENTITIES``), + the returned ``pa.DataType`` is ``pa.list_(pa.field("item", item_arrow_type, nullable=True, metadata={"xsd_type": ...}))`` + and the second element is ``None`` because ``"xsd_type"`` is attached to the inner ``"item"`` field. """ - list_paths: set[tuple[str, str]] = set() - single_paths: set[tuple[str, str]] = set() - visiting: set[int] = set() - - def walk(element: xmlschema.XsdElement) -> None: - elem_type = element.type - if elem_type is None: - return - - model_group = elem_type.model_group - if model_group is None: - return - - # guards against infinite recursion for self-referential types - type_key = id(elem_type) - if type_key in visiting: - return - visiting.add(type_key) - - parent_name = element.local_name - for child in model_group.iter_elements(): - if not isinstance(child, xmlschema.XsdElement): + builtin_name = _builtin_xsd_name(simple_type) + if builtin_name in BUILTIN_LIST_ITEM_TYPES: + item_xsd = BUILTIN_LIST_ITEM_TYPES[builtin_name] + item_field = pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": item_xsd}) + return pa.list_(item_field), None + + if simple_type.is_list(): + item_type = _unwrap_item_type(simple_type) + if item_type is not None: + item_arrow, item_xsd = _resolve_scalar_type(item_type) + else: + item_arrow, item_xsd = pa.string(), "xs:string" + item_field = pa.field("item", item_arrow, nullable=True, metadata={"xsd_type": item_xsd}) + return pa.list_(item_field), None + + return _resolve_scalar_type(simple_type) + + +def _iter_simple_builtin_names(simple_type: XsdType) -> list[str]: + """Return all constituent built-in ``xs:*`` type names for an atomic, list, or union simpleType.""" + builtin_name = _builtin_xsd_name(simple_type) + if builtin_name in BUILTIN_LIST_ITEM_TYPES: + return [BUILTIN_LIST_ITEM_TYPES[builtin_name]] + if simple_type.is_list(): + item_type = _unwrap_item_type(simple_type) + return _iter_simple_builtin_names(item_type) if item_type is not None else ["xs:string"] + if simple_type.is_union(): + names: list[str] = [] + for member in _unwrap_member_types(simple_type): + names.extend(_iter_simple_builtin_names(member)) + return names + return [builtin_name] + + +@dataclass(frozen=True, slots=True) +class _SchemaContext: + """Immutable traversal context for PyArrow schema generation.""" + + schema: xmlschema.XMLSchema + type_counts: Mapping[int, int] + max_recursion_depth: int + + +@dataclass(frozen=True, slots=True) +class _ConvertContext: + """Immutable traversal context for XML-to-dict conversion.""" + + schema: xmlschema.XMLSchema + type_counts: Mapping[int, int] + max_recursion_depth: int | None + + +def _check_complex_type_forbidden_recursion(root_type: XsdType, current: XsdComplexType) -> list[XsdType]: + """Validate attributes on ``current`` and return child ``XsdType``s to inspect for forbidden recursive scalars.""" + for attr in current.attributes.values(): + if isinstance(attr, XsdAttribute) and attr.type is not None: + for xsd_name in _iter_simple_builtin_names(attr.type): + if xsd_name in FORBIDDEN_RECURSIVE_XSD_TYPES: + msg = ( + f"Unsupported XSD feature: recursive type {root_type.local_name or root_type.name!r} " + f"contains attribute @{attr.local_name} of type {xsd_name!r}" + ) + raise ValueError(msg) + + if current.has_simple_content() and isinstance(current.content, XsdType): + return [current.content] + if not current.mixed and current.model_group is not None: + return [ + child.type + for child in current.model_group.iter_elements() + if isinstance(child, XsdElement) and child.type is not None and child.max_occurs != 0 + ] + return [] + + +def _check_no_forbidden_types_under_recursion(root_type: XsdType) -> None: + """Raise ValueError if a recursive type or any descendant type uses date, dateTime, or binary scalars.""" + visited: set[int] = set() + stack: list[XsdType] = [root_type] + + while stack: + current = stack.pop() + if id(current) in visited: + continue + visited.add(id(current)) + + if not isinstance(current, XsdComplexType): + for xsd_name in _iter_simple_builtin_names(current): + if xsd_name in FORBIDDEN_RECURSIVE_XSD_TYPES: + msg = ( + f"Unsupported XSD feature: recursive type {root_type.local_name or root_type.name!r} " + f"contains {xsd_name!r}, which cannot be serialized to JSON when recursion is truncated" + ) + raise ValueError(msg) + continue + + stack.extend(_check_complex_type_forbidden_recursion(root_type, current)) + + +def _group_min_occurs_by_name(group: XsdGroup, local_name: str) -> int: + """Compute minimum occurrences of elements with ``local_name`` across a model group.""" + if group.min_occurs == 0 or group.max_occurs == 0: + return 0 + target_group = group.ref if group.ref is not None else group + counts: list[int] = [] + for item in target_group.content: + if isinstance(item, XsdGroup): + counts.append(_group_min_occurs_by_name(item, local_name)) + elif isinstance(item, XsdElement) and item.max_occurs != 0 and item.local_name == local_name: + counts.append(item.min_occurs) + else: + counts.append(0) + if not counts: + return 0 + branch_min = min(counts) if target_group.model == "choice" else sum(counts) + return group.min_occurs * branch_min + + +def _particle_max_occurs_by_name(item: object, local_name: str) -> int | None: + """Compute maximum occurrences (None for unbounded) of ``local_name`` for a single group particle.""" + if isinstance(item, XsdGroup): + return _group_max_occurs_by_name(item, local_name) + if isinstance(item, XsdElement) and item.max_occurs != 0 and item.local_name == local_name: + return item.max_occurs + return 0 + + +def _group_max_occurs_by_name(group: XsdGroup, local_name: str) -> int | None: + """Compute maximum occurrences (None for unbounded) of elements with ``local_name`` across a model group.""" + if group.max_occurs == 0: + return 0 + target_group = group.ref if group.ref is not None else group + branch_maxes = [_particle_max_occurs_by_name(item, local_name) for item in target_group.content] + if not branch_maxes: + return 0 + if any(m is None for m in branch_maxes): + inner_max: int | None = None + elif target_group.model == "choice": + inner_max = max(m for m in branch_maxes if m is not None) + else: + inner_max = sum(m for m in branch_maxes if m is not None) + if inner_max == 0: + return 0 + if inner_max is None or group.max_occurs is None: + return None + return group.max_occurs * inner_max + + +def _is_any_element(item: object) -> bool: + """Return True if ``item`` is an ``xs:any`` wildcard particle.""" + return isinstance(item, XsdAnyElement) + + +def _is_any_attribute(attr: object) -> bool: + """Return True if ``attr`` is an ``xs:anyAttribute`` wildcard declaration.""" + return isinstance(attr, XsdAnyAttribute) + + +def _validate_group_particles(schema: xmlschema.XMLSchema, group: XsdGroup) -> None: + """Validate that a model group contains no wildcards, abstract elements/types, or substitution groups.""" + target_group = group.ref if group.ref is not None else group + for item in target_group.content: + if _is_any_element(item): + msg = "Unsupported XSD feature: xs:any wildcard element is not supported" + raise ValueError(msg) + if isinstance(item, XsdGroup): + _validate_group_particles(schema, item) + elif isinstance(item, XsdElement): + target_el = item.ref if item.ref is not None else item + if getattr(item, "abstract", False) or getattr(target_el, "abstract", False): + msg = f"Unsupported XSD feature: abstract element {item.local_name!r} is not supported" + raise ValueError(msg) + if ( + item.name in schema.substitution_groups + or target_el.name in schema.substitution_groups + or getattr(item, "substitution_group", None) is not None + or getattr(target_el, "substitution_group", None) is not None + ): + msg = f"Unsupported XSD feature: substitutionGroup on element {item.local_name!r} is not supported" + raise ValueError(msg) + if item.type is not None and getattr(item.type, "abstract", False): + msg = f"Unsupported XSD feature: abstract type {item.type.local_name!r} is not supported" + raise ValueError(msg) + + +def _collect_unique_children(schema: xmlschema.XMLSchema, complex_type: XsdComplexType) -> list[XsdElement]: + """Validate and return unique child ``XsdElement`` declarations in content-model order.""" + model_group = complex_type.model_group + if model_group is None: + return [] + _validate_group_particles(schema, model_group) + seen_names: set[str] = set() + children: list[XsdElement] = [] + for child in model_group.iter_elements(): + if not isinstance(child, XsdElement): + continue + max_occurs = _group_max_occurs_by_name(model_group, child.local_name) + if max_occurs == 0 or child.local_name in seen_names: + continue + seen_names.add(child.local_name) + children.append(child) + return children + + +def _collect_attributes(complex_type: XsdComplexType) -> list[XsdAttribute]: + """Validate and return declared attributes on ``complex_type`` in declaration order.""" + attrs: list[XsdAttribute] = [] + for attr in complex_type.attributes.values(): + if _is_any_attribute(attr): + msg = "Unsupported XSD feature: xs:anyAttribute wildcard is not supported" + raise ValueError(msg) + if isinstance(attr, XsdAttribute) and getattr(attr, "use", None) != "prohibited": + if attr.local_name is None: continue + attrs.append(attr) + return attrs + + +def _is_empty_marker_complex_type(complex_type: XsdComplexType) -> bool: + """Return True if ``complex_type`` has no attributes, no child elements, not simpleContent, and not mixed.""" + if complex_type.has_simple_content() or complex_type.mixed: + return False + if any( + isinstance(a, XsdAttribute) and getattr(a, "use", None) != "prohibited" + for a in complex_type.attributes.values() + ): + return False + if complex_type.model_group is None: + return True + return not any( + isinstance(el, XsdElement) and _group_max_occurs_by_name(complex_type.model_group, el.local_name) != 0 + for el in complex_type.model_group.iter_elements() + ) + + +def _build_complex_struct_fields( + ctx: _SchemaContext, + complex_type: XsdComplexType, + *, + parent_nillable: bool, +) -> list[pa.Field]: + """Build the ordered list of ``pa.Field``s for a struct-mapped ``XsdComplexType``.""" + fields: list[pa.Field] = [] + attr_field_names: set[str] = set() + + for attr in _collect_attributes(complex_type): + field_name = f"_{attr.local_name}" + attr_field_names.add(field_name) + attr_type = attr.type + if attr_type is not None: + pa_type, xsd_type_str = _simple_type_to_arrow(attr_type) + else: + pa_type, xsd_type_str = pa.string(), "xs:string" + comment = _extract_doc_comment(attr, attr_type) + nullable = parent_nillable or getattr(attr, "use", "optional") != "required" + fields.append( + pa.field( + field_name, + pa_type, + nullable=nullable, + metadata=_build_field_metadata(xsd_type=xsd_type_str, comment=comment), + ) + ) + + if complex_type.has_simple_content(): + content_type = complex_type.content + if isinstance(content_type, XsdType): + pa_type, xsd_type_str = _simple_type_to_arrow(content_type) + else: + pa_type, xsd_type_str = pa.string(), "xs:string" + fields.append( + pa.field( + "#text", + pa_type, + nullable=True, + metadata=_build_field_metadata(xsd_type=xsd_type_str), + ) + ) + return fields + + if complex_type.mixed: + fields.append( + pa.field( + "#text", + pa.string(), + nullable=True, + metadata=_build_field_metadata(xsd_type="xs:string"), + ) + ) + return fields + + model_group = complex_type.model_group + if model_group is None: + return fields + + for child in _collect_unique_children(ctx.schema, complex_type): + child_name = child.local_name + if child_name in attr_field_names: + msg = ( + f"Field name collision in {complex_type.local_name or 'complexType'!r}: " + f"attribute and child element both map to {child_name!r}" + ) + raise ValueError(msg) + + max_occurs = _group_max_occurs_by_name(model_group, child_name) + min_occurs = _group_min_occurs_by_name(model_group, child_name) + is_list = max_occurs is None or max_occurs > 1 + nullable = parent_nillable or child.nillable or min_occurs < 1 + + fields.append( + _build_element_field( + ctx, + child, + is_list=is_list, + nullable=nullable, + ) + ) + + return fields + + +def _resolve_non_struct_element_type(elem_type: XsdType | None) -> tuple[pa.DataType, str | None] | None: + """Return ``(pa_type, xsd_type_or_None)`` if ``elem_type`` maps to a leaf scalar rather than a struct.""" + if not isinstance(elem_type, XsdComplexType): + return _simple_type_to_arrow(elem_type) if elem_type is not None else (pa.string(), "xs:string") + + attrs = _collect_attributes(elem_type) + if elem_type.has_simple_content() and not attrs: + content_type = elem_type.content + return _simple_type_to_arrow(content_type) if isinstance(content_type, XsdType) else (pa.string(), "xs:string") + + if elem_type.mixed and not attrs: + return pa.string(), "xs:string" - max_occurs = elem_type.overall_max_occurs(child) - # never occurs => skip it - if max_occurs == 0: + if _is_empty_marker_complex_type(elem_type): + return pa.string(), None + + return None + + +def _build_element_field( + ctx: _SchemaContext, + element: XsdElement, + *, + is_list: bool, + nullable: bool, +) -> pa.Field: + """Build a ``pa.Field`` for a child ``XsdElement`` given its cardinality and nullability.""" + elem_type = element.type + comment = _extract_doc_comment(element, elem_type) + + leaf_type_info = _resolve_non_struct_element_type(elem_type) + if leaf_type_info is not None: + return _wrap_scalar_or_list_field( + element.local_name, + leaf_type_info, + comment=comment, + is_list=is_list, + nullable=nullable, + ) + + assert isinstance(elem_type, XsdComplexType) # noqa: S101 + type_id = id(elem_type) + if type_id in ctx.type_counts: + _check_no_forbidden_types_under_recursion(elem_type) + + next_count = ctx.type_counts.get(type_id, 0) + 1 + if next_count > ctx.max_recursion_depth: + return _wrap_scalar_or_list_field( + element.local_name, + (pa.string(), None), + comment=comment, + is_list=is_list, + nullable=nullable, + ) + + next_ctx = _SchemaContext( + schema=ctx.schema, + type_counts={**ctx.type_counts, type_id: next_count}, + max_recursion_depth=ctx.max_recursion_depth, + ) + struct_fields = _build_complex_struct_fields( + next_ctx, + elem_type, + parent_nillable=element.nillable, + ) + struct_type = pa.struct(struct_fields) + field_type = pa.list_(pa.field("item", struct_type, nullable=True)) if is_list else struct_type + return pa.field( + element.local_name, + field_type, + nullable=nullable, + metadata=_build_field_metadata(comment=comment), + ) + + +def _wrap_scalar_or_list_field( + name: str, + type_info: tuple[pa.DataType, str | None], + *, + comment: str | None, + is_list: bool, + nullable: bool, +) -> pa.Field: + """Create a scalar or list ``pa.Field`` with ``xsd_type`` on leaf scalar fields and ``comment`` on the outer field.""" + pa_type, xsd_type = type_info + if is_list: + item_field = pa.field("item", pa_type, nullable=True, metadata=_build_field_metadata(xsd_type=xsd_type)) + return pa.field( + name, + pa.list_(item_field), + nullable=nullable, + metadata=_build_field_metadata(comment=comment), + ) + return pa.field( + name, + pa_type, + nullable=nullable, + metadata=_build_field_metadata(xsd_type=xsd_type, comment=comment), + ) + + +def _resolve_target_element(schema: xmlschema.XMLSchema, xml_tag: str) -> XsdElement: + """Resolve ``xml_tag`` (local name or Clark notation) to a unique ``XsdElement`` declaration.""" + if xml_tag.startswith("{") and "}" in xml_tag: + ns_filter: str | None = xml_tag + local_target = xml_tag.rsplit("}", maxsplit=1)[-1] + else: + ns_filter = None + local_target = xml_tag + + def _matches(el: XsdElement) -> bool: + if el.local_name != local_target: + return False + return ns_filter is None or el.name == ns_filter + + # Depth 0: global elements + depth0 = [el for el in schema.elements.values() if isinstance(el, XsdElement) and _matches(el)] + if depth0: + return _deduplicate_target_elements(depth0, xml_tag) + + # BFS from depth 1 onwards through complexType model groups + current_level: list[XsdElement] = [el for el in schema.elements.values() if isinstance(el, XsdElement)] + visited_types: set[int] = set() + while current_level: + matches: list[XsdElement] = [] + next_level: list[XsdElement] = [] + for parent in current_level: + p_type = parent.type + if not isinstance(p_type, XsdComplexType) or id(p_type) in visited_types: + continue + visited_types.add(id(p_type)) + if p_type.model_group is None: continue + for child in p_type.model_group.iter_elements(): + if not isinstance(child, XsdElement): + continue + if _matches(child): + matches.append(child) + next_level.append(child) + if matches: + return _deduplicate_target_elements(matches, xml_tag) + current_level = next_level + + msg = f"Element {xml_tag!r} not found in XSD schema" + raise ValueError(msg) + + +def _deduplicate_target_elements(matches: list[XsdElement], xml_tag: str) -> XsdElement: + """Verify all matching ``XsdElement`` declarations at the resolution depth share a single ``XsdType``.""" + by_type: dict[int, XsdElement] = {id(el.type): el for el in matches} + if len(by_type) > 1: + msg = f"Ambiguous xml_tag {xml_tag!r}: found {len(by_type)} declarations with different types" + raise ValueError(msg) + target = next(iter(by_type.values())) + target_ref = target.ref if target.ref is not None else target + if getattr(target, "abstract", False) or getattr(target_ref, "abstract", False): + msg = f"Unsupported XSD feature: abstract element {xml_tag!r} is not supported" + raise ValueError(msg) + if target.type is not None and getattr(target.type, "abstract", False): + msg = f"Unsupported XSD feature: abstract type on {xml_tag!r} is not supported" + raise ValueError(msg) + return target + + +def _require_complex_type(elem_type: XsdType | None, xml_tag: str) -> XsdComplexType: + """Ensure that the resolved target element has an ``XsdComplexType``.""" + if elem_type is None or not elem_type.is_complex(): + msg = f"Top-level xml_tag {xml_tag!r} must have a complex type with attributes or child elements" + raise ValueError(msg) + assert isinstance(elem_type, XsdComplexType) # noqa: S101 + return elem_type + + +def generate_pyarrow_schema( + schema: xmlschema.XMLSchema, + xml_tag: str, + *, + max_recursion_depth: int = DEFAULT_MAX_RECURSION_DEPTH, +) -> pa.Schema: + """Generate a PyArrow schema for the target XML element from an XSD schema. + + :param schema: loaded XMLSchema instance + :type schema: xmlschema.XMLSchema + :param xml_tag: target XML element name (local name or Clark-notation ``{ns}local``) + :type xml_tag: str + :param max_recursion_depth: maximum structural recursion depth before JSON serialization + :type max_recursion_depth: int + :return: PyArrow schema whose top-level fields correspond to the target element's fields + :rtype : pa.Schema + """ + target_elem = _resolve_target_element(schema, xml_tag) + elem_type = _require_complex_type(target_elem.type, xml_tag) + + if _is_empty_marker_complex_type(elem_type): + msg = f"Top-level xml_tag {xml_tag!r} has an empty complex type with no attributes or child elements" + raise ValueError(msg) + + attrs = _collect_attributes(elem_type) + if (elem_type.has_simple_content() or elem_type.mixed) and not attrs: + msg = f"Top-level xml_tag {xml_tag!r} must have a complex type that resolves to a struct" + raise ValueError(msg) + + ctx = _SchemaContext( + schema=schema, + type_counts={id(elem_type): 1}, + max_recursion_depth=max_recursion_depth, + ) + fields = _build_complex_struct_fields( + ctx, + elem_type, + parent_nillable=target_elem.nillable, + ) + if not fields: + msg = f"Top-level xml_tag {xml_tag!r} must have a complex type with at least one field" + raise ValueError(msg) + return pa.schema(fields) + + +def _is_ignored_attr(attr_name: str) -> bool: + """Return True if an XML attribute is an ``xmlns`` or ``xsi:*`` attribute that should be stripped.""" + return attr_name.startswith((XSI_NAMESPACE, "xsi:", "xmlns", "{http://www.w3.org/2000/xmlns/}")) + + +def _is_xsi_nil(element: Element) -> bool: + """Return True if ``element`` carries ``xsi:nil='true'`` or ``xsi:nil='1'``.""" + val = element.attrib.get(XSI_NIL) or element.attrib.get("xsi:nil") + return val is not None and val.strip() in {"true", "1"} + + +def _escape_xml_text(text: str) -> str: + """Escape XML character entities ``&``, ``<``, and ``>`` in mixed-content text.""" + return text.replace("&", "&").replace("<", "<").replace(">", ">") + + +def _escape_xml_attr(value: str) -> str: + """Escape XML character entities and double quotes in mixed-content attribute values.""" + return _escape_xml_text(value).replace('"', """) + - path = (parent_name, child.local_name) - if max_occurs is None or max_occurs > 1: - list_paths.add(path) - else: - single_paths.add(path) +def _serialize_mixed_node(node: Element) -> str: + """Serialize an inline child element inside a ``mixed='true'`` element without namespace declarations.""" + local = _local_name(node.tag) + attrs = "".join( + f' {_local_name(k)}="{_escape_xml_attr(v)}"' for k, v in node.attrib.items() if not _is_ignored_attr(k) + ) + inner = _serialize_inner_xml(node) + if not inner and len(node) == 0 and node.text is None: + return f"<{local}{attrs}/>" + return f"<{local}{attrs}>{inner}" - walk(child) - visiting.discard(type_key) +def _serialize_inner_xml(element: Element) -> str: + """Serialize the raw inner XML (text and inline child elements) of a ``mixed='true'`` element.""" + parts: list[str] = [] + if element.text: + parts.append(_escape_xml_text(element.text)) + for child in element: + if isinstance(child.tag, str): + parts.append(_serialize_mixed_node(child)) + if child.tail: + parts.append(_escape_xml_text(child.tail)) + return "".join(parts) - for element in schema.elements.values(): - walk(element) - return list_paths, single_paths +def _coerce_resolved_scalar(stripped: str, arrow_type: pa.DataType, xsd_name: str) -> Any: # noqa: ANN401 + """Coerce a stripped lexical string using its resolved ``arrow_type`` and ``xsd_name``.""" + if pa.types.is_boolean(arrow_type): + return stripped in {"true", "1"} + if pa.types.is_integer(arrow_type): + return int(stripped) + if pa.types.is_floating(arrow_type): + return float(stripped) + if pa.types.is_date32(arrow_type): + return dt.date.fromisoformat(stripped[:10]) + if pa.types.is_binary(arrow_type): + return bytes.fromhex(stripped) if xsd_name == "xs:hexBinary" else base64.b64decode(stripped) + return stripped + + +def _coerce_scalar_text(raw_text: str, simple_type: XsdType | None) -> Any: # noqa: ANN401 + """Coerce a non-empty lexical XML string according to its XSD ``simple_type``.""" + if simple_type is None: + return raw_text + + builtin_name = _builtin_xsd_name(simple_type) + if builtin_name in BUILTIN_LIST_ITEM_TYPES: + return raw_text.split() + + if simple_type.is_list(): + item_type = _unwrap_item_type(simple_type) + return [_coerce_scalar_text(token, item_type) for token in raw_text.split()] + + arrow_type, xsd_name = _resolve_scalar_type(simple_type) + return _coerce_resolved_scalar(raw_text.strip(), arrow_type, xsd_name) + + +def _extract_struct_attributes(element: Element, complex_type: XsdComplexType) -> dict[str, Any]: + """Extract and coerce declared XML attributes from ``element`` into ``_`` dictionary entries.""" + declared_attrs = {attr.local_name: attr for attr in _collect_attributes(complex_type) if attr.local_name} + raw_attrs = {_local_name(k): v for k, v in element.attrib.items() if not _is_ignored_attr(k)} + return { + f"_{attr_name}": _coerce_scalar_text(raw_attrs[attr_name], attr_decl.type) + for attr_name, attr_decl in declared_attrs.items() + if attr_name in raw_attrs + } + + +def _group_child_elements_by_name(element: Element) -> dict[str, list[Element]]: + """Group direct XML child elements by their local tag name in document order.""" + children_by_name: dict[str, list[Element]] = {} + for child_el in element: + if isinstance(child_el.tag, str): + children_by_name.setdefault(_local_name(child_el.tag), []).append(child_el) + return children_by_name + + +def _convert_complex_struct( + ctx: _ConvertContext, + element: Element, + complex_type: XsdComplexType, +) -> dict[str, Any] | None: + """Convert an XML ``element`` with a struct-mapped ``complex_type`` into a sparse dictionary.""" + result = _extract_struct_attributes(element, complex_type) + + if _is_xsi_nil(element): + return result or None + + if complex_type.has_simple_content(): + text_val = (element.text or "").strip() + if text_val: + content_type = complex_type.content if isinstance(complex_type.content, XsdType) else None + result["#text"] = _coerce_scalar_text(text_val, content_type) + return result or None + + if complex_type.mixed: + inner_xml = _serialize_inner_xml(element).strip() + if inner_xml: + result["#text"] = inner_xml + return result or None + + model_group = complex_type.model_group + if model_group is None: + return result or None + + children_by_name = _group_child_elements_by_name(element) + + for child_decl in _collect_unique_children(ctx.schema, complex_type): + child_name = child_decl.local_name + matching_elements = children_by_name.get(child_name) + if not matching_elements: + continue + + max_occurs = _group_max_occurs_by_name(model_group, child_name) + min_occurs = _group_min_occurs_by_name(model_group, child_name) + is_list = max_occurs is None or max_occurs > 1 + nullable = child_decl.nillable or min_occurs < 1 + + if is_list: + result[child_name] = [ + _convert_child_occurrence(ctx, el, child_decl, nullable=True) for el in matching_elements + ] + else: + val = _convert_child_occurrence(ctx, matching_elements[0], child_decl, nullable=nullable) + if val is not None: + result[child_name] = val + + return result or None + + +def _convert_text_or_empty_leaf( + element: Element, + *, + is_mixed: bool, + simple_type: XsdType | None, + nullable: bool, +) -> Any: # noqa: ANN401 + """Convert an XML element that maps to a scalar or mixed-text leaf field.""" + if _is_xsi_nil(element): + return None + raw_val = _serialize_inner_xml(element).strip() if is_mixed else (element.text or "").strip() + if not raw_val: + return "" if not nullable else None + return raw_val if is_mixed else _coerce_scalar_text(raw_val, simple_type) + + +def _convert_child_occurrence( + ctx: _ConvertContext, + element: Element, + xsd_element: XsdElement, + *, + nullable: bool, +) -> Any: # noqa: ANN401 + """Convert a single XML element occurrence according to ``xsd_element``.""" + elem_type = xsd_element.type + + if not isinstance(elem_type, XsdComplexType): + return _convert_text_or_empty_leaf(element, is_mixed=False, simple_type=elem_type, nullable=nullable) + + attrs = _collect_attributes(elem_type) + if elem_type.has_simple_content() and not attrs: + content_type = elem_type.content if isinstance(elem_type.content, XsdType) else None + return _convert_text_or_empty_leaf(element, is_mixed=False, simple_type=content_type, nullable=nullable) + + if elem_type.mixed and not attrs: + return _convert_text_or_empty_leaf(element, is_mixed=True, simple_type=None, nullable=nullable) + + if _is_empty_marker_complex_type(elem_type): + return None if _is_xsi_nil(element) else "" + + type_id = id(elem_type) + next_count = ctx.type_counts.get(type_id, 0) + 1 + if ctx.max_recursion_depth is not None and next_count > ctx.max_recursion_depth: + unbounded_ctx = _ConvertContext(schema=ctx.schema, type_counts={}, max_recursion_depth=None) + nested = _convert_complex_struct(unbounded_ctx, element, elem_type) + return json.dumps(nested) if nested is not None else None + + next_ctx = _ConvertContext( + schema=ctx.schema, + type_counts={**ctx.type_counts, type_id: next_count}, + max_recursion_depth=ctx.max_recursion_depth, + ) + return _convert_complex_struct(next_ctx, element, elem_type) + + +def convert_xml_element( + element: Element, + schema: xmlschema.XMLSchema, + xml_tag: str, + *, + max_recursion_depth: int = DEFAULT_MAX_RECURSION_DEPTH, +) -> dict[str, Any]: + """Convert an lxml Element matching ``xml_tag`` into a typed dictionary according to the XSD. + + :param element: XML element to convert + :type element: Element + :param schema: loaded XMLSchema instance + :type schema: xmlschema.XMLSchema + :param xml_tag: target XML element name (local name or Clark-notation ``{ns}local``) + :type xml_tag: str + :param max_recursion_depth: maximum structural recursion depth before JSON serialization + :type max_recursion_depth: int + :return: sparse dictionary of coerced field values matching the generated PyArrow schema + :rtype : dict[str, Any] + """ + target_elem = _resolve_target_element(schema, xml_tag) + elem_type = _require_complex_type(target_elem.type, xml_tag) + + ctx = _ConvertContext( + schema=schema, + type_counts={id(elem_type): 1}, + max_recursion_depth=max_recursion_depth, + ) + converted = _convert_complex_struct(ctx, element, elem_type) + return converted or {} diff --git a/tests/cdm_data_loaders/pipelines/framework.py b/tests/cdm_data_loaders/pipelines/framework.py index e684970b..66d11659 100644 --- a/tests/cdm_data_loaders/pipelines/framework.py +++ b/tests/cdm_data_loaders/pipelines/framework.py @@ -77,6 +77,7 @@ "uniprot_cdm": ("uniprot_cdm", "cdm_data_loaders.pipelines.uniprot_kb_cdm"), "uniref_cdm": ("uniref_cdm", "cdm_data_loaders.pipelines.uniref_cdm"), "xmltodict": ("xmltodict", "cdm_data_loaders.pipelines.xmltodict.pipeline"), + "xmltopyarrow": ("xmltopyarrow", "cdm_data_loaders.pipelines.xmltopyarrow.pipeline"), "xml2db": ("xml2db", "cdm_data_loaders.pipelines.xml2db.pipeline"), "xsv": ("xsv", "cdm_data_loaders.pipelines.xsv.pipeline"), } diff --git a/tests/cdm_data_loaders/pipelines/xmltodict/integration/test_xmltodict_reference_tests.py b/tests/cdm_data_loaders/pipelines/xmltodict/integration/test_xmltodict_reference_tests.py index ff7e45ea..9727c356 100644 --- a/tests/cdm_data_loaders/pipelines/xmltodict/integration/test_xmltodict_reference_tests.py +++ b/tests/cdm_data_loaders/pipelines/xmltodict/integration/test_xmltodict_reference_tests.py @@ -45,6 +45,14 @@ def test_xml_pass_reference_dataset_has_nested_structures( ) -> None: """The DuckDB reference stores nested JSON as STRUCT/LIST types, not JSON text columns.""" reference_xml_table = reference_xml_dataset.table("reference") + reference_xml_df = reference_xml_table.df() + entry = reference_xml_df["entry"].iloc[0] + assert isinstance(entry, dict) + assert isinstance(entry["property"], list) + assert isinstance(entry["representativeMember"], dict) + assert len(reference_xml_df) == EXPECTED_ENTRY_COUNT + + # same thing, but using native duckdb entry_type = reference_xml_table.types[0] entry_fields = dict(entry_type.children) @@ -52,7 +60,6 @@ def test_xml_pass_reference_dataset_has_nested_structures( member_type = entry_fields["representativeMember"] assert property_type.id == "list" assert member_type.id == "struct" - count = reference_xml_table.count("*").fetchone()[0] assert count == EXPECTED_ENTRY_COUNT diff --git a/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_pipeline.py b/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_pipeline.py index 74e788fb..d39392af 100644 --- a/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_pipeline.py +++ b/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_pipeline.py @@ -59,18 +59,6 @@ def test_run_xmltodict_pipeline_pass_binds_reader_before_run_pipeline( resource = mock_run_pipeline.call_args.kwargs["resource"] assert resource.name == "xmltodict_reader" assert resource._pipe.parent.name == "filesystem" # noqa: SLF001 - - -def test_run_xmltodict_pipeline_pass_resource_is_reader_piped_into_source( - settings_factory: Callable[..., XmlToDictSettings], -) -> None: - """The resource passed to run_pipeline combines the filesystem source with the bound reader.""" - settings = settings_factory() - - with patch.object(xmltodict_module, "run_pipeline") as mock_run_pipeline: - run_xmltodict_pipeline(settings) - - resource = mock_run_pipeline.call_args.kwargs["resource"] assert isinstance(resource, dlt.sources.DltResource) diff --git a/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_settings.py b/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_settings.py index 437f5d0a..ae0e53b6 100644 --- a/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_settings.py +++ b/tests/cdm_data_loaders/pipelines/xmltodict/test_xmltodict_settings.py @@ -1,6 +1,5 @@ """Tests for XmlToDictSettings field validation and defaults.""" -import shutil from collections.abc import Callable from pathlib import Path @@ -12,19 +11,6 @@ from cdm_data_loaders.pipelines.xmltodict.settings import XmlToDictSettings from tests.conftest import S3 -# All schema-driven tests below use tests/data/xsd/uniref_like.xsd, which is already exercised -# (and its list_paths/single_paths independently verified) by test_xsd.py: -# test_find_list_and_single_child_paths_pass_uniref_like_schema. -UNIREF_LIKE_XSD: str = "uniref_like.xsd" - - -def _copy_xsd_fixture(test_data_dir: Path, input_dir: Path, filename: str) -> str: - """Copy an XSD fixture from tests/data/xsd into input_dir; return its path.""" - input_dir.mkdir(parents=True, exist_ok=True) - dest = input_dir / filename - shutil.copy(test_data_dir / "xsd" / filename, dest) - return str(dest) - def test_xmltodict_settings_pass_defaults(settings_factory: Callable[..., XmlToDictSettings]) -> None: """file_glob defaults to '*.xml*'. buffer_size and log_interval default to the common CTS defaults.""" @@ -113,109 +99,6 @@ def test_xmltodict_settings_pass_unknown_use_destination_rejected( assert settings.use_destination == "not_a_destination" -def test_xmltodict_settings_pass_xmltodict_args_empty_without_xsd_file( - settings_factory: Callable[..., XmlToDictSettings], -) -> None: - """Without xsd_file (the default), xmltodict_args contributes no extra xmltodict.parse() kwargs.""" - settings = settings_factory() - assert settings.xsd_file is None - assert settings.xmltodict_args == {} - - -def test_xmltodict_settings_pass_xmltodict_args_has_force_list_callable_when_xsd_file_set( - tmp_path: Path, test_data_dir: Path, settings_factory: Callable[..., XmlToDictSettings] -) -> None: - """With xsd_file set, xmltodict_args exposes exactly one force_list callable kwarg.""" - xsd_file = _copy_xsd_fixture(test_data_dir, tmp_path / "input", UNIREF_LIKE_XSD) - settings = settings_factory(xsd_file=xsd_file) - - assert settings.xmltodict_args.keys() == {"force_list"} - assert callable(settings.xmltodict_args["force_list"]) - - -def test_xmltodict_settings_pass_xmltodict_args_is_cached( - tmp_path: Path, test_data_dir: Path, settings_factory: Callable[..., XmlToDictSettings] -) -> None: - """xmltodict_args is a cached_property: repeated access returns the same dict, not a rebuild.""" - xsd_file = _copy_xsd_fixture(test_data_dir, tmp_path / "input", UNIREF_LIKE_XSD) - settings = settings_factory(xsd_file=xsd_file) - assert settings.xmltodict_args is settings.xmltodict_args - - -def test_xmltodict_settings_fail_xsd_file_not_found_raises_runtime_error( - settings_factory: Callable[..., XmlToDictSettings], -) -> None: - """An xsd_file that does not exist raises ValidationError at construction.""" - with pytest.raises(ValidationError, match="Path does not point to a file"): - settings_factory(xsd_file="does_not_exist.xsd") - - -@pytest.fixture -def force_list_from_uniref_like_xsd( - tmp_path: Path, test_data_dir: Path, settings_factory: Callable[..., XmlToDictSettings] -) -> Callable[..., bool]: - """Build a real XmlToDictSettings from uniref_like.xsd and return its force_list callable.""" - xsd_file = _copy_xsd_fixture(test_data_dir, tmp_path / "input", UNIREF_LIKE_XSD) - settings = settings_factory(xsd_file=xsd_file) - return settings.xmltodict_args["force_list"] - - -@pytest.mark.parametrize( - ("path", "key", "expected"), - [ - pytest.param([], "UniRef50", False, id="document-root-is-never-forced-into-a-list"), - pytest.param([("UniRef50", None)], "entry", True, id="direct-child-of-root-list-path"), - pytest.param([("UniRef50", None), ("entry", None)], "property", True, id="two-levels-deep-list-path"), - pytest.param( - [("UniRef50", None), ("entry", None)], - "representativeMember", - False, - id="two-levels-deep-single-path", - ), - pytest.param( - [("UniRef50", None), ("entry", None), ("representativeMember", None)], - "dbReference", - False, - id="three-levels-deep-single-path-uses-immediate-parent-not-full-ancestor-chain", - ), - pytest.param( - [("UniRef50", None), ("entry", None), ("representativeMember", None), ("dbReference", None)], - "property", - True, - id="four-levels-deep-list-path-uses-immediate-parent-not-full-ancestor-chain", - ), - pytest.param( - [("UniRef50", None)], - "totallyUnmodelledChild", - True, - id="child-not-covered-by-schema-defaults-to-list", - ), - ], -) -def test_xmltodict_settings_pass_force_list_classifies_by_immediate_parent( - force_list_from_uniref_like_xsd: Callable[..., bool], - path: list[tuple[str, dict[str, str] | None]], - key: str, - expected: bool, -) -> None: - """force_list classifies every (parent, key) pair correctly, using only the immediate parent. - - xmltodict calls force_list(path, key, value) with `path` set to the full ancestor chain (a - list of (name, attrs) tuples) down to and including the immediate parent -- not just the - parent's own name -- and with an empty path for the top-level parsed element itself. A - correct implementation must: - - - Extract the immediate parent's name from the end of that chain, not compare the whole - chain (which would never match at any nesting depth beyond the root's direct children, - since find_list_and_single_child_paths stores plain (parent_name, child_name) pairs). - - Never force the top-level element into a list, since a parsed document only ever has one - root. - - Not raise: the ancestor chain is a list, so comparing it directly (unconverted) against a - set of (parent, key) string pairs raises `TypeError: unhashable type: 'list'`. - """ - assert force_list_from_uniref_like_xsd(path, key, "some-value") is expected - - @pytest.mark.parametrize("file_glob_arg", ["--file-glob", "-g"]) def test_xmltodict_settings_pass_xml_tag_shortcut(tmp_path: Path, file_glob_arg: str) -> None: """The file_glob field is exposed as both --file-glob and -g on the command line.""" diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/__init__.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/__init__.py @@ -0,0 +1 @@ + diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/conftest.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/conftest.py new file mode 100644 index 00000000..fc225dc4 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/conftest.py @@ -0,0 +1,53 @@ +"""Shared fixtures for xmltopyarrow pipeline tests.""" + +import shutil +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pytest + +from cdm_data_loaders.pipelines.xmltopyarrow.settings import XmlToPyArrowSettings +from tests.cdm_data_loaders.pipelines.helpers import make_cts_settings +from tests.xml_samples import LIBRARY_XSD + + +@pytest.fixture +def library_xsd(tmp_path: Path) -> Path: + """Write LIBRARY_XSD to tmp_path and return its path.""" + xsd_path = tmp_path / "library.xsd" + xsd_path.write_text(LIBRARY_XSD, encoding="utf-8") + return xsd_path + + +@pytest.fixture +def settings_factory(tmp_path: Path, library_xsd: Path) -> Callable[..., XmlToPyArrowSettings]: + """Return a factory that builds a valid XmlToPyArrowSettings, with fields open to override.""" + + def _factory(**overrides: Any) -> XmlToPyArrowSettings: # noqa: ANN401 + defaults: dict[str, Any] = { + "dlt_dev_mode": True, + "dataset_name": "xml_test_dataset", + "table_name": "book", + "xml_tag": "book", + "xsd_file": str(library_xsd), + "file_glob": "*.xml*", + } + defaults.update(overrides) + return make_cts_settings(XmlToPyArrowSettings, tmp_path, **defaults) + + return _factory + + +@pytest.fixture +def copy_xsd_fixture(tmp_path: Path, test_data_dir: Path) -> Callable[[str], str]: + """Return a function that copies an XSD from tests/data/xsd into the input dir and returns its path.""" + + def _copy(filename: str) -> str: + input_dir = tmp_path / "input" + input_dir.mkdir(exist_ok=True) + dest = input_dir / filename + shutil.copy(test_data_dir / "xsd" / filename, dest) + return str(dest) + + return _copy diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/__init__.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/__init__.py new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/__init__.py @@ -0,0 +1 @@ + diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/conftest.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/conftest.py new file mode 100644 index 00000000..7e295bd7 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/conftest.py @@ -0,0 +1,206 @@ +"""Shared fixtures for xmltopyarrow pipeline integration tests.""" + +import gzip +import json +from collections.abc import Callable, Generator +from pathlib import Path +from typing import Any + +import duckdb +import pytest +import xmltodict +from dlt.common.pipeline import LoadInfo +from frozendict import frozendict + +from cdm_data_loaders.pipelines.xmltopyarrow.pipeline import run_xmltopyarrow_pipeline +from cdm_data_loaders.pipelines.xmltopyarrow.settings import XmlToPyArrowSettings +from cdm_data_loaders.readers.xml import DEFAULT_XMLTODICT_ARGS +from tests.cdm_data_loaders.pipelines.helpers import ( + denormalize_identifier, + duckdb_reference_from_jsonl, + make_cts_settings, + read_jsonl_file, + strip_metadata_columns, +) +from tests.conftest import N_REFERENCE_XML_ENTRIES, REFERENCE_XML_TAG +from tests.xml_samples import LIBRARY_XSD + +DEFAULT_XMLTOPYARROW_SETTINGS: frozendict = frozendict( + { + "buffer_size": 10, + "dlt_dev_mode": False, + "file_glob": "*.xml*", + "log_interval": 1000, + "use_output_dir_for_pipeline_metadata": False, + } +) + + +@pytest.fixture +def library_xsd(tmp_path: Path) -> Path: + """Write LIBRARY_XSD to tmp_path and return its path.""" + xsd_path = tmp_path / "library.xsd" + xsd_path.write_text(LIBRARY_XSD, encoding="utf-8") + return xsd_path + + +@pytest.fixture +def settings_factory( + tmp_path: Path, + dlt_destination_config: str, + library_xsd: Path, +) -> Callable[..., XmlToPyArrowSettings]: + """Return a factory that builds a valid XmlToPyArrowSettings, with fields open to override.""" + + def _factory(**overrides: Any) -> XmlToPyArrowSettings: # noqa: ANN401 + defaults: dict[str, Any] = { + **DEFAULT_XMLTOPYARROW_SETTINGS, + "dataset_name": "xml_test_dataset", + "table_name": "book", + "use_destination": dlt_destination_config, + "xml_tag": "book", + "xsd_file": str(library_xsd), + } + defaults.update(overrides) + return make_cts_settings(XmlToPyArrowSettings, tmp_path, **defaults) + + return _factory + + +def import_jsonl_data(jsonl_path: Path) -> list[dict[str, Any]]: + """Read the reference JSONL into a list of canonical nested entry dicts.""" + return read_jsonl_file(jsonl_path) + + +def write_reference_xml_as_jsonl(source_xml: Path, jsonl_path: Path) -> int: + """Parse the UniRef XML with xmltodict and write the reference JSONL.""" + with gzip.open(source_xml, "rb") as fh: + document = xmltodict.parse(fh.read(), **DEFAULT_XMLTODICT_ARGS) + + entries = document["UniRef50"]["entry"] + with jsonl_path.open("w", encoding="utf-8") as out: + for entry in entries: + out.write(json.dumps({"entry": entry}, ensure_ascii=False) + "\n") + return len(entries) + + +@pytest.fixture(scope="session") +def reference_xml_as_jsonl(reference_xml_data_dir: Path) -> Path: + """Generate the reference JSONL from the chunk_100_el source, if not present.""" + jsonl_path = reference_xml_data_dir / "jsonl" / "uniref_100.jsonl" + if not jsonl_path.exists(): + source_xml = reference_xml_data_dir / "chunk_100_el" / "uniref_100_part_01.xml.gz" + n_entries = write_reference_xml_as_jsonl(source_xml, jsonl_path) + assert n_entries == N_REFERENCE_XML_ENTRIES + + return jsonl_path + + +@pytest.fixture(scope="session") +def reference_xml_entries(reference_xml_as_jsonl: Path) -> list[dict[str, Any]]: + """The canonical nested entry dicts read back from the reference JSONL.""" + ref_json = import_jsonl_data(reference_xml_as_jsonl) + assert isinstance(ref_json, list) + for line in ref_json: + assert list(line.keys()) == ["entry"] + entry = line.get("entry", {}) + for key in ("_id", "name", "property", "representativeMember"): + assert key in entry + assert entry.get("_id") is not None + assert entry.get("_id").startswith("UniRef50_") + assert len(ref_json) == N_REFERENCE_XML_ENTRIES + return ref_json + + +@pytest.fixture(scope="session") +def sorted_reference_xml_entries(reference_xml_entries: list[dict[str, Any]]) -> list[str]: + """Sorted XSD-normalized entries for the uniref reference dataset.""" + return sorted( + json.dumps(normalize_uniref_entry_for_xsd(entry), sort_keys=True, default=str) + for entry in reference_xml_entries + ) + + +@pytest.fixture(scope="session") +def reference_xml_dataset(reference_xml_as_jsonl: Path) -> Generator[duckdb.DuckDBPyConnection]: + """Load the reference JSONL into an in-memory DuckDB connection with nested structures.""" + with duckdb_reference_from_jsonl(reference_xml_as_jsonl) as conn: + yield conn + + +@pytest.fixture +def run_xmltopyarrow_pipeline_fxt( + reference_xml_data_dir: Path, + dlt_destination_config: str, + isolated_run_factory: Callable[..., Callable[..., tuple[LoadInfo | None, Path]]], +) -> Callable[..., Any]: + """Return a factory that runs the xmltopyarrow pipeline on a chunk dir with full isolation. + + Each run gets a unique output_dir and dataset_name under tmp_path, so + parallel-safe repeated runs never collide or accumulate rows. + """ + base_kwargs = { + **DEFAULT_XMLTOPYARROW_SETTINGS, + "settings_cls": XmlToPyArrowSettings, + "table_name": "entry", + "use_destination": dlt_destination_config, + "xml_tag": REFERENCE_XML_TAG, + "xsd_file": str(reference_xml_data_dir / "uniref.xsd"), + } + runner = isolated_run_factory( + dataset_prefix="reference_run", + run_fn=run_xmltopyarrow_pipeline, + base_kwargs=base_kwargs, + ) + + def _factory(chunk_dir: str, **overrides: Any) -> tuple[LoadInfo | None, Path]: # noqa: ANN401 + return runner(input_dir=str(reference_xml_data_dir / chunk_dir), **overrides) + + return _factory + + +def _strip_none_values(value: Any) -> Any: # noqa: ANN401 + """Recursively remove keys with None values from nested dicts and lists.""" + if isinstance(value, dict): + return {k: _strip_none_values(v) for k, v in value.items() if v is not None} + if isinstance(value, list): + return [_strip_none_values(item) for item in value] + return value + + +def strip_xsd_arrow_row(row: dict[str, Any]) -> dict[str, Any]: + """Strip dlt metadata columns, unmangle top-level column names, and drop None struct fields.""" + unmangled = {denormalize_identifier(k): v for k, v in strip_metadata_columns(row).items()} + return _strip_none_values(unmangled) + + +def _normalize_xsd_reference_member(member: dict[str, Any]) -> dict[str, Any]: + """Normalize an xmltodict member dict to match XSD-typed cardinality and scalar types.""" + normalized = dict(member) + db_ref = dict(normalized["dbReference"]) + if "property" in db_ref and isinstance(db_ref["property"], dict): + db_ref["property"] = [db_ref["property"]] + normalized["dbReference"] = db_ref + if "sequence" in normalized and isinstance(normalized["sequence"], dict): + seq = dict(normalized["sequence"]) + if "_length" in seq: + seq["_length"] = int(seq["_length"]) + normalized["sequence"] = seq + return normalized + + +def normalize_uniref_entry_for_xsd(raw_entry: dict[str, Any]) -> dict[str, Any]: + """Normalize a raw xmltodict UniRef entry dict to match XSD-backed Arrow output.""" + entry = dict(raw_entry.get("entry", raw_entry)) + if "property" in entry and isinstance(entry["property"], dict): + entry["property"] = [entry["property"]] + entry["representativeMember"] = _normalize_xsd_reference_member(entry["representativeMember"]) + if "member" in entry: + members = entry["member"] if isinstance(entry["member"], list) else [entry["member"]] + entry["member"] = [_normalize_xsd_reference_member(m) for m in members] + return entry + + +def reconstruct_xsd_entries(tables: dict[str, list[dict[str, Any]]]) -> list[dict[str, Any]]: + """Extract normalized XSD entry dicts from the single Arrow-backed entry table.""" + return [strip_xsd_arrow_row(row) for row in tables.get("entry", [])] diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/test_xmltopyarrow_end_to_end.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/test_xmltopyarrow_end_to_end.py new file mode 100644 index 00000000..8c7724cb --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/test_xmltopyarrow_end_to_end.py @@ -0,0 +1,144 @@ +"""End-to-end tests for the xmltopyarrow pipeline.""" + +from collections.abc import Callable +from pathlib import Path +from typing import Any + +import pytest +from dlt.common.pipeline import LoadInfo +from pandas import DataFrame + +from cdm_data_loaders.pipelines.xmltopyarrow.settings import PIPELINE_NAME, XmlToPyArrowSettings +from tests.cdm_data_loaders.pipelines.conftest import DEFAULT_DLT_TABLES +from tests.cdm_data_loaders.pipelines.framework import PipelineTestRunner +from tests.cdm_data_loaders.pipelines.helpers import ( + LOADER_FILE_FORMATS, + assert_pipeline_report_written, + assert_run_ok, + settings_to_argv, + sorted_json, +) +from tests.cdm_data_loaders.pipelines.xmltopyarrow.integration.conftest import ( + normalize_uniref_entry_for_xsd, + strip_xsd_arrow_row, +) +from tests.conftest import N_REFERENCE_XML_ENTRIES +from tests.xml_samples import SIMPLE_LIBRARY_XML + + +def check_book_list_results(load_info: LoadInfo | None, table_name: str | None = None) -> DataFrame: + """Check over the output of parsing SIMPLE_LIBRARY_XML with LIBRARY_XSD.""" + load_info = assert_run_ok(load_info) + dataset = load_info.pipeline.dataset() + table_name = table_name or "book" + assert set(dataset.tables) == {table_name, *DEFAULT_DLT_TABLES} + book_df = dataset.table(table_name).df() + assert set(book_df.columns.tolist()) >= {"_id", "title"} + return book_df + + +@pytest.mark.parametrize("loader_file_format", LOADER_FILE_FORMATS) +def test_run_xmltopyarrow_pipeline_pass_writes_expected_output( + settings_factory: Callable[..., XmlToPyArrowSettings], + loader_file_format: str, + pipeline_runner: PipelineTestRunner, +) -> None: + """Running the pipeline on xml files loads one row per matching element into the configured table.""" + settings = settings_factory(loader_file_format=loader_file_format) + input_dir = Path(settings.input_dir) + input_dir.mkdir(exist_ok=True) + (input_dir / "library.xml").write_text(SIMPLE_LIBRARY_XML, encoding="utf-8") + + load_info = pipeline_runner.run(PIPELINE_NAME, settings=settings) + book_df = check_book_list_results(load_info) + assert sorted(book_df["_id"].tolist()) == ["1", "2", "3"] + assert_pipeline_report_written(Path(settings.output_dir), load_info) + + +def test_run_xmltopyarrow_pipeline_pass_gzip_files_are_loaded( + settings_factory: Callable[..., XmlToPyArrowSettings], + write_gzip_file: Callable[[Path, str, str], Path], + pipeline_runner: PipelineTestRunner, +) -> None: + """Gzip-compressed xml files matching the glob are decompressed and loaded.""" + settings = settings_factory() + input_dir = Path(settings.input_dir) + input_dir.mkdir(exist_ok=True) + (input_dir / "plain.xml").write_text(SIMPLE_LIBRARY_XML, encoding="utf-8") + write_gzip_file(input_dir, "library.xml.gz", SIMPLE_LIBRARY_XML) + + load_info = pipeline_runner.run(PIPELINE_NAME, settings=settings) + book_df = check_book_list_results(load_info) + # one copy of the three books from the plain file, one from the gzipped copy + assert sorted(book_df["_id"].tolist()) == ["1", "1", "2", "2", "3", "3"] + + +def test_run_xmltopyarrow_pipeline_pass_no_matching_files_yields_no_data_table( + settings_factory: Callable[..., XmlToPyArrowSettings], + pipeline_runner: PipelineTestRunner, +) -> None: + """An input dir with no matching xml files does not fail the run and creates no data table.""" + settings = settings_factory() + Path(settings.input_dir).mkdir(exist_ok=True) + + load_info = assert_run_ok(pipeline_runner.run(PIPELINE_NAME, settings=settings)) + dataset = load_info.pipeline.dataset() + # only dlt's own metadata tables exist; the data table is never created + assert set(dataset.tables) == DEFAULT_DLT_TABLES + assert_pipeline_report_written(Path(settings.output_dir), load_info) + + +def test_run_xmltopyarrow_pipeline_pass_custom_table_name_is_respected( + settings_factory: Callable[..., XmlToPyArrowSettings], + pipeline_runner: PipelineTestRunner, +) -> None: + """Rows land in the table named by table_name, not in a name derived from the xml tag.""" + settings = settings_factory(table_name="library_entries") + input_dir = Path(settings.input_dir) + input_dir.mkdir(exist_ok=True) + (input_dir / "library.xml").write_text(SIMPLE_LIBRARY_XML, encoding="utf-8") + + load_info = pipeline_runner.run(PIPELINE_NAME, settings=settings) + book_df = check_book_list_results(load_info, "library_entries") + assert sorted(book_df["_id"].tolist()) == ["1", "2", "3"] + + +def test_cli_pass_runs_end_to_end_from_command_line_arguments( + settings_factory: Callable[..., XmlToPyArrowSettings], + pipeline_runner: PipelineTestRunner, +) -> None: + """cli() reads command-line arguments and runs the pipeline, producing the expected output.""" + settings = settings_factory(dataset_name="cli_dataset") + input_dir = Path(settings.input_dir) + input_dir.mkdir(exist_ok=True) + (input_dir / "library.xml").write_text(SIMPLE_LIBRARY_XML, encoding="utf-8") + + load_info = assert_run_ok(pipeline_runner.run(PIPELINE_NAME, cli_args=settings_to_argv(settings))) + book_df = check_book_list_results(load_info) + assert sorted(book_df["_id"].tolist()) == ["1", "2", "3"] + assert_pipeline_report_written(Path(settings.output_dir), load_info) + + +def test_run_xmltopyarrow_pipeline_pass_uniref_chunk_5_el( + settings_factory: Callable[..., XmlToPyArrowSettings], + reference_xml_data_dir: Path, + reference_xml_entries: list[dict[str, Any]], + pipeline_runner: PipelineTestRunner, +) -> None: + """Running the pipeline on uniref chunk_5_el with uniref.xsd loads all 100 entries with Arrow types.""" + settings = settings_factory( + input_dir=str(reference_xml_data_dir / "chunk_5_el"), + xsd_file=str(reference_xml_data_dir / "uniref.xsd"), + xml_tag="entry", + table_name="entry", + ) + + load_info = assert_run_ok(pipeline_runner.run(PIPELINE_NAME, settings=settings)) + dataset = load_info.pipeline.dataset() + assert set(dataset.tables) == {"entry", *DEFAULT_DLT_TABLES} + + loaded_rows = [strip_xsd_arrow_row(row) for row in dataset.table("entry").arrow().to_pylist()] + expected_rows = [normalize_uniref_entry_for_xsd(entry) for entry in reference_xml_entries] + assert len(loaded_rows) == N_REFERENCE_XML_ENTRIES + assert sorted_json(loaded_rows) == sorted_json(expected_rows) + assert_pipeline_report_written(Path(settings.output_dir), load_info) diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/test_xmltopyarrow_reference_tests.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/test_xmltopyarrow_reference_tests.py new file mode 100644 index 00000000..3372d831 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/integration/test_xmltopyarrow_reference_tests.py @@ -0,0 +1,170 @@ +"""Integration tests for the xmltopyarrow pipeline against the uniref reference dataset.""" + +from collections.abc import Callable +from typing import Any + +import dlt +import duckdb +import pytest + +from cdm_data_loaders.core.fields import PARQUET +from tests.cdm_data_loaders.pipelines.helpers import ( + BUFFER_SIZES, + LOADER_FILE_FORMATS, + WORKER_CONFIGS, + assert_dataset_matches_reference, + assert_datasets_equal, + read_pipeline_tables, + reference_entries_from_duckdb, + run_and_compare_variations, + run_and_read, + sorted_json, +) +from tests.cdm_data_loaders.pipelines.xmltopyarrow.integration.conftest import ( + normalize_uniref_entry_for_xsd, + reconstruct_xsd_entries, + strip_xsd_arrow_row, +) + +CHUNK_DIRS = ("chunk_5_el", "chunk_20_el", "chunk_100_el") +EXPECTED_ENTRY_COUNT = 100 + + +def test_xmltopyarrow_pass_diff_format_same_results( + run_xmltopyarrow_pipeline_fxt: Callable[..., Any], + sorted_reference_xml_entries: list[str], +) -> None: + """Ensure that the save format does not affect the reconstructed output.""" + reconstructed_by_format: dict[str, list[str]] = {} + for output_format in LOADER_FILE_FORMATS: + fmt = str(output_format) + load_info, output_dir = run_and_read( + run_xmltopyarrow_pipeline_fxt, + "chunk_20_el", + loader_file_format=output_format, + ) + tables = read_pipeline_tables(output_dir / load_info.dataset_name, fmt) + reconstructed_by_format[fmt] = sorted_json(reconstruct_xsd_entries(tables)) + + formats = list(reconstructed_by_format.keys()) + for fmt in formats[1:]: + assert reconstructed_by_format[fmt] == reconstructed_by_format[formats[0]] + assert reconstructed_by_format[formats[0]] == sorted_reference_xml_entries + + +@pytest.mark.parametrize("loader_file_format", LOADER_FILE_FORMATS) +def test_xmltopyarrow_pass_output_identical_across_chunk_dirs( + run_xmltopyarrow_pipeline_fxt: Callable[..., Any], + sorted_reference_xml_entries: list[str], + loader_file_format: str, +) -> None: + """Pipeline output is identical across all three chunk dirs and matches the reference.""" + run_and_compare_variations( + run_xmltopyarrow_pipeline_fxt, + {f"{key}_{loader_file_format}": (key, {"loader_file_format": loader_file_format}) for key in CHUNK_DIRS}, + sorted_reference_json=sorted_reference_xml_entries, + output_format=loader_file_format, + reconstruct=reconstruct_xsd_entries, + ) + + +@pytest.mark.parametrize("loader_file_format", LOADER_FILE_FORMATS) +def test_xmltopyarrow_pass_output_identical_across_buffer_sizes( + run_xmltopyarrow_pipeline_fxt: Callable[..., Any], + sorted_reference_xml_entries: list[str], + loader_file_format: str, +) -> None: + """ArrowListBuffer size does not affect pipeline output for any chunk dir.""" + for chunk_dir in CHUNK_DIRS: + run_and_compare_variations( + run_xmltopyarrow_pipeline_fxt, + { + f"{buffer_size}_{loader_file_format}": ( + chunk_dir, + {"buffer_size": buffer_size, "loader_file_format": loader_file_format}, + ) + for buffer_size in BUFFER_SIZES + }, + sorted_reference_json=sorted_reference_xml_entries, + output_format=loader_file_format, + reconstruct=reconstruct_xsd_entries, + ) + + +@pytest.mark.parametrize( + ("extract_workers", "normalize_workers", "load_workers"), + WORKER_CONFIGS, + ids=[f"extract_{e}-normalize_{n}-load_{l}" for e, n, l in WORKER_CONFIGS], # noqa: E741 +) +@pytest.mark.parametrize("loader_file_format", LOADER_FILE_FORMATS, ids=str) +def test_xmltopyarrow_pass_no_data_lost_across_worker_configs( + *, + run_xmltopyarrow_pipeline_fxt: Callable[..., Any], + sorted_reference_xml_entries: list[str], + extract_workers: int, + normalize_workers: int, + load_workers: int, + loader_file_format: str, +) -> None: + """No data is lost regardless of how the job is split across extract/normalize/load workers.""" + load_info_and_dir = {} + for worker_key in ("baseline", "current"): + worker_config = ( + {} + if worker_key == "baseline" + else { + "extract.workers": extract_workers, + "normalize.workers": normalize_workers, + "load.workers": load_workers, + } + ) + with dlt.config.values(worker_config): + load_info_and_dir[worker_key] = run_and_read( + run_xmltopyarrow_pipeline_fxt, "chunk_5_el", loader_file_format=loader_file_format + ) + + assert_datasets_equal(load_info_and_dir) + assert_dataset_matches_reference( + load_info_and_dir, + sorted_reference_xml_entries, + loader_file_format, + reconstruct=reconstruct_xsd_entries, + ) + + +def test_xmltopyarrow_pass_uniref_chunk_5_el_matches_reference( + run_xmltopyarrow_pipeline_fxt: Callable[..., Any], + reference_xml_entries: list[dict[str, Any]], +) -> None: + """Running on chunk_5_el with uniref.xsd produces a single Arrow-typed entry table matching the 100 reference entries.""" + (load_info, output_dir) = run_and_read( + run_xmltopyarrow_pipeline_fxt, + "chunk_5_el", + ) + tables = read_pipeline_tables(output_dir / load_info.dataset_name, PARQUET) + assert set(tables) == {"entry"} + loaded_rows = [strip_xsd_arrow_row(row) for row in tables["entry"]] + expected_rows = [normalize_uniref_entry_for_xsd(entry) for entry in reference_xml_entries] + assert len(loaded_rows) == EXPECTED_ENTRY_COUNT + assert sorted_json(loaded_rows) == sorted_json(expected_rows) + + +@pytest.mark.parametrize("loader_file_format", LOADER_FILE_FORMATS) +def test_xmltopyarrow_pass_reconstructed_matches_duckdb_reference( + run_xmltopyarrow_pipeline_fxt: Callable[..., Any], + reference_xml_dataset: duckdb.DuckDBPyConnection, + loader_file_format: str, +) -> None: + """Reconstructed pipeline entries equal the entries loaded from the DuckDB reference.""" + (load_info, output_dir) = run_and_read( + run_xmltopyarrow_pipeline_fxt, + "chunk_100_el", + loader_file_format=loader_file_format, + ) + tables = read_pipeline_tables(output_dir / load_info.dataset_name, loader_file_format) + reconstructed = reconstruct_xsd_entries(tables) + from_duckdb = [ + normalize_uniref_entry_for_xsd(entry) for entry in reference_entries_from_duckdb(reference_xml_dataset) + ] + assert len(from_duckdb) == EXPECTED_ENTRY_COUNT + assert sorted_json(reconstructed) == sorted_json(from_duckdb) diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xmltopyarrow_pipeline.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xmltopyarrow_pipeline.py new file mode 100644 index 00000000..e6053cd9 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xmltopyarrow_pipeline.py @@ -0,0 +1,240 @@ +"""Unit tests for run_xmltopyarrow_pipeline and cli wiring.""" + +import sys +from collections.abc import Callable +from pathlib import Path +from typing import Any +from unittest.mock import patch +from uuid import uuid4 + +import dlt +import pytest +from dlt.common.pipeline import LoadInfo +from dlt.extract import DltResource + +import cdm_data_loaders.pipelines.xmltopyarrow.pipeline as xmltopyarrow_module +from cdm_data_loaders.core.fields import LoaderFileFormatEnum +from cdm_data_loaders.pipelines.xmltopyarrow.pipeline import ( + cli, + run_xmltopyarrow_pipeline, +) +from cdm_data_loaders.pipelines.xmltopyarrow.settings import PIPELINE_NAME, XmlToPyArrowSettings +from tests.cdm_data_loaders.pipelines.conftest import duckdb_pipeline +from tests.xml_samples import TWO_BOOK_LIBRARY_XML + + +@pytest.mark.parametrize("loader_file_format", list(LoaderFileFormatEnum), ids=str) +def test_run_xmltopyarrow_pipeline_pass_sets_core_run_pipeline_args_correctly( + settings_factory: Callable[..., XmlToPyArrowSettings], + loader_file_format: str, +) -> None: + """run_xmltopyarrow_pipeline builds the reader and delegates to run_pipeline with the correct args.""" + settings = settings_factory(loader_file_format=loader_file_format) + + with patch.object(xmltopyarrow_module, "run_pipeline") as mock_run_pipeline: + run_xmltopyarrow_pipeline(settings) + + assert mock_run_pipeline.call_count == 1 + _, kwargs = mock_run_pipeline.call_args + assert kwargs.keys() == {"settings", "resource", "pipeline_kwargs", "pipeline_run_kwargs"} + assert kwargs["settings"] == settings + assert kwargs["pipeline_kwargs"] == { + "pipeline_name": PIPELINE_NAME, + "dataset_name": settings.dataset_name, + } + assert kwargs["pipeline_run_kwargs"] == {"loader_file_format": loader_file_format} + assert isinstance(kwargs["resource"], DltResource) + + +def test_run_xmltopyarrow_pipeline_pass_binds_reader_before_run_pipeline( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """The resource passed to run_pipeline is the bound reader piped into the filesystem source.""" + settings = settings_factory() + + with patch.object(xmltopyarrow_module, "run_pipeline") as mock_run_pipeline: + run_xmltopyarrow_pipeline(settings) + + mock_run_pipeline.assert_called_once() + resource = mock_run_pipeline.call_args.kwargs["resource"] + assert resource.name == "xmltopyarrow_reader" + assert resource._pipe.parent.name == "filesystem" # noqa: SLF001 + assert isinstance(resource, dlt.sources.DltResource) + + +def test_cli_pass_runs_end_to_end_from_command_line_arguments( + tmp_path: Path, + library_xsd: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """cli() reads command-line arguments and runs the pipeline via run_xmltopyarrow_pipeline. + + The real XmlToPyArrowSettings parses argv (seeded by the autouse dlt + config isolation fixture), and core.run_pipeline is redirected to a real + DuckDB pipeline so the full flow is validated. + """ + input_dir = tmp_path / "cli_input" + input_dir.mkdir() + (input_dir / "library.xml").write_text(TWO_BOOK_LIBRARY_XML, encoding="utf-8") + output_dir = tmp_path / "cli_output" + output_dir.mkdir() + log_config_file = tmp_path / "logging.json" + log_config_file.write_text('{"version": 1}') + + argv = [ + PIPELINE_NAME, + "--input-dir", + str(input_dir), + "--output-dir", + str(output_dir), + "--use-destination", + "local_fs", + "--log-config-file", + str(log_config_file), + "--dataset-name", + "cli_dataset", + "--table-name", + "book", + "--xml-tag", + "book", + "--xsd-file", + str(library_xsd), + "--use-output-dir-for-pipeline-metadata", + "false", + "--dlt-dev-mode", + "false", + ] + monkeypatch.setattr(sys, "argv", argv) + + # unique pipeline name: the duckdb database file is derived from it, so + # repeated runs of the suite do not see rows from previous runs + pipeline_name = f"test_xmltopyarrow_cli_pipeline_{uuid4().hex}" + + def fake_run_pipeline( + *, + resource: DltResource, + pipeline_kwargs: dict[str, Any], + **_: dict[str, Any], + ) -> LoadInfo | None: + assert pipeline_kwargs == {"pipeline_name": PIPELINE_NAME, "dataset_name": "cli_dataset"} + pipeline = duckdb_pipeline(tmp_path, pipeline_name, "cli_dataset") + return pipeline.run(resource) + + with patch.object(xmltopyarrow_module, "run_pipeline", fake_run_pipeline): + load_info = cli.run() + + assert load_info is not None + assert not load_info.has_failed_jobs + + with ( + load_info.pipeline.sql_client() as client, + client.execute_query("SELECT COUNT(*) FROM book") as cur, + ): + (row_count,) = cur.fetchone() + assert row_count == 2 # noqa: PLR2004 + + +def test_cli_pass_writes_rows_for_matching_xml_files( + tmp_path: Path, + library_xsd: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """cli() with xml files in the input dir loads one row per matching element.""" + input_dir = tmp_path / "xml_input" + input_dir.mkdir() + (input_dir / "data.xml").write_text(TWO_BOOK_LIBRARY_XML, encoding="utf-8") + + output_dir = tmp_path / "cli_output" + output_dir.mkdir() + log_config_file = tmp_path / "logging.json" + log_config_file.write_text('{"version": 1}') + + argv = [ + PIPELINE_NAME, + "--input-dir", + str(input_dir), + "--output-dir", + str(output_dir), + "--use-destination", + "local_fs", + "--log-config-file", + str(log_config_file), + "--dataset-name", + "cli_dataset", + "--table-name", + "book", + "--xml-tag", + "book", + "--xsd-file", + str(library_xsd), + "--use-output-dir-for-pipeline-metadata", + "false", + "--dlt-dev-mode", + "false", + ] + monkeypatch.setattr(sys, "argv", argv) + + pipeline_name = f"test_xmltopyarrow_cli_rows_pipeline_{uuid4().hex}" + + def fake_run_pipeline( + *, + resource: Any, # noqa: ANN401 + **_: Any, # noqa: ANN401 + ) -> LoadInfo | None: + pipeline = duckdb_pipeline(tmp_path, pipeline_name, "cli_dataset") + return pipeline.run(resource) + + with patch.object(xmltopyarrow_module, "run_pipeline", fake_run_pipeline): + load_info = cli.run() + + assert load_info is not None + assert not load_info.has_failed_jobs + pipeline = load_info.pipeline + with ( + pipeline.sql_client() as client, + client.execute_query("SELECT COUNT(*) FROM book") as cur, + ): + (row_count,) = cur.fetchone() + assert row_count == 2 # noqa: PLR2004 + + +@pytest.mark.parametrize( + ("missing_args", "expected_fragment"), + [ + (["--table-name", "book", "--xml-tag", "book", "--xsd-file", "{xsd}"], "dataset_name"), + (["--dataset-name", "cli_dataset", "--xml-tag", "book", "--xsd-file", "{xsd}"], "table_name"), + (["--dataset-name", "cli_dataset", "--table-name", "book", "--xsd-file", "{xsd}"], "xml_tag"), + (["--dataset-name", "cli_dataset", "--table-name", "book", "--xml-tag", "book"], "xsd_file"), + ], +) +def test_cli_fail_missing_required_args_raise( + tmp_path: Path, + library_xsd: Path, + monkeypatch: pytest.MonkeyPatch, + missing_args: list[str], + expected_fragment: str, +) -> None: + """cli() without required pipeline-specific args raises before running the pipeline.""" + log_config_file = tmp_path / "logging.json" + log_config_file.write_text('{"version": 1}') + + resolved_args = [arg.format(xsd=library_xsd) for arg in missing_args] + argv = [ + PIPELINE_NAME, + "--input-dir", + str(tmp_path / "input"), + "--output-dir", + str(tmp_path / "output"), + "--log-config-file", + str(log_config_file), + *resolved_args, + ] + monkeypatch.setattr(sys, "argv", argv) + + with ( + patch.object(xmltopyarrow_module, "run_pipeline") as mock_run_pipeline, + pytest.raises(Exception, match=expected_fragment), + ): + cli() + + mock_run_pipeline.assert_not_called() diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xmltopyarrow_settings.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xmltopyarrow_settings.py new file mode 100644 index 00000000..df4ae695 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xmltopyarrow_settings.py @@ -0,0 +1,213 @@ +"""Tests for XmlToPyArrowSettings field validation and defaults.""" + +from collections.abc import Callable +from pathlib import Path + +import pyarrow as pa +import pytest +import xmlschema +from pydantic import ValidationError +from pydantic_settings import CliApp + +from cdm_data_loaders.core.fields import DEFAULTS +from cdm_data_loaders.pipelines.xmltopyarrow.settings import XmlToPyArrowSettings +from cdm_data_loaders.readers.xsd import DEFAULT_MAX_RECURSION_DEPTH, generate_pyarrow_schema, load_schema +from tests.conftest import S3 + +UNIREF_LIKE_XSD: str = "uniref_like.xsd" + + +def test_xmltopyarrow_settings_pass_defaults(settings_factory: Callable[..., XmlToPyArrowSettings]) -> None: + """file_glob defaults to '*.xml*'. buffer_size, log_interval, and max_recursion_depth default to expected values.""" + settings = settings_factory() + assert settings.file_glob == "*.xml*" + assert settings.buffer_size == DEFAULTS["buffer_size"] + assert settings.log_interval == DEFAULTS["log_interval"] + assert settings.max_recursion_depth == DEFAULT_MAX_RECURSION_DEPTH + + +def test_xmltopyarrow_settings_pass_custom_file_glob( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """file_glob accepts a custom glob pattern.""" + assert settings_factory(file_glob="*.rdf").file_glob == "*.rdf" + + +def test_xmltopyarrow_settings_pass_custom_xml_tag( + settings_factory: Callable[..., XmlToPyArrowSettings], + copy_xsd_fixture: Callable[[str], str], +) -> None: + """xml_tag accepts any tag name present in the XSD.""" + xsd_file = copy_xsd_fixture(UNIREF_LIKE_XSD) + settings = settings_factory(xsd_file=xsd_file, xml_tag="entry") + assert settings.xml_tag == "entry" + + +def test_xmltopyarrow_settings_fail_missing_xml_tag( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """Omitting xml_tag raises ValidationError.""" + with pytest.raises(ValidationError): + settings_factory(xml_tag=None) + + +def test_xmltopyarrow_settings_fail_missing_table_name( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """Omitting table_name raises ValidationError.""" + with pytest.raises(ValidationError): + settings_factory(table_name=None) + + +def test_xmltopyarrow_settings_fail_missing_dataset_name( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """Omitting dataset_name raises ValidationError.""" + with pytest.raises(ValidationError): + settings_factory(dataset_name=None) + + +def test_xmltopyarrow_settings_fail_missing_xsd_file( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """Omitting xsd_file raises ValidationError.""" + with pytest.raises(ValidationError): + settings_factory(xsd_file=None) + + +def test_xmltopyarrow_settings_fail_non_positive_buffer_size( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """buffer_size must be a positive integer.""" + with pytest.raises(ValidationError): + settings_factory(buffer_size=0) + + +def test_xmltopyarrow_settings_fail_non_positive_log_interval( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """log_interval must be a positive integer.""" + with pytest.raises(ValidationError): + settings_factory(log_interval=0) + + +def test_xmltopyarrow_settings_fail_local_destination_with_s3_output_dir( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """An s3:// output_dir raises ValidationError at construction because output_dir must be a local path.""" + with pytest.raises(ValidationError): + settings_factory(output_dir="s3://some-bucket/path", use_destination="local_fs") + + +def test_xmltopyarrow_settings_fail_local_destination_with_s3_flag( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """An s3 destination with a local output_dir is accepted at construction.""" + settings = settings_factory(use_destination=S3) + assert settings.use_destination == S3 + + +def test_xmltopyarrow_settings_pass_unknown_use_destination_rejected( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """use_destination not present in the dlt config is accepted at construction.""" + settings = settings_factory(use_destination="not_a_destination") + assert settings.use_destination == "not_a_destination" + + +def test_xmltopyarrow_settings_pass_schema_properties_populated_when_xsd_file_set( + settings_factory: Callable[..., XmlToPyArrowSettings], + copy_xsd_fixture: Callable[[str], str], +) -> None: + """With xsd_file set, xsd_schema and pyarrow_schema are populated for xml_tag.""" + xsd_file = copy_xsd_fixture(UNIREF_LIKE_XSD) + settings = settings_factory(xsd_file=xsd_file, xml_tag="entry", table_name="entry") + + assert isinstance(settings.xsd_schema, xmlschema.XMLSchema) + assert isinstance(settings.pyarrow_schema, pa.Schema) + expected_schema = generate_pyarrow_schema( + load_schema(xsd_file), + "entry", + max_recursion_depth=DEFAULT_MAX_RECURSION_DEPTH, + ) + assert settings.pyarrow_schema.equals(expected_schema, check_metadata=True) + + +def test_xmltopyarrow_settings_pass_custom_max_recursion_depth_controls_pyarrow_schema( + settings_factory: Callable[..., XmlToPyArrowSettings], + copy_xsd_fixture: Callable[[str], str], +) -> None: + """Custom max_recursion_depth is forwarded to generate_pyarrow_schema.""" + xsd_file = copy_xsd_fixture("self_referential.xsd") + settings_d1 = settings_factory(xsd_file=xsd_file, xml_tag="node", table_name="node", max_recursion_depth=1) + settings_d2 = settings_factory(xsd_file=xsd_file, xml_tag="node", table_name="node", max_recursion_depth=2) + + assert settings_d1.pyarrow_schema is not None + assert settings_d2.pyarrow_schema is not None + assert not settings_d1.pyarrow_schema.equals(settings_d2.pyarrow_schema) + + +def test_xmltopyarrow_settings_fail_non_positive_max_recursion_depth( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """max_recursion_depth must be a positive integer.""" + with pytest.raises(ValidationError): + settings_factory(max_recursion_depth=0) + + +def test_xmltopyarrow_settings_fail_xsd_file_not_found_raises_validation_error( + settings_factory: Callable[..., XmlToPyArrowSettings], +) -> None: + """An xsd_file that does not exist raises ValidationError at construction.""" + with pytest.raises(ValidationError, match="Path does not point to a file"): + settings_factory(xsd_file="does_not_exist.xsd") + + +def test_xmltopyarrow_settings_fail_xml_tag_not_in_xsd_raises_validation_error( + settings_factory: Callable[..., XmlToPyArrowSettings], + copy_xsd_fixture: Callable[[str], str], +) -> None: + """An xml_tag not present in xsd_file raises ValidationError at construction.""" + xsd_file = copy_xsd_fixture(UNIREF_LIKE_XSD) + with pytest.raises(ValidationError, match="Could not generate PyArrow schema from XSD"): + settings_factory(xsd_file=xsd_file, xml_tag="nonExistentTag") + + +@pytest.mark.parametrize("file_glob_arg", ["--file-glob", "-g"]) +def test_xmltopyarrow_settings_pass_xml_tag_shortcut( + tmp_path: Path, + library_xsd: Path, + file_glob_arg: str, +) -> None: + """The file_glob field is exposed as both --file-glob and -g on the command line.""" + log_config_file = tmp_path / "logging.json" + log_config_file.write_text('{"version": 1}') + input_dir = tmp_path / "input" + input_dir.mkdir() + output_dir = tmp_path / "output" + output_dir.mkdir() + + argv = [ + "--input-dir", + str(input_dir), + "--output-dir", + str(output_dir), + "--use-destination", + "local_fs", + "--log-config-file", + str(log_config_file), + file_glob_arg, + "*.xml.gz", + "--dataset-name", + "cli_dataset", + "--table-name", + "book", + "--xml-tag", + "book", + "--xsd-file", + str(library_xsd), + ] + + settings = CliApp.run(XmlToPyArrowSettings, cli_args=argv) + assert settings.file_glob == "*.xml.gz" + assert settings.xml_tag == "book" diff --git a/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xsd_to_arrow_end_to_end.py b/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xsd_to_arrow_end_to_end.py new file mode 100644 index 00000000..11bc8e59 --- /dev/null +++ b/tests/cdm_data_loaders/pipelines/xmltopyarrow/test_xsd_to_arrow_end_to_end.py @@ -0,0 +1,197 @@ +"""End-to-end specification tests: XSD + XML -> process_xml_file_to_pyarrow -> PyArrow Table batches.""" + +import datetime as dt +import json +from collections.abc import Callable +from pathlib import Path + +import pyarrow as pa +import pytest + +from cdm_data_loaders.pipelines.xmltopyarrow.settings import XmlToPyArrowSettings +from cdm_data_loaders.readers.xml import process_xml_file_to_pyarrow +from cdm_data_loaders.readers.xsd import generate_pyarrow_schema, load_schema + + +def _json_default(obj: object) -> str: + """Serialize Python date and bytes values from Arrow tables for comparison with expected JSON.""" + if isinstance(obj, dt.date): + return obj.isoformat() + if isinstance(obj, bytes): + return obj.decode("utf-8") + msg = f"Object of type {type(obj)} is not JSON serializable" + raise TypeError(msg) + + +def _strip_none(value: object) -> object: + """Recursively remove keys with None values from dicts to compare Arrow-materialized rows with sparse expected.json.""" + if isinstance(value, dict): + return {k: _strip_none(v) for k, v in value.items() if v is not None} + if isinstance(value, list): + return [_strip_none(item) for item in value] + return value + + +@pytest.mark.parametrize( + ("xsd_rel_path", "xml_rel_path", "expected_rel_path", "xml_tag"), + [ + pytest.param( + "xsd/spec/01_scalars_and_simple_types/schema.xsd", + "xsd/spec/01_scalars_and_simple_types/data.xml", + "xsd/spec/01_scalars_and_simple_types/expected.json", + "record", + id="spec-01-scalars-and-simple-types", + ), + pytest.param( + "xsd/spec/02_namespaces_and_nillable/schema.xsd", + "xsd/spec/02_namespaces_and_nillable/data.xml", + "xsd/spec/02_namespaces_and_nillable/expected.json", + "entry", + id="spec-02-namespaces-and-nillable", + ), + pytest.param( + "xsd/spec/03_mixed_content_raw_text/schema.xsd", + "xsd/spec/03_mixed_content_raw_text/data.xml", + "xsd/spec/03_mixed_content_raw_text/expected.json", + "article", + id="spec-03-mixed-content-raw-text", + ), + pytest.param( + "xsd/spec/04_recursion_depth/schema.xsd", + "xsd/spec/04_recursion_depth/data.xml", + "xsd/spec/04_recursion_depth/expected.json", + "record", + id="spec-04-recursion-depth", + ), + pytest.param( + "xsd/spec/05_groups_and_compositors/schema.xsd", + "xsd/spec/05_groups_and_compositors/data.xml", + "xsd/spec/05_groups_and_compositors/expected.json", + "item", + id="spec-05-groups-and-compositors", + ), + pytest.param( + "xsd/spec/06_context_dependent_paths/schema.xsd", + "xsd/spec/06_context_dependent_paths/data.xml", + "xsd/spec/06_context_dependent_paths/expected.json", + "root", + id="spec-06-context-dependent-paths", + ), + pytest.param( + "xsd/spec/07_multi_root_shared_type/schema.xsd", + "xsd/spec/07_multi_root_shared_type/data.xml", + "xsd/spec/07_multi_root_shared_type/expected.json", + "entry", + id="spec-07-multi-root-shared-type", + ), + pytest.param( + "xsd/spec/08_nullability_and_empty_tags/schema.xsd", + "xsd/spec/08_nullability_and_empty_tags/data.xml", + "xsd/spec/08_nullability_and_empty_tags/expected.json", + "entry", + id="spec-08-nullability-and-empty-tags", + ), + pytest.param( + "xsd/spec/09_empty_marker_elements/schema.xsd", + "xsd/spec/09_empty_marker_elements/data.xml", + "xsd/spec/09_empty_marker_elements/expected.json", + "book", + id="spec-09-empty-marker-elements", + ), + pytest.param( + "xsd/spec/10_combinations/schema.xsd", + "xsd/spec/10_combinations/data.xml", + "xsd/spec/10_combinations/expected.json", + "record", + id="spec-10-combinations", + ), + pytest.param( + "uniprot/uniref/uniref.xsd", + "xsd/examples/uniref/minimal.xml", + "xsd/examples/uniref/minimal_expected.json", + "entry", + id="example-uniref-minimal", + ), + pytest.param( + "uniprot/uniref/uniref.xsd", + "xsd/examples/uniref/maximal.xml", + "xsd/examples/uniref/maximal_expected.json", + "entry", + id="example-uniref-maximal", + ), + pytest.param( + "uniprot/uniprot_kb/uniprot.xsd", + "xsd/examples/uniprot/minimal.xml", + "xsd/examples/uniprot/minimal_expected.json", + "entry", + id="example-uniprot-minimal", + ), + pytest.param( + "uniprot/uniprot_kb/uniprot.xsd", + "xsd/examples/uniprot/maximal.xml", + "xsd/examples/uniprot/maximal_expected.json", + "entry", + id="example-uniprot-maximal", + ), + pytest.param( + "pubmed/pubmed.xsd", + "xsd/examples/pubmed_article/minimal.xml", + "xsd/examples/pubmed_article/minimal_expected.json", + "PubmedArticle", + id="example-pubmed-article-minimal", + ), + pytest.param( + "pubmed/pubmed.xsd", + "xsd/examples/pubmed_article/maximal.xml", + "xsd/examples/pubmed_article/maximal_expected.json", + "PubmedArticle", + id="example-pubmed-article-maximal", + ), + pytest.param( + "pubmed/pubmed.xsd", + "xsd/examples/pubmed_book_article/minimal.xml", + "xsd/examples/pubmed_book_article/minimal_expected.json", + "PubmedBookArticle", + id="example-pubmed-book-article-minimal", + ), + pytest.param( + "pubmed/pubmed.xsd", + "xsd/examples/pubmed_book_article/maximal.xml", + "xsd/examples/pubmed_book_article/maximal_expected.json", + "PubmedBookArticle", + id="example-pubmed-book-article-maximal", + ), + ], +) +def test_process_xml_file_to_pyarrow_pass_yields_arrow_tables_for_spec_and_examples( + settings_factory: Callable[..., XmlToPyArrowSettings], + test_data_dir: Path, + *, + xsd_rel_path: str, + xml_rel_path: str, + expected_rel_path: str, + xml_tag: str, +) -> None: + """process_xml_file_to_pyarrow yields pa.Table batches matching the XSD PyArrow schema and expected JSON.""" + xsd_path = test_data_dir / xsd_rel_path + xml_path = test_data_dir / xml_rel_path + expected = json.loads((test_data_dir / expected_rel_path).read_text(encoding="utf-8")) + + settings = settings_factory( + table_name=xml_tag, + xml_tag=xml_tag, + xsd_file=str(xsd_path), + ) + + items = list(process_xml_file_to_pyarrow(settings, file_path=xml_path)) + assert len(items) == 1 + assert items[0].meta.table_name == xml_tag + assert isinstance(items[0].data, pa.Table) + + table: pa.Table = items[0].data + expected_pa_schema = generate_pyarrow_schema(load_schema(xsd_path), xml_tag, max_recursion_depth=2) + assert table.schema.equals(expected_pa_schema, check_metadata=True) + assert table.num_rows == len(expected) + + actual_rows = json.loads(json.dumps(_strip_none(table.to_pylist()), default=_json_default)) + assert actual_rows == expected diff --git a/tests/cdm_data_loaders/readers/conftest.py b/tests/cdm_data_loaders/readers/conftest.py deleted file mode 100644 index 02a9d990..00000000 --- a/tests/cdm_data_loaders/readers/conftest.py +++ /dev/null @@ -1,17 +0,0 @@ -"""Shared fixtures and functions for cdm_data_loaders.readers.""" - -from collections.abc import Callable -from pathlib import Path - -import pytest -import xmlschema - - -@pytest.fixture -def load_schema(test_data_dir: Path) -> Callable[[str], xmlschema.XMLSchema]: - """Return a function that builds an XMLSchema from a file name in the data directory.""" - - def _load(filename: str) -> xmlschema.XMLSchema: - return xmlschema.XMLSchema(str(test_data_dir / "xsd" / filename)) - - return _load diff --git a/tests/cdm_data_loaders/readers/test_xsd.py b/tests/cdm_data_loaders/readers/test_xsd.py deleted file mode 100644 index 2fe2529d..00000000 --- a/tests/cdm_data_loaders/readers/test_xsd.py +++ /dev/null @@ -1,203 +0,0 @@ -"""Tests for the xsd reader module. - -Test XSDs live under tests/data/xsd and are loaded through the load_schema -fixture defined in conftest.py. -""" - -from collections.abc import Callable -from pathlib import Path - -import pytest -import xmlschema - -from cdm_data_loaders.readers.xsd import find_list_and_single_child_paths - - -def test_find_list_and_single_child_paths_pass_no_children( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A root element with a simple type has no children, so both result sets are empty.""" - schema = load_schema("no_children.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == set() - assert single_paths == set() - - -def test_find_list_and_single_child_paths_pass_empty_schema( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A schema with no global elements yields no paths at all.""" - schema = load_schema("empty_schema.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == set() - assert single_paths == set() - - -def test_find_list_and_single_child_paths_pass_attribute_only_type_no_children( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A complex type with only attributes and no child elements yields no paths.""" - schema = load_schema("attribute_only.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == set() - assert single_paths == set() - - -@pytest.mark.parametrize( - ("child_name", "expected_set_name"), - [ - pytest.param("requiredSingle", "single", id="required-single-occurs-once"), - pytest.param("optionalSingle", "single", id="optional-single-occurs-zero-or-one"), - pytest.param("optionalRepeated", "list", id="optional-repeated-occurs-zero-or-more"), - pytest.param("requiredRepeated", "list", id="required-repeated-occurs-one-or-more"), - pytest.param("boundedRepeated", "list", id="bounded-repeated-max-occurs-two"), - ], -) -def test_find_list_and_single_child_paths_pass_occurs_variety( - load_schema: Callable[[str], xmlschema.XMLSchema], - child_name: str, - expected_set_name: str, -) -> None: - """Children are sorted into list_paths or single_paths based on their maxOccurs.""" - schema = load_schema("occurs_variety.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - path = ("root", child_name) - if expected_set_name == "list": - assert path in list_paths - assert path not in single_paths - else: - assert path in single_paths - assert path not in list_paths - - -def test_find_list_and_single_child_paths_pass_occurs_variety_excludes_impossible_child( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A child with maxOccurs=0 can never appear, so it is excluded from both result sets.""" - schema = load_schema("occurs_variety.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - path = ("root", "neverOccurs") - assert path not in list_paths - assert path not in single_paths - - -def test_find_list_and_single_child_paths_pass_choice_group_repeated( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """Children of a repeatable choice group are list paths even with their own maxOccurs=1.""" - schema = load_schema("nested_choice.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == {("root", "optionA"), ("root", "optionB")} - assert single_paths == set() - - -def test_find_list_and_single_child_paths_pass_nested_sequence_flattened( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A child nested two sequence levels deep is still reported against its direct parent element.""" - schema = load_schema("nested_sequence_deep.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == {("root", "deepChild")} - assert single_paths == set() - - -def test_find_list_and_single_child_paths_pass_ref_element_resolved( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A child declared with ref= resolves to the referenced global element's local name.""" - schema = load_schema("ref_element.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert ("root", "shared") in list_paths - assert ("root", "shared") not in single_paths - - -def test_find_list_and_single_child_paths_pass_self_referential_type_no_infinite_recursion( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """A type that nests itself as its own child is recorded once and does not recurse forever.""" - schema = load_schema("self_referential.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == {("node", "child")} - assert single_paths == set() - - -def test_find_list_and_single_child_paths_pass_mutual_recursion_no_infinite_recursion( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """Two types that reference each other are both traversed once without infinite recursion.""" - schema = load_schema("mutual_recursion.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == {("a", "b")} - assert single_paths == {("b", "a")} - - -def test_find_list_and_single_child_paths_pass_uniref_like_schema( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """Test the function on a more realistic schema.""" - schema = load_schema("uniref_like.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths == { - ("UniRef50", "entry"), - ("UniRef90", "entry"), - ("UniRef100", "entry"), - ("entry", "property"), - ("entry", "member"), - ("dbReference", "property"), - } - assert single_paths == { - ("entry", "representativeMember"), - ("member", "dbReference"), - ("member", "sequence"), - ("representativeMember", "dbReference"), - ("representativeMember", "sequence"), - } - - -def test_find_list_and_single_child_paths_pass_returns_disjoint_sets( - load_schema: Callable[[str], xmlschema.XMLSchema], -) -> None: - """No path can be classified as both a list path and a single path at the same time.""" - schema = load_schema("uniref_like.xsd") - - list_paths, single_paths = find_list_and_single_child_paths(schema) - - assert list_paths.isdisjoint(single_paths) - - -def test_find_list_and_single_child_paths_fail_invalid_schema_type() -> None: - """Passing an object without a schema-like elements mapping raises AttributeError.""" - not_a_schema = "this is a string, not an XMLSchema" - - with pytest.raises(AttributeError): - find_list_and_single_child_paths(not_a_schema) # type: ignore[arg-type] - - -def test_find_list_and_single_child_paths_fail_nonexistent_file_path(test_data_dir: Path) -> None: - """Loading a schema from a path that does not exist raises an OSError from xmlschema.""" - missing_path = test_data_dir / "does_not_exist.xsd" - - with pytest.raises(OSError): - xmlschema.XMLSchema(str(missing_path)) diff --git a/tests/cdm_data_loaders/readers/test_xsd_spec.py b/tests/cdm_data_loaders/readers/test_xsd_spec.py new file mode 100644 index 00000000..ca4358b8 --- /dev/null +++ b/tests/cdm_data_loaders/readers/test_xsd_spec.py @@ -0,0 +1,536 @@ +"""Specification tests for XSD-to-PyArrow schema generation and deterministic XML-to-Arrow conversion.""" + +import copy +import datetime as dt +import importlib.util +import io +import json +from collections import defaultdict +from pathlib import Path +from typing import Any + +import pyarrow as pa +import pyarrow.parquet as pq +import pytest +import xmlschema +from _pytest.mark.structures import ParameterSet +from lxml import etree + +from cdm_data_loaders.readers.xsd import ( + convert_xml_element, + generate_pyarrow_schema, + load_schema, +) + + +def _list_field(name: str, item_type: pa.DataType, *, nullable: bool = True) -> pa.Field: + """Build a PyArrow list field with standard nullable item field.""" + return pa.field(name, pa.list_(pa.field("item", item_type, nullable=True)), nullable=nullable) + + +def _load_expected_schema(schema_file: Path, attr_name: str = "EXPECTED_SCHEMA") -> pa.Schema: + """Load a pa.Schema constant from an expected_schema.py fixture file.""" + spec = importlib.util.spec_from_file_location(schema_file.stem, schema_file) + if spec is None or spec.loader is None: + msg = f"Could not load schema module from {schema_file}" + raise RuntimeError(msg) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + schema = getattr(module, attr_name) + if not isinstance(schema, pa.Schema): + msg = f"Expected {attr_name} in {schema_file} to be a pa.Schema, got {type(schema)}" + raise TypeError(msg) + return schema + + +def _coerce_json_rows_for_arrow(value: Any, dtype: pa.DataType) -> Any: # noqa: ANN401 + """Coerce ISO date strings and binary strings from expected JSON into Python objects for pa.Table.from_pylist.""" + if value is None: + return None + if pa.types.is_date32(dtype) and isinstance(value, str): + return dt.date.fromisoformat(value) + if pa.types.is_binary(dtype) and isinstance(value, str): + return value.encode("utf-8") + if pa.types.is_list(dtype) and isinstance(value, list): + return [_coerce_json_rows_for_arrow(item, dtype.value_type) for item in value] + if pa.types.is_struct(dtype) and isinstance(value, dict): + return { + field.name: _coerce_json_rows_for_arrow(value[field.name], field.type) + for field in dtype + if field.name in value + } + return value + + +def _coerce_rows_to_schema(rows: list[dict[str, Any]], schema: pa.Schema) -> list[dict[str, Any]]: + """Coerce JSON-loaded rows using field types in schema so pa.Table.from_pylist can materialize them.""" + struct_type = pa.struct(list(schema)) + return [_coerce_json_rows_for_arrow(row, struct_type) for row in rows] + + +def _json_default(obj: object) -> str: + """Serialize Python date and bytes values from Arrow tables for comparison with expected JSON.""" + if isinstance(obj, dt.date): + return obj.isoformat() + if isinstance(obj, bytes): + return obj.decode("utf-8") + msg = f"Object of type {type(obj)} is not JSON serializable" + raise TypeError(msg) + + +def _strip_none(value: object) -> object: + """Recursively remove keys with None values from dicts to compare Arrow-materialized rows with sparse expected.json.""" + if isinstance(value, dict): + return {k: _strip_none(v) for k, v in value.items() if v is not None} + if isinstance(value, list): + return [_strip_none(item) for item in value] + return value + + +def _extract_matching_elements(xml_path: Path, xml_tag: str) -> list[etree._Element]: + """Parse xml_path and return all elements matching xml_tag at the root or as a direct child of root.""" + local_tag = xml_tag.rsplit("}", maxsplit=1)[-1] if "}" in xml_tag else xml_tag + tree = etree.parse(str(xml_path)) + root = tree.getroot() + root_local = root.tag.rsplit("}", maxsplit=1)[-1] if "}" in root.tag else root.tag + if root_local == local_tag: + return [root] + return [ + child + for child in root + if isinstance(child.tag, str) + and (child.tag.rsplit("}", maxsplit=1)[-1] if "}" in child.tag else child.tag) == local_tag + ] + + +SPEC_CASES: list[ParameterSet] = [ + pytest.param("01_scalars_and_simple_types", "record", id="01-scalars-and-simple-types"), + pytest.param("02_namespaces_and_nillable", "entry", id="02-namespaces-and-nillable"), + pytest.param("03_mixed_content_raw_text", "article", id="03-mixed-content-raw-text"), + pytest.param("04_recursion_depth", "record", id="04-recursion-depth"), + pytest.param("05_groups_and_compositors", "item", id="05-groups-and-compositors"), + pytest.param("06_context_dependent_paths", "root", id="06-context-dependent-paths"), + pytest.param("07_multi_root_shared_type", "entry", id="07-multi-root-shared-type"), + pytest.param("08_nullability_and_empty_tags", "entry", id="08-nullability-and-empty-tags"), + pytest.param("09_empty_marker_elements", "book", id="09-empty-marker-elements"), + pytest.param("10_combinations", "record", id="10-combinations"), +] + +REAL_WORLD_SCHEMAS: list[ParameterSet] = [ + pytest.param("uniref", "uniprot/uniref/uniref.xsd", "entry", id="uniref"), + pytest.param("uniprot", "uniprot/uniprot_kb/uniprot.xsd", "entry", id="uniprot"), + pytest.param("pubmed_article", "pubmed/pubmed.xsd", "PubmedArticle", id="pubmed-article"), + pytest.param("pubmed_book_article", "pubmed/pubmed.xsd", "PubmedBookArticle", id="pubmed-book-article"), +] + +REAL_WORLD_EXAMPLES: list[ParameterSet] = [ + pytest.param("uniref", "uniprot/uniref/uniref.xsd", "entry", "minimal", id="uniref-minimal"), + pytest.param("uniref", "uniprot/uniref/uniref.xsd", "entry", "maximal", id="uniref-maximal"), + pytest.param("uniprot", "uniprot/uniprot_kb/uniprot.xsd", "entry", "minimal", id="uniprot-minimal"), + pytest.param("uniprot", "uniprot/uniprot_kb/uniprot.xsd", "entry", "maximal", id="uniprot-maximal"), + pytest.param("pubmed_article", "pubmed/pubmed.xsd", "PubmedArticle", "minimal", id="pubmed-article-minimal"), + pytest.param("pubmed_article", "pubmed/pubmed.xsd", "PubmedArticle", "maximal", id="pubmed-article-maximal"), + pytest.param( + "pubmed_book_article", + "pubmed/pubmed.xsd", + "PubmedBookArticle", + "minimal", + id="pubmed-book-article-minimal", + ), + pytest.param( + "pubmed_book_article", + "pubmed/pubmed.xsd", + "PubmedBookArticle", + "maximal", + id="pubmed-book-article-maximal", + ), +] + + +@pytest.mark.parametrize(("case_dir", "xml_tag"), SPEC_CASES) +def test_spec_fixtures_pass_xml_and_expected_schema_validate( + test_data_dir: Path, + case_dir: str, + xml_tag: str, +) -> None: + """Every specification data.xml validates against schema.xsd and expected.json materializes with expected_schema.py.""" + fixture_dir = test_data_dir / "xsd" / "spec" / case_dir + schema = load_schema(fixture_dir / "schema.xsd") + schema.validate(str(fixture_dir / "data.xml")) + + elements = _extract_matching_elements(fixture_dir / "data.xml", xml_tag) + expected = json.loads((fixture_dir / "expected.json").read_text(encoding="utf-8")) + assert len(elements) == len(expected) + + expected_pa_schema = _load_expected_schema(fixture_dir / "expected_schema.py") + coerced_rows = _coerce_rows_to_schema(expected, expected_pa_schema) + table = pa.Table.from_pylist(coerced_rows, schema=expected_pa_schema) + table.validate(full=True) + assert table.num_rows == len(expected) + + dumped_rows = json.loads(json.dumps(_strip_none(table.to_pylist()), default=_json_default)) + assert dumped_rows == expected + + buf = io.BytesIO() + pq.write_table(table, buf) + buf.seek(0) + roundtrip_table = pq.read_table(buf) + roundtrip_rows = json.loads(json.dumps(_strip_none(roundtrip_table.to_pylist()), default=_json_default)) + assert roundtrip_rows == expected + + +@pytest.mark.parametrize( + ("example_dir", "xsd_rel_path", "xml_tag", "variant"), + REAL_WORLD_EXAMPLES, +) +def test_real_world_examples_pass_xml_and_expected_schema_validate( + test_data_dir: Path, + example_dir: str, + xsd_rel_path: str, + xml_tag: str, + variant: str, +) -> None: + """Every minimal and maximal real-world XML validates against its XSD and materializes with expected_schema.py.""" + schema = load_schema(test_data_dir / xsd_rel_path) + example_base = test_data_dir / "xsd" / "examples" / example_dir + xml_path = example_base / f"{variant}.xml" + expected_path = example_base / f"{variant}_expected.json" + + schema.validate(str(xml_path)) + elements = _extract_matching_elements(xml_path, xml_tag) + expected = json.loads(expected_path.read_text(encoding="utf-8")) + assert len(elements) == 1 + assert len(expected) == 1 + + expected_pa_schema = _load_expected_schema(example_base / "expected_schema.py") + coerced_rows = _coerce_rows_to_schema(expected, expected_pa_schema) + table = pa.Table.from_pylist(coerced_rows, schema=expected_pa_schema) + table.validate(full=True) + assert table.num_rows == 1 + + dumped_rows = json.loads(json.dumps(_strip_none(table.to_pylist()), default=_json_default)) + assert dumped_rows == expected + + buf = io.BytesIO() + pq.write_table(table, buf) + buf.seek(0) + roundtrip_table = pq.read_table(buf) + roundtrip_rows = json.loads(json.dumps(_strip_none(roundtrip_table.to_pylist()), default=_json_default)) + assert roundtrip_rows == expected + + +@pytest.mark.parametrize( + "example_dir", + [ + pytest.param("uniref", id="uniref-maximal"), + pytest.param("uniprot", id="uniprot-maximal"), + pytest.param("pubmed_article", id="pubmed-article-maximal"), + pytest.param("pubmed_book_article", id="pubmed-book-article-maximal"), + ], +) +def test_real_world_maximal_examples_pass_repeated_elements_have_distinct_content( + test_data_dir: Path, + example_dir: str, +) -> None: + """Every repeated element in a maximal XML example contains distinct data.""" + xml_path = test_data_dir / "xsd" / "examples" / example_dir / "maximal.xml" + root = etree.parse(str(xml_path)).getroot() + + by_tag: dict[str, list[str]] = defaultdict(list) + for elem in root.iter(): + if elem is root or not isinstance(elem.tag, str): + continue + clone = copy.deepcopy(elem) + for node in clone.iter(): + if node.text is not None: + node.text = node.text.strip() + if node.tail is not None: + node.tail = node.tail.strip() + local = elem.tag.rsplit("}", maxsplit=1)[-1] if "}" in elem.tag else elem.tag + by_tag[local].append(etree.tostring(clone, encoding="unicode")) + + duplicates = {tag: sigs for tag, sigs in by_tag.items() if len(sigs) > 1 and len(set(sigs)) < len(sigs)} + assert not duplicates + + +@pytest.mark.parametrize(("case_dir", "xml_tag"), SPEC_CASES) +def test_generate_pyarrow_schema_pass_spec_cases( + test_data_dir: Path, + case_dir: str, + xml_tag: str, +) -> None: + """Verify XSD specification schemas generate the exact expected PyArrow schema in expected_schema.py.""" + fixture_dir = test_data_dir / "xsd" / "spec" / case_dir + schema = load_schema(fixture_dir / "schema.xsd") + pa_schema = generate_pyarrow_schema(schema, xml_tag, max_recursion_depth=2) + expected_schema = _load_expected_schema(fixture_dir / "expected_schema.py") + assert pa_schema.equals(expected_schema, check_metadata=True) + + +def test_generate_pyarrow_schema_pass_clark_notation_tag(test_data_dir: Path) -> None: + """Verify Clark-notation {namespace}local_name resolves identically to local_name.""" + fixture_dir = test_data_dir / "xsd" / "spec" / "02_namespaces_and_nillable" + schema = load_schema(fixture_dir / "schema.xsd") + pa_clark = generate_pyarrow_schema(schema, "{http://example.org/ns}entry") + expected_schema = _load_expected_schema(fixture_dir / "expected_schema.py") + assert pa_clark.equals(expected_schema, check_metadata=True) + + +def test_generate_pyarrow_schema_pass_04_recursion_depth_1(test_data_dir: Path) -> None: + """Verify structural recursion with max_recursion_depth=1 matches EXPECTED_SCHEMA_DEPTH_1.""" + fixture_dir = test_data_dir / "xsd" / "spec" / "04_recursion_depth" + schema = load_schema(fixture_dir / "schema.xsd") + schema_d1 = generate_pyarrow_schema(schema, "record", max_recursion_depth=1) + expected_d1 = _load_expected_schema(fixture_dir / "expected_schema.py", "EXPECTED_SCHEMA_DEPTH_1") + assert schema_d1.equals(expected_d1, check_metadata=True) + + +def test_generate_pyarrow_schema_pass_mutual_and_self_referential_xsds(test_data_dir: Path) -> None: + """Verify existing self_referential.xsd and mutual_recursion.xsd schemas terminate at max_recursion_depth.""" + self_ref_schema = load_schema(test_data_dir / "xsd" / "self_referential.xsd") + pa_self = generate_pyarrow_schema(self_ref_schema, "node", max_recursion_depth=2) + expected_self = pa.schema( + [ + _list_field( + "child", + pa.struct( + [ + _list_field("child", pa.string(), nullable=True), + ] + ), + nullable=True, + ), + ] + ) + assert pa_self.equals(expected_self, check_metadata=True) + + mutual_schema = load_schema(test_data_dir / "xsd" / "mutual_recursion.xsd") + pa_mutual = generate_pyarrow_schema(mutual_schema, "a", max_recursion_depth=1) + expected_mutual = pa.schema( + [ + _list_field( + "b", + pa.struct( + [ + pa.field("a", pa.string(), nullable=True), + ] + ), + nullable=True, + ), + ] + ) + assert pa_mutual.equals(expected_mutual, check_metadata=True) + + +@pytest.mark.parametrize( + ("example_dir", "xsd_rel_path", "xml_tag"), + REAL_WORLD_SCHEMAS, +) +def test_generate_pyarrow_schema_pass_real_world_schemas( + test_data_dir: Path, + example_dir: str, + xsd_rel_path: str, + xml_tag: str, +) -> None: + """Verify PyArrow schema generation on UniRef, UniProtKB, and PubMed XSDs matches expected_schema.py.""" + schema = load_schema(test_data_dir / xsd_rel_path) + pa_schema = generate_pyarrow_schema(schema, xml_tag, max_recursion_depth=2) + expected_schema = _load_expected_schema(test_data_dir / "xsd" / "examples" / example_dir / "expected_schema.py") + assert pa_schema.equals(expected_schema, check_metadata=True) + + +@pytest.mark.parametrize( + ("error_xsd_filename", "xml_tag", "expected_match"), + [ + pytest.param("ambiguous_xml_tag.xsd", "entry", "Ambiguous", id="E2-ambiguous-xml-tag"), + pytest.param("scalar_top_level.xsd", "leaf", "complex type", id="E3-scalar-top-level"), + pytest.param("empty_marker_top_level.xsd", "marker", "complex type", id="E3-empty-marker-top-level"), + pytest.param("wildcard_any_element.xsd", "entry", "xs:any", id="E4-wildcard-any-element"), + pytest.param("wildcard_any_attribute.xsd", "entry", "xs:anyAttribute", id="E5-wildcard-any-attribute"), + pytest.param("abstract_element.xsd", "entry", "abstract", id="E6-abstract-element"), + pytest.param("abstract_type.xsd", "entry", "abstract", id="E6-abstract-type"), + pytest.param("substitution_group.xsd", "entry", "substitutionGroup", id="E7-substitution-group"), + pytest.param("attr_child_name_collision.xsd", "entry", "collision", id="E8-attr-child-name-collision"), + ], +) +def test_generate_pyarrow_schema_fail_unsupported_or_invalid_xsd( + test_data_dir: Path, + error_xsd_filename: str, + xml_tag: str, + expected_match: str, +) -> None: + """Verify unsupported or ambiguous XSD features raise a descriptive ValueError.""" + schema = load_schema(test_data_dir / "xsd" / "spec" / "errors" / error_xsd_filename) + with pytest.raises(ValueError, match=expected_match): + generate_pyarrow_schema(schema, xml_tag) + + +@pytest.mark.parametrize( + ("xsd_filename", "xml_tag", "expected_match"), + [ + pytest.param("empty_schema.xsd", "root", "not found", id="E1-empty-schema"), + pytest.param("no_children.xsd", "leaf", "complex type", id="E3-no-children-scalar-root"), + pytest.param("cdata_variety.xsd", "root", "xs:anyAttribute", id="E5-cdata-variety-has-anyAttribute"), + pytest.param("combinations.xsd", "record", "xs:anyAttribute", id="E5-legacy-combinations-has-anyAttribute"), + ], +) +def test_generate_pyarrow_schema_fail_existing_invalid_schemas( + test_data_dir: Path, + xsd_filename: str, + xml_tag: str, + expected_match: str, +) -> None: + """Verify existing XSDs with scalar roots, empty schemas, or xs:anyAttribute raise ValueError.""" + schema = load_schema(test_data_dir / "xsd" / xsd_filename) + with pytest.raises(ValueError, match=expected_match): + generate_pyarrow_schema(schema, xml_tag) + + +def test_generate_pyarrow_schema_fail_nonexistent_xml_tag(test_data_dir: Path) -> None: + """Verify requesting an xml_tag that does not exist in the XSD raises ValueError.""" + schema = load_schema(test_data_dir / "uniprot" / "uniref" / "uniref.xsd") + with pytest.raises(ValueError, match="not found"): + generate_pyarrow_schema(schema, "nonExistentElement") + + +@pytest.mark.parametrize( + "forbidden_xsd_type", + [ + pytest.param("xs:date", id="xs-date"), + pytest.param("xs:dateTime", id="xs-dateTime"), + pytest.param("xs:hexBinary", id="xs-hexBinary"), + pytest.param("xs:base64Binary", id="xs-base64Binary"), + ], +) +def test_generate_pyarrow_schema_fail_non_json_serializable_type_under_recursion( + forbidden_xsd_type: str, +) -> None: + """Verify xs:date, xs:dateTime, xs:hexBinary, and xs:base64Binary under a recursive element raise ValueError.""" + xsd_with_child = f""" + + + + + + + + + + """ + schema_child = xmlschema.XMLSchema(xsd_with_child) + with pytest.raises(ValueError, match="recursive type"): + generate_pyarrow_schema(schema_child, "root") + + xsd_with_attr = f""" + + + + + + + + + + """ + schema_attr = xmlschema.XMLSchema(xsd_with_attr) + with pytest.raises(ValueError, match="recursive type"): + generate_pyarrow_schema(schema_attr, "root") + + +@pytest.mark.parametrize(("case_dir", "xml_tag"), SPEC_CASES) +def test_convert_xml_element_and_arrow_table_pass_spec_fixtures( + test_data_dir: Path, + case_dir: str, + xml_tag: str, +) -> None: + """Verify XML elements convert to expected JSON and build a valid PyArrow Table and Parquet file.""" + fixture_dir = test_data_dir / "xsd" / "spec" / case_dir + schema = load_schema(fixture_dir / "schema.xsd") + pa_schema = generate_pyarrow_schema(schema, xml_tag, max_recursion_depth=2) + + elements = _extract_matching_elements(fixture_dir / "data.xml", xml_tag) + rows = [convert_xml_element(elem, schema, xml_tag, max_recursion_depth=2) for elem in elements] + + expected = json.loads((fixture_dir / "expected.json").read_text(encoding="utf-8")) + serialized_rows = json.loads(json.dumps(rows, default=_json_default)) + assert serialized_rows == expected + + table = pa.Table.from_pylist(rows, schema=pa_schema) + table.validate(full=True) + assert table.num_rows == len(expected) + + dumped_rows = json.loads(json.dumps(_strip_none(table.to_pylist()), default=_json_default)) + assert dumped_rows == expected + + buf = io.BytesIO() + pq.write_table(table, buf) + buf.seek(0) + roundtrip_table = pq.read_table(buf) + roundtrip_rows = json.loads(json.dumps(_strip_none(roundtrip_table.to_pylist()), default=_json_default)) + assert roundtrip_rows == expected + + +def test_convert_xml_element_and_arrow_table_pass_recursion_depth_1(test_data_dir: Path) -> None: + """Verify max_recursion_depth=1 truncates recursive structures one level earlier than depth=2.""" + fixture_dir = test_data_dir / "xsd" / "spec" / "04_recursion_depth" + schema = load_schema(fixture_dir / "schema.xsd") + pa_schema = generate_pyarrow_schema(schema, "record", max_recursion_depth=1) + + elements = _extract_matching_elements(fixture_dir / "data.xml", "record") + rows = [convert_xml_element(elem, schema, "record", max_recursion_depth=1) for elem in elements] + + expected = json.loads((fixture_dir / "expected_depth_1.json").read_text(encoding="utf-8")) + serialized_rows = json.loads(json.dumps(rows, default=_json_default)) + assert serialized_rows == expected + + table = pa.Table.from_pylist(rows, schema=pa_schema) + table.validate(full=True) + dumped_rows = json.loads(json.dumps(_strip_none(table.to_pylist()), default=_json_default)) + assert dumped_rows == expected + + buf = io.BytesIO() + pq.write_table(table, buf) + buf.seek(0) + roundtrip_table = pq.read_table(buf) + roundtrip_rows = json.loads(json.dumps(_strip_none(roundtrip_table.to_pylist()), default=_json_default)) + assert roundtrip_rows == expected + + +@pytest.mark.parametrize( + ("example_dir", "xsd_rel_path", "xml_tag", "variant"), + REAL_WORLD_EXAMPLES, +) +def test_convert_xml_element_and_arrow_table_pass_real_world_examples( + test_data_dir: Path, + example_dir: str, + xsd_rel_path: str, + xml_tag: str, + variant: str, +) -> None: + """Verify minimal and maximal UniRef, UniProt, and PubMed entries convert to expected JSON and Arrow tables.""" + schema = load_schema(test_data_dir / xsd_rel_path) + xml_path = test_data_dir / "xsd" / "examples" / example_dir / f"{variant}.xml" + expected_path = test_data_dir / "xsd" / "examples" / example_dir / f"{variant}_expected.json" + + pa_schema = generate_pyarrow_schema(schema, xml_tag, max_recursion_depth=2) + elements = _extract_matching_elements(xml_path, xml_tag) + rows = [convert_xml_element(elem, schema, xml_tag, max_recursion_depth=2) for elem in elements] + + expected = json.loads(expected_path.read_text(encoding="utf-8")) + serialized_rows = json.loads(json.dumps(rows, default=_json_default)) + assert serialized_rows == expected + + table = pa.Table.from_pylist(rows, schema=pa_schema) + table.validate(full=True) + assert table.num_rows == 1 + + dumped_rows = json.loads(json.dumps(_strip_none(table.to_pylist()), default=_json_default)) + assert dumped_rows == expected + + buf = io.BytesIO() + pq.write_table(table, buf) + buf.seek(0) + roundtrip_table = pq.read_table(buf) + assert roundtrip_table.num_rows == 1 + roundtrip_rows = json.loads(json.dumps(_strip_none(roundtrip_table.to_pylist()), default=_json_default)) + assert roundtrip_rows == expected diff --git a/tests/cdm_data_loaders/readers/xml/test_xml_process_xml_file_and_batches.py b/tests/cdm_data_loaders/readers/xml/test_xml_process_xml_file_and_batches.py index 8659678f..e7caac24 100644 --- a/tests/cdm_data_loaders/readers/xml/test_xml_process_xml_file_and_batches.py +++ b/tests/cdm_data_loaders/readers/xml/test_xml_process_xml_file_and_batches.py @@ -1,4 +1,4 @@ -"""Tests for `process_xml_file`, `process_xml_file_items`, `build_xml_file_resource`, and `process_xml_file_to_dict` from cdm_data_loaders.readers.xml.""" +"""Tests for `process_xml_file`, `process_xml_file_items`, `build_xml_file_resource`, `process_xml_file_to_dict`, and `process_xml_file_to_pyarrow` from cdm_data_loaders.readers.xml.""" import gzip import logging @@ -7,7 +7,9 @@ from typing import Any, Final from unittest.mock import MagicMock +import pyarrow as pa import pytest +import xmlschema import xmltodict from dlt.common.storages.fsspec_filesystem import FileItemDict from dlt.extract.items import DataItemWithMeta, TableNameMeta @@ -19,13 +21,16 @@ from cdm_data_loaders.core.fields import BUFFER_SIZE, DEFAULT_XML_FILE_GLOB, DEFAULTS, LOG_INTERVAL from cdm_data_loaders.core.settings import CtsSettings from cdm_data_loaders.pipelines.xmltodict.settings import XmlToDictSettings +from cdm_data_loaders.pipelines.xmltopyarrow.settings import XmlToPyArrowSettings from cdm_data_loaders.readers.xml import ( DEFAULT_XMLTODICT_ARGS, process_xml_file, process_xml_file_batches, process_xml_file_items, process_xml_file_to_dict, + process_xml_file_to_pyarrow, ) +from cdm_data_loaders.readers.xsd import DEFAULT_MAX_RECURSION_DEPTH, generate_pyarrow_schema ParseFn = Callable[..., dict[str, list[dict[str, Any]]]] @@ -84,6 +89,28 @@ {"person": {"_id": p["id"], "name": p["name"], "email": p["email"]}} for p in PEOPLE_PARSED ] +PEOPLE_XML_TO_PYARROW_PARSED = [{"_id": p["id"], "name": p["name"], "email": p["email"]} for p in PEOPLE_PARSED] + +PEOPLE_XSD: Final[str] = """ + + + + + + + + + + + + + + + + + +""" + PEOPLE_XML_EMPTY: Final[str] = """ @@ -248,6 +275,32 @@ def fake_xmltodict_settings( return settings +def fake_xmltopyarrow_settings( + *, + table_name: str = "people", + xml_tag: str = "person", + xsd_text: str = PEOPLE_XSD, + buffer_size: int = DEFAULTS[BUFFER_SIZE], + log_interval: int = DEFAULTS[LOG_INTERVAL], + max_recursion_depth: int = DEFAULT_MAX_RECURSION_DEPTH, +) -> XmlToPyArrowSettings: + """Build a MagicMock XmlToPyArrowSettings with xsd_schema/pyarrow_schema set, for process_xml_file_to_pyarrow tests.""" + settings: XmlToPyArrowSettings = fake_settings( # pyright: ignore[reportAssignmentType] + buffer_size=buffer_size, log_interval=log_interval, settings_class=XmlToPyArrowSettings + ) + settings.table_name = table_name + settings.xml_tag = xml_tag + settings.max_recursion_depth = max_recursion_depth + schema = xmlschema.XMLSchema(xsd_text) + settings.xsd_schema = schema # pyright: ignore[reportAttributeAccessIssue] + settings.pyarrow_schema = generate_pyarrow_schema( # pyright: ignore[reportAttributeAccessIssue] + schema, + xml_tag, + max_recursion_depth=max_recursion_depth, + ) + return settings + + def test_process_xml_file_pass_single_table_single_entry(tmp_path: Path) -> None: """Verify a single matching element is parsed into one correctly table-tagged row.""" file_path = _write_xml(tmp_path, "one.xml", PEOPLE_XML_2) @@ -660,33 +713,44 @@ def test_process_xml_file_to_dict_fail_malformed_xml(tmp_path: Path) -> None: """ -def _build_real_xmltodict_settings(tmp_path: Path, test_data_dir: Path, xsd_filename: str | None) -> XmlToDictSettings: - """Build a real (non-mock) XmlToDictSettings, optionally with an xsd_file copied from tests/data/xsd. +def _build_real_xmltodict_settings(tmp_path: Path) -> XmlToDictSettings: + """Build a real (non-mock) XmlToDictSettings.""" + input_dir = tmp_path / "input" + input_dir.mkdir(exist_ok=True) + log_config_file = tmp_path / "logging.json" + log_config_file.write_text('{"version": 1}', encoding="utf-8") - A real settings object is required here, rather than fake_xmltodict_settings, because - xmltodict_args is a computed property with real schema-derived logic to exercise: a mock - would either need that same logic re-implemented by hand (proving nothing about the actual - implementation) or would unpack to `{}` by default, silently skipping the feature entirely. - """ + return XmlToDictSettings( + input_dir=str(input_dir), + log_config_file=str(log_config_file), + dataset_name="xsd_test_dataset", + table_name="entry", + xml_tag="entry", + buffer_size=100, + log_interval=1000, + output_dir=tmp_path, + use_destination="whatever", + ) # pyright: ignore[reportCallIssue] + + +def _build_real_xmltopyarrow_settings(tmp_path: Path, test_data_dir: Path, xsd_filename: str) -> XmlToPyArrowSettings: + """Build a real (non-mock) XmlToPyArrowSettings with an xsd_file copied from tests/data/xsd.""" input_dir = tmp_path / "input" input_dir.mkdir(exist_ok=True) log_config_file = tmp_path / "logging.json" log_config_file.write_text('{"version": 1}', encoding="utf-8") - xsd_file = None - if xsd_filename is not None: - xsd_content = (test_data_dir / "xsd" / xsd_filename).read_text(encoding="utf-8") - xsd_path = input_dir / xsd_filename - xsd_path.write_text(xsd_content, encoding="utf-8") - xsd_file = str(xsd_path) + xsd_content = (test_data_dir / "xsd" / xsd_filename).read_text(encoding="utf-8") + xsd_path = input_dir / xsd_filename + xsd_path.write_text(xsd_content, encoding="utf-8") - return XmlToDictSettings( + return XmlToPyArrowSettings( input_dir=str(input_dir), log_config_file=str(log_config_file), dataset_name="xsd_test_dataset", table_name="entry", xml_tag="entry", - xsd_file=xsd_file, + xsd_file=str(xsd_path), buffer_size=100, log_interval=1000, output_dir=tmp_path, @@ -694,51 +758,229 @@ def _build_real_xmltodict_settings(tmp_path: Path, test_data_dir: Path, xsd_file ) # pyright: ignore[reportCallIssue] +def test_process_xml_file_to_dict_pass_keeps_xmltodict_default_dicts( + tmp_path: Path, +) -> None: + """process_xml_file_to_dict uses xmltodict's ordinary single-occurrence dicts.""" + file_path = _write_xml(tmp_path, "entry.xml", UNIREF_LIKE_ENTRY_XML) + settings = _build_real_xmltodict_settings(tmp_path) + + items = _table_and_data(process_xml_file_to_dict(settings, file_path=file_path)) + + assert items == [ + ( + "entry", + [ + { + "entry": { + "property": {"_type": "member count", "_value": "1"}, + "representativeMember": { + "dbReference": {"property": {"_type": "isSeed", "_value": "true"}}, + }, + } + } + ], + ) + ] + + +"""process_xml_file_to_pyarrow""" + + +def test_process_xml_file_to_pyarrow_pass_custom_table_name_and_xml_tag_are_respected(tmp_path: Path) -> None: + """Verify rows land under settings.table_name as a PyArrow Table, matched by settings.xml_tag.""" + file_path = _write_xml(tmp_path, "two.xml", PEOPLE_XML_2) + settings = fake_xmltopyarrow_settings(table_name="library_entries", xml_tag="person") + + tagged = _table_and_data(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + assert [table_name for table_name, _ in tagged] == ["library_entries"] + assert isinstance(tagged[0][1], pa.Table) + assert tagged[0][1].to_pylist() == PEOPLE_XML_TO_PYARROW_PARSED[:2] + + +def test_process_xml_file_to_pyarrow_pass_resolves_default_namespace_on_unprefixed_tag(tmp_path: Path) -> None: + """Verify an unprefixed xml_tag resolves the root default namespace via parse_head_matter.""" + namespace = "http://example.com/people" + namespaced_xsd = f""" + + + + + + + + + + + + + + + + + +""" + namespaced_xml = f""" + + Anne Examplea@example.com + Belinda Carlisleb@example.com + +""" + file_path = _write_xml(tmp_path, "namespaced.xml", namespaced_xml) + settings = fake_xmltopyarrow_settings(table_name="people", xml_tag="person", xsd_text=namespaced_xsd) + + tagged = _table_and_data(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + assert [table_name for table_name, _ in tagged] == ["people"] + assert tagged[0][1].to_pylist() == PEOPLE_XML_TO_PYARROW_PARSED[:2] + + @pytest.mark.parametrize( - ("xsd_filename", "expected_entry"), + ("buffer_size", "expected_batch_sizes"), [ - pytest.param( - "uniref_like.xsd", - { - "property": [{"_type": "member count", "_value": "1"}], - "representativeMember": { - "dbReference": {"property": [{"_type": "isSeed", "_value": "true"}]}, - }, - }, - id="with-xsd-file-forces-repeatable-schema-children-into-lists", - ), - pytest.param( - None, - { - "property": {"_type": "member count", "_value": "1"}, - "representativeMember": { - "dbReference": {"property": {"_type": "isSeed", "_value": "true"}}, - }, - }, - id="without-xsd-file-keeps-xmltodicts-default-single-occurrence-dicts", - ), + pytest.param(1, [1, 1, 1, 1, 1], id="one-row-batches"), + pytest.param(2, [2, 2, 1], id="two-row-batches-with-remainder"), + pytest.param(3, [3, 2], id="three-row-batches-with-remainder"), + pytest.param(10, [5], id="larger-than-input"), + ], +) +def test_process_xml_file_to_pyarrow_pass_buffer_size_controls_batching( + tmp_path: Path, buffer_size: int, expected_batch_sizes: list[int] +) -> None: + """Verify rows are yielded in pa.Table batches of at most `buffer_size`, with a final partial batch.""" + file_path = _write_xml(tmp_path, "five.xml", PEOPLE_XML_5) + settings = fake_xmltopyarrow_settings(buffer_size=buffer_size) + + items = _table_and_data(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + assert [table.num_rows for _, table in items] == expected_batch_sizes + assert [row for _, table in items for row in table.to_pylist()] == PEOPLE_XML_TO_PYARROW_PARSED + assert {table_name for table_name, _ in items} == {"people"} + + +def test_process_xml_file_to_pyarrow_pass_no_matching_elements_yields_nothing_and_logs_no_progress( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """Verify a well-formed file with no elements matching xml_tag yields nothing and logs no progress.""" + caplog.set_level(logging.DEBUG) + file_path = _write_xml(tmp_path, "empty.xml", PEOPLE_XML_EMPTY) + settings = fake_xmltopyarrow_settings() + + items = list(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + assert items == [] + assert [r for r in caplog.records if r.msg.startswith("Processed")] == [] + + +def test_process_xml_file_to_pyarrow_pass_logs_reading_info_message_once( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """Verify the INFO-level 'Reading from ' message is logged exactly once with the correct path.""" + caplog.set_level(logging.INFO) + file_path = _write_xml(tmp_path, "five.xml", PEOPLE_XML_5) + settings = fake_xmltopyarrow_settings() + + list(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + info_records = [r for r in caplog.records if r.levelno == logging.INFO] + assert len(info_records) == 1 + assert info_records[0].msg == "Reading from %s" + assert info_records[0].args == (str(file_path),) + + +@pytest.mark.parametrize( + ("log_interval", "expected_interim_counts", "expected_final_count"), + [ + pytest.param(1, [1, 2, 3, 4, 5], None, id="divides_evenly_every_entry"), + pytest.param(5, [5], None, id="divides_evenly_equal_to_count"), + pytest.param(2, [2, 4], 5, id="remainder_leaves_trailing_entries"), + pytest.param(10, [], 5, id="interval_larger_than_entry_count"), ], ) -def test_process_xml_file_to_dict_pass_force_list_shapes_children_by_schema_occurs( +def test_process_xml_file_to_pyarrow_pass_log_interval_boundary( tmp_path: Path, - test_data_dir: Path, - xsd_filename: str | None, - expected_entry: dict[str, Any], + caplog: pytest.LogCaptureFixture, + log_interval: int, + expected_interim_counts: list[int], + expected_final_count: int | None, ) -> None: - """Verify children the schema allows to repeat become lists even with a single occurrence. + """Verify the exact entry counts logged at each interim checkpoint and in the trailing summary.""" + caplog.set_level(logging.DEBUG) + file_path = _write_xml(tmp_path, "five.xml", PEOPLE_XML_5) + settings = fake_xmltopyarrow_settings(log_interval=log_interval) - With xsd_file set, every child the schema allows to repeat must be a list, regardless of how - many times it actually occurs in this particular document, while children the schema allows - only once stay plain dicts -- at every nesting depth. Without xsd_file, xmltodict's ordinary - behaviour applies instead: a lone occurrence is never turned into a list, whether or not the - schema would allow more. - """ + list(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + interim_counts = [r.args[0] for r in caplog.records if r.msg == "Processed %d entries"] + final_records = [r for r in caplog.records if r.msg == "Processed %d entries from %s"] + + assert interim_counts == expected_interim_counts + if expected_final_count is None: + assert final_records == [] + else: + assert len(final_records) == 1 + assert final_records[0].args == (expected_final_count, file_path.name) + + +def test_process_xml_file_to_pyarrow_pass_gzip_file(tmp_path: Path) -> None: + """Verify gzip-compressed XML input (.gz suffix) is transparently decompressed and parsed.""" + plain_path = _write_xml(tmp_path, "plain.xml", PEOPLE_XML_5) + gz_path = _write_xml(tmp_path, "compressed.xml.gz", PEOPLE_XML_5, gzip_compress=True) + + plain_items = _table_and_data(process_xml_file_to_pyarrow(fake_xmltopyarrow_settings(), file_path=plain_path)) + gz_items = _table_and_data(process_xml_file_to_pyarrow(fake_xmltopyarrow_settings(), file_path=gz_path)) + + plain_rows = [row for _, table in plain_items for row in table.to_pylist()] + gz_rows = [row for _, table in gz_items for row in table.to_pylist()] + assert gz_rows == plain_rows == PEOPLE_XML_TO_PYARROW_PARSED + + +def test_process_xml_file_to_pyarrow_fail_missing_file(tmp_path: Path) -> None: + """Verify a nonexistent file path raises FileNotFoundError when the generator is consumed.""" + missing_path = tmp_path / "does_not_exist.xml" + settings = fake_xmltopyarrow_settings() + with pytest.raises(FileNotFoundError, match="No such file or directory"): + list(process_xml_file_to_pyarrow(settings, file_path=missing_path)) + + +def test_process_xml_file_to_pyarrow_fail_malformed_xml(tmp_path: Path) -> None: + """Verify malformed (not well-formed) XML raises lxml's XMLSyntaxError during streaming.""" + file_path = _write_xml(tmp_path, "malformed.xml", PEOPLE_XML_MALFORMED) + settings = fake_xmltopyarrow_settings() + with pytest.raises(XMLSyntaxError, match="Opening and ending tag mismatch"): + list(process_xml_file_to_pyarrow(settings, file_path=file_path)) + + +def test_process_xml_file_to_pyarrow_pass_with_xsd_file_yields_arrow_table_with_schema_lists( + tmp_path: Path, + test_data_dir: Path, +) -> None: + """process_xml_file_to_pyarrow yields a PyArrow Table with schema-driven list and struct fields.""" file_path = _write_xml(tmp_path, "entry.xml", UNIREF_LIKE_ENTRY_XML) - settings = _build_real_xmltodict_settings(tmp_path, test_data_dir, xsd_filename) + settings = _build_real_xmltopyarrow_settings(tmp_path, test_data_dir, "uniref_like.xsd") - items = _table_and_data(process_xml_file_to_dict(settings, file_path=file_path)) + items = _table_and_data(process_xml_file_to_pyarrow(settings, file_path=file_path)) - assert items == [("entry", [{"entry": expected_entry}])] + assert len(items) == 1 + table_name, table = items[0] + assert table_name == "entry" + assert isinstance(table, pa.Table) + assert settings.pyarrow_schema is not None + assert table.schema.equals(settings.pyarrow_schema, check_metadata=True) + assert table.to_pylist() == [ + { + "property": [{"_type": "member count", "_value": "1"}], + "representativeMember": { + "dbReference": {"property": [{"_type": "isSeed", "_value": "true"}]}, + "sequence": None, + }, + "member": None, + } + ] """process_xml_file_items""" diff --git a/tests/cdm_data_loaders/readers/xml/test_xml_stream_xml_file.py b/tests/cdm_data_loaders/readers/xml/test_xml_stream_xml_file.py index 956ec4ba..ec0b68e5 100644 --- a/tests/cdm_data_loaders/readers/xml/test_xml_stream_xml_file.py +++ b/tests/cdm_data_loaders/readers/xml/test_xml_stream_xml_file.py @@ -5,7 +5,7 @@ from pathlib import Path import pytest -from lxml.etree import XMLSyntaxError +from lxml.etree import XMLSyntaxError, tostring from cdm_data_loaders.readers.xml import stream_xml_file from tests.xml_samples import SIMPLE_LIBRARY_XML @@ -205,3 +205,76 @@ def test_stream_xml_file_fail_non_gzip_content_with_gz_extension_raises_error(tm with pytest.raises(OSError, match="Not a gzipped file"): list(stream_xml_file(fake_gz_path, "book")) + + +@pytest.mark.parametrize( + ("xml", "tag", "expected"), + [ + pytest.param("", "node", [], id="no-match"), + pytest.param( + '', + "node", + [b'', b''], + id="siblings", + ), + pytest.param( + '', + "node", + [b'', b''], + id="matches-under-different-parents", + ), + pytest.param( + '', + "node", + [b''], + id="nested-once", + ), + pytest.param( + 'x', + "node", + [b'x'], + id="nested-with-earlier-siblings", + ), + pytest.param( + '', + "node", + [b''], + id="nested-three-levels", + ), + pytest.param( + '', + "node", + [b'', b''], + id="nested-then-sibling", + ), + pytest.param( + '', + "node", + [b''], + id="root-is-a-match", + ), + pytest.param( + '', + "{urn:x}node", + [b''], + id="nested-in-default-namespace", + ), + ], +) +@pytest.mark.parametrize("compressed", [False, True], ids=["plain", "gzip"]) +def test_stream_xml_file_pass_yields_outermost_matches( # noqa: PLR0917 + tmp_path: Path, + write_gzip_file: Callable[[Path, str, str], Path], + xml: str, + tag: str, + expected: list[bytes], + compressed: bool, +) -> None: + """Only the outermost matching elements are yielded, each with all of its content.""" + if compressed: + file_path = write_gzip_file(tmp_path, "doc.xml.gz", xml) + else: + file_path = tmp_path / "doc.xml" + file_path.write_text(xml, encoding="utf-8") + + assert [tostring(elem, with_tail=False) for elem in stream_xml_file(file_path, tag)] == expected diff --git a/tests/data/pubmed/pubmed.xsd b/tests/data/pubmed/pubmed.xsd new file mode 100644 index 00000000..9bd92936 --- /dev/null +++ b/tests/data/pubmed/pubmed.xsd @@ -0,0 +1,1473 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/uniprot/uniprot_kb/uniprot.xsd b/tests/data/uniprot/uniprot_kb/uniprot.xsd new file mode 100644 index 00000000..e79ba5ea --- /dev/null +++ b/tests/data/uniprot/uniprot_kb/uniprot.xsd @@ -0,0 +1,1125 @@ + + + + + + + + + + Describes a collection of UniProtKB entries. + + + + + + + + + + + + + + Describes a UniProtKB entry. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes the names for the protein and parts thereof. + Equivalent to the flat file DE-line. + + + + + + Describes names of "domains". + Equivalent to the flat file DE-line Includes: section. + + + + + + + + Describes names of processed products. + Equivalent to the flat file DE-line Contains: section. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes a gene. + Equivalent to the flat file GN-line. + + + + + + + + Describes different types of gene designations. + Equivalent to the flat file GN-line. + + + + + + + + + + + + + + + + + + + + + + + Describes the source organism. + + + + + Describes the names of the source organism. + Equivalent to the flat file OS-line. + + + + + Describes a cross-reference to the NCBI taxonomy database. + Equivalent to the flat file OX-line. + + + + + Describes the lineage of the source organism. + Equivalent to the flat file OC-line. + + + + + + + + + + + + + Describes different types of source organism names. + + + + + + + + + + + + + + + + + + + + + + + Describes non-nuclear gene locations (organelles and plasmids). + Equivalent to the flat file OG-line. + + + + + + + + + + + + + + + + + + + + + + + + + Indicates whether the name of a plasmid is known or unknown. + + + + + + + + + + + + + + + + + + + + Describes a citation and a summary of its content. + Equivalent to the flat file RN-, RP-, RC-, RX-, RG-, RA-, RT- and RL-lines. + + + + + + + + + + + + + + Describes different types of citations. + Equivalent to the flat file RX-, RG-, RA-, RT- and RL-lines. + + + + + Describes the title of a citation. + Equivalent to the flat file RT-line. + + + + + Describes the editors of a book (only used for books). + Equivalent to part of the flat file RL-line of books. + + + + + Describes the authors of a citation. + Equivalent to the flat file RA-line. + + + + + Describes the location (URL) of an online journal article. + No flat file equivalent. + + + + + Describes cross-references to bibliography databases (MEDLINE, PubMed, AGRICOLA) or other online resources (DOI). + Equivalent to the flat file RX-line. + + + + + + Describes the type of a citation. + + + + + + + + + + + + + + + + + + + + + Describes the name of an (online) journal or book. + + + + + Describes the volume of a journal or book. + + + + + Describes the first page of an article. + + + + + Describes the last page of an article. + + + + + Describes the publisher of a book. + + + + + Describes the city where a book was published. + + + + + Describes the database name of submissions. + + + + + Describes a patent number. + + + + + Describes the institute where a thesis was made. + + + + + Describes the country where a thesis was made. + + + + + + Describes the authors of a citation when these are represented by a consortium. + Equivalent to the flat file RG-line. + + + + + + + + + + + + + + + + + + Groups a citation's scope and source descriptions. + + + + + Describes the scope of a citation. + Equivalent to the flat file RP-line. + + + + + + + + Describes the source of the sequence according to the citation. + Equivalent to the flat file RC-line. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes different types of general annotations. + Equivalent to the flat file CC-line. + + + + + + + + + Used in 'catalytic activity' annotations. + + + + + + + + Used in 'cofactor' annotations. + + + + + + + Used in 'subcellular location' annotations. + + + + + + + Used in 'sequence caution' annotations. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Refers to the 'key' attribute of a 'reference' element. + + + + + + + + + Used in 'online information' annotations. + + + + + + + + + + Used in 'alternative products' annotations. + + + + + + + + Used in 'interaction' annotations. + + + + + + + + + Used in 'disease' annotations. + + + + + + + + + + + + + + + + + Used in 'mass spectrometry' and 'sequence caution' annotations. + + + + + + + + + + Describes the type of a general annotation. + Equivalent to the flat file CC comment topics (except for "DATABASE" which is translated to "online information"). + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes the type of sequence location in 'RNA editing' annotations. Common values are "Not_applicable" and "Undetermined". + + + + + + Describes an optional name for an 'online information'. + + + + + + Describes the molecular mass in 'mass spectrometry' annotations. + + + + + Describes the error of the mass measurement in 'mass spectrometry' annotations. + + + + + Describes the experimental method in 'mass spectrometry' annotations. + + + + + + + + + Describes different types of biophysicochemical properties. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes a chemical reaction. + + + + + + + + + + + Describes a physiological reaction. + + + + + + + + + + + + + + + + + + Describes a cofactor. + + + + + + + + + + + Describes the subcellular location and optionally the topology and orientation of a molecule. + + + + + + + + + + + Describes the type of events that cause alternative products. + + + + + + + + + + + + + + + + Describes isoforms in 'alternative products' annotations. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes a database cross-reference. + Equivalent to the flat file DR-line. + + + + + + + + + Describes the name of the database. + + + + + Describes a unique database identifier. + + + + + + + + + + + + + + + + Describes the evidence for the protein's existence. + Equivalent to the flat file PE-line. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes different types of sequence annotations. + Equivalent to the flat file FT-line. + + + + + Describes the original sequence in annotations that describe natural or artifical sequence variations. + + + + + Describes the variant sequence in annotations that describe natural or artifical sequence variations. + + + + + Describes the sequence coordinates of the annotation. + + + + + Describes the chemical entity that is bound in annotations that describe binding sites. + + + + + Describes the specific part of a molecule that is bound in annotations that describe binding sites. + + + + + + Describes the type of a sequence annotation. + Equivalent to the flat file FT feature keys, but using full terms instead of acronyms. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes a sequence location as either a range with a begin and end or as a position. The 'sequence' attribute is only used when the location is not on the canonical sequence displayed in the current entry. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes a ligand. + + + + + + + + + + + + Describes a ligand part. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Describes a molecule by name or unique identifier. + + + + + + + + + + + + + Describes the evidence for an annotation. + No flat file equivalent. + + + + + + + + Describes the type of an evidence using the Evidence Code Ontology (http://www.obofoundry.org/cgi-bin/detail.cgi?id=evidence_code). + + + + + A unique key to link annotations (via 'evidence' attributes) to evidences. + + + + + + Describes the source of the data using a database cross-reference (or a 'ref' attribute when the source cannot be found in a public data source, such as PubMed, and is cited only within the UniProtKB entry). + + + + + + + + + Describes the source of the evidence, when it is not assigned by UniProt, but imported from an external database. + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/tests/data/xsd/attribute_only.xsd b/tests/data/xsd/attribute_only.xsd deleted file mode 100644 index 516e2f56..00000000 --- a/tests/data/xsd/attribute_only.xsd +++ /dev/null @@ -1,7 +0,0 @@ - - - - - - - diff --git a/tests/data/xsd/cdata_variety.xsd b/tests/data/xsd/cdata_variety.xsd new file mode 100644 index 00000000..5310cf36 --- /dev/null +++ b/tests/data/xsd/cdata_variety.xsd @@ -0,0 +1,63 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/combinations.xsd b/tests/data/xsd/combinations.xsd new file mode 100644 index 00000000..6ae69962 --- /dev/null +++ b/tests/data/xsd/combinations.xsd @@ -0,0 +1,245 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/examples/pubmed_article/expected_schema.py b/tests/data/xsd/examples/pubmed_article/expected_schema.py new file mode 100644 index 00000000..a5dad39c --- /dev/null +++ b/tests/data/xsd/examples/pubmed_article/expected_schema.py @@ -0,0 +1,1584 @@ +"""Expected PyArrow schema for pubmed_article (xml_tag='PubmedArticle').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field( + "MedlineCitation", + pa.struct( + [ + pa.field("_Owner", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_Status", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("_VersionID", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_VersionDate", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_IndexingMethod", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "PMID", + pa.struct( + [ + pa.field("_Version", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + ), + pa.field( + "DateCompleted", + pa.struct( + [ + pa.field("Year", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Month", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Day", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "DateRevised", + pa.struct( + [ + pa.field("Year", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Month", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Day", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "Article", + pa.struct( + [ + pa.field("_PubModel", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "Journal", + pa.struct( + [ + pa.field( + "ISSN", + pa.struct( + [ + pa.field( + "_IssnType", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "JournalIssue", + pa.struct( + [ + pa.field( + "_CitedMedium", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Volume", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Issue", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "PubDate", + pa.struct( + [ + pa.field( + "Year", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Month", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Day", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Season", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "MedlineDate", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "Title", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "ISOAbbreviation", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "ArticleTitle", + pa.struct( + [ + pa.field( + "_book", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_part", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_sec", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=False, + ), + pa.field( + "Pagination", + pa.struct( + [ + pa.field( + "StartPage", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "EndPage", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "MedlinePgn", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "ELocationID", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_EIdType", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "Abstract", + pa.struct( + [ + pa.field( + "AbstractText", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Label", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_NlmCategory", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + pa.field( + "CopyrightInformation", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "AuthorList", + pa.struct( + [ + pa.field( + "_CompleteYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_Type", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Author", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_EqualContrib", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "LastName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "CollectiveName", + pa.struct( + [ + pa.field( + "_Investigators", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:IDREF"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "AffiliationInfo", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Affiliation", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "Language", + pa.list_( + pa.field( + "item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ), + nullable=False, + ), + pa.field( + "DataBankList", + pa.struct( + [ + pa.field( + "_CompleteYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "DataBank", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "DataBankName", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "AccessionNumberList", + pa.struct( + [ + pa.field( + "AccessionNumber", + pa.list_( + pa.field( + "item", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "GrantList", + pa.struct( + [ + pa.field( + "_CompleteYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Grant", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "GrantID", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Acronym", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Agency", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Country", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "PublicationTypeList", + pa.struct( + [ + pa.field( + "PublicationType", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_UI", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "VernacularTitle", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "ArticleDate", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_DateType", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Year", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Month", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Day", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + ), + pa.field( + "MedlineJournalInfo", + pa.struct( + [ + pa.field("Country", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("MedlineTA", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("NlmUniqueID", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("ISSNLinking", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + ), + pa.field( + "ChemicalList", + pa.struct( + [ + pa.field( + "Chemical", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "RegistryNumber", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "NameOfSubstance", + pa.struct( + [ + pa.field( + "_UI", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "SupplMeshList", + pa.struct( + [ + pa.field( + "SupplMeshName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_UI", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "CitationSubset", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "CommentsCorrectionsList", + pa.struct( + [ + pa.field( + "CommentsCorrections", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_RefType", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "RefSource", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "PMID", + pa.struct( + [ + pa.field( + "_Version", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "Note", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "GeneSymbolList", + pa.struct( + [ + pa.field( + "GeneSymbol", + pa.list_( + pa.field( + "item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "MeshHeadingList", + pa.struct( + [ + pa.field( + "MeshHeading", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "DescriptorName", + pa.struct( + [ + pa.field( + "_MajorTopicYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_AutoHM", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_Type", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_UI", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "QualifierName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_AutoHM", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_MajorTopicYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_UI", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field("NumberOfReferences", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "PersonalNameSubjectList", + pa.struct( + [ + pa.field( + "PersonalNameSubject", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "LastName", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "OtherID", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "OtherAbstract", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Type", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_Language", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "AbstractText", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Label", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_NlmCategory", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + pa.field( + "CopyrightInformation", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "KeywordList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Owner", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Keyword", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_MajorTopicYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field("CoiStatement", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "SpaceFlightMission", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "InvestigatorList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_ID", pa.string(), nullable=True, metadata={"xsd_type": "xs:ID"}), + pa.field( + "Investigator", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "LastName", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "AffiliationInfo", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Affiliation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "GeneralNote", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Owner", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + ), + pa.field( + "PubmedData", + pa.struct( + [ + pa.field( + "History", + pa.struct( + [ + pa.field( + "PubMedPubDate", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_PubStatus", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Year", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Month", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Day", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Hour", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Minute", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Second", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field("PublicationStatus", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "ObjectList", + pa.struct( + [ + pa.field( + "Object", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Param", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "ReferenceList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Title", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Reference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Citation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ReferenceList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Title", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Reference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Citation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ReferenceList", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/examples/pubmed_article/maximal.xml b/tests/data/xsd/examples/pubmed_article/maximal.xml new file mode 100644 index 00000000..2e4c380c --- /dev/null +++ b/tests/data/xsd/examples/pubmed_article/maximal.xml @@ -0,0 +1,433 @@ + + + + + 38123456 + + 2024 + 05 + 29 + + + 2025 + 01 + 16 + +
+ + 1476-4687 + + 629 + 8012 + + 2023 + 09 + 15 + + + Nature + Nature + + Structural basis of TP53 tetramerization and Ca2+ signaling + + 410 + 418 + 410-418 + + 10.1038/s41586-024-07123-x + S0028-0836(24)00123-4 + + The tumor suppressor p53 regulates transcription in response to <5 Gy stress and H2O2. + We undertook 20 semi-structured interviews with participants aged >18 years studying Ca2+ signaling & DNA repair and underlined text. + + Copyright (c) 2024 Springer Nature Limited. + + + + Morris + Charlotte + C + Jr + 0000-0002-9047-0481 + 0000-0001-6450-5815 + + The University of Manchester, Division of Population Health, Manchester, United Kingdom. + 0000-0003-2804-1181 + 0000-0002-0939-9842 + + + National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. + 0000-0001-5784-7403 + 0000-0002-1825-0097 + + + + International Cancer Genome Consortium + https://ror.org/027m9bs27 + https://ror.org/01ej9dk98 + + Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. + https://ror.org/00za53h95 + https://ror.org/03vek6s52 + + + Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. + 0000-0002-9047-0481 (11) + 0000-0001-6450-5815 (12) + + + + eng + fre + + + PDB + + 1TUP + 2OCJ + + + + GenBank + + NM_000546 + AF307851 + + + + + + R01-CA043460 + CA + NCI NIH HHS + United States + + + MR/T021975/1 + MRC + Medical Research Council + United Kingdom + + + + Journal Article + Research Support, Non-U.S. Gov't + + Bases structurales de la tetramerisation de TP53 + + 2022 + 11 + 04 + + + 2021 + 03 + 21 + +
+ + Belgium + Nature + 0410462 + 0028-0836 + + + + EC 2.7.11.1 + Tumor Suppressor Protein p53 + + + EC 6.3.2.19 + Proto-Oncogene Proteins c-mdm2 + + + + Li-Fraumeni Syndrome + Chromatin Immunoprecipitation + + IM + AIM + + + Nature. 2000 Nov 16;408(6810):307-310. + 11099028 + Foundational review of the p53 signaling network. + + + Science. 2024 Jun 7;384(6700):1020-1021. + 38843210 + Perspective highlighting TP53 tetramerization dynamics. + + + + TP53 + MDM2 + + + + Tumor Suppressor Protein p53 + genetics + metabolism + + + DNA Repair + physiology + drug effects + + + 42 + + + Kontopantelis + Evangelos + E + III + + + Robinson + Louise + L + Sr + + + NTRS-20240001234 + PIP-987654 + + Tetramerization affinity increased in vitro by 3.5-fold (p < 0.001) upon phosphorylation. + Clarifying TP53 & MDM2 kinetics improves targeted cancer therapy. + Copyright (c) 2024 The Authors. Published under CC-BY 4.0. + + + French summary: Le facteur de transcription p53 maintient la stabilite du genome. + Plain-language summary: How cells repair damaged DNA when exposed to &gt;2 Gy radiation. + Copyright (c) 2024 National Library of Medicine. + + + p53 signaling + tetramerization + + + DNA damage response + apoptosis + + The authors declare no competing financial or non-financial interests. + ISS Expedition 68 + Artemis I BioSentinel + + + Ashcroft + Darren + D + II + 0000-0003-2804-1181 (13) + 0000-0002-0939-9842 (14) + + MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom. + 0000-0001-5784-7403 (15) + 0000-0002-1825-0097 (16) + + + Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA. + https://ror.org/027m9bs27 (17) + https://ror.org/01ej9dk98 (18) + + + + Blakeman + Thomas + T + IV + https://ror.org/00za53h95 (19) + https://ror.org/03vek6s52 (20) + + Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA. + 0000-0002-9047-0481 (21) + 0000-0001-6450-5815 (22) + + + Department of Molecular Biology, Princeton University, Princeton, NJ, USA. + 0000-0003-2804-1181 (23) + 0000-0002-0939-9842 (24) + + + + + + Tucker + Joseph + J + PhD + 0000-0001-5784-7403 (25) + 0000-0002-1825-0097 (26) + + The University of Manchester, Division of Population Health, Manchester, United Kingdom. (9) + https://ror.org/027m9bs27 (27) + https://ror.org/01ej9dk98 (28) + + + National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. (10) + https://ror.org/00za53h95 (29) + https://ror.org/03vek6s52 (30) + + + + Mullens + Amy + A + MD + 0000-0002-9047-0481 (31) + 0000-0001-6450-5815 (32) + + Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. (11) + 0000-0003-2804-1181 (33) + 0000-0002-0939-9842 (34) + + + Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. (12) + 0000-0001-5784-7403 (35) + 0000-0002-1825-0097 (36) + + + + Indexed by automated curation. + Supported by NASA Space Biology grant 80NSSC22K0123. + + + + + 2020 + 07 + 12 + 14 + 30 + 00 + + + 2019 + 02 + 08 + 09 + 15 + 25 + + + ppublish + + 38123456 + 10.1038/s41586-024-07123-x + + + + supp_table_1.pdf + Crystallographic data collection and refinement statistics + + + GSE245678 + NCBI Gene Expression Omnibus + + + + Primary Literature + + Vogelstein B, Lane D, Levine AJ. Surfing the p53 network. Nature. 2000;408(6810):307-310. + + PMC10987654 + S0028-0836(24)00123-4 + + + + Cho Y, Gorina S, Jeffrey PD, Pavletich NP. Crystal structure of a p53 tumor suppressor-DNA complex. Science. 1994;265(5170):346-355. + + 11099028 + 10.1038/35042675 + + + + Structural Biology References + + Baker SJ, Fearon ER, Nigro JM, et al. Chromosome 17 deletions and p53 gene mutations in colorectal carcinomas. Science. 1989;244(4901):217-221. + + 7585406 + 10.1126/science.7878469 + + + + Momand J, Zambetti GP, Olson DC, George D, Levine AJ. The mdm-2 oncogene product forms a complex with the p53 protein. Cell. 1992;69(7):1237-1245. + + 2694717 + 10.1016/0092-8674(89)90984-7 + + + + Clinical &amp; Genetic Studies + + Kussie PH, Gorina S, Marechal V, et al. Structure of the MDM2 oncoprotein bound to the p53 tumor suppressor transactivation domain. Science. 1996;274(5289):948-953. + + 8183552 + 10.1126/science.8183552 + + + + Haupt Y, Maya R, Kazaz A, Oren M. Mdm2 promotes the rapid degradation of p53. Nature. 1997;387(6630):296-299. + + 9367981 + 10.1038/387296a0 + + + + + + + Historical Background + + Cohen GN, Barbe V, Flament D, et al. An integrated analysis of the genome of Pyrococcus abyssi. Mol Microbiol. 2003;47(6):1495-1512. + + 10430560 + 10.1093/genetics/152.4.1299 + + + + Erkel C, Kube M, Reinhardt R, Liesack W. Genome of rice cluster I archaea. Science. 2006;313(5785):370-372. + + 12622808 + 10.1046/j.1365-2958.2003.03381.x + + + + Methodological References + + Anderson IJ, Sieprawska-Lupa M, Lapidus A, et al. Complete genome sequence of Methanoculleus marisnigri JR1. Stand Genomic Sci. 2009;1(2):189-196. + + 16857943 + 10.1126/science.1127062 + + + + Gao J, Wang J. Re-annotation of two hyperthermophilic archaea Pyrococcus abyssi & P. furiosus. Curr Microbiol. 2012;64(2):118-129. + + 21304656 + 10.4056/sigs.32535 + + + + Supplementary Citations + + Morris C, Robinson L, Ashcroft D, et al. Primary care experiences of people with dementia. Br J Gen Pract. 2025;75(751):e102-e110. + + 22057919 + 10.1007/s00284-011-0035-x + + + + Tucker J, Mullens A, Morroni C, Ong J. The HIV response at an inflection point: resilience &amp; equity. Sex Health. 2026;23(1):SH25236. + + 31234567 + 10.1016/j.cell.2019.05.012 + + + + + + + + diff --git a/tests/data/xsd/examples/pubmed_article/maximal_expected.json b/tests/data/xsd/examples/pubmed_article/maximal_expected.json new file mode 100644 index 00000000..fdbdb6c3 --- /dev/null +++ b/tests/data/xsd/examples/pubmed_article/maximal_expected.json @@ -0,0 +1,894 @@ +[ + { + "MedlineCitation": { + "_Owner": "NLM", + "_Status": "MEDLINE", + "_VersionID": "1", + "_VersionDate": "2024-05-29", + "_IndexingMethod": "Automated", + "PMID": { + "_Version": "1", + "#text": "38123456" + }, + "DateCompleted": { + "Year": "2024", + "Month": "05", + "Day": "29" + }, + "DateRevised": { + "Year": "2025", + "Month": "01", + "Day": "16" + }, + "Article": { + "_PubModel": "Print-Electronic", + "Journal": { + "ISSN": { + "_IssnType": "Electronic", + "#text": "1476-4687" + }, + "JournalIssue": { + "_CitedMedium": "Internet", + "Volume": "629", + "Issue": "8012", + "PubDate": { + "Year": "2023", + "Month": "09", + "Day": "15" + } + }, + "Title": "Nature", + "ISOAbbreviation": "Nature" + }, + "ArticleTitle": { + "_book": "nbk21054", + "_part": "part1", + "_sec": "sec1.1", + "#text": "Structural basis of TP53 tetramerization and Ca2+ signaling" + }, + "Pagination": { + "StartPage": "410", + "EndPage": "418", + "MedlinePgn": "410-418" + }, + "ELocationID": [ + { + "_EIdType": "doi", + "_ValidYN": "Y", + "#text": "10.1038/s41586-024-07123-x" + }, + { + "_EIdType": "pii", + "_ValidYN": "N", + "#text": "S0028-0836(24)00123-4" + } + ], + "Abstract": { + "AbstractText": [ + { + "_Label": "BACKGROUND", + "_NlmCategory": "BACKGROUND", + "#text": "The tumor suppressor p53 regulates transcription in response to <5 Gy stress and H2O2." + }, + { + "_Label": "METHODS", + "_NlmCategory": "METHODS", + "#text": "We undertook 20 semi-structured interviews with participants aged >18 years studying Ca2+ signaling & DNA repair and underlined text." + }, + { + "_Label": "UNLABELLED", + "_NlmCategory": "UNASSIGNED" + } + ], + "CopyrightInformation": "Copyright (c) 2024 Springer Nature Limited." + }, + "AuthorList": { + "_CompleteYN": "Y", + "_Type": "authors", + "Author": [ + { + "_ValidYN": "Y", + "_EqualContrib": "Y", + "LastName": "Morris", + "ForeName": "Charlotte", + "Initials": "C", + "Suffix": "Jr", + "Identifier": [ + { + "_Source": "ORCID", + "#text": "0000-0002-9047-0481" + }, + { + "_Source": "ISNI", + "#text": "0000-0001-6450-5815" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "The University of Manchester, Division of Population Health, Manchester, United Kingdom.", + "Identifier": [ + { + "_Source": "VIAF", + "#text": "0000-0003-2804-1181" + }, + { + "_Source": "GND", + "#text": "0000-0002-0939-9842" + } + ] + }, + { + "Affiliation": "National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom.", + "Identifier": [ + { + "_Source": "RINGGOLD", + "#text": "0000-0001-5784-7403" + }, + { + "_Source": "GRID", + "#text": "0000-0002-1825-0097" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "_EqualContrib": "N", + "CollectiveName": { + "_Investigators": "INV_1", + "#text": "International Cancer Genome Consortium" + }, + "Identifier": [ + { + "_Source": "ROR", + "#text": "https://ror.org/027m9bs27" + }, + { + "_Source": "ORCID (8)", + "#text": "https://ror.org/01ej9dk98" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom.", + "Identifier": [ + { + "_Source": "ISNI (9)", + "#text": "https://ror.org/00za53h95" + }, + { + "_Source": "VIAF (10)", + "#text": "https://ror.org/03vek6s52" + } + ] + }, + { + "Affiliation": "Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA.", + "Identifier": [ + { + "_Source": "GND (11)", + "#text": "0000-0002-9047-0481 (11)" + }, + { + "_Source": "RINGGOLD (12)", + "#text": "0000-0001-6450-5815 (12)" + } + ] + } + ] + } + ] + }, + "Language": [ + "eng", + "fre" + ], + "DataBankList": { + "_CompleteYN": "Y", + "DataBank": [ + { + "DataBankName": "PDB", + "AccessionNumberList": { + "AccessionNumber": [ + "1TUP", + "2OCJ" + ] + } + }, + { + "DataBankName": "GenBank", + "AccessionNumberList": { + "AccessionNumber": [ + "NM_000546", + "AF307851" + ] + } + } + ] + }, + "GrantList": { + "_CompleteYN": "Y", + "Grant": [ + { + "GrantID": "R01-CA043460", + "Acronym": "CA", + "Agency": "NCI NIH HHS", + "Country": "United States" + }, + { + "GrantID": "MR/T021975/1", + "Acronym": "MRC", + "Agency": "Medical Research Council", + "Country": "United Kingdom" + } + ] + }, + "PublicationTypeList": { + "PublicationType": [ + { + "_UI": "D016428", + "#text": "Journal Article" + }, + { + "_UI": "D013485", + "#text": "Research Support, Non-U.S. Gov't" + } + ] + }, + "VernacularTitle": "Bases structurales de la tetramerisation de TP53", + "ArticleDate": [ + { + "_DateType": "Electronic", + "Year": "2022", + "Month": "11", + "Day": "04" + }, + { + "_DateType": "Electronic", + "Year": "2021", + "Month": "03", + "Day": "21" + } + ] + }, + "MedlineJournalInfo": { + "Country": "Belgium", + "MedlineTA": "Nature", + "NlmUniqueID": "0410462", + "ISSNLinking": "0028-0836" + }, + "ChemicalList": { + "Chemical": [ + { + "RegistryNumber": "EC 2.7.11.1", + "NameOfSubstance": { + "_UI": "Sample_UI_1", + "#text": "Tumor Suppressor Protein p53" + } + }, + { + "RegistryNumber": "EC 6.3.2.19", + "NameOfSubstance": { + "_UI": "Sample_UI_2", + "#text": "Proto-Oncogene Proteins c-mdm2" + } + } + ] + }, + "SupplMeshList": { + "SupplMeshName": [ + { + "_Type": "Disease", + "_UI": "C536599", + "#text": "Li-Fraumeni Syndrome" + }, + { + "_Type": "Protocol", + "_UI": "C045812", + "#text": "Chromatin Immunoprecipitation" + } + ] + }, + "CitationSubset": [ + "IM", + "AIM" + ], + "CommentsCorrectionsList": { + "CommentsCorrections": [ + { + "_RefType": "Cites", + "RefSource": "Nature. 2000 Nov 16;408(6810):307-310.", + "PMID": { + "_Version": "2", + "#text": "11099028" + }, + "Note": "Foundational review of the p53 signaling network." + }, + { + "_RefType": "CommentIn", + "RefSource": "Science. 2024 Jun 7;384(6700):1020-1021.", + "PMID": { + "_Version": "3", + "#text": "38843210" + }, + "Note": "Perspective highlighting TP53 tetramerization dynamics." + } + ] + }, + "GeneSymbolList": { + "GeneSymbol": [ + "TP53", + "MDM2" + ] + }, + "MeshHeadingList": { + "MeshHeading": [ + { + "DescriptorName": { + "_MajorTopicYN": "Y", + "_AutoHM": "Y", + "_Type": "Geographic", + "_UI": "D016159", + "#text": "Tumor Suppressor Protein p53" + }, + "QualifierName": [ + { + "_AutoHM": "Y", + "_MajorTopicYN": "N", + "_UI": "Q000235", + "#text": "genetics" + }, + { + "_AutoHM": "Y", + "_MajorTopicYN": "Y", + "_UI": "Q000378", + "#text": "metabolism" + } + ] + }, + { + "DescriptorName": { + "_MajorTopicYN": "N", + "_AutoHM": "Y", + "_Type": "Geographic", + "_UI": "D004260", + "#text": "DNA Repair" + }, + "QualifierName": [ + { + "_AutoHM": "Y", + "_MajorTopicYN": "N", + "_UI": "Q000502", + "#text": "physiology" + }, + { + "_AutoHM": "Y", + "_MajorTopicYN": "Y", + "_UI": "Q000187", + "#text": "drug effects" + } + ] + } + ] + }, + "NumberOfReferences": "42", + "PersonalNameSubjectList": { + "PersonalNameSubject": [ + { + "LastName": "Kontopantelis", + "ForeName": "Evangelos", + "Initials": "E", + "Suffix": "III" + }, + { + "LastName": "Robinson", + "ForeName": "Louise", + "Initials": "L", + "Suffix": "Sr" + } + ] + }, + "OtherID": [ + { + "_Source": "NASA", + "#text": "NTRS-20240001234" + }, + { + "_Source": "PIP", + "#text": "PIP-987654" + } + ], + "OtherAbstract": [ + { + "_Type": "Publisher", + "_Language": "eng", + "AbstractText": [ + { + "_Label": "RESULTS", + "_NlmCategory": "RESULTS", + "#text": "Tetramerization affinity increased in vitro by 3.5-fold (p < 0.001) upon phosphorylation." + }, + { + "_Label": "CONCLUSIONS", + "_NlmCategory": "CONCLUSIONS", + "#text": "Clarifying TP53 & MDM2 kinetics improves targeted cancer therapy." + } + ], + "CopyrightInformation": "Copyright (c) 2024 The Authors. Published under CC-BY 4.0." + }, + { + "_Type": "AAMC", + "_Language": "fre", + "AbstractText": [ + { + "_Label": "OBJECTIVE", + "_NlmCategory": "OBJECTIVE", + "#text": "French summary: Le facteur de transcription p53 maintient la stabilite du genome." + }, + { + "_Label": "BACKGROUND (6)", + "_NlmCategory": "UNASSIGNED", + "#text": "Plain-language summary: How cells repair damaged DNA when exposed to &gt;2 Gy radiation." + } + ], + "CopyrightInformation": "Copyright (c) 2024 National Library of Medicine." + } + ], + "KeywordList": [ + { + "_Owner": "NOTNLM", + "Keyword": [ + { + "_MajorTopicYN": "Y", + "#text": "p53 signaling" + }, + { + "_MajorTopicYN": "N", + "#text": "tetramerization" + } + ] + }, + { + "_Owner": "NLM", + "Keyword": [ + { + "_MajorTopicYN": "Y", + "#text": "DNA damage response" + }, + { + "_MajorTopicYN": "N", + "#text": "apoptosis" + } + ] + } + ], + "CoiStatement": "The authors declare no competing financial or non-financial interests.", + "SpaceFlightMission": [ + "ISS Expedition 68", + "Artemis I BioSentinel" + ], + "InvestigatorList": [ + { + "_ID": "INV_1", + "Investigator": [ + { + "_ValidYN": "Y", + "LastName": "Ashcroft", + "ForeName": "Darren", + "Initials": "D", + "Suffix": "II", + "Identifier": [ + { + "_Source": "GRID (13)", + "#text": "0000-0003-2804-1181 (13)" + }, + { + "_Source": "ROR (14)", + "#text": "0000-0002-0939-9842 (14)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom.", + "Identifier": [ + { + "_Source": "ORCID (15)", + "#text": "0000-0001-5784-7403 (15)" + }, + { + "_Source": "ISNI (16)", + "#text": "0000-0002-1825-0097 (16)" + } + ] + }, + { + "Affiliation": "Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA.", + "Identifier": [ + { + "_Source": "VIAF (17)", + "#text": "https://ror.org/027m9bs27 (17)" + }, + { + "_Source": "GND (18)", + "#text": "https://ror.org/01ej9dk98 (18)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "LastName": "Blakeman", + "ForeName": "Thomas", + "Initials": "T", + "Suffix": "IV", + "Identifier": [ + { + "_Source": "RINGGOLD (19)", + "#text": "https://ror.org/00za53h95 (19)" + }, + { + "_Source": "GRID (20)", + "#text": "https://ror.org/03vek6s52 (20)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA.", + "Identifier": [ + { + "_Source": "ROR (21)", + "#text": "0000-0002-9047-0481 (21)" + }, + { + "_Source": "ORCID (22)", + "#text": "0000-0001-6450-5815 (22)" + } + ] + }, + { + "Affiliation": "Department of Molecular Biology, Princeton University, Princeton, NJ, USA.", + "Identifier": [ + { + "_Source": "ISNI (23)", + "#text": "0000-0003-2804-1181 (23)" + }, + { + "_Source": "VIAF (24)", + "#text": "0000-0002-0939-9842 (24)" + } + ] + } + ] + } + ] + }, + { + "_ID": "INV_2", + "Investigator": [ + { + "_ValidYN": "Y", + "LastName": "Tucker", + "ForeName": "Joseph", + "Initials": "J", + "Suffix": "PhD", + "Identifier": [ + { + "_Source": "GND (25)", + "#text": "0000-0001-5784-7403 (25)" + }, + { + "_Source": "RINGGOLD (26)", + "#text": "0000-0002-1825-0097 (26)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "The University of Manchester, Division of Population Health, Manchester, United Kingdom. (9)", + "Identifier": [ + { + "_Source": "GRID (27)", + "#text": "https://ror.org/027m9bs27 (27)" + }, + { + "_Source": "ROR (28)", + "#text": "https://ror.org/01ej9dk98 (28)" + } + ] + }, + { + "Affiliation": "National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. (10)", + "Identifier": [ + { + "_Source": "ORCID (29)", + "#text": "https://ror.org/00za53h95 (29)" + }, + { + "_Source": "ISNI (30)", + "#text": "https://ror.org/03vek6s52 (30)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "LastName": "Mullens", + "ForeName": "Amy", + "Initials": "A", + "Suffix": "MD", + "Identifier": [ + { + "_Source": "VIAF (31)", + "#text": "0000-0002-9047-0481 (31)" + }, + { + "_Source": "GND (32)", + "#text": "0000-0001-6450-5815 (32)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. (11)", + "Identifier": [ + { + "_Source": "RINGGOLD (33)", + "#text": "0000-0003-2804-1181 (33)" + }, + { + "_Source": "GRID (34)", + "#text": "0000-0002-0939-9842 (34)" + } + ] + }, + { + "Affiliation": "Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. (12)", + "Identifier": [ + { + "_Source": "ROR (35)", + "#text": "0000-0001-5784-7403 (35)" + }, + { + "_Source": "ORCID (36)", + "#text": "0000-0002-1825-0097 (36)" + } + ] + } + ] + } + ] + } + ], + "GeneralNote": [ + { + "_Owner": "NLM", + "#text": "Indexed by automated curation." + }, + { + "_Owner": "NASA", + "#text": "Supported by NASA Space Biology grant 80NSSC22K0123." + } + ] + }, + "PubmedData": { + "History": { + "PubMedPubDate": [ + { + "_PubStatus": "received", + "Year": "2020", + "Month": "07", + "Day": "12", + "Hour": "14", + "Minute": "30", + "Second": "00" + }, + { + "_PubStatus": "revised", + "Year": "2019", + "Month": "02", + "Day": "08", + "Hour": "09", + "Minute": "15", + "Second": "25" + } + ] + }, + "PublicationStatus": "ppublish", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pubmed", + "#text": "38123456" + }, + { + "_IdType": "doi", + "#text": "10.1038/s41586-024-07123-x" + } + ] + }, + "ObjectList": { + "Object": [ + { + "_Type": "SupplementaryMaterial", + "Param": [ + { + "_Name": "id", + "#text": "supp_table_1.pdf" + }, + { + "_Name": "caption", + "#text": "Crystallographic data collection and refinement statistics" + } + ] + }, + { + "_Type": "Dataset", + "Param": [ + { + "_Name": "accession", + "#text": "GSE245678" + }, + { + "_Name": "repository", + "#text": "NCBI Gene Expression Omnibus" + } + ] + } + ] + }, + "ReferenceList": [ + { + "Title": "Primary Literature", + "Reference": [ + { + "Citation": "Vogelstein B, Lane D, Levine AJ. Surfing the p53 network. Nature. 2000;408(6810):307-310.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmc", + "#text": "PMC10987654" + }, + { + "_IdType": "pii", + "#text": "S0028-0836(24)00123-4" + } + ] + } + }, + { + "Citation": "Cho Y, Gorina S, Jeffrey PD, Pavletich NP. Crystal structure of a p53 tumor suppressor-DNA complex. Science. 1994;265(5170):346-355.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "mid", + "#text": "11099028" + }, + { + "_IdType": "pmcid", + "#text": "10.1038/35042675" + } + ] + } + } + ], + "ReferenceList": [ + { + "Title": "Structural Biology References", + "Reference": [ + { + "Citation": "Baker SJ, Fearon ER, Nigro JM, et al. Chromosome 17 deletions and p53 gene mutations in colorectal carcinomas. Science. 1989;244(4901):217-221.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmcpid", + "#text": "7585406" + }, + { + "_IdType": "pmpid", + "#text": "10.1126/science.7878469" + } + ] + } + }, + { + "Citation": "Momand J, Zambetti GP, Olson DC, George D, Levine AJ. The mdm-2 oncogene product forms a complex with the p53 protein. Cell. 1992;69(7):1237-1245.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "sici", + "#text": "2694717" + }, + { + "_IdType": "medline", + "#text": "10.1016/0092-8674(89)90984-7" + } + ] + } + } + ], + "ReferenceList": [ + "{\"Title\": \"Clinical &amp; Genetic Studies\", \"Reference\": [{\"Citation\": \"Kussie PH, Gorina S, Marechal V, et al. Structure of the MDM2 oncoprotein bound to the p53 tumor suppressor transactivation domain. Science. 1996;274(5289):948-953.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pmcbook\", \"#text\": \"8183552\"}, {\"_IdType\": \"bookaccession\", \"#text\": \"10.1126/science.8183552\"}]}}, {\"Citation\": \"Haupt Y, Maya R, Kazaz A, Oren M. Mdm2 promotes the rapid degradation of p53. Nature. 1997;387(6630):296-299.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pubmed\", \"#text\": \"9367981\"}, {\"_IdType\": \"doi\", \"#text\": \"10.1038/387296a0\"}]}}]}" + ] + } + ] + }, + { + "Title": "Historical Background", + "Reference": [ + { + "Citation": "Cohen GN, Barbe V, Flament D, et al. An integrated analysis of the genome of Pyrococcus abyssi. Mol Microbiol. 2003;47(6):1495-1512.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmc", + "#text": "10430560" + }, + { + "_IdType": "pii", + "#text": "10.1093/genetics/152.4.1299" + } + ] + } + }, + { + "Citation": "Erkel C, Kube M, Reinhardt R, Liesack W. Genome of rice cluster I archaea. Science. 2006;313(5785):370-372.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "mid", + "#text": "12622808" + }, + { + "_IdType": "pmcid", + "#text": "10.1046/j.1365-2958.2003.03381.x" + } + ] + } + } + ], + "ReferenceList": [ + { + "Title": "Methodological References", + "Reference": [ + { + "Citation": "Anderson IJ, Sieprawska-Lupa M, Lapidus A, et al. Complete genome sequence of Methanoculleus marisnigri JR1. Stand Genomic Sci. 2009;1(2):189-196.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmcpid", + "#text": "16857943" + }, + { + "_IdType": "pmpid", + "#text": "10.1126/science.1127062" + } + ] + } + }, + { + "Citation": "Gao J, Wang J. Re-annotation of two hyperthermophilic archaea Pyrococcus abyssi & P. furiosus. Curr Microbiol. 2012;64(2):118-129.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "sici", + "#text": "21304656" + }, + { + "_IdType": "medline", + "#text": "10.4056/sigs.32535" + } + ] + } + } + ], + "ReferenceList": [ + "{\"Title\": \"Supplementary Citations\", \"Reference\": [{\"Citation\": \"Morris C, Robinson L, Ashcroft D, et al. Primary care experiences of people with dementia. Br J Gen Pract. 2025;75(751):e102-e110.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pmcbook\", \"#text\": \"22057919\"}, {\"_IdType\": \"bookaccession\", \"#text\": \"10.1007/s00284-011-0035-x\"}]}}, {\"Citation\": \"Tucker J, Mullens A, Morroni C, Ong J. The HIV response at an inflection point: resilience &amp; equity. Sex Health. 2026;23(1):SH25236.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pubmed\", \"#text\": \"31234567\"}, {\"_IdType\": \"doi\", \"#text\": \"10.1016/j.cell.2019.05.012\"}]}}]}" + ] + } + ] + } + ] + } + } +] diff --git a/tests/data/xsd/examples/pubmed_article/minimal.xml b/tests/data/xsd/examples/pubmed_article/minimal.xml new file mode 100644 index 00000000..ef0c1aef --- /dev/null +++ b/tests/data/xsd/examples/pubmed_article/minimal.xml @@ -0,0 +1,26 @@ + + + + + 38123456 +
+ + + + Sample_MedlineDate_1 + + + + Structural basis of TP53 tetramerization and Ca2+ signaling + 10.1038/s41586-024-07123-x + eng + + Journal Article + +
+ + Nature + + + + diff --git a/tests/data/xsd/examples/pubmed_article/minimal_expected.json b/tests/data/xsd/examples/pubmed_article/minimal_expected.json new file mode 100644 index 00000000..e5b4d516 --- /dev/null +++ b/tests/data/xsd/examples/pubmed_article/minimal_expected.json @@ -0,0 +1,45 @@ +[ + { + "MedlineCitation": { + "_Status": "MEDLINE", + "PMID": { + "_Version": "1", + "#text": "38123456" + }, + "Article": { + "_PubModel": "Print-Electronic", + "Journal": { + "JournalIssue": { + "_CitedMedium": "Internet", + "PubDate": { + "MedlineDate": "Sample_MedlineDate_1" + } + } + }, + "ArticleTitle": { + "#text": "Structural basis of TP53 tetramerization and Ca2+ signaling" + }, + "ELocationID": [ + { + "_EIdType": "doi", + "#text": "10.1038/s41586-024-07123-x" + } + ], + "Language": [ + "eng" + ], + "PublicationTypeList": { + "PublicationType": [ + { + "_UI": "D016428", + "#text": "Journal Article" + } + ] + } + }, + "MedlineJournalInfo": { + "MedlineTA": "Nature" + } + } + } +] diff --git a/tests/data/xsd/examples/pubmed_book_article/expected_schema.py b/tests/data/xsd/examples/pubmed_book_article/expected_schema.py new file mode 100644 index 00000000..47354c9c --- /dev/null +++ b/tests/data/xsd/examples/pubmed_book_article/expected_schema.py @@ -0,0 +1,1600 @@ +"""Expected PyArrow schema for pubmed_book_article (xml_tag='PubmedBookArticle').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field( + "BookDocument", + pa.struct( + [ + pa.field( + "PMID", + pa.struct( + [ + pa.field("_Version", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + ), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "Book", + pa.struct( + [ + pa.field( + "Publisher", + pa.struct( + [ + pa.field( + "PublisherName", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "PublisherLocation", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "BookTitle", + pa.struct( + [ + pa.field( + "_book", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_part", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_sec", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=False, + ), + pa.field( + "PubDate", + pa.struct( + [ + pa.field( + "Year", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Month", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Day", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Season", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "MedlineDate", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "BeginningDate", + pa.struct( + [ + pa.field( + "Year", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Month", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Day", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Season", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + pa.field( + "EndingDate", + pa.struct( + [ + pa.field( + "Year", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Month", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Day", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Season", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + pa.field( + "AuthorList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_CompleteYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_Type", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Author", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_EqualContrib", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "LastName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "CollectiveName", + pa.struct( + [ + pa.field( + "_Investigators", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:IDREF" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "AffiliationInfo", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Affiliation", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "InvestigatorList", + pa.struct( + [ + pa.field("_ID", pa.string(), nullable=True, metadata={"xsd_type": "xs:ID"}), + pa.field( + "Investigator", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "LastName", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "AffiliationInfo", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Affiliation", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field("Volume", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("VolumeTitle", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("Edition", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "CollectionTitle", + pa.struct( + [ + pa.field( + "_book", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_part", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_sec", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + pa.field( + "Isbn", + pa.list_( + pa.field( + "item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ), + nullable=True, + ), + pa.field( + "ELocationID", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_EIdType", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field("Medium", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "ReportNumber", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=False, + ), + pa.field( + "LocationLabel", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Type", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ArticleTitle", + pa.struct( + [ + pa.field("_book", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_part", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_sec", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field("VernacularTitle", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "Pagination", + pa.struct( + [ + pa.field("StartPage", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("EndPage", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("MedlinePgn", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "Language", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "AuthorList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_CompleteYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_Type", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Author", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_EqualContrib", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "LastName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "CollectiveName", + pa.struct( + [ + pa.field( + "_Investigators", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:IDREF"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "AffiliationInfo", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Affiliation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "InvestigatorList", + pa.struct( + [ + pa.field("_ID", pa.string(), nullable=True, metadata={"xsd_type": "xs:ID"}), + pa.field( + "Investigator", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ValidYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "LastName", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ForeName", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Initials", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Suffix", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "AffiliationInfo", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Affiliation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Identifier", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Source", + pa.string(), + nullable=False, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "PublicationType", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_UI", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "Abstract", + pa.struct( + [ + pa.field( + "AbstractText", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Label", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_NlmCategory", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + pa.field( + "CopyrightInformation", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "Sections", + pa.struct( + [ + pa.field( + "Section", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "LocationLabel", + pa.struct( + [ + pa.field( + "_Type", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "SectionTitle", + pa.struct( + [ + pa.field( + "_book", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_part", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_sec", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "Section", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "LocationLabel", + pa.struct( + [ + pa.field( + "_Type", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + pa.field( + "SectionTitle", + pa.struct( + [ + pa.field( + "_book", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "_part", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "_sec", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=False, + ), + pa.field( + "Section", + pa.list_( + pa.field( + "item", pa.string(), nullable=True + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "KeywordList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Owner", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Keyword", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_MajorTopicYN", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ContributionDate", + pa.struct( + [ + pa.field("Year", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Month", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("Day", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("Season", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "DateRevised", + pa.struct( + [ + pa.field("Year", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Month", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("Day", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "GrantList", + pa.struct( + [ + pa.field("_CompleteYN", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "Grant", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "GrantID", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Acronym", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Agency", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Country", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "ItemList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_ListType", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Item", + pa.list_( + pa.field( + "item", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ReferenceList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Title", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "Reference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Citation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ReferenceList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Title", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Reference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "Citation", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={ + "xsd_type": "xs:string" + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ReferenceList", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + ), + pa.field( + "PubmedBookData", + pa.struct( + [ + pa.field( + "History", + pa.struct( + [ + pa.field( + "PubMedPubDate", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_PubStatus", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Year", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Month", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Day", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Hour", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Minute", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Second", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field("PublicationStatus", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "ArticleIdList", + pa.struct( + [ + pa.field( + "ArticleId", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_IdType", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "ObjectList", + pa.struct( + [ + pa.field( + "Object", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Param", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_Name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/examples/pubmed_book_article/maximal.xml b/tests/data/xsd/examples/pubmed_book_article/maximal.xml new file mode 100644 index 00000000..47126081 --- /dev/null +++ b/tests/data/xsd/examples/pubmed_book_article/maximal.xml @@ -0,0 +1,482 @@ + + + + + 38123456 + + 38123456 + 10.1038/s41586-024-07123-x + + + + Cold Spring Harbor Laboratory Press + Cold Spring Harbor (NY) + + Molecular Biology of the Cell, 7th Edition + + 2024 + 05 + 29 + + + 2025 + 01 + 16 + + + 2023 + 09 + 15 + + + + Morris + Charlotte + C + Jr + 0000-0002-9047-0481 + 0000-0001-6450-5815 + + The University of Manchester, Division of Population Health, Manchester, United Kingdom. + 0000-0003-2804-1181 + 0000-0002-0939-9842 + + + National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. + 0000-0001-5784-7403 + 0000-0002-1825-0097 + + + + International Cancer Genome Consortium + https://ror.org/027m9bs27 + https://ror.org/01ej9dk98 + + Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. + https://ror.org/00za53h95 + https://ror.org/03vek6s52 + + + Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. + 0000-0002-9047-0481 (11) + 0000-0001-6450-5815 (12) + + + + + + UK Primary Care Dementia Research Group + 0000-0003-2804-1181 (13) + 0000-0002-0939-9842 (14) + + MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom. + 0000-0001-5784-7403 (15) + 0000-0002-1825-0097 (16) + + + Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA. + https://ror.org/027m9bs27 (17) + https://ror.org/01ej9dk98 (18) + + + + Kontopantelis + Evangelos + E + III + https://ror.org/00za53h95 (19) + https://ror.org/03vek6s52 (20) + + Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA. + 0000-0002-9047-0481 (21) + 0000-0001-6450-5815 (22) + + + Department of Molecular Biology, Princeton University, Princeton, NJ, USA. + 0000-0003-2804-1181 (23) + 0000-0002-0939-9842 (24) + + + + + + Robinson + Louise + L + Sr + 0000-0001-5784-7403 (25) + 0000-0002-1825-0097 (26) + + The University of Manchester, Division of Population Health, Manchester, United Kingdom. (9) + https://ror.org/027m9bs27 (27) + https://ror.org/01ej9dk98 (28) + + + National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. (10) + https://ror.org/00za53h95 (29) + https://ror.org/03vek6s52 (30) + + + + Ashcroft + Darren + D + II + 0000-0002-9047-0481 (31) + 0000-0001-6450-5815 (32) + + Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. (11) + 0000-0003-2804-1181 (33) + 0000-0002-0939-9842 (34) + + + Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. (12) + 0000-0001-5784-7403 (35) + 0000-0002-1825-0097 (36) + + + + 629 + Part I: Introduction to the Cell + 7th + NCBI Bookshelf Classics + 978-0-8153-4432-2 + 978-0-8153-4464-3 + 10.1038/s41586-024-07123-x + S0028-0836(24)00123-4 + Internet + NBK21054 + + Chapter 1 + Part I + Structural basis of TP53 tetramerization and Ca2+ signaling + Bases structurales de la tetramerisation de TP53 + + 410 + 418 + 410-418 + + eng + fre + + + Blakeman + Thomas + T + IV + https://ror.org/027m9bs27 (37) + https://ror.org/01ej9dk98 (38) + + MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom. (13) + https://ror.org/00za53h95 (39) + https://ror.org/03vek6s52 (40) + + + Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA. (14) + 0000-0002-9047-0481 (41) + 0000-0001-6450-5815 (42) + + + + Asia-Pacific HIV Prevention Network + 0000-0003-2804-1181 (43) + 0000-0002-0939-9842 (44) + + Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA. (15) + 0000-0001-5784-7403 (45) + 0000-0002-1825-0097 (46) + + + Department of Molecular Biology, Princeton University, Princeton, NJ, USA. (16) + https://ror.org/027m9bs27 (47) + https://ror.org/01ej9dk98 (48) + + + + + + European TP53 Structural Biology Consortium + https://ror.org/00za53h95 (49) + https://ror.org/03vek6s52 (50) + + The University of Manchester, Division of Population Health, Manchester, United Kingdom. (17) + 0000-0002-9047-0481 (51) + 0000-0001-6450-5815 (52) + + + National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. (18) + 0000-0003-2804-1181 (53) + 0000-0002-0939-9842 (54) + + + + Tucker + Joseph + J + PhD + 0000-0001-5784-7403 (55) + 0000-0002-1825-0097 (56) + + Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. (19) + https://ror.org/027m9bs27 (57) + https://ror.org/01ej9dk98 (58) + + + Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. (20) + https://ror.org/00za53h95 (59) + https://ror.org/03vek6s52 (60) + + + + + + Mullens + Amy + A + MD + 0000-0002-9047-0481 (61) + 0000-0001-6450-5815 (62) + + MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom. (21) + 0000-0003-2804-1181 (63) + 0000-0002-0939-9842 (64) + + + Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA. (22) + 0000-0001-5784-7403 (65) + 0000-0002-1825-0097 (66) + + + + Morroni + Chelsea + CM + FRS + https://ror.org/027m9bs27 (67) + https://ror.org/01ej9dk98 (68) + + Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA. (23) + https://ror.org/00za53h95 (69) + https://ror.org/03vek6s52 (70) + + + Department of Molecular Biology, Princeton University, Princeton, NJ, USA. (24) + 0000-0002-9047-0481 (71) + 0000-0001-6450-5815 (72) + + + + Journal Article + Research Support, Non-U.S. Gov't + + The tumor suppressor p53 regulates transcription in response to <5 Gy stress and H2O2. + We undertook 20 semi-structured interviews with participants aged >18 years studying Ca2+ signaling & DNA repair and underlined text. + + Copyright (c) 2024 Springer Nature Limited. + + +
+ Chapter 1 + Chapter 1: Cells and Genomes & Evolution +
+ Section 1.1 + Section 1.1: The Universal Features of Cells on Earth +
+ Subsection 1.1.1 + Subsection 1.1.1: All Cells Store Their Hereditary Information in DNA +
+
+
+
+ Chapter 2 + Chapter 2: Cell Chemistry and Bioenergetics +
+ Section 2.1 + Section 2.1: The Chemical Components of a Cell & CO2 Fixation +
+ Subsection 2.1.1 + Subsection 2.1.1: Chemical Bonds and Ca2+ Ions +
+
+
+ + + p53 signaling + tetramerization + + + DNA damage response + apoptosis + + + 2022 + 11 + 04 + + + 2021 + 03 + 21 + + + + R01-CA043460 + CA + NCI NIH HHS + United States + + + MR/T021975/1 + MRC + Medical Research Council + United Kingdom + + + + G1/S DNA damage checkpoint + G2/M mitotic spindle checkpoint + + + Ataxia-telangiectasia mutated (ATM) kinase activation + CHK2-mediated phosphorylation of Ser-20 + + + Nature + + Vogelstein B, Lane D, Levine AJ. Surfing the p53 network. Nature. 2000;408(6810):307-310. + + PMC10987654 + S0028-0836(24)00123-4 + + + + Cho Y, Gorina S, Jeffrey PD, Pavletich NP. Crystal structure of a p53 tumor suppressor-DNA complex. Science. 1994;265(5170):346-355. + + 11099028 + 10.1038/35042675 + + + + Primary Literature + + Baker SJ, Fearon ER, Nigro JM, et al. Chromosome 17 deletions and p53 gene mutations in colorectal carcinomas. Science. 1989;244(4901):217-221. + + 7585406 + 10.1126/science.7878469 + + + + Momand J, Zambetti GP, Olson DC, George D, Levine AJ. The mdm-2 oncogene product forms a complex with the p53 protein. Cell. 1992;69(7):1237-1245. + + 2694717 + 10.1016/0092-8674(89)90984-7 + + + + Structural Biology References + + Kussie PH, Gorina S, Marechal V, et al. Structure of the MDM2 oncoprotein bound to the p53 tumor suppressor transactivation domain. Science. 1996;274(5289):948-953. + + 8183552 + 10.1126/science.8183552 + + + + Haupt Y, Maya R, Kazaz A, Oren M. Mdm2 promotes the rapid degradation of p53. Nature. 1997;387(6630):296-299. + + 9367981 + 10.1038/387296a0 + + + + + + + Clinical &amp; Genetic Studies + + Cohen GN, Barbe V, Flament D, et al. An integrated analysis of the genome of Pyrococcus abyssi. Mol Microbiol. 2003;47(6):1495-1512. + + 10430560 + 10.1093/genetics/152.4.1299 + + + + Erkel C, Kube M, Reinhardt R, Liesack W. Genome of rice cluster I archaea. Science. 2006;313(5785):370-372. + + 12622808 + 10.1046/j.1365-2958.2003.03381.x + + + + Historical Background + + Anderson IJ, Sieprawska-Lupa M, Lapidus A, et al. Complete genome sequence of Methanoculleus marisnigri JR1. Stand Genomic Sci. 2009;1(2):189-196. + + 16857943 + 10.1126/science.1127062 + + + + Gao J, Wang J. Re-annotation of two hyperthermophilic archaea Pyrococcus abyssi & P. furiosus. Curr Microbiol. 2012;64(2):118-129. + + 21304656 + 10.4056/sigs.32535 + + + + Methodological References + + Morris C, Robinson L, Ashcroft D, et al. Primary care experiences of people with dementia. Br J Gen Pract. 2025;75(751):e102-e110. + + 22057919 + 10.1007/s00284-011-0035-x + + + + Tucker J, Mullens A, Morroni C, Ong J. The HIV response at an inflection point: resilience &amp; equity. Sex Health. 2026;23(1):SH25236. + + 31234567 + 10.1016/j.cell.2019.05.012 + + + + + + + + + + 2020 + 07 + 12 + 14 + 30 + 00 + + + 2019 + 02 + 08 + 09 + 15 + 25 + + + ppublish + + 38123456 (27) + 10.1038/s41586-024-07123-x (28) + + + + supp_table_1.pdf + Crystallographic data collection and refinement statistics + + + GSE245678 + NCBI Gene Expression Omnibus + + + + + diff --git a/tests/data/xsd/examples/pubmed_book_article/maximal_expected.json b/tests/data/xsd/examples/pubmed_book_article/maximal_expected.json new file mode 100644 index 00000000..28ff4c61 --- /dev/null +++ b/tests/data/xsd/examples/pubmed_book_article/maximal_expected.json @@ -0,0 +1,1069 @@ +[ + { + "BookDocument": { + "PMID": { + "_Version": "1", + "#text": "38123456" + }, + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pubmed", + "#text": "38123456" + }, + { + "_IdType": "doi", + "#text": "10.1038/s41586-024-07123-x" + } + ] + }, + "Book": { + "Publisher": { + "PublisherName": "Cold Spring Harbor Laboratory Press", + "PublisherLocation": "Cold Spring Harbor (NY)" + }, + "BookTitle": { + "_book": "nbk21054", + "_part": "part1", + "_sec": "sec1.1", + "#text": "Molecular Biology of the Cell, 7th Edition" + }, + "PubDate": { + "Year": "2024", + "Month": "05", + "Day": "29" + }, + "BeginningDate": { + "Year": "2025", + "Month": "01", + "Day": "16" + }, + "EndingDate": { + "Year": "2023", + "Month": "09", + "Day": "15" + }, + "AuthorList": [ + { + "_CompleteYN": "Y", + "_Type": "authors", + "Author": [ + { + "_ValidYN": "Y", + "_EqualContrib": "Y", + "LastName": "Morris", + "ForeName": "Charlotte", + "Initials": "C", + "Suffix": "Jr", + "Identifier": [ + { + "_Source": "ORCID", + "#text": "0000-0002-9047-0481" + }, + { + "_Source": "ISNI", + "#text": "0000-0001-6450-5815" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "The University of Manchester, Division of Population Health, Manchester, United Kingdom.", + "Identifier": [ + { + "_Source": "VIAF", + "#text": "0000-0003-2804-1181" + }, + { + "_Source": "GND", + "#text": "0000-0002-0939-9842" + } + ] + }, + { + "Affiliation": "National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom.", + "Identifier": [ + { + "_Source": "RINGGOLD", + "#text": "0000-0001-5784-7403" + }, + { + "_Source": "GRID", + "#text": "0000-0002-1825-0097" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "_EqualContrib": "N", + "CollectiveName": { + "_Investigators": "INV_1", + "#text": "International Cancer Genome Consortium" + }, + "Identifier": [ + { + "_Source": "ROR", + "#text": "https://ror.org/027m9bs27" + }, + { + "_Source": "ORCID (8)", + "#text": "https://ror.org/01ej9dk98" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom.", + "Identifier": [ + { + "_Source": "ISNI (9)", + "#text": "https://ror.org/00za53h95" + }, + { + "_Source": "VIAF (10)", + "#text": "https://ror.org/03vek6s52" + } + ] + }, + { + "Affiliation": "Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA.", + "Identifier": [ + { + "_Source": "GND (11)", + "#text": "0000-0002-9047-0481 (11)" + }, + { + "_Source": "RINGGOLD (12)", + "#text": "0000-0001-6450-5815 (12)" + } + ] + } + ] + } + ] + }, + { + "_CompleteYN": "N", + "_Type": "editors", + "Author": [ + { + "_ValidYN": "Y", + "_EqualContrib": "Y", + "CollectiveName": { + "_Investigators": "INV_1", + "#text": "UK Primary Care Dementia Research Group" + }, + "Identifier": [ + { + "_Source": "GRID (13)", + "#text": "0000-0003-2804-1181 (13)" + }, + { + "_Source": "ROR (14)", + "#text": "0000-0002-0939-9842 (14)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom.", + "Identifier": [ + { + "_Source": "ORCID (15)", + "#text": "0000-0001-5784-7403 (15)" + }, + { + "_Source": "ISNI (16)", + "#text": "0000-0002-1825-0097 (16)" + } + ] + }, + { + "Affiliation": "Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA.", + "Identifier": [ + { + "_Source": "VIAF (17)", + "#text": "https://ror.org/027m9bs27 (17)" + }, + { + "_Source": "GND (18)", + "#text": "https://ror.org/01ej9dk98 (18)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "_EqualContrib": "N", + "LastName": "Kontopantelis", + "ForeName": "Evangelos", + "Initials": "E", + "Suffix": "III", + "Identifier": [ + { + "_Source": "RINGGOLD (19)", + "#text": "https://ror.org/00za53h95 (19)" + }, + { + "_Source": "GRID (20)", + "#text": "https://ror.org/03vek6s52 (20)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA.", + "Identifier": [ + { + "_Source": "ROR (21)", + "#text": "0000-0002-9047-0481 (21)" + }, + { + "_Source": "ORCID (22)", + "#text": "0000-0001-6450-5815 (22)" + } + ] + }, + { + "Affiliation": "Department of Molecular Biology, Princeton University, Princeton, NJ, USA.", + "Identifier": [ + { + "_Source": "ISNI (23)", + "#text": "0000-0003-2804-1181 (23)" + }, + { + "_Source": "VIAF (24)", + "#text": "0000-0002-0939-9842 (24)" + } + ] + } + ] + } + ] + } + ], + "InvestigatorList": { + "_ID": "INV_1", + "Investigator": [ + { + "_ValidYN": "Y", + "LastName": "Robinson", + "ForeName": "Louise", + "Initials": "L", + "Suffix": "Sr", + "Identifier": [ + { + "_Source": "GND (25)", + "#text": "0000-0001-5784-7403 (25)" + }, + { + "_Source": "RINGGOLD (26)", + "#text": "0000-0002-1825-0097 (26)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "The University of Manchester, Division of Population Health, Manchester, United Kingdom. (9)", + "Identifier": [ + { + "_Source": "GRID (27)", + "#text": "https://ror.org/027m9bs27 (27)" + }, + { + "_Source": "ROR (28)", + "#text": "https://ror.org/01ej9dk98 (28)" + } + ] + }, + { + "Affiliation": "National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. (10)", + "Identifier": [ + { + "_Source": "ORCID (29)", + "#text": "https://ror.org/00za53h95 (29)" + }, + { + "_Source": "ISNI (30)", + "#text": "https://ror.org/03vek6s52 (30)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "LastName": "Ashcroft", + "ForeName": "Darren", + "Initials": "D", + "Suffix": "II", + "Identifier": [ + { + "_Source": "VIAF (31)", + "#text": "0000-0002-9047-0481 (31)" + }, + { + "_Source": "GND (32)", + "#text": "0000-0001-6450-5815 (32)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. (11)", + "Identifier": [ + { + "_Source": "RINGGOLD (33)", + "#text": "0000-0003-2804-1181 (33)" + }, + { + "_Source": "GRID (34)", + "#text": "0000-0002-0939-9842 (34)" + } + ] + }, + { + "Affiliation": "Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. (12)", + "Identifier": [ + { + "_Source": "ROR (35)", + "#text": "0000-0001-5784-7403 (35)" + }, + { + "_Source": "ORCID (36)", + "#text": "0000-0002-1825-0097 (36)" + } + ] + } + ] + } + ] + }, + "Volume": "629", + "VolumeTitle": "Part I: Introduction to the Cell", + "Edition": "7th", + "CollectionTitle": { + "_book": "nbk21054", + "_part": "seriesA", + "_sec": "vol1", + "#text": "NCBI Bookshelf Classics" + }, + "Isbn": [ + "978-0-8153-4432-2", + "978-0-8153-4464-3" + ], + "ELocationID": [ + { + "_EIdType": "doi", + "_ValidYN": "Y", + "#text": "10.1038/s41586-024-07123-x" + }, + { + "_EIdType": "pii", + "_ValidYN": "N", + "#text": "S0028-0836(24)00123-4" + } + ], + "Medium": "Internet", + "ReportNumber": "NBK21054" + }, + "LocationLabel": [ + { + "_Type": "chapter", + "#text": "Chapter 1" + }, + { + "_Type": "section", + "#text": "Part I" + } + ], + "ArticleTitle": { + "_book": "nbk21054", + "_part": "part1", + "_sec": "sec1.1", + "#text": "Structural basis of TP53 tetramerization and Ca2+ signaling" + }, + "VernacularTitle": "Bases structurales de la tetramerisation de TP53", + "Pagination": { + "StartPage": "410", + "EndPage": "418", + "MedlinePgn": "410-418" + }, + "Language": [ + "eng", + "fre" + ], + "AuthorList": [ + { + "_CompleteYN": "Y", + "_Type": "authors", + "Author": [ + { + "_ValidYN": "Y", + "_EqualContrib": "Y", + "LastName": "Blakeman", + "ForeName": "Thomas", + "Initials": "T", + "Suffix": "IV", + "Identifier": [ + { + "_Source": "ISNI (37)", + "#text": "https://ror.org/027m9bs27 (37)" + }, + { + "_Source": "VIAF (38)", + "#text": "https://ror.org/01ej9dk98 (38)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom. (13)", + "Identifier": [ + { + "_Source": "GND (39)", + "#text": "https://ror.org/00za53h95 (39)" + }, + { + "_Source": "RINGGOLD (40)", + "#text": "https://ror.org/03vek6s52 (40)" + } + ] + }, + { + "Affiliation": "Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA. (14)", + "Identifier": [ + { + "_Source": "GRID (41)", + "#text": "0000-0002-9047-0481 (41)" + }, + { + "_Source": "ROR (42)", + "#text": "0000-0001-6450-5815 (42)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "_EqualContrib": "N", + "CollectiveName": { + "_Investigators": "INV_1", + "#text": "Asia-Pacific HIV Prevention Network" + }, + "Identifier": [ + { + "_Source": "ORCID (43)", + "#text": "0000-0003-2804-1181 (43)" + }, + { + "_Source": "ISNI (44)", + "#text": "0000-0002-0939-9842 (44)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA. (15)", + "Identifier": [ + { + "_Source": "VIAF (45)", + "#text": "0000-0001-5784-7403 (45)" + }, + { + "_Source": "GND (46)", + "#text": "0000-0002-1825-0097 (46)" + } + ] + }, + { + "Affiliation": "Department of Molecular Biology, Princeton University, Princeton, NJ, USA. (16)", + "Identifier": [ + { + "_Source": "RINGGOLD (47)", + "#text": "https://ror.org/027m9bs27 (47)" + }, + { + "_Source": "GRID (48)", + "#text": "https://ror.org/01ej9dk98 (48)" + } + ] + } + ] + } + ] + }, + { + "_CompleteYN": "N", + "_Type": "editors", + "Author": [ + { + "_ValidYN": "Y", + "_EqualContrib": "Y", + "CollectiveName": { + "_Investigators": "INV_1", + "#text": "European TP53 Structural Biology Consortium" + }, + "Identifier": [ + { + "_Source": "ROR (49)", + "#text": "https://ror.org/00za53h95 (49)" + }, + { + "_Source": "ORCID (50)", + "#text": "https://ror.org/03vek6s52 (50)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "The University of Manchester, Division of Population Health, Manchester, United Kingdom. (17)", + "Identifier": [ + { + "_Source": "ISNI (51)", + "#text": "0000-0002-9047-0481 (51)" + }, + { + "_Source": "VIAF (52)", + "#text": "0000-0001-6450-5815 (52)" + } + ] + }, + { + "Affiliation": "National Institute for Health and Care Research School for Primary Care Research, Manchester, United Kingdom. (18)", + "Identifier": [ + { + "_Source": "GND (53)", + "#text": "0000-0003-2804-1181 (53)" + }, + { + "_Source": "RINGGOLD (54)", + "#text": "0000-0002-0939-9842 (54)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "_EqualContrib": "N", + "LastName": "Tucker", + "ForeName": "Joseph", + "Initials": "J", + "Suffix": "PhD", + "Identifier": [ + { + "_Source": "GRID (55)", + "#text": "0000-0001-5784-7403 (55)" + }, + { + "_Source": "ROR (56)", + "#text": "0000-0002-1825-0097 (56)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Newcastle University Faculty of Medical Sciences, Population Health Sciences Institute, Newcastle upon Tyne, United Kingdom. (19)", + "Identifier": [ + { + "_Source": "ORCID (57)", + "#text": "https://ror.org/027m9bs27 (57)" + }, + { + "_Source": "ISNI (58)", + "#text": "https://ror.org/01ej9dk98 (58)" + } + ] + }, + { + "Affiliation": "Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University, Baltimore, MD, USA. (20)", + "Identifier": [ + { + "_Source": "VIAF (59)", + "#text": "https://ror.org/00za53h95 (59)" + }, + { + "_Source": "GND (60)", + "#text": "https://ror.org/03vek6s52 (60)" + } + ] + } + ] + } + ] + } + ], + "InvestigatorList": { + "_ID": "INV_2", + "Investigator": [ + { + "_ValidYN": "Y", + "LastName": "Mullens", + "ForeName": "Amy", + "Initials": "A", + "Suffix": "MD", + "Identifier": [ + { + "_Source": "RINGGOLD (61)", + "#text": "0000-0002-9047-0481 (61)" + }, + { + "_Source": "GRID (62)", + "#text": "0000-0001-6450-5815 (62)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "MRC Laboratory of Molecular Biology, Francis Crick Avenue, Cambridge, United Kingdom. (21)", + "Identifier": [ + { + "_Source": "ROR (63)", + "#text": "0000-0003-2804-1181 (63)" + }, + { + "_Source": "ORCID (64)", + "#text": "0000-0002-0939-9842 (64)" + } + ] + }, + { + "Affiliation": "Cold Spring Harbor Laboratory, One Bungtown Road, Cold Spring Harbor, NY, USA. (22)", + "Identifier": [ + { + "_Source": "ISNI (65)", + "#text": "0000-0001-5784-7403 (65)" + }, + { + "_Source": "VIAF (66)", + "#text": "0000-0002-1825-0097 (66)" + } + ] + } + ] + }, + { + "_ValidYN": "N", + "LastName": "Morroni", + "ForeName": "Chelsea", + "Initials": "CM", + "Suffix": "FRS", + "Identifier": [ + { + "_Source": "GND (67)", + "#text": "https://ror.org/027m9bs27 (67)" + }, + { + "_Source": "RINGGOLD (68)", + "#text": "https://ror.org/01ej9dk98 (68)" + } + ], + "AffiliationInfo": [ + { + "Affiliation": "Institute for Advanced Study, Einstein Drive, Princeton, NJ, USA. (23)", + "Identifier": [ + { + "_Source": "GRID (69)", + "#text": "https://ror.org/00za53h95 (69)" + }, + { + "_Source": "ROR (70)", + "#text": "https://ror.org/03vek6s52 (70)" + } + ] + }, + { + "Affiliation": "Department of Molecular Biology, Princeton University, Princeton, NJ, USA. (24)", + "Identifier": [ + { + "_Source": "ORCID (71)", + "#text": "0000-0002-9047-0481 (71)" + }, + { + "_Source": "ISNI (72)", + "#text": "0000-0001-6450-5815 (72)" + } + ] + } + ] + } + ] + }, + "PublicationType": [ + { + "_UI": "D016428", + "#text": "Journal Article" + }, + { + "_UI": "D013485", + "#text": "Research Support, Non-U.S. Gov't" + } + ], + "Abstract": { + "AbstractText": [ + { + "_Label": "BACKGROUND", + "_NlmCategory": "BACKGROUND", + "#text": "The tumor suppressor p53 regulates transcription in response to <5 Gy stress and H2O2." + }, + { + "_Label": "METHODS", + "_NlmCategory": "METHODS", + "#text": "We undertook 20 semi-structured interviews with participants aged >18 years studying Ca2+ signaling & DNA repair and underlined text." + }, + { + "_Label": "UNLABELLED", + "_NlmCategory": "UNASSIGNED" + } + ], + "CopyrightInformation": "Copyright (c) 2024 Springer Nature Limited." + }, + "Sections": { + "Section": [ + { + "LocationLabel": { + "_Type": "part", + "#text": "Chapter 1" + }, + "SectionTitle": { + "_book": "nbk21054", + "_part": "part1", + "_sec": "sec1", + "#text": "Chapter 1: Cells and Genomes & Evolution" + }, + "Section": [ + { + "LocationLabel": { + "_Type": "appendix", + "#text": "Section 1.1" + }, + "SectionTitle": { + "_book": "nbk21055", + "_part": "part1", + "_sec": "sec1.1", + "#text": "Section 1.1: The Universal Features of Cells on Earth" + }, + "Section": [ + "{\"LocationLabel\": {\"_Type\": \"table\", \"#text\": \"Subsection 1.1.1\"}, \"SectionTitle\": {\"_book\": \"nbk21056\", \"_part\": \"part1\", \"_sec\": \"sec1.1.1\", \"#text\": \"Subsection 1.1.1: All Cells Store Their Hereditary Information in DNA\"}}" + ] + } + ] + }, + { + "LocationLabel": { + "_Type": "figure", + "#text": "Chapter 2" + }, + "SectionTitle": { + "_book": "nbk21057", + "_part": "part2", + "_sec": "sec2", + "#text": "Chapter 2: Cell Chemistry and Bioenergetics" + }, + "Section": [ + { + "LocationLabel": { + "_Type": "box", + "#text": "Section 2.1" + }, + "SectionTitle": { + "_book": "nbk21058", + "_part": "part2", + "_sec": "sec2.1", + "#text": "Section 2.1: The Chemical Components of a Cell & CO2 Fixation" + }, + "Section": [ + "{\"LocationLabel\": {\"_Type\": \"chapter\", \"#text\": \"Subsection 2.1.1\"}, \"SectionTitle\": {\"_book\": \"nbk21059\", \"_part\": \"part2\", \"_sec\": \"sec2.1.1\", \"#text\": \"Subsection 2.1.1: Chemical Bonds and Ca2+ Ions\"}}" + ] + } + ] + } + ] + }, + "KeywordList": [ + { + "_Owner": "NOTNLM", + "Keyword": [ + { + "_MajorTopicYN": "Y", + "#text": "p53 signaling" + }, + { + "_MajorTopicYN": "N", + "#text": "tetramerization" + } + ] + }, + { + "_Owner": "NLM", + "Keyword": [ + { + "_MajorTopicYN": "Y", + "#text": "DNA damage response" + }, + { + "_MajorTopicYN": "N", + "#text": "apoptosis" + } + ] + } + ], + "ContributionDate": { + "Year": "2022", + "Month": "11", + "Day": "04" + }, + "DateRevised": { + "Year": "2021", + "Month": "03", + "Day": "21" + }, + "GrantList": { + "_CompleteYN": "Y", + "Grant": [ + { + "GrantID": "R01-CA043460", + "Acronym": "CA", + "Agency": "NCI NIH HHS", + "Country": "United States" + }, + { + "GrantID": "MR/T021975/1", + "Acronym": "MRC", + "Agency": "Medical Research Council", + "Country": "United Kingdom" + } + ] + }, + "ItemList": [ + { + "_ListType": "list", + "Item": [ + "G1/S DNA damage checkpoint", + "G2/M mitotic spindle checkpoint" + ] + }, + { + "_ListType": "ordered", + "Item": [ + "Ataxia-telangiectasia mutated (ATM) kinase activation", + "CHK2-mediated phosphorylation of Ser-20" + ] + } + ], + "ReferenceList": [ + { + "Title": "Nature", + "Reference": [ + { + "Citation": "Vogelstein B, Lane D, Levine AJ. Surfing the p53 network. Nature. 2000;408(6810):307-310.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmc", + "#text": "PMC10987654" + }, + { + "_IdType": "pii", + "#text": "S0028-0836(24)00123-4" + } + ] + } + }, + { + "Citation": "Cho Y, Gorina S, Jeffrey PD, Pavletich NP. Crystal structure of a p53 tumor suppressor-DNA complex. Science. 1994;265(5170):346-355.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "mid", + "#text": "11099028" + }, + { + "_IdType": "pmcid", + "#text": "10.1038/35042675" + } + ] + } + } + ], + "ReferenceList": [ + { + "Title": "Primary Literature", + "Reference": [ + { + "Citation": "Baker SJ, Fearon ER, Nigro JM, et al. Chromosome 17 deletions and p53 gene mutations in colorectal carcinomas. Science. 1989;244(4901):217-221.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmcpid", + "#text": "7585406" + }, + { + "_IdType": "pmpid", + "#text": "10.1126/science.7878469" + } + ] + } + }, + { + "Citation": "Momand J, Zambetti GP, Olson DC, George D, Levine AJ. The mdm-2 oncogene product forms a complex with the p53 protein. Cell. 1992;69(7):1237-1245.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "sici", + "#text": "2694717" + }, + { + "_IdType": "medline", + "#text": "10.1016/0092-8674(89)90984-7" + } + ] + } + } + ], + "ReferenceList": [ + "{\"Title\": \"Structural Biology References\", \"Reference\": [{\"Citation\": \"Kussie PH, Gorina S, Marechal V, et al. Structure of the MDM2 oncoprotein bound to the p53 tumor suppressor transactivation domain. Science. 1996;274(5289):948-953.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pmcbook\", \"#text\": \"8183552\"}, {\"_IdType\": \"bookaccession\", \"#text\": \"10.1126/science.8183552\"}]}}, {\"Citation\": \"Haupt Y, Maya R, Kazaz A, Oren M. Mdm2 promotes the rapid degradation of p53. Nature. 1997;387(6630):296-299.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pubmed\", \"#text\": \"9367981\"}, {\"_IdType\": \"doi\", \"#text\": \"10.1038/387296a0\"}]}}]}" + ] + } + ] + }, + { + "Title": "Clinical &amp; Genetic Studies", + "Reference": [ + { + "Citation": "Cohen GN, Barbe V, Flament D, et al. An integrated analysis of the genome of Pyrococcus abyssi. Mol Microbiol. 2003;47(6):1495-1512.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmc", + "#text": "10430560" + }, + { + "_IdType": "pii", + "#text": "10.1093/genetics/152.4.1299" + } + ] + } + }, + { + "Citation": "Erkel C, Kube M, Reinhardt R, Liesack W. Genome of rice cluster I archaea. Science. 2006;313(5785):370-372.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "mid", + "#text": "12622808" + }, + { + "_IdType": "pmcid", + "#text": "10.1046/j.1365-2958.2003.03381.x" + } + ] + } + } + ], + "ReferenceList": [ + { + "Title": "Historical Background", + "Reference": [ + { + "Citation": "Anderson IJ, Sieprawska-Lupa M, Lapidus A, et al. Complete genome sequence of Methanoculleus marisnigri JR1. Stand Genomic Sci. 2009;1(2):189-196.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmcpid", + "#text": "16857943" + }, + { + "_IdType": "pmpid", + "#text": "10.1126/science.1127062" + } + ] + } + }, + { + "Citation": "Gao J, Wang J. Re-annotation of two hyperthermophilic archaea Pyrococcus abyssi & P. furiosus. Curr Microbiol. 2012;64(2):118-129.", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "sici", + "#text": "21304656" + }, + { + "_IdType": "medline", + "#text": "10.4056/sigs.32535" + } + ] + } + } + ], + "ReferenceList": [ + "{\"Title\": \"Methodological References\", \"Reference\": [{\"Citation\": \"Morris C, Robinson L, Ashcroft D, et al. Primary care experiences of people with dementia. Br J Gen Pract. 2025;75(751):e102-e110.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pmcbook\", \"#text\": \"22057919\"}, {\"_IdType\": \"bookaccession\", \"#text\": \"10.1007/s00284-011-0035-x\"}]}}, {\"Citation\": \"Tucker J, Mullens A, Morroni C, Ong J. The HIV response at an inflection point: resilience &amp; equity. Sex Health. 2026;23(1):SH25236.\", \"ArticleIdList\": {\"ArticleId\": [{\"_IdType\": \"pubmed\", \"#text\": \"31234567\"}, {\"_IdType\": \"doi\", \"#text\": \"10.1016/j.cell.2019.05.012\"}]}}]}" + ] + } + ] + } + ] + }, + "PubmedBookData": { + "History": { + "PubMedPubDate": [ + { + "_PubStatus": "received", + "Year": "2020", + "Month": "07", + "Day": "12", + "Hour": "14", + "Minute": "30", + "Second": "00" + }, + { + "_PubStatus": "revised", + "Year": "2019", + "Month": "02", + "Day": "08", + "Hour": "09", + "Minute": "15", + "Second": "25" + } + ] + }, + "PublicationStatus": "ppublish", + "ArticleIdList": { + "ArticleId": [ + { + "_IdType": "pmc", + "#text": "38123456 (27)" + }, + { + "_IdType": "pii", + "#text": "10.1038/s41586-024-07123-x (28)" + } + ] + }, + "ObjectList": { + "Object": [ + { + "_Type": "SupplementaryMaterial", + "Param": [ + { + "_Name": "id", + "#text": "supp_table_1.pdf" + }, + { + "_Name": "caption", + "#text": "Crystallographic data collection and refinement statistics" + } + ] + }, + { + "_Type": "Dataset", + "Param": [ + { + "_Name": "accession", + "#text": "GSE245678" + }, + { + "_Name": "repository", + "#text": "NCBI Gene Expression Omnibus" + } + ] + } + ] + } + } + } +] diff --git a/tests/data/xsd/examples/pubmed_book_article/minimal.xml b/tests/data/xsd/examples/pubmed_book_article/minimal.xml new file mode 100644 index 00000000..e4688d09 --- /dev/null +++ b/tests/data/xsd/examples/pubmed_book_article/minimal.xml @@ -0,0 +1,20 @@ + + + + + 38123456 + + 38123456 + + + + Cold Spring Harbor Laboratory Press + + Molecular Biology of the Cell, 7th Edition + + Sample_MedlineDate_1 + + + + + diff --git a/tests/data/xsd/examples/pubmed_book_article/minimal_expected.json b/tests/data/xsd/examples/pubmed_book_article/minimal_expected.json new file mode 100644 index 00000000..77d6ab90 --- /dev/null +++ b/tests/data/xsd/examples/pubmed_book_article/minimal_expected.json @@ -0,0 +1,28 @@ +[ + { + "BookDocument": { + "PMID": { + "_Version": "1", + "#text": "38123456" + }, + "ArticleIdList": { + "ArticleId": [ + { + "#text": "38123456" + } + ] + }, + "Book": { + "Publisher": { + "PublisherName": "Cold Spring Harbor Laboratory Press" + }, + "BookTitle": { + "#text": "Molecular Biology of the Cell, 7th Edition" + }, + "PubDate": { + "MedlineDate": "Sample_MedlineDate_1" + } + } + } + } +] diff --git a/tests/data/xsd/examples/uniprot/expected_schema.py b/tests/data/xsd/examples/uniprot/expected_schema.py new file mode 100644 index 00000000..ceb7a289 --- /dev/null +++ b/tests/data/xsd/examples/uniprot/expected_schema.py @@ -0,0 +1,4346 @@ +"""Expected PyArrow schema for uniprot (xml_tag='entry').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_dataset", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("_created", pa.date32(), nullable=False, metadata={"xsd_type": "xs:date"}), + pa.field("_modified", pa.date32(), nullable=False, metadata={"xsd_type": "xs:date"}), + pa.field("_version", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field( + "accession", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + ), + pa.field( + "name", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + ), + pa.field( + "protein", + pa.struct( + [ + pa.field( + "recommendedName", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=False, + ), + pa.field( + "shortName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "alternativeName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "shortName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "submittedName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "allergenName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "biotechName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "cdAntigenName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"} + ), + ), + nullable=True, + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "innName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"} + ), + ), + nullable=True, + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "domain", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "recommendedName", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "shortName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "alternativeName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "shortName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={ + "xsd_type": "xs:int" + }, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={ + "xsd_type": "xs:int" + }, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "submittedName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={ + "xsd_type": "xs:int" + }, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "allergenName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "biotechName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "cdAntigenName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "innName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": 'Describes names of "domains". Equivalent to the flat file DE-line Includes: section.' + }, + ), + pa.field( + "component", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "recommendedName", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "shortName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "alternativeName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "shortName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={ + "xsd_type": "xs:int" + }, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={ + "xsd_type": "xs:int" + }, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "submittedName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "fullName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "ecNumber", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={ + "xsd_type": "xs:int" + }, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "allergenName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "biotechName", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "cdAntigenName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "innName", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes names of processed products. Equivalent to the flat file DE-line Contains: section." + }, + ), + ] + ), + nullable=False, + metadata={ + "comment": "Describes the names for the protein and parts thereof. Equivalent to the flat file DE-line." + }, + ), + pa.field( + "gene", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "name", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes different types of gene designations. Equivalent to the flat file GN-line." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes a gene. Equivalent to the flat file GN-line."}, + ), + pa.field( + "organism", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field( + "name", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes the names of the source organism. Equivalent to the flat file OS-line." + }, + ), + pa.field( + "dbReference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"} + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={"comment": "Describes a molecule by name or unique identifier."}, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes a cross-reference to the NCBI taxonomy database. Equivalent to the flat file OX-line." + }, + ), + pa.field( + "lineage", + pa.struct( + [ + pa.field( + "taxon", + pa.list_( + pa.field( + "item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the lineage of the source organism. Equivalent to the flat file OC-line." + }, + ), + ] + ), + nullable=False, + metadata={"comment": "Describes the source organism."}, + ), + pa.field( + "organismHost", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field( + "name", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes the names of the source organism. Equivalent to the flat file OS-line." + }, + ), + pa.field( + "dbReference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes a cross-reference to the NCBI taxonomy database. Equivalent to the flat file OX-line." + }, + ), + pa.field( + "lineage", + pa.struct( + [ + pa.field( + "taxon", + pa.list_( + pa.field( + "item", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the lineage of the source organism. Equivalent to the flat file OC-line." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes the source organism."}, + ), + pa.field( + "geneLocation", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field( + "name", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Indicates whether the name of a plasmid is known or unknown."}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes non-nuclear gene locations (organelles and plasmids). Equivalent to the flat file OG-line." + }, + ), + pa.field( + "reference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field("_key", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "citation", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the type of a citation.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_date", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:date | xs:gYearMonth | xs:gYear"}, + ), + pa.field( + "_name", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the name of an (online) journal or book.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_volume", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the volume of a journal or book.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_first", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the first page of an article.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_last", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the last page of an article.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_publisher", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the publisher of a book.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_city", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the city where a book was published.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_db", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the database name of submissions.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_number", + pa.string(), + nullable=True, + metadata={"comment": "Describes a patent number.", "xsd_type": "xs:string"}, + ), + pa.field( + "_institute", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the institute where a thesis was made.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_country", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the country where a thesis was made.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "title", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the title of a citation. Equivalent to the flat file RT-line.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "editorList", + pa.struct( + [ + pa.field( + "consortium", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes the authors of a citation when these are represented by a consortium. Equivalent to the flat file RG-line." + }, + ), + pa.field( + "person", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the editors of a book (only used for books). Equivalent to part of the flat file RL-line of books." + }, + ), + pa.field( + "authorList", + pa.struct( + [ + pa.field( + "consortium", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes the authors of a citation when these are represented by a consortium. Equivalent to the flat file RG-line." + }, + ), + pa.field( + "person", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the authors of a citation. Equivalent to the flat file RA-line." + }, + ), + pa.field( + "locator", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the location (URL) of an online journal article. No flat file equivalent.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "dbReference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes cross-references to bibliography databases (MEDLINE, PubMed, AGRICOLA) or other online resources (DOI). Equivalent to the flat file RX-line." + }, + ), + ] + ), + nullable=False, + metadata={ + "comment": "Describes different types of citations. Equivalent to the flat file RX-, RG-, RA-, RT- and RL-lines." + }, + ), + pa.field( + "scope", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + metadata={ + "comment": "Describes the scope of a citation. Equivalent to the flat file RP-line." + }, + ), + pa.field( + "source", + pa.struct( + [ + pa.field( + "strain", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "plasmid", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "transposon", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "tissue", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the source of the sequence according to the citation. Equivalent to the flat file RC-line." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes a citation and a summary of its content. Equivalent to the flat file RN-, RP-, RC-, RX-, RG-, RA-, RT- and RL-lines." + }, + ), + pa.field( + "comment", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=True, + metadata={ + "comment": 'Describes the type of a general annotation. Equivalent to the flat file CC comment topics (except for "DATABASE" which is translated to "online information").', + "xsd_type": "xs:string", + }, + ), + pa.field( + "_locationType", + pa.string(), + nullable=True, + metadata={ + "comment": 'Describes the type of sequence location in \'RNA editing\' annotations. Common values are "Not_applicable" and "Undetermined".', + "xsd_type": "xs:string", + }, + ), + pa.field( + "_name", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes an optional name for an 'online information'.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_mass", + pa.float64(), + nullable=True, + metadata={ + "comment": "Describes the molecular mass in 'mass spectrometry' annotations.", + "xsd_type": "xs:float", + }, + ), + pa.field( + "_error", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the error of the mass measurement in 'mass spectrometry' annotations.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_method", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the experimental method in 'mass spectrometry' annotations.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field("_id", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + metadata={"comment": "Describes a molecule by name or unique identifier."}, + ), + pa.field( + "absorption", + pa.struct( + [ + pa.field( + "max", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "kinetics", + pa.struct( + [ + pa.field( + "KM", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "Vmax", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "phDependence", + pa.struct( + [ + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "redoxPotential", + pa.struct( + [ + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "temperatureDependence", + pa.struct( + [ + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "reaction", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"} + ), + ), + nullable=True, + ), + pa.field( + "text", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "dbReference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + metadata={"comment": "Describes a chemical reaction."}, + ), + pa.field( + "physiologicalReaction", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_direction", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes a physiological reaction."}, + ), + pa.field( + "cofactor", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "name", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes a cofactor."}, + ), + pa.field( + "subcellularLocation", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "location", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + pa.field( + "topology", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "orientation", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes the subcellular location and optionally the topology and orientation of a molecule." + }, + ), + pa.field( + "conflict", + pa.struct( + [ + pa.field( + "_type", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "_ref", + pa.string(), + nullable=True, + metadata={ + "comment": "Refers to the 'key' attribute of a 'reference' element.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "sequence", + pa.struct( + [ + pa.field( + "_resource", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_version", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={"comment": "Used in 'sequence caution' annotations."}, + ), + pa.field( + "link", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_uri", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:anyURI"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Used in 'online information' annotations."}, + ), + pa.field( + "event", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes the type of events that cause alternative products."}, + ), + pa.field( + "isoform", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "id", + pa.list_( + pa.field( + "item", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ), + nullable=False, + ), + pa.field( + "name", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + pa.field( + "sequence", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_ref", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=False, + ), + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes isoforms in 'alternative products' annotations."}, + ), + pa.field( + "interactant", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_intactId", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "id", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "label", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field("organismsDiffer", pa.bool_(), nullable=True, metadata={"xsd_type": "xs:boolean"}), + pa.field("experiments", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field( + "disease", + pa.struct( + [ + pa.field( + "_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "acronym", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "description", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + metadata={"comment": "Used in 'disease' annotations."}, + ), + pa.field( + "location", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_sequence", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "begin", + pa.struct( + [ + pa.field( + "_position", + pa.uint64(), + nullable=True, + metadata={"xsd_type": "xs:unsignedLong"}, + ), + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "end", + pa.struct( + [ + pa.field( + "_position", + pa.uint64(), + nullable=True, + metadata={"xsd_type": "xs:unsignedLong"}, + ), + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "position", + pa.struct( + [ + pa.field( + "_position", + pa.uint64(), + nullable=True, + metadata={"xsd_type": "xs:unsignedLong"}, + ), + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Used in 'mass spectrometry' and 'sequence caution' annotations."}, + ), + pa.field( + "text", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes different types of general annotations. Equivalent to the flat file CC-line." + }, + ), + pa.field( + "dbReference", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"comment": "Describes the name of the database.", "xsd_type": "xs:string"}, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field("_id", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + metadata={"comment": "Describes a molecule by name or unique identifier."}, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes a database cross-reference. Equivalent to the flat file DR-line."}, + ), + pa.field( + "proteinExistence", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + metadata={ + "comment": "Describes the evidence for the protein's existence. Equivalent to the flat file PE-line." + }, + ), + pa.field( + "keyword", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "feature", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the type of a sequence annotation. Equivalent to the flat file FT feature keys, but using full terms instead of acronyms.", + "xsd_type": "xs:string", + }, + ), + pa.field("_id", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_description", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field("_ref", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "original", + pa.string(), + nullable=True, + metadata={ + "comment": "Describes the original sequence in annotations that describe natural or artifical sequence variations.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "variation", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + metadata={ + "comment": "Describes the variant sequence in annotations that describe natural or artifical sequence variations." + }, + ), + pa.field( + "location", + pa.struct( + [ + pa.field( + "_sequence", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "begin", + pa.struct( + [ + pa.field( + "_position", + pa.uint64(), + nullable=True, + metadata={"xsd_type": "xs:unsignedLong"}, + ), + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "end", + pa.struct( + [ + pa.field( + "_position", + pa.uint64(), + nullable=True, + metadata={"xsd_type": "xs:unsignedLong"}, + ), + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "position", + pa.struct( + [ + pa.field( + "_position", + pa.uint64(), + nullable=True, + metadata={"xsd_type": "xs:unsignedLong"}, + ), + pa.field( + "_status", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={"comment": "Describes the sequence coordinates of the annotation."}, + ), + pa.field( + "ligand", + pa.struct( + [ + pa.field( + "name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + pa.field( + "label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "note", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the chemical entity that is bound in annotations that describe binding sites." + }, + ), + pa.field( + "ligandPart", + pa.struct( + [ + pa.field( + "name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + pa.field( + "label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "note", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the specific part of a molecule that is bound in annotations that describe binding sites." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={ + "comment": "Describes different types of sequence annotations. Equivalent to the flat file FT-line." + }, + ), + pa.field( + "evidence", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the type of an evidence using the Evidence Code Ontology (http://www.obofoundry.org/cgi-bin/detail.cgi?id=evidence_code).", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_key", + pa.int64(), + nullable=False, + metadata={ + "comment": "A unique key to link annotations (via 'evidence' attributes) to evidences.", + "xsd_type": "xs:integer", + }, + ), + pa.field( + "source", + pa.struct( + [ + pa.field( + "_ref", pa.int64(), nullable=True, metadata={"xsd_type": "xs:integer"} + ), + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the source of the data using a database cross-reference (or a 'ref' attribute when the source cannot be found in a public data source, such as PubMed, and is cited only within the UniProtKB entry)." + }, + ), + pa.field( + "importedFrom", + pa.struct( + [ + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes the name of the database.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={ + "comment": "Describes a unique database identifier.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_evidence", + pa.list_( + pa.field( + "item", + pa.int32(), + nullable=True, + metadata={"xsd_type": "xs:int"}, + ), + ), + nullable=True, + ), + pa.field( + "molecule", + pa.struct( + [ + pa.field( + "_id", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes a molecule by name or unique identifier." + }, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={ + "comment": "Describes a database cross-reference. Equivalent to the flat file DR-line." + }, + ), + ] + ), + nullable=True, + metadata={ + "comment": "Describes the source of the evidence, when it is not assigned by UniProt, but imported from an external database." + }, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "Describes the evidence for an annotation. No flat file equivalent."}, + ), + pa.field( + "sequence", + pa.struct( + [ + pa.field("_length", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field("_mass", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field("_checksum", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("_modified", pa.date32(), nullable=False, metadata={"xsd_type": "xs:date"}), + pa.field("_version", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field("_precursor", pa.bool_(), nullable=True, metadata={"xsd_type": "xs:boolean"}), + pa.field("_fragment", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + ), + ] +) diff --git a/tests/data/xsd/examples/uniprot/maximal.xml b/tests/data/xsd/examples/uniprot/maximal.xml new file mode 100644 index 00000000..4567e786 --- /dev/null +++ b/tests/data/xsd/examples/uniprot/maximal.xml @@ -0,0 +1,693 @@ + + + + P04637 + Q9V2L2 + P53_HUMAN + 1A1D_PYRAB + + + Cellular tumor antigen p53 + p53 + TP53-alpha + 2.7.11.1 + 3.5.99.7 + + + Antigen NY-CO-13 + NY-CO-13 + Pp53 + 3.1.3.16 + 3.1.3.48 + + + Phosphoprotein p53 + TRP53 + p53-iso1 + 2.7.10.2 + 2.7.12.1 + + + Cellular phosphoprotein p53 + 2.7.11.22 + 2.7.11.24 + + + Mutant-type p53 antigen + 3.2.2.6 + 3.2.2.20 + + Hom s p53 + Recombinant human p53 (rHu-p53) + CD-p53a + CD-p53b + p53-alfa + p53-beta + + + DNA-binding core domain + p53-DBD + DBD-core + 3.2.2.- + 3.1.3.16 + + + Tumor suppressor p53 + CD-A1 + CD-A2 + 3.4.19.12 + 3.4.22.68 + + + Transformation-related protein 53 + CD-B1 + CD-B2 + 2.4.2.30 + 2.1.1.43 + + + Zinc-coordinated DNA binding region + 3.5.1.98 + 3.5.1.104 + + + Conserved box II-V domain + 6.3.2.25 + 6.3.2.27 + + Hom s p53-DBD + Engineered p53 core domain + CD-DBD1 + CD-DBD2 + p53-gamma + p53-delta + + + + Tetramerization domain + p53-TD + TD-domain + 2.3.1.48 + 3.4.21.- + + + Core domain isoform A + OR-1a + OR-1b + 1.1.1.1 + 1.2.1.12 + + + Core domain isoform B + OR-2a + OR-2b + 3.1.3.16 (11) + 3.1.3.48 (12) + + + C-terminal tetramerization helix + 2.7.1.37 + 2.7.4.3 + + + Nuclear export signal region + 2.7.11.22 (11) + 2.7.11.24 (12) + + Hom s p53-TD + Stabilized p53 tetramer domain + CD-TD1 + CD-TD2 + p53-epsilon + p53-zeta + + + + Cleaved p53 N-terminal peptide + p53-N + N-pep + 2.7.7.7 + 3.6.4.12 + + + Oligomerization region 1 + TA-1a + TA-1b + 2.7.10.2 (13) + 2.7.12.1 (14) + + + Oligomerization region 2 + TA-2a + TA-2b + 3.4.19.12 (15) + 3.4.22.68 (16) + + + N-terminal TAD1 peptide + 3.2.2.6 (13) + 3.2.2.20 (14) + + + N-terminal TAD2 peptide + 3.5.1.98 (15) + 3.5.1.104 (16) + + Hom s p53-N + Synthetic TAD peptide + CD-TAD1 + CD-TAD2 + p53-eta + p53-theta + + + + Cleaved p53 C-terminal regulatory chain + p53-C + C-chain + 1.14.11.- + 6.3.2.19 + + + Transactivation subunit alpha + CT-1a + CT-1b + 2.4.2.30 (17) + 2.1.1.43 (18) + + + Transactivation subunit beta + CT-2a + CT-2b + 1.1.1.1 (19) + 1.2.1.12 (20) + + + Basic regulatory region + 6.3.2.25 (17) + 6.3.2.27 (18) + + + Apoptosis cofactor peptide + 2.7.1.37 (19) + 2.7.4.3 (20) + + Hom s p53-C + Fusion C-terminal peptide + CD-REG1 + CD-REG2 + p53-iota + p53-kappa + + + + TP53 + P53 + + + PYRAB00630 + PAB2303 + + + Homo sapiens + Human + + Isoform 1 + + + + + Isoform 2 + + + + + Eukaryota + Metazoa + + + + Pyrococcus abyssi (strain GE5 / Orsay) + Pyrococcus abyssi + + Isoform 3 + + + + + Chain A + + + + + Archaea + Thermococci + + + + Methanocella arvoryzae (strain DSM 22066 / NBRC 105507 / MRE50) + Rice cluster I archaeon + + Chain B + + + + + Monomer + + + + + Methanomicrobia + Methanocellales + + + + 17p13.1 + chr17:7668402-7687550 + + + pBR322 + pMRE50 + + + + An integrated analysis of the genome of the hyperthermophilic archaeon Pyrococcus abyssi. + + + + + + + + + 10.1046/j.1365-2958.2003.03381.x + + Homotetramer + + + + + Core domain + + + + + NUCLEOTIDE SEQUENCE [LARGE SCALE GENOMIC DNA] + X-RAY CRYSTALLOGRAPHY (2.2 ANGSTROMS) OF 102-292 + + GE5 / Orsay + pAB1 + Tn10 + Liver + + + + + Genome of rice cluster I archaea -- the key methane producers in the rice rhizosphere. + + + + + + + + + 10.1126/science.1127062 + + Precursor + + + + + Mature peptide + + + + + GENOME REANNOTATION + FUNCTION AND CATALYTIC ACTIVITY + + DSM 22066 / NBRC 105507 / MRE50 + pMRE50 + Tn5 + Colon epithelium + + + + + NUCLEOTIDE SEQUENCE [GENOMIC DNA] + + + Isoform 1 (11) + + 280 + Absorbs UV light with a primary peak at 280 nm. + Secondary shoulder observed at 292 nm in native buffer. + + + 12.5 uM for ATP + 48.0 uM for 1-aminocyclopropane-1-carboxylate + 450 nmol/min/mg enzyme + 1280 nmol/min/mg in presence of PLP cofactor + Follows classical Michaelis-Menten kinetics at pH 7.4. + Shows positive cooperativity upon tetramerization. + + + Optimum pH is 7.4-7.8 in phosphate buffer. + Rapidly loses DNA-binding activity below pH 6.0. + + + Midpoint potential E0 is -240 mV at pH 7.0. + Oxidation of Cys-277 abolishes sequence-specific binding. + + + Optimum temperature is 37 degrees Celsius. + Unfolds irreversibly above 44 degrees Celsius (Tm = 42.5 C). + + + + + + + + + Biophysical measurements performed on purified recombinant core domain. + Exhibits high-affinity sequence-specific DNA binding. + + + Isoform 2 (12) + + 1-aminocyclopropane-1-carboxylate + H2O = 2-oxobutanoate + NH4(+) + + Isoform 3 (13) + + + + + Chain A (14) + + + + + + + Chain B (15) + + + + + + + Monomer (16) + + + + + + + + + + + + Catalyzes the ring-opening deamination of ACC. + Activity is strictly dependent on divalent cations. + + + Homotetramer (17) + + pyridoxal 5'-phosphate + + Core domain (18) + + + + + + Zn(2+) + + Precursor (19) + + + + + + + + + + + + Zinc coordination stabilizes loop-sheet-helix motif. + Depletion of zinc induces molten-globule conformation. + + + Mature peptide (20) + + Nucleus + Nucleolus + Peripheral membrane protein + Single-pass membrane protein + Cytoplasmic side + Extracellular side + + + Cytoplasm + Mitochondrion + Multi-pass membrane protein + Lipid-anchor + Matrix side + Lumenal side + + + + + + + + + Shuttles between nucleus and cytoplasm in a CRM1-dependent manner. + Accumulates in the nucleus following ionizing radiation. + + + Isoform 1 (21) + + + + + + + + + + + Sequence differs from genomic reference at codon 112. + Verified by mass spectrometry of tryptic peptides. + + + Isoform 2 (22) + + + + + + + + + + Full structural and clinical annotations available online. + Cross-referenced with PDBe macromolecular database. + + + Isoform 3 (23) + + + + P04637-1 + P04637-alpha + Alpha + Canonical + + Chosen as the canonical sequence. + Expressed ubiquitously in normal tissues. + + + P04637-2 + P04637-beta + Beta + Alternative + + Produced by alternative splicing at intron 9. + Lacks the C-terminal oligomerization domain. + + + + + + + + + Contains 12 characterized splice isoforms. + Alternative promoter usage produces delta133 isoforms. + + + Chain A (24) + + P04637 + + + Chain B (25) + + + + + + Q00987 + + + Monomer (26) + + + + + false + 14 + + + + + + + + Forms a tight complex with E3 ubiquitin-protein ligase MDM2. + Phosphorylation at Ser-15 disrupts MDM2 binding. + + + Homotetramer (27) + + Li-Fraumeni syndrome + LFS1 + An autosomal dominant familial cancer syndrome caused by germline TP53 mutations. + + Core domain (28) + + + + + + + + + + + + Mutations in the DNA-binding domain account for 80% of LFS cases. + Associated with early-onset sarcomas and breast carcinomas. + + + + Precursor (29) + + + + + Mature peptide (30) + + + + + 3D-structure + DNA damage + + R + H + Q + + + + + + Mg(2+) + + Isoform 1 (31) + + + + + Coordinated by Cys-176, His-179, Cys-238, and Cys-242. + + + catalytic zinc site + + Isoform 2 (32) + + + + + Essential for structural integrity of L2/L3 loops. + + + + S + A + E + + + + + S-adenosyl-L-methionine + + Isoform 3 (33) + + + + + Covalently linked via Schiff base to Lys-54. + + + phosphate binding pocket + + Chain A (34) + + + + + Stabilized by hydrogen bonding with Gly-188 and Thr-190. + + + + + + Chain B (35) + + + + + + + Monomer (36) + + + + + + + + + Homotetramer (37) + + + + + + + Core domain (38) + + + + + + MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP + + diff --git a/tests/data/xsd/examples/uniprot/maximal_expected.json b/tests/data/xsd/examples/uniprot/maximal_expected.json new file mode 100644 index 00000000..4a61b48c --- /dev/null +++ b/tests/data/xsd/examples/uniprot/maximal_expected.json @@ -0,0 +1,3525 @@ +[ + { + "_dataset": "Swiss-Prot", + "_created": "1986-08-13", + "_modified": "2024-05-29", + "_version": 285, + "accession": [ + "P04637", + "Q9V2L2" + ], + "name": [ + "P53_HUMAN", + "1A1D_PYRAB" + ], + "protein": { + "recommendedName": { + "fullName": { + "_evidence": [ + 1, + 2 + ], + "#text": "Cellular tumor antigen p53" + }, + "shortName": [ + { + "_evidence": [ + 1, + 3 + ], + "#text": "p53" + }, + { + "_evidence": [ + 2, + 3 + ], + "#text": "TP53-alpha" + } + ], + "ecNumber": [ + { + "_evidence": [ + 1, + 4 + ], + "#text": "2.7.11.1" + }, + { + "_evidence": [ + 2, + 4 + ], + "#text": "3.5.99.7" + } + ] + }, + "alternativeName": [ + { + "fullName": { + "_evidence": [ + 3, + 4 + ], + "#text": "Antigen NY-CO-13" + }, + "shortName": [ + { + "_evidence": [ + 1, + 2, + 3 + ], + "#text": "NY-CO-13" + }, + { + "_evidence": [ + 1, + 2, + 4 + ], + "#text": "Pp53" + } + ], + "ecNumber": [ + { + "_evidence": [ + 2, + 3, + 4 + ], + "#text": "3.1.3.16" + }, + { + "_evidence": [ + 1, + 5 + ], + "#text": "3.1.3.48" + } + ] + }, + { + "fullName": { + "_evidence": [ + 2, + 5 + ], + "#text": "Phosphoprotein p53" + }, + "shortName": [ + { + "_evidence": [ + 3, + 5 + ], + "#text": "TRP53" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "p53-iso1" + } + ], + "ecNumber": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "2.7.10.2" + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "2.7.12.1" + } + ] + } + ], + "submittedName": [ + { + "fullName": { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Cellular phosphoprotein p53" + }, + "ecNumber": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "2.7.11.22" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "2.7.11.24" + } + ] + }, + { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Mutant-type p53 antigen" + }, + "ecNumber": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "3.2.2.6" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "3.2.2.20" + } + ] + } + ], + "allergenName": { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Hom s p53" + }, + "biotechName": { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Recombinant human p53 (rHu-p53)" + }, + "cdAntigenName": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "CD-p53a" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CD-p53b" + } + ], + "innName": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "p53-alfa" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "p53-beta" + } + ], + "domain": [ + { + "recommendedName": { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "DNA-binding core domain" + }, + "shortName": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "p53-DBD" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "DBD-core" + } + ], + "ecNumber": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "3.2.2.-" + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "3.1.3.16" + } + ] + }, + "alternativeName": [ + { + "fullName": { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Tumor suppressor p53" + }, + "shortName": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CD-A1" + }, + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "CD-A2" + } + ], + "ecNumber": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "3.4.19.12" + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "3.4.22.68" + } + ] + }, + { + "fullName": { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Transformation-related protein 53" + }, + "shortName": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "CD-B1" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "CD-B2" + } + ], + "ecNumber": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "2.4.2.30" + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "2.1.1.43" + } + ] + } + ], + "submittedName": [ + { + "fullName": { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Zinc-coordinated DNA binding region" + }, + "ecNumber": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "3.5.1.98" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "3.5.1.104" + } + ] + }, + { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Conserved box II-V domain" + }, + "ecNumber": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "6.3.2.25" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "6.3.2.27" + } + ] + } + ], + "allergenName": { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Hom s p53-DBD" + }, + "biotechName": { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Engineered p53 core domain" + }, + "cdAntigenName": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "CD-DBD1" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CD-DBD2" + } + ], + "innName": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "p53-gamma" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "p53-delta" + } + ] + }, + { + "recommendedName": { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Tetramerization domain" + }, + "shortName": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "p53-TD" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "TD-domain" + } + ], + "ecNumber": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "2.3.1.48" + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "3.4.21.-" + } + ] + }, + "alternativeName": [ + { + "fullName": { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Core domain isoform A" + }, + "shortName": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "OR-1a" + }, + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "OR-1b" + } + ], + "ecNumber": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "1.1.1.1" + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "1.2.1.12" + } + ] + }, + { + "fullName": { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Core domain isoform B" + }, + "shortName": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "OR-2a" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "OR-2b" + } + ], + "ecNumber": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "3.1.3.16 (11)" + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "3.1.3.48 (12)" + } + ] + } + ], + "submittedName": [ + { + "fullName": { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "C-terminal tetramerization helix" + }, + "ecNumber": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "2.7.1.37" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "2.7.4.3" + } + ] + }, + { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Nuclear export signal region" + }, + "ecNumber": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "2.7.11.22 (11)" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "2.7.11.24 (12)" + } + ] + } + ], + "allergenName": { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Hom s p53-TD" + }, + "biotechName": { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Stabilized p53 tetramer domain" + }, + "cdAntigenName": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "CD-TD1" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CD-TD2" + } + ], + "innName": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "p53-epsilon" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "p53-zeta" + } + ] + } + ], + "component": [ + { + "recommendedName": { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Cleaved p53 N-terminal peptide" + }, + "shortName": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "p53-N" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "N-pep" + } + ], + "ecNumber": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "2.7.7.7" + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "3.6.4.12" + } + ] + }, + "alternativeName": [ + { + "fullName": { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Oligomerization region 1" + }, + "shortName": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "TA-1a" + }, + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "TA-1b" + } + ], + "ecNumber": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "2.7.10.2 (13)" + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "2.7.12.1 (14)" + } + ] + }, + { + "fullName": { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Oligomerization region 2" + }, + "shortName": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "TA-2a" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "TA-2b" + } + ], + "ecNumber": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "3.4.19.12 (15)" + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "3.4.22.68 (16)" + } + ] + } + ], + "submittedName": [ + { + "fullName": { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "N-terminal TAD1 peptide" + }, + "ecNumber": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "3.2.2.6 (13)" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "3.2.2.20 (14)" + } + ] + }, + { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "N-terminal TAD2 peptide" + }, + "ecNumber": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "3.5.1.98 (15)" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "3.5.1.104 (16)" + } + ] + } + ], + "allergenName": { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Hom s p53-N" + }, + "biotechName": { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Synthetic TAD peptide" + }, + "cdAntigenName": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "CD-TAD1" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CD-TAD2" + } + ], + "innName": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "p53-eta" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "p53-theta" + } + ] + }, + { + "recommendedName": { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Cleaved p53 C-terminal regulatory chain" + }, + "shortName": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "p53-C" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "C-chain" + } + ], + "ecNumber": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "1.14.11.-" + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "6.3.2.19" + } + ] + }, + "alternativeName": [ + { + "fullName": { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Transactivation subunit alpha" + }, + "shortName": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CT-1a" + }, + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "CT-1b" + } + ], + "ecNumber": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "2.4.2.30 (17)" + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "2.1.1.43 (18)" + } + ] + }, + { + "fullName": { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Transactivation subunit beta" + }, + "shortName": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "CT-2a" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "CT-2b" + } + ], + "ecNumber": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "1.1.1.1 (19)" + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "1.2.1.12 (20)" + } + ] + } + ], + "submittedName": [ + { + "fullName": { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Basic regulatory region" + }, + "ecNumber": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "6.3.2.25 (17)" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "6.3.2.27 (18)" + } + ] + }, + { + "fullName": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Apoptosis cofactor peptide" + }, + "ecNumber": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "2.7.1.37 (19)" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "2.7.4.3 (20)" + } + ] + } + ], + "allergenName": { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Hom s p53-C" + }, + "biotechName": { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Fusion C-terminal peptide" + }, + "cdAntigenName": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "CD-REG1" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "CD-REG2" + } + ], + "innName": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "p53-iota" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "p53-kappa" + } + ] + } + ] + }, + "gene": [ + { + "name": [ + { + "_evidence": [ + 1, + 3, + 6 + ], + "_type": "primary", + "#text": "TP53" + }, + { + "_evidence": [ + 2, + 4, + 7 + ], + "_type": "synonym", + "#text": "P53" + } + ] + }, + { + "name": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "_type": "ordered locus", + "#text": "PYRAB00630" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "_type": "ORF", + "#text": "PAB2303" + } + ] + } + ], + "organism": { + "_evidence": [ + 1 + ], + "name": [ + { + "_type": "synonym", + "#text": "Homo sapiens" + }, + { + "_type": "synonym", + "#text": "Human" + } + ], + "dbReference": [ + { + "_type": "UniProtKB ID", + "_id": "P53_HUMAN", + "_evidence": [ + 5, + 7, + 1 + ], + "molecule": { + "_id": "P04637-1", + "#text": "Isoform 1" + }, + "property": [ + { + "_type": "member count", + "_value": "3" + }, + { + "_type": "common taxon", + "_value": "Homo sapiens" + } + ] + }, + { + "_type": "UniParc ID", + "_id": "UPI000002ED67", + "_evidence": [ + 6, + 8, + 2 + ], + "molecule": { + "_id": "P04637-2", + "#text": "Isoform 2" + }, + "property": [ + { + "_type": "common taxon ID", + "_value": "9606" + }, + { + "_type": "GO Molecular Function", + "_value": "GO:0003677" + } + ] + } + ], + "lineage": { + "taxon": [ + "Eukaryota", + "Metazoa" + ] + } + }, + "organismHost": [ + { + "_evidence": [ + 1, + 2 + ], + "name": [ + { + "_type": "synonym", + "#text": "Pyrococcus abyssi (strain GE5 / Orsay)" + }, + { + "_type": "synonym", + "#text": "Pyrococcus abyssi" + } + ], + "dbReference": [ + { + "_type": "RefSeq", + "_id": "NP_000537.3", + "_evidence": [ + 7, + 9, + 3 + ], + "molecule": { + "_id": "P04637-3", + "#text": "Isoform 3" + }, + "property": [ + { + "_type": "UniProtKB accession", + "_value": "P04637" + }, + { + "_type": "UniParc ID", + "_value": "UPI000002ED67" + } + ] + }, + { + "_type": "NCBI Taxonomy", + "_id": "9606", + "_evidence": [ + 8, + 1, + 4 + ], + "molecule": { + "_id": "Q9V2L2-1", + "#text": "Chain A" + }, + "property": [ + { + "_type": "UniRef100 ID", + "_value": "UniRef100_P04637" + }, + { + "_type": "UniRef90 ID", + "_value": "UniRef90_P04637" + } + ] + } + ], + "lineage": { + "taxon": [ + "Archaea", + "Thermococci" + ] + } + }, + { + "_evidence": [ + 2, + 3 + ], + "name": [ + { + "_type": "synonym", + "#text": "Methanocella arvoryzae (strain DSM 22066 / NBRC 105507 / MRE50)" + }, + { + "_type": "synonym", + "#text": "Rice cluster I archaeon" + } + ], + "dbReference": [ + { + "_type": "Proteomes", + "_id": "UP000005640", + "_evidence": [ + 9, + 2, + 5 + ], + "molecule": { + "_id": "Q0W5C8-1", + "#text": "Chain B" + }, + "property": [ + { + "_type": "protein name", + "_value": "Cellular tumor antigen p53" + }, + { + "_type": "source organism", + "_value": "Homo sapiens (Human)" + } + ] + }, + { + "_type": "EMBL", + "_id": "X02469", + "_evidence": [ + 1, + 3, + 6 + ], + "molecule": { + "_id": "A3CTY6-1", + "#text": "Monomer" + }, + "property": [ + { + "_type": "NCBI taxonomy", + "_value": "9606" + }, + { + "_type": "length", + "_value": "393" + } + ] + } + ], + "lineage": { + "taxon": [ + "Methanomicrobia", + "Methanocellales" + ] + } + } + ], + "geneLocation": [ + { + "_type": "mitochondrion", + "_evidence": [ + 2, + 4, + 7 + ], + "name": [ + { + "_status": "known", + "#text": "17p13.1" + }, + { + "_status": "known", + "#text": "chr17:7668402-7687550" + } + ] + }, + { + "_type": "plasmid", + "_evidence": [ + 3, + 5, + 8 + ], + "name": [ + { + "_status": "known", + "#text": "pBR322" + }, + { + "_status": "known", + "#text": "pMRE50" + } + ] + } + ], + "reference": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "_key": "1", + "citation": { + "_type": "journal article", + "_date": "2003-09-15", + "_name": "Mol. Microbiol.", + "_volume": "47", + "_first": "1495", + "_last": "1512", + "_publisher": "Wiley-Blackwell", + "_city": "Oxford", + "_db": "EMBL/GenBank/DDBJ", + "_number": "US20230123456", + "_institute": "MRC Laboratory of Molecular Biology", + "_country": "United Kingdom", + "title": "An integrated analysis of the genome of the hyperthermophilic archaeon Pyrococcus abyssi.", + "editorList": { + "consortium": [ + { + "_name": "The UniProt Consortium" + } + ], + "person": [ + { + "_name": "Cohen G.N." + } + ] + }, + "authorList": { + "consortium": [ + { + "_name": "International Cancer Genome Consortium" + } + ], + "person": [ + { + "_name": "Forterre P." + } + ] + }, + "locator": "10.1046/j.1365-2958.2003.03381.x", + "dbReference": [ + { + "_type": "GO", + "_id": "GO:0003677", + "_evidence": [ + 5, + 7, + 1 + ], + "molecule": { + "_id": "P04637-1 (7)", + "#text": "Homotetramer" + }, + "property": [ + { + "_type": "isSeed", + "_value": "true" + }, + { + "_type": "protein sequence ID", + "_value": "CAB48986.1" + } + ] + }, + { + "_type": "Pfam", + "_id": "PF00870", + "_evidence": [ + 6, + 8, + 2 + ], + "molecule": { + "_id": "P04637-2 (8)", + "#text": "Core domain" + }, + "property": [ + { + "_type": "molecule type", + "_value": "Genomic_DNA" + }, + { + "_type": "entry name", + "_value": "ACCD_DCysDesulf" + } + ] + } + ] + }, + "scope": [ + "NUCLEOTIDE SEQUENCE [LARGE SCALE GENOMIC DNA]", + "X-RAY CRYSTALLOGRAPHY (2.2 ANGSTROMS) OF 102-292" + ], + "source": { + "strain": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "GE5 / Orsay" + } + ], + "plasmid": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "pAB1" + } + ], + "transposon": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "Tn10" + } + ], + "tissue": [ + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Liver" + } + ] + } + }, + { + "_evidence": [ + 2, + 4, + 7 + ], + "_key": "2", + "citation": { + "_type": "book", + "_date": "1999", + "_name": "Curr. Microbiol.", + "_volume": "64", + "_first": "118", + "_last": "129", + "_publisher": "Springer", + "_city": "Heidelberg", + "_db": "UniProtKB", + "_number": "EP3987654A1", + "_institute": "Max Planck Institute for Biochemistry", + "_country": "Germany", + "title": "Genome of rice cluster I archaea -- the key methane producers in the rice rhizosphere.", + "editorList": { + "consortium": [ + { + "_name": "Archaea Genome Sequencing Project" + } + ], + "person": [ + { + "_name": "Erkel C." + } + ] + }, + "authorList": { + "consortium": [ + { + "_name": "The UniProt Consortium (4)" + } + ], + "person": [ + { + "_name": "Liesack W." + } + ] + }, + "locator": "10.1126/science.1127062", + "dbReference": [ + { + "_type": "InterPro", + "_id": "IPR011615", + "_evidence": [ + 3, + 5, + 8 + ], + "molecule": { + "_id": "P04637-3 (9)", + "#text": "Precursor" + }, + "property": [ + { + "_type": "match status", + "_value": "1" + }, + { + "_type": "nucleotide sequence ID", + "_value": "NC_000868.1" + } + ] + }, + { + "_type": "PDB", + "_id": "1TUP", + "_evidence": [ + 4, + 6, + 9 + ], + "molecule": { + "_id": "Q9V2L2-1 (10)", + "#text": "Mature peptide" + }, + "property": [ + { + "_type": "taxonomic scope", + "_value": "Archaea" + }, + { + "_type": "component", + "_value": "Chromosome" + } + ] + } + ] + }, + "scope": [ + "GENOME REANNOTATION", + "FUNCTION AND CATALYTIC ACTIVITY" + ], + "source": { + "strain": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "DSM 22066 / NBRC 105507 / MRE50" + } + ], + "plasmid": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "pMRE50" + } + ], + "transposon": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Tn5" + } + ], + "tissue": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "Colon epithelium" + } + ] + } + }, + { + "_evidence": [ + 1, + 5, + 9 + ], + "_key": "3", + "citation": { + "_type": "submission", + "_date": "2007-10", + "_db": "EMBL/GenBank/DDBJ databases" + }, + "scope": [ + "NUCLEOTIDE SEQUENCE [GENOMIC DNA]" + ] + } + ], + "comment": [ + { + "_type": "biophysicochemical properties", + "_locationType": "topology", + "_name": "UV-Vis absorption profile", + "_mass": 43653.5, + "_error": "Frameshift at position 112", + "_method": "Electrospray ionization", + "_evidence": [ + 9, + 2, + 5 + ], + "molecule": { + "_id": "Q0W5C8-1 (11)", + "#text": "Isoform 1 (11)" + }, + "absorption": { + "max": { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "280" + }, + "text": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Absorbs UV light with a primary peak at 280 nm." + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "Secondary shoulder observed at 292 nm in native buffer." + } + ] + }, + "kinetics": { + "KM": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "12.5 uM for ATP" + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "48.0 uM for 1-aminocyclopropane-1-carboxylate" + } + ], + "Vmax": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "450 nmol/min/mg enzyme" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "1280 nmol/min/mg in presence of PLP cofactor" + } + ], + "text": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "Follows classical Michaelis-Menten kinetics at pH 7.4." + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "Shows positive cooperativity upon tetramerization." + } + ] + }, + "phDependence": { + "text": [ + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Optimum pH is 7.4-7.8 in phosphate buffer." + }, + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Rapidly loses DNA-binding activity below pH 6.0." + } + ] + }, + "redoxPotential": { + "text": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "Midpoint potential E0 is -240 mV at pH 7.0." + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Oxidation of Cys-277 abolishes sequence-specific binding." + } + ] + }, + "temperatureDependence": { + "text": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Optimum temperature is 37 degrees Celsius." + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Unfolds irreversibly above 44 degrees Celsius (Tm = 42.5 C)." + } + ] + }, + "location": [ + { + "_sequence": "VSP_006535", + "begin": { + "_position": 1, + "_status": "certain", + "_evidence": [ + 7, + 9, + 3 + ] + }, + "end": { + "_position": 42, + "_status": "certain", + "_evidence": [ + 8, + 1, + 4 + ] + } + }, + { + "_sequence": "VSP_006536", + "position": { + "_position": 15, + "_status": "certain", + "_evidence": [ + 9, + 2, + 5 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Biophysical measurements performed on purified recombinant core domain." + }, + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Exhibits high-affinity sequence-specific DNA binding." + } + ] + }, + { + "_type": "catalytic activity", + "_locationType": "transmembrane", + "_name": "ACC deaminase reaction", + "_mass": 35755.25, + "_error": "Erroneous initiation at residue 1", + "_method": "MALDI-TOF", + "_evidence": [ + 3, + 5, + 8 + ], + "molecule": { + "_id": "A3CTY6-1 (12)", + "#text": "Isoform 2 (12)" + }, + "reaction": { + "_evidence": [ + 4, + 6, + 9 + ], + "text": "1-aminocyclopropane-1-carboxylate + H2O = 2-oxobutanoate + NH4(+)", + "dbReference": [ + { + "_type": "PubMed", + "_id": "12622808", + "_evidence": [ + 5, + 7, + 1 + ], + "molecule": { + "_id": "P04637-1 (13)", + "#text": "Isoform 3 (13)" + }, + "property": [ + { + "_type": "term", + "_value": "F:DNA-binding transcription factor activity" + }, + { + "_type": "evidence", + "_value": "ECO:0007669" + } + ] + }, + { + "_type": "DOI", + "_id": "10.1046/j.1365-2958.2003.03381.x", + "_evidence": [ + 6, + 8, + 2 + ], + "molecule": { + "_id": "P04637-2 (14)", + "#text": "Chain A (14)" + }, + "property": [ + { + "_type": "project", + "_value": "UniProtKB-KW" + }, + { + "_type": "chains", + "_value": "A/B=102-292" + } + ] + } + ] + }, + "physiologicalReaction": [ + { + "_direction": "left-to-right", + "_evidence": [ + 7, + 9, + 3 + ], + "dbReference": { + "_type": "Rhea", + "_id": "RHEA:16933", + "_evidence": [ + 8, + 1, + 4 + ], + "molecule": { + "_id": "P04637-3 (15)", + "#text": "Chain B (15)" + }, + "property": [ + { + "_type": "resolution", + "_value": "2.20 A" + }, + { + "_type": "method", + "_value": "X-ray" + } + ] + } + }, + { + "_direction": "right-to-left", + "_evidence": [ + 9, + 2, + 5 + ], + "dbReference": { + "_type": "ChEBI", + "_id": "CHEBI:15377", + "_evidence": [ + 1, + 3, + 6 + ], + "molecule": { + "_id": "Q9V2L2-1 (16)", + "#text": "Monomer (16)" + }, + "property": [ + { + "_type": "member count (27)", + "_value": "Q9V2L2" + }, + { + "_type": "common taxon (28)", + "_value": "UPI0000061B2A" + } + ] + } + } + ], + "location": [ + { + "_sequence": "VSP_006537", + "position": { + "_position": 20, + "_status": "uncertain", + "_evidence": [ + 2, + 4, + 7 + ] + } + }, + { + "_sequence": "VSP_006538", + "begin": { + "_position": 43, + "_status": "uncertain", + "_evidence": [ + 3, + 5, + 8 + ] + }, + "end": { + "_position": 63, + "_status": "uncertain", + "_evidence": [ + 4, + 6, + 9 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Catalyzes the ring-opening deamination of ACC." + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Activity is strictly dependent on divalent cations." + } + ] + }, + { + "_type": "cofactor", + "_locationType": "intramembrane", + "_name": "Zinc-binding site", + "_mass": 21879.75, + "_error": "Stop codon readthrough at residue 341", + "_method": "Analytical ultracentrifugation", + "_evidence": [ + 7, + 9, + 3 + ], + "molecule": { + "_id": "Q0W5C8-1 (17)", + "#text": "Homotetramer (17)" + }, + "cofactor": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "name": "pyridoxal 5'-phosphate", + "dbReference": { + "_type": "EC", + "_id": "3.5.99.7", + "_evidence": [ + 9, + 2, + 5 + ], + "molecule": { + "_id": "A3CTY6-1 (18)", + "#text": "Core domain (18)" + }, + "property": [ + { + "_type": "common taxon ID (29)", + "_value": "UniRef100_Q9V2L2" + }, + { + "_type": "GO Molecular Function (30)", + "_value": "Putative 1-aminocyclopropane-1-carboxylate deaminase" + } + ] + } + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "name": "Zn(2+)", + "dbReference": { + "_type": "IntAct", + "_id": "EBI-366083", + "_evidence": [ + 2, + 4, + 7 + ], + "molecule": { + "_id": "P04637-1 (19)", + "#text": "Precursor (19)" + }, + "property": [ + { + "_type": "UniProtKB accession (31)", + "_value": "Pyrococcus abyssi (strain GE5 / Orsay)" + }, + { + "_type": "UniParc ID (32)", + "_value": "272844" + } + ] + } + } + ], + "location": [ + { + "_sequence": "VSP_006539", + "begin": { + "_position": 65, + "_status": "less than", + "_evidence": [ + 3, + 5, + 8 + ] + }, + "end": { + "_position": 92, + "_status": "less than", + "_evidence": [ + 4, + 6, + 9 + ] + } + }, + { + "_sequence": "VSP_006540", + "position": { + "_position": 37, + "_status": "less than", + "_evidence": [ + 5, + 7, + 1 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Zinc coordination stabilizes loop-sheet-helix motif." + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Depletion of zinc induces molten-globule conformation." + } + ] + }, + { + "_type": "subcellular location", + "_locationType": "topological domain", + "_name": "Nuclear-cytoplasmic shuttling", + "_mass": 42110.125, + "_error": "Intron retention in exon 4", + "_method": "Size-exclusion chromatography", + "_evidence": [ + 8, + 1, + 4 + ], + "molecule": { + "_id": "P04637-2 (20)", + "#text": "Mature peptide (20)" + }, + "subcellularLocation": [ + { + "location": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "Nucleus" + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Nucleolus" + } + ], + "topology": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Peripheral membrane protein" + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "Single-pass membrane protein" + } + ], + "orientation": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Cytoplasmic side" + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Extracellular side" + } + ] + }, + { + "location": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Cytoplasm" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Mitochondrion" + } + ], + "topology": [ + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "Multi-pass membrane protein" + }, + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "Lipid-anchor" + } + ], + "orientation": [ + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Matrix side" + }, + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Lumenal side" + } + ] + } + ], + "location": [ + { + "_sequence": "VSP_006541", + "position": { + "_position": 54, + "_status": "greater than", + "_evidence": [ + 3, + 5, + 8 + ] + } + }, + { + "_sequence": "VSP_006542", + "begin": { + "_position": 102, + "_status": "greater than", + "_evidence": [ + 4, + 6, + 9 + ] + }, + "end": { + "_position": 292, + "_status": "greater than", + "_evidence": [ + 5, + 7, + 1 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Shuttles between nucleus and cytoplasm in a CRM1-dependent manner." + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Accumulates in the nucleus following ionizing radiation." + } + ] + }, + { + "_type": "sequence caution", + "_locationType": "chain", + "_name": "Frameshift discrepancy", + "_mass": 39850.0, + "_error": "Sequencing artifact at position 215", + "_method": "X-ray crystallography", + "_evidence": [ + 8, + 1, + 4 + ], + "molecule": { + "_id": "P04637-3 (21)", + "#text": "Isoform 1 (21)" + }, + "conflict": { + "_type": "frameshift", + "_ref": "1", + "sequence": { + "_resource": "EMBL-CDS", + "_id": "CAB48986", + "_version": 1 + } + }, + "location": [ + { + "_sequence": "VSP_006543", + "begin": { + "_position": 113, + "_status": "unknown", + "_evidence": [ + 9, + 2, + 5 + ] + }, + "end": { + "_position": 185, + "_status": "unknown", + "_evidence": [ + 1, + 3, + 6 + ] + } + }, + { + "_sequence": "VSP_006544", + "position": { + "_position": 72, + "_status": "unknown", + "_evidence": [ + 2, + 4, + 7 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "Sequence differs from genomic reference at codon 112." + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Verified by mass spectrometry of tryptic peptides." + } + ] + }, + { + "_type": "online information", + "_locationType": "domain", + "_name": "UniProtKB entry page", + "_mass": 41200.5, + "_error": "Mismatch at codon 72", + "_method": "Cryo-electron microscopy", + "_evidence": [ + 5, + 7, + 1 + ], + "molecule": { + "_id": "Q9V2L2-1 (22)", + "#text": "Isoform 2 (22)" + }, + "link": [ + { + "_uri": "https://www.uniprot.org/uniprotkb/P04637" + }, + { + "_uri": "https://www.ebi.ac.uk/pdbe/entry/pdb/1tup" + } + ], + "location": [ + { + "_sequence": "VSP_006535 (11)", + "position": { + "_position": 175, + "_status": "certain", + "_evidence": [ + 6, + 8, + 2 + ] + } + }, + { + "_sequence": "VSP_006536 (12)", + "begin": { + "_position": 195, + "_status": "certain", + "_evidence": [ + 7, + 9, + 3 + ] + }, + "end": { + "_position": 240, + "_status": "certain", + "_evidence": [ + 8, + 1, + 4 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "Full structural and clinical annotations available online." + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Cross-referenced with PDBe macromolecular database." + } + ] + }, + { + "_type": "alternative products", + "_locationType": "region", + "_name": "TP53 splice variants", + "_mass": 38915.25, + "_error": "Alternative exon boundary", + "_method": "NMR spectroscopy", + "_evidence": [ + 2, + 4, + 7 + ], + "molecule": { + "_id": "Q0W5C8-1 (23)", + "#text": "Isoform 3 (23)" + }, + "event": [ + { + "_type": "alternative splicing" + }, + { + "_type": "alternative initiation" + } + ], + "isoform": [ + { + "id": [ + "P04637-1", + "P04637-alpha" + ], + "name": [ + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "Alpha" + }, + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Canonical" + } + ], + "sequence": { + "_type": "displayed", + "_ref": "VSP_006535" + }, + "text": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Chosen as the canonical sequence." + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Expressed ubiquitously in normal tissues." + } + ] + }, + { + "id": [ + "P04637-2", + "P04637-beta" + ], + "name": [ + { + "_evidence": [ + 7, + 9, + 3 + ], + "#text": "Beta" + }, + { + "_evidence": [ + 8, + 1, + 4 + ], + "#text": "Alternative" + } + ], + "sequence": { + "_type": "external", + "_ref": "VSP_006536" + }, + "text": [ + { + "_evidence": [ + 9, + 2, + 5 + ], + "#text": "Produced by alternative splicing at intron 9." + }, + { + "_evidence": [ + 1, + 3, + 6 + ], + "#text": "Lacks the C-terminal oligomerization domain." + } + ] + } + ], + "location": [ + { + "_sequence": "VSP_006537 (13)", + "begin": { + "_position": 248, + "_status": "uncertain", + "_evidence": [ + 2, + 4, + 7 + ] + }, + "end": { + "_position": 270, + "_status": "uncertain", + "_evidence": [ + 3, + 5, + 8 + ] + } + }, + { + "_sequence": "VSP_006538 (14)", + "position": { + "_position": 245, + "_status": "uncertain", + "_evidence": [ + 4, + 6, + 9 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Contains 12 characterized splice isoforms." + }, + { + "_evidence": [ + 6, + 8, + 2 + ], + "#text": "Alternative promoter usage produces delta133 isoforms." + } + ] + }, + { + "_type": "interaction", + "_locationType": "motif", + "_name": "p53-MDM2 complex", + "_mass": 44102.75, + "_error": "Premature termination at 305", + "_method": "Surface plasmon resonance", + "_evidence": [ + 7, + 9, + 3 + ], + "molecule": { + "_id": "A3CTY6-1 (24)", + "#text": "Chain A (24)" + }, + "interactant": [ + { + "_intactId": "EBI-366083", + "id": "P04637", + "label": "TP53", + "dbReference": { + "_type": "MIM", + "_id": "151623", + "_evidence": [ + 8, + 1, + 4 + ], + "molecule": { + "_id": "P04637-1 (25)", + "#text": "Chain B (25)" + }, + "property": [ + { + "_type": "UniRef100 ID (33)", + "_value": "330" + }, + { + "_type": "UniRef90 ID (34)", + "_value": "false" + } + ] + } + }, + { + "_intactId": "EBI-389668", + "id": "Q00987", + "label": "MDM2", + "dbReference": { + "_type": "Reactome", + "_id": "R-HSA-3700989", + "_evidence": [ + 9, + 2, + 5 + ], + "molecule": { + "_id": "P04637-2 (26)", + "#text": "Monomer (26)" + }, + "property": [ + { + "_type": "protein name (35)", + "_value": "CAJ36415.1" + }, + { + "_type": "source organism (36)", + "_value": "mRNA" + } + ] + } + } + ], + "organismsDiffer": false, + "experiments": 14, + "location": [ + { + "_sequence": "VSP_006539 (15)", + "position": { + "_position": 248, + "_status": "less than", + "_evidence": [ + 1, + 3, + 6 + ] + } + }, + { + "_sequence": "VSP_006540 (16)", + "begin": { + "_position": 273, + "_status": "less than", + "_evidence": [ + 2, + 4, + 7 + ] + }, + "end": { + "_position": 299, + "_status": "less than", + "_evidence": [ + 3, + 5, + 8 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 4, + 6, + 9 + ], + "#text": "Forms a tight complex with E3 ubiquitin-protein ligase MDM2." + }, + { + "_evidence": [ + 5, + 7, + 1 + ], + "#text": "Phosphorylation at Ser-15 disrupts MDM2 binding." + } + ] + }, + { + "_type": "disease", + "_locationType": "site", + "_name": "Li-Fraumeni predisposition", + "_mass": 43590.875, + "_error": "Missense substitution R175H", + "_method": "Isothermal titration calorimetry", + "_evidence": [ + 6, + 8, + 2 + ], + "molecule": { + "_id": "P04637-3 (27)", + "#text": "Homotetramer (27)" + }, + "disease": { + "_id": "DI-00682", + "name": "Li-Fraumeni syndrome", + "acronym": "LFS1", + "description": "An autosomal dominant familial cancer syndrome caused by germline TP53 mutations.", + "dbReference": { + "_type": "AlphaFoldDB", + "_id": "AF-P04637-F1", + "_evidence": [ + 7, + 9, + 3 + ], + "molecule": { + "_id": "Q9V2L2-1 (28)", + "#text": "Core domain (28)" + }, + "property": [ + { + "_type": "NCBI taxonomy (37)", + "_value": "Pur_DNA_glyco" + }, + { + "_type": "length (38)", + "_value": "2" + } + ] + } + }, + "location": [ + { + "_sequence": "VSP_006541 (17)", + "begin": { + "_position": 301, + "_status": "greater than", + "_evidence": [ + 8, + 1, + 4 + ] + }, + "end": { + "_position": 322, + "_status": "greater than", + "_evidence": [ + 9, + 2, + 5 + ] + } + }, + { + "_sequence": "VSP_006542 (18)", + "position": { + "_position": 249, + "_status": "greater than", + "_evidence": [ + 1, + 3, + 6 + ] + } + } + ], + "text": [ + { + "_evidence": [ + 2, + 4, + 7 + ], + "#text": "Mutations in the DNA-binding domain account for 80% of LFS cases." + }, + { + "_evidence": [ + 3, + 5, + 8 + ], + "#text": "Associated with early-onset sarcomas and breast carcinomas." + } + ] + }, + { + "_type": "miscellaneous" + } + ], + "dbReference": [ + { + "_type": "SMART", + "_id": "SM00325", + "_evidence": [ + 4, + 6, + 9 + ], + "molecule": { + "_id": "Q0W5C8-1 (29)", + "#text": "Precursor (29)" + }, + "property": [ + { + "_type": "isSeed (39)", + "_value": "NC_009464.1" + }, + { + "_type": "protein sequence ID (40)", + "_value": "Eukaryota" + } + ] + }, + { + "_type": "SUPFAM", + "_id": "SSF49417", + "_evidence": [ + 5, + 7, + 1 + ], + "molecule": { + "_id": "A3CTY6-1 (30)", + "#text": "Mature peptide (30)" + }, + "property": [ + { + "_type": "molecule type (41)", + "_value": "Mitochondrion" + }, + { + "_type": "entry name (42)", + "_value": "P:regulation of cell cycle" + } + ] + } + ], + "proteinExistence": { + "_type": "evidence at protein level" + }, + "keyword": [ + { + "_evidence": [ + 6, + 8, + 2 + ], + "_id": "KW-0002", + "#text": "3D-structure" + }, + { + "_evidence": [ + 7, + 9, + 3 + ], + "_id": "KW-0227", + "#text": "DNA damage" + } + ], + "feature": [ + { + "_type": "domain", + "_id": "PRO_0000080560", + "_description": "p53 DNA-binding core domain", + "_evidence": [ + 8, + 1, + 4 + ], + "_ref": "1", + "original": "R", + "variation": [ + "H", + "Q" + ], + "location": { + "_sequence": "VSP_006543 (19)", + "begin": { + "_position": 325, + "_status": "unknown", + "_evidence": [ + 9, + 2, + 5 + ] + }, + "end": { + "_position": 355, + "_status": "unknown", + "_evidence": [ + 1, + 3, + 6 + ] + } + }, + "ligand": { + "name": "Mg(2+)", + "dbReference": { + "_type": "PANTHER", + "_id": "PTHR11447", + "_evidence": [ + 2, + 4, + 7 + ], + "molecule": { + "_id": "P04637-1 (31)", + "#text": "Isoform 1 (31)" + }, + "property": [ + { + "_type": "match status (43)", + "_value": "ECO:0000314" + }, + { + "_type": "nucleotide sequence ID (44)", + "_value": "GO_Central" + } + ] + }, + "label": "ZN-1", + "note": "Coordinated by Cys-176, His-179, Cys-238, and Cys-242." + }, + "ligandPart": { + "name": "catalytic zinc site", + "dbReference": { + "_type": "Gene3D", + "_id": "2.60.40.720", + "_evidence": [ + 3, + 5, + 8 + ], + "molecule": { + "_id": "P04637-2 (32)", + "#text": "Isoform 2 (32)" + }, + "property": [ + { + "_type": "taxonomic scope (45)", + "_value": "3 (45)" + }, + { + "_type": "component (46)", + "_value": "Homo sapiens (46)" + } + ] + }, + "label": "ZN-coord", + "note": "Essential for structural integrity of L2/L3 loops." + } + }, + { + "_type": "mutagenesis site", + "_id": "PRO_0000184521", + "_description": "Loss of sequence-specific DNA binding", + "_evidence": [ + 4, + 6, + 9 + ], + "_ref": "2", + "original": "S", + "variation": [ + "A", + "E" + ], + "location": { + "_sequence": "VSP_006544 (20)", + "position": { + "_position": 273, + "_status": "unknown", + "_evidence": [ + 5, + 7, + 1 + ] + } + }, + "ligand": { + "name": "S-adenosyl-L-methionine", + "dbReference": { + "_type": "CDD", + "_id": "cd08367", + "_evidence": [ + 6, + 8, + 2 + ], + "molecule": { + "_id": "P04637-3 (33)", + "#text": "Isoform 3 (33)" + }, + "property": [ + { + "_type": "term (47)", + "_value": "9606 (47)" + }, + { + "_type": "evidence (48)", + "_value": "GO:0003677 (48)" + } + ] + }, + "label": "PLP-54", + "note": "Covalently linked via Schiff base to Lys-54." + }, + "ligandPart": { + "name": "phosphate binding pocket", + "dbReference": { + "_type": "HAMAP-Rule", + "_id": "MF_00527", + "_evidence": [ + 7, + 9, + 3 + ], + "molecule": { + "_id": "Q9V2L2-1 (34)", + "#text": "Chain A (34)" + }, + "property": [ + { + "_type": "project (49)", + "_value": "P04637 (49)" + }, + { + "_type": "chains (50)", + "_value": "UPI000002ED67 (50)" + } + ] + }, + "label": "PLP-OPO3", + "note": "Stabilized by hydrogen bonding with Gly-188 and Thr-190." + } + } + ], + "evidence": [ + { + "_type": "ECO:0000269", + "_key": 1, + "source": { + "_ref": 1, + "dbReference": { + "_type": "PIR", + "_id": "A25224", + "_evidence": [ + 8, + 1, + 4 + ], + "molecule": { + "_id": "Q0W5C8-1 (35)", + "#text": "Chain B (35)" + }, + "property": [ + { + "_type": "resolution (51)", + "_value": "UniRef100_P04637 (51)" + }, + { + "_type": "method (52)", + "_value": "UniRef90_P04637 (52)" + } + ] + } + }, + "importedFrom": { + "dbReference": { + "_type": "KEGG", + "_id": "hsa:7157", + "_evidence": [ + 9, + 2, + 5 + ], + "molecule": { + "_id": "A3CTY6-1 (36)", + "#text": "Monomer (36)" + }, + "property": [ + { + "_type": "member count (53)", + "_value": "Cellular tumor antigen p53 (53)" + }, + { + "_type": "common taxon (54)", + "_value": "Homo sapiens (Human) (54)" + } + ] + } + } + }, + { + "_type": "ECO:0000305", + "_key": 2, + "source": { + "_ref": 2, + "dbReference": { + "_type": "STRING", + "_id": "9606.ENSP00000269305", + "_evidence": [ + 1, + 3, + 6 + ], + "molecule": { + "_id": "P04637-1 (37)", + "#text": "Homotetramer (37)" + }, + "property": [ + { + "_type": "common taxon ID (55)", + "_value": "9606 (55)" + }, + { + "_type": "GO Molecular Function (56)", + "_value": "393 (56)" + } + ] + } + }, + "importedFrom": { + "dbReference": { + "_type": "OrthoDB", + "_id": "119234at2759", + "_evidence": [ + 2, + 4, + 7 + ], + "molecule": { + "_id": "P04637-2 (38)", + "#text": "Core domain (38)" + }, + "property": [ + { + "_type": "UniProtKB accession (57)", + "_value": "true (57)" + }, + { + "_type": "UniParc ID (58)", + "_value": "CAB48986.1 (58)" + } + ] + } + } + } + ], + "sequence": { + "_length": 393, + "_mass": 43653, + "_checksum": "AD5C149FD8106131", + "_modified": "2009-11-24", + "_version": 2, + "_precursor": true, + "_fragment": "single", + "#text": "MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP" + } + } +] diff --git a/tests/data/xsd/examples/uniprot/minimal.xml b/tests/data/xsd/examples/uniprot/minimal.xml new file mode 100644 index 00000000..6fb4ac89 --- /dev/null +++ b/tests/data/xsd/examples/uniprot/minimal.xml @@ -0,0 +1,22 @@ + + + + P04637 + P53_HUMAN + + + Cellular tumor antigen p53 + + + + Homo sapiens + + + + + NUCLEOTIDE SEQUENCE [LARGE SCALE GENOMIC DNA] + + + MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP + + diff --git a/tests/data/xsd/examples/uniprot/minimal_expected.json b/tests/data/xsd/examples/uniprot/minimal_expected.json new file mode 100644 index 00000000..b30aba42 --- /dev/null +++ b/tests/data/xsd/examples/uniprot/minimal_expected.json @@ -0,0 +1,57 @@ +[ + { + "_dataset": "Swiss-Prot", + "_created": "1986-08-13", + "_modified": "2024-05-29", + "_version": 285, + "accession": [ + "P04637" + ], + "name": [ + "P53_HUMAN" + ], + "protein": { + "recommendedName": { + "fullName": { + "#text": "Cellular tumor antigen p53" + } + } + }, + "organism": { + "name": [ + { + "_type": "synonym", + "#text": "Homo sapiens" + } + ], + "dbReference": [ + { + "_type": "UniProtKB ID", + "_id": "P53_HUMAN" + } + ] + }, + "reference": [ + { + "_key": "1", + "citation": { + "_type": "journal article" + }, + "scope": [ + "NUCLEOTIDE SEQUENCE [LARGE SCALE GENOMIC DNA]" + ] + } + ], + "proteinExistence": { + "_type": "evidence at protein level" + }, + "sequence": { + "_length": 393, + "_mass": 43653, + "_checksum": "AD5C149FD8106131", + "_modified": "2009-11-24", + "_version": 1, + "#text": "MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP" + } + } +] diff --git a/tests/data/xsd/examples/uniref/expected_schema.py b/tests/data/xsd/examples/uniref/expected_schema.py new file mode 100644 index 00000000..cdd3e680 --- /dev/null +++ b/tests/data/xsd/examples/uniref/expected_schema.py @@ -0,0 +1,188 @@ +"""Expected PyArrow schema for uniref (xml_tag='entry').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field( + "_id", pa.string(), nullable=False, metadata={"comment": "The entry identifier.", "xsd_type": "xs:ID"} + ), + pa.field( + "_updated", + pa.date32(), + nullable=False, + metadata={"comment": "The update date for the entry.", "xsd_type": "xs:date"}, + ), + pa.field("name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("_value", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "representativeMember", + pa.struct( + [ + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "The name of the database this cross-reference is referring to.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={"comment": "The ID referred to.", "xsd_type": "xs:string"}, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={"comment": "Database cross-references"}, + ), + pa.field( + "sequence", + pa.struct( + [ + pa.field("_length", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field("_checksum", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={"comment": "UniRef50/90/100 cluster members"}, + ), + pa.field( + "member", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "dbReference", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={ + "comment": "The name of the database this cross-reference is referring to.", + "xsd_type": "xs:string", + }, + ), + pa.field( + "_id", + pa.string(), + nullable=False, + metadata={"comment": "The ID referred to.", "xsd_type": "xs:string"}, + ), + pa.field( + "property", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_type", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=False, + metadata={"comment": "Database cross-references"}, + ), + pa.field( + "sequence", + pa.struct( + [ + pa.field( + "_length", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"} + ), + pa.field( + "_checksum", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + ] + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + metadata={"comment": "UniRef50/90/100 cluster members"}, + ), + ] +) diff --git a/tests/data/xsd/examples/uniref/maximal.xml b/tests/data/xsd/examples/uniref/maximal.xml new file mode 100644 index 00000000..6e5a3d58 --- /dev/null +++ b/tests/data/xsd/examples/uniref/maximal.xml @@ -0,0 +1,29 @@ + + + + Cluster: Glycogen synthase + + + + + + + + MHPKVDALLSRFPRITLIPWETPIQYLPRISRELGVDVYVKRDDLTGLGIGGNKIRKLEF + + + + + + + MPVLPRDFYDRPTLEVARDLLGKTLVRQLPAGRVALRIVETEAYIGENDKACHASKGMTA + + + + + + + MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP + + + diff --git a/tests/data/xsd/examples/uniref/maximal_expected.json b/tests/data/xsd/examples/uniref/maximal_expected.json new file mode 100644 index 00000000..9b6883a8 --- /dev/null +++ b/tests/data/xsd/examples/uniref/maximal_expected.json @@ -0,0 +1,82 @@ +[ + { + "_id": "UniRef100_P04637", + "_updated": "2025-02-05", + "name": "Cluster: Glycogen synthase", + "property": [ + { + "_type": "member count", + "_value": "3" + }, + { + "_type": "common taxon", + "_value": "Homo sapiens" + } + ], + "representativeMember": { + "dbReference": { + "_type": "UniProtKB ID", + "_id": "P53_HUMAN", + "property": [ + { + "_type": "common taxon ID", + "_value": "9606" + }, + { + "_type": "GO Molecular Function", + "_value": "GO:0003677" + } + ] + }, + "sequence": { + "_length": 393, + "_checksum": "AD5C149FD8106131", + "#text": "MHPKVDALLSRFPRITLIPWETPIQYLPRISRELGVDVYVKRDDLTGLGIGGNKIRKLEF" + } + }, + "member": [ + { + "dbReference": { + "_type": "UniParc ID", + "_id": "UPI000002ED67", + "property": [ + { + "_type": "UniProtKB accession", + "_value": "P04637" + }, + { + "_type": "UniParc ID", + "_value": "UPI000002ED67" + } + ] + }, + "sequence": { + "_length": 330, + "_checksum": "CC8699923C0B11CD", + "#text": "MPVLPRDFYDRPTLEVARDLLGKTLVRQLPAGRVALRIVETEAYIGENDKACHASKGMTA" + } + }, + { + "dbReference": { + "_type": "RefSeq", + "_id": "NP_000537.3", + "property": [ + { + "_type": "UniRef100 ID", + "_value": "UniRef100_P04637" + }, + { + "_type": "UniRef90 ID", + "_value": "UniRef90_P04637" + } + ] + }, + "sequence": { + "_length": 200, + "_checksum": "8DA31EA44EEB8636", + "#text": "MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP" + } + } + ] + } +] diff --git a/tests/data/xsd/examples/uniref/minimal.xml b/tests/data/xsd/examples/uniref/minimal.xml new file mode 100644 index 00000000..7a1904bd --- /dev/null +++ b/tests/data/xsd/examples/uniref/minimal.xml @@ -0,0 +1,9 @@ + + + + Cluster: Glycogen synthase + + + + + diff --git a/tests/data/xsd/examples/uniref/minimal_expected.json b/tests/data/xsd/examples/uniref/minimal_expected.json new file mode 100644 index 00000000..1c78c485 --- /dev/null +++ b/tests/data/xsd/examples/uniref/minimal_expected.json @@ -0,0 +1,13 @@ +[ + { + "_id": "UniRef100_P04637", + "_updated": "2025-02-05", + "name": "Cluster: Glycogen synthase", + "representativeMember": { + "dbReference": { + "_type": "UniProtKB ID", + "_id": "P53_HUMAN" + } + } + } +] diff --git a/tests/data/xsd/nested_choice.xsd b/tests/data/xsd/nested_choice.xsd deleted file mode 100644 index 4fe1fdd0..00000000 --- a/tests/data/xsd/nested_choice.xsd +++ /dev/null @@ -1,10 +0,0 @@ - - - - - - - - - - diff --git a/tests/data/xsd/nested_sequence_deep.xsd b/tests/data/xsd/nested_sequence_deep.xsd deleted file mode 100644 index 46d867bb..00000000 --- a/tests/data/xsd/nested_sequence_deep.xsd +++ /dev/null @@ -1,11 +0,0 @@ - - - - - - - - - - - diff --git a/tests/data/xsd/occurs_variety.xsd b/tests/data/xsd/occurs_variety.xsd deleted file mode 100644 index 52427754..00000000 --- a/tests/data/xsd/occurs_variety.xsd +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - - - - - - - - - diff --git a/tests/data/xsd/ref_element.xsd b/tests/data/xsd/ref_element.xsd deleted file mode 100644 index 52ab6632..00000000 --- a/tests/data/xsd/ref_element.xsd +++ /dev/null @@ -1,10 +0,0 @@ - - - - - - - - - - diff --git a/tests/data/xsd/spec/01_scalars_and_simple_types/data.xml b/tests/data/xsd/spec/01_scalars_and_simple_types/data.xml new file mode 100644 index 00000000..e89fcd3d --- /dev/null +++ b/tests/data/xsd/spec/01_scalars_and_simple_types/data.xml @@ -0,0 +1,50 @@ + + + + true + -5 + 1024 + 100000 + 999999999999 + 255 + 65535 + 4000000000 + 18446744073709551615 + 1.5 + 2.718281828 + 3.14 + 2025-06-15 + 2025-06-15T12:30:00Z + 2025-06 + 42000000000 + 10 20 30 + alpha beta gamma + 48656c6c6f + SGVsbG8= + + + 0 + + + 1 + -128 + -32768 + -2147483648 + -999999999999 + 0 + 0 + 0 + 0 + -0.5 + 1.25e-4 + 42 + 2025-06-15Z + 2025-06-15T12:30:00.123+02:00 + 1999 + -42 + 100 200 + one two + 48656C6C6F + SGVsbG8= + + diff --git a/tests/data/xsd/spec/01_scalars_and_simple_types/expected.json b/tests/data/xsd/spec/01_scalars_and_simple_types/expected.json new file mode 100644 index 00000000..42319478 --- /dev/null +++ b/tests/data/xsd/spec/01_scalars_and_simple_types/expected.json @@ -0,0 +1,81 @@ +[ + { + "_id": "rec1", + "_status": "active", + "_evidence": [ + 1, + 4, + 9 + ], + "_version": 2, + "flag": true, + "smallInt": -5, + "medInt": 1024, + "stdInt": 100000, + "bigInt": 999999999999, + "uByte": 255, + "uShort": 65535, + "uInt": 4000000000, + "uLong": 18446744073709551615, + "f32": 1.5, + "f64": 2.718281828, + "dec": 3.14, + "created": "2025-06-15", + "updatedAt": "2025-06-15T12:30:00Z", + "published": "2025-06", + "homogeneousUnion": 42000000000, + "scores": [ + 10, + 20, + 30 + ], + "tokens": [ + "alpha", + "beta", + "gamma" + ], + "rawHex": "Hello", + "rawB64": "Hello" + }, + { + "_id": "rec2", + "_status": "archived", + "_version": 1, + "flag": false + }, + { + "_id": "rec3", + "_status": "active", + "_evidence": [ + 10, + 20 + ], + "_version": -42, + "flag": true, + "smallInt": -128, + "medInt": -32768, + "stdInt": -2147483648, + "bigInt": -999999999999, + "uByte": 0, + "uShort": 0, + "uInt": 0, + "uLong": 0, + "f32": -0.5, + "f64": 0.000125, + "dec": 42.0, + "created": "2025-06-15", + "updatedAt": "2025-06-15T12:30:00.123+02:00", + "published": "1999", + "homogeneousUnion": -42, + "scores": [ + 100, + 200 + ], + "tokens": [ + "one", + "two" + ], + "rawHex": "Hello", + "rawB64": "Hello" + } +] diff --git a/tests/data/xsd/spec/01_scalars_and_simple_types/expected_schema.py b/tests/data/xsd/spec/01_scalars_and_simple_types/expected_schema.py new file mode 100644 index 00000000..aac1da5b --- /dev/null +++ b/tests/data/xsd/spec/01_scalars_and_simple_types/expected_schema.py @@ -0,0 +1,61 @@ +"""Expected PyArrow schema for 01_scalars_and_simple_types (xml_tag='record').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field( + "_id", pa.string(), nullable=False, metadata={"comment": "Unique record identifier.", "xsd_type": "xs:ID"} + ), + pa.field( + "_status", + pa.string(), + nullable=False, + metadata={"comment": "Record lifecycle status.", "xsd_type": "xs:string"}, + ), + pa.field( + "_evidence", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field("_version", pa.int32(), nullable=False, metadata={"xsd_type": "xs:int"}), + pa.field( + "flag", pa.bool_(), nullable=False, metadata={"comment": "Primary boolean flag.", "xsd_type": "xs:boolean"} + ), + pa.field("smallInt", pa.int8(), nullable=True, metadata={"xsd_type": "xs:byte"}), + pa.field("medInt", pa.int16(), nullable=True, metadata={"xsd_type": "xs:short"}), + pa.field("stdInt", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field("bigInt", pa.int64(), nullable=True, metadata={"xsd_type": "xs:integer"}), + pa.field("uByte", pa.uint8(), nullable=True, metadata={"xsd_type": "xs:unsignedByte"}), + pa.field("uShort", pa.uint16(), nullable=True, metadata={"xsd_type": "xs:unsignedShort"}), + pa.field("uInt", pa.uint32(), nullable=True, metadata={"xsd_type": "xs:unsignedInt"}), + pa.field("uLong", pa.uint64(), nullable=True, metadata={"xsd_type": "xs:unsignedLong"}), + pa.field("f32", pa.float64(), nullable=True, metadata={"xsd_type": "xs:float"}), + pa.field("f64", pa.float64(), nullable=True, metadata={"xsd_type": "xs:double"}), + pa.field("dec", pa.float64(), nullable=True, metadata={"xsd_type": "xs:decimal"}), + pa.field("created", pa.date32(), nullable=True, metadata={"xsd_type": "xs:date"}), + pa.field("updatedAt", pa.string(), nullable=True, metadata={"xsd_type": "xs:dateTime"}), + pa.field("published", pa.string(), nullable=True, metadata={"xsd_type": "xs:date | xs:gYearMonth | xs:gYear"}), + pa.field("homogeneousUnion", pa.int64(), nullable=True, metadata={"xsd_type": "xs:long | xs:integer"}), + pa.field( + "scores", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=True, + ), + pa.field( + "tokens", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:NMTOKEN"}), + ), + nullable=True, + ), + pa.field("rawHex", pa.binary(), nullable=True, metadata={"xsd_type": "xs:hexBinary"}), + pa.field("rawB64", pa.binary(), nullable=True, metadata={"xsd_type": "xs:base64Binary"}), + ] +) diff --git a/tests/data/xsd/spec/01_scalars_and_simple_types/schema.xsd b/tests/data/xsd/spec/01_scalars_and_simple_types/schema.xsd new file mode 100644 index 00000000..31364311 --- /dev/null +++ b/tests/data/xsd/spec/01_scalars_and_simple_types/schema.xsd @@ -0,0 +1,69 @@ + + + + + + + + + + + + + + + + + Record lifecycle status. + + + + + + + + + + + + + + + + Primary boolean flag. + + + + + + + + + + + + + + + + + + + + + + + + + Unique record identifier. + + + + + + + + + + + diff --git a/tests/data/xsd/spec/02_namespaces_and_nillable/data.xml b/tests/data/xsd/spec/02_namespaces_and_nillable/data.xml new file mode 100644 index 00000000..4eda01ef --- /dev/null +++ b/tests/data/xsd/spec/02_namespaces_and_nillable/data.xml @@ -0,0 +1,15 @@ + + + + + <label lang="en" xsi:nil="true"/> + <comment type="function" xsi:nil="true"/> + <comment type="subunit"> + <text>Homodimer</text> + </comment> + </entry> + <entry id="e2"> + <title>Normal Title + + diff --git a/tests/data/xsd/spec/02_namespaces_and_nillable/expected.json b/tests/data/xsd/spec/02_namespaces_and_nillable/expected.json new file mode 100644 index 00000000..58ff76bd --- /dev/null +++ b/tests/data/xsd/spec/02_namespaces_and_nillable/expected.json @@ -0,0 +1,21 @@ +[ + { + "_id": "e1", + "label": { + "_lang": "en" + }, + "comment": [ + { + "_type": "function" + }, + { + "_type": "subunit", + "text": "Homodimer" + } + ] + }, + { + "_id": "e2", + "title": "Normal Title" + } +] diff --git a/tests/data/xsd/spec/02_namespaces_and_nillable/expected_schema.py b/tests/data/xsd/spec/02_namespaces_and_nillable/expected_schema.py new file mode 100644 index 00000000..a2c573cf --- /dev/null +++ b/tests/data/xsd/spec/02_namespaces_and_nillable/expected_schema.py @@ -0,0 +1,38 @@ +"""Expected PyArrow schema for 02_namespaces_and_nillable (xml_tag='entry').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("title", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "label", + pa.struct( + [ + pa.field("_lang", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "comment", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/02_namespaces_and_nillable/schema.xsd b/tests/data/xsd/spec/02_namespaces_and_nillable/schema.xsd new file mode 100644 index 00000000..e6058f32 --- /dev/null +++ b/tests/data/xsd/spec/02_namespaces_and_nillable/schema.xsd @@ -0,0 +1,37 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/spec/03_mixed_content_raw_text/data.xml b/tests/data/xsd/spec/03_mixed_content_raw_text/data.xml new file mode 100644 index 00000000..b63a0ff7 --- /dev/null +++ b/tests/data/xsd/spec/03_mixed_content_raw_text/data.xml @@ -0,0 +1,13 @@ + + +
+ Ca<sup>2+</sup> signaling in <i>E. <b>coli</b></i> cells + Studied bold & italic text where x < y. + Unlabelled paragraph with 2O and empty tag. + + +
+
+ + </article> +</articles> diff --git a/tests/data/xsd/spec/03_mixed_content_raw_text/expected.json b/tests/data/xsd/spec/03_mixed_content_raw_text/expected.json new file mode 100644 index 00000000..65f443d2 --- /dev/null +++ b/tests/data/xsd/spec/03_mixed_content_raw_text/expected.json @@ -0,0 +1,21 @@ +[ + { + "title": "Ca<sup>2+</sup> signaling in <i>E. <b>coli</b></i> cells", + "abstractText": [ + { + "_Label": "BACKGROUND", + "#text": "Studied <b>bold</b> & <i>italic</i> text where x < y." + }, + { + "#text": "Unlabelled paragraph with <sub>2</sub>O and empty <sup/> tag." + }, + { + "_Label": "EMPTY" + }, + null + ] + }, + { + "title": "" + } +] diff --git a/tests/data/xsd/spec/03_mixed_content_raw_text/expected_schema.py b/tests/data/xsd/spec/03_mixed_content_raw_text/expected_schema.py new file mode 100644 index 00000000..b372cb8f --- /dev/null +++ b/tests/data/xsd/spec/03_mixed_content_raw_text/expected_schema.py @@ -0,0 +1,27 @@ +"""Expected PyArrow schema for 03_mixed_content_raw_text (xml_tag='article').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("title", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "abstractText", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_Label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/03_mixed_content_raw_text/schema.xsd b/tests/data/xsd/spec/03_mixed_content_raw_text/schema.xsd new file mode 100644 index 00000000..2f200b4b --- /dev/null +++ b/tests/data/xsd/spec/03_mixed_content_raw_text/schema.xsd @@ -0,0 +1,35 @@ +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:group name="inlineFormatting"> + <xs:choice> + <xs:element name="b" type="formattedText"/> + <xs:element name="i" type="formattedText"/> + <xs:element name="sup" type="formattedText"/> + <xs:element name="sub" type="formattedText"/> + </xs:choice> + </xs:group> + + <xs:complexType name="formattedText" mixed="true"> + <xs:group ref="inlineFormatting" minOccurs="0" maxOccurs="unbounded"/> + </xs:complexType> + + <xs:complexType name="abstractTextType" mixed="true"> + <xs:group ref="inlineFormatting" minOccurs="0" maxOccurs="unbounded"/> + <xs:attribute name="Label" type="xs:string"/> + </xs:complexType> + + <xs:element name="articles"> + <xs:complexType> + <xs:sequence> + <xs:element name="article" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="title" type="formattedText"/> + <xs:element name="abstractText" type="abstractTextType" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> diff --git a/tests/data/xsd/spec/04_recursion_depth/data.xml b/tests/data/xsd/spec/04_recursion_depth/data.xml new file mode 100644 index 00000000..f330d7d4 --- /dev/null +++ b/tests/data/xsd/spec/04_recursion_depth/data.xml @@ -0,0 +1,22 @@ +<?xml version="1.0" encoding="UTF-8"?> +<doc> + <record id="r1"> + <tree label="L1" weight="10"> + <branch label="L2"> + <branch label="L3" weight="30"> + <branch label="L4"/> + </branch> + <branch label="L3_sibling"/> + </branch> + </tree> + <Section> + <SectionTitle>Ch 1</SectionTitle> + <Section> + <SectionTitle>Sec 1.1</SectionTitle> + <Section> + <SectionTitle>Subsec 1.1.1</SectionTitle> + </Section> + </Section> + </Section> + </record> +</doc> diff --git a/tests/data/xsd/spec/04_recursion_depth/expected.json b/tests/data/xsd/spec/04_recursion_depth/expected.json new file mode 100644 index 00000000..44e8d693 --- /dev/null +++ b/tests/data/xsd/spec/04_recursion_depth/expected.json @@ -0,0 +1,29 @@ +[ + { + "_id": "r1", + "tree": { + "_label": "L1", + "_weight": 10, + "branch": [ + { + "_label": "L2", + "branch": [ + "{\"_label\": \"L3\", \"_weight\": 30, \"branch\": [{\"_label\": \"L4\"}]}", + "{\"_label\": \"L3_sibling\"}" + ] + } + ] + }, + "Section": { + "SectionTitle": "Ch 1", + "Section": [ + { + "SectionTitle": "Sec 1.1", + "Section": [ + "{\"SectionTitle\": \"Subsec 1.1.1\"}" + ] + } + ] + } + } +] diff --git a/tests/data/xsd/spec/04_recursion_depth/expected_depth_1.json b/tests/data/xsd/spec/04_recursion_depth/expected_depth_1.json new file mode 100644 index 00000000..d364b862 --- /dev/null +++ b/tests/data/xsd/spec/04_recursion_depth/expected_depth_1.json @@ -0,0 +1,18 @@ +[ + { + "_id": "r1", + "tree": { + "_label": "L1", + "_weight": 10, + "branch": [ + "{\"_label\": \"L2\", \"branch\": [{\"_label\": \"L3\", \"_weight\": 30, \"branch\": [{\"_label\": \"L4\"}]}, {\"_label\": \"L3_sibling\"}]}" + ] + }, + "Section": { + "SectionTitle": "Ch 1", + "Section": [ + "{\"SectionTitle\": \"Sec 1.1\", \"Section\": [{\"SectionTitle\": \"Subsec 1.1.1\"}]}" + ] + } + } +] diff --git a/tests/data/xsd/spec/04_recursion_depth/expected_schema.py b/tests/data/xsd/spec/04_recursion_depth/expected_schema.py new file mode 100644 index 00000000..d7f6d57f --- /dev/null +++ b/tests/data/xsd/spec/04_recursion_depth/expected_schema.py @@ -0,0 +1,121 @@ +"""Expected PyArrow schema for 04_recursion_depth (xml_tag='record').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "tree", + pa.struct( + [ + pa.field("_label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_weight", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field( + "branch", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field("_weight", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field( + "branch", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "Section", + pa.struct( + [ + pa.field("SectionTitle", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "Section", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "SectionTitle", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "Section", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] +) + +EXPECTED_SCHEMA_DEPTH_1 = pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "tree", + pa.struct( + [ + pa.field("_label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_weight", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field( + "branch", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "Section", + pa.struct( + [ + pa.field("SectionTitle", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "Section", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/04_recursion_depth/schema.xsd b/tests/data/xsd/spec/04_recursion_depth/schema.xsd new file mode 100644 index 00000000..e79b9051 --- /dev/null +++ b/tests/data/xsd/spec/04_recursion_depth/schema.xsd @@ -0,0 +1,35 @@ +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:complexType name="TreeType"> + <xs:sequence> + <xs:element name="branch" type="TreeType" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + <xs:attribute name="label" type="xs:string"/> + <xs:attribute name="weight" type="xs:int"/> + </xs:complexType> + + <xs:element name="Section"> + <xs:complexType> + <xs:sequence> + <xs:element name="SectionTitle" type="xs:string"/> + <xs:element ref="Section" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + </xs:element> + + <xs:element name="doc"> + <xs:complexType> + <xs:sequence> + <xs:element name="record" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="tree" type="TreeType" minOccurs="0"/> + <xs:element ref="Section" minOccurs="0"/> + </xs:sequence> + <xs:attribute name="id" type="xs:string" use="required"/> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> diff --git a/tests/data/xsd/spec/05_groups_and_compositors/data.xml b/tests/data/xsd/spec/05_groups_and_compositors/data.xml new file mode 100644 index 00000000..302a7b2b --- /dev/null +++ b/tests/data/xsd/spec/05_groups_and_compositors/data.xml @@ -0,0 +1,29 @@ +<?xml version="1.0" encoding="UTF-8"?> +<dataset> + <item createdBy="alice" version="1"> + <pagination> + <StartPage>10</StartPage> + <EndPage>20</EndPage> + <MedlinePgn>10-20</MedlinePgn> + </pagination> + <interaction> + <interactant>P12345</interactant> + <interactant>Q67890</interactant> + </interaction> + <meta> + <year>2025</year> + <author>Bob</author> + </meta> + <pairs> + <k>a</k> + <v>1</v> + <k>b</k> + <v>2</v> + </pairs> + </item> + <item> + <pagination> + <MedlinePgn>e12345</MedlinePgn> + </pagination> + </item> +</dataset> diff --git a/tests/data/xsd/spec/05_groups_and_compositors/expected.json b/tests/data/xsd/spec/05_groups_and_compositors/expected.json new file mode 100644 index 00000000..538b9710 --- /dev/null +++ b/tests/data/xsd/spec/05_groups_and_compositors/expected.json @@ -0,0 +1,36 @@ +[ + { + "_createdBy": "alice", + "_version": 1, + "pagination": { + "StartPage": "10", + "EndPage": "20", + "MedlinePgn": "10-20" + }, + "interaction": { + "interactant": [ + "P12345", + "Q67890" + ] + }, + "meta": { + "author": "Bob", + "year": 2025 + }, + "pairs": { + "k": [ + "a", + "b" + ], + "v": [ + 1, + 2 + ] + } + }, + { + "pagination": { + "MedlinePgn": "e12345" + } + } +] diff --git a/tests/data/xsd/spec/05_groups_and_compositors/expected_schema.py b/tests/data/xsd/spec/05_groups_and_compositors/expected_schema.py new file mode 100644 index 00000000..390b5318 --- /dev/null +++ b/tests/data/xsd/spec/05_groups_and_compositors/expected_schema.py @@ -0,0 +1,70 @@ +"""Expected PyArrow schema for 05_groups_and_compositors (xml_tag='item').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_createdBy", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_version", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field( + "pagination", + pa.struct( + [ + pa.field("StartPage", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("EndPage", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("MedlinePgn", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "interaction", + pa.struct( + [ + pa.field( + "interactant", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "meta", + pa.struct( + [ + pa.field("author", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("year", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ] + ), + nullable=True, + ), + pa.field( + "pairs", + pa.struct( + [ + pa.field( + "k", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + ), + pa.field( + "v", + pa.list_( + pa.field("item", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/05_groups_and_compositors/schema.xsd b/tests/data/xsd/spec/05_groups_and_compositors/schema.xsd new file mode 100644 index 00000000..d382efdf --- /dev/null +++ b/tests/data/xsd/spec/05_groups_and_compositors/schema.xsd @@ -0,0 +1,61 @@ +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:attributeGroup name="auditAttrs"> + <xs:attribute name="createdBy" type="xs:string"/> + <xs:attribute name="version" type="xs:int"/> + </xs:attributeGroup> + + <xs:group name="kvGroup"> + <xs:sequence> + <xs:element name="k" type="xs:string"/> + <xs:element name="v" type="xs:int"/> + </xs:sequence> + </xs:group> + + <xs:complexType name="paginationType"> + <xs:choice> + <xs:sequence> + <xs:element name="StartPage" type="xs:string"/> + <xs:element name="EndPage" type="xs:string" minOccurs="0"/> + <xs:element name="MedlinePgn" type="xs:string" minOccurs="0"/> + </xs:sequence> + <xs:element name="MedlinePgn" type="xs:string"/> + </xs:choice> + </xs:complexType> + + <xs:complexType name="interactionType"> + <xs:sequence> + <xs:element name="interactant" type="xs:string"/> + <xs:element name="interactant" type="xs:string"/> + </xs:sequence> + </xs:complexType> + + <xs:complexType name="metadataAllType"> + <xs:all> + <xs:element name="author" type="xs:string"/> + <xs:element name="year" type="xs:int" minOccurs="0"/> + </xs:all> + </xs:complexType> + + <xs:element name="dataset"> + <xs:complexType> + <xs:sequence> + <xs:element name="item" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="pagination" type="paginationType" minOccurs="0"/> + <xs:element name="interaction" type="interactionType" minOccurs="0"/> + <xs:element name="meta" type="metadataAllType" minOccurs="0"/> + <xs:element name="pairs" minOccurs="0"> + <xs:complexType> + <xs:group ref="kvGroup" maxOccurs="unbounded"/> + </xs:complexType> + </xs:element> + </xs:sequence> + <xs:attributeGroup ref="auditAttrs"/> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> diff --git a/tests/data/xsd/spec/06_context_dependent_paths/data.xml b/tests/data/xsd/spec/06_context_dependent_paths/data.xml new file mode 100644 index 00000000..48246436 --- /dev/null +++ b/tests/data/xsd/spec/06_context_dependent_paths/data.xml @@ -0,0 +1,14 @@ +<?xml version="1.0" encoding="UTF-8"?> +<root> + <left> + <node> + <leaf>l1</leaf> + </node> + </left> + <right> + <node> + <leaf>r1</leaf> + <leaf lang="en">r2</leaf> + </node> + </right> +</root> diff --git a/tests/data/xsd/spec/06_context_dependent_paths/expected.json b/tests/data/xsd/spec/06_context_dependent_paths/expected.json new file mode 100644 index 00000000..9551adce --- /dev/null +++ b/tests/data/xsd/spec/06_context_dependent_paths/expected.json @@ -0,0 +1,22 @@ +[ + { + "left": { + "node": { + "leaf": "l1" + } + }, + "right": { + "node": { + "leaf": [ + { + "#text": "r1" + }, + { + "_lang": "en", + "#text": "r2" + } + ] + } + } + } +] diff --git a/tests/data/xsd/spec/06_context_dependent_paths/expected_schema.py b/tests/data/xsd/spec/06_context_dependent_paths/expected_schema.py new file mode 100644 index 00000000..753f5621 --- /dev/null +++ b/tests/data/xsd/spec/06_context_dependent_paths/expected_schema.py @@ -0,0 +1,69 @@ +"""Expected PyArrow schema for 06_context_dependent_paths (xml_tag='root').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field( + "left", + pa.struct( + [ + pa.field( + "node", + pa.struct( + [ + pa.field("leaf", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + pa.field( + "right", + pa.struct( + [ + pa.field( + "node", + pa.struct( + [ + pa.field( + "leaf", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_lang", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + ] +) diff --git a/tests/data/xsd/spec/06_context_dependent_paths/schema.xsd b/tests/data/xsd/spec/06_context_dependent_paths/schema.xsd new file mode 100644 index 00000000..79e9a9e6 --- /dev/null +++ b/tests/data/xsd/spec/06_context_dependent_paths/schema.xsd @@ -0,0 +1,43 @@ +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:element name="root"> + <xs:complexType> + <xs:sequence> + <xs:element name="left"> + <xs:complexType> + <xs:sequence> + <xs:element name="node"> + <xs:complexType> + <xs:sequence> + <xs:element name="leaf" type="xs:string"/> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> + <xs:element name="right"> + <xs:complexType> + <xs:sequence> + <xs:element name="node"> + <xs:complexType> + <xs:sequence> + <xs:element name="leaf" maxOccurs="unbounded"> + <xs:complexType> + <xs:simpleContent> + <xs:extension base="xs:string"> + <xs:attribute name="lang" type="xs:string"/> + </xs:extension> + </xs:simpleContent> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> diff --git a/tests/data/xsd/spec/07_multi_root_shared_type/data.xml b/tests/data/xsd/spec/07_multi_root_shared_type/data.xml new file mode 100644 index 00000000..b6946958 --- /dev/null +++ b/tests/data/xsd/spec/07_multi_root_shared_type/data.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="UTF-8"?> +<Cluster50> + <entry id="C1"> + <name>First Cluster</name> + </entry> + <entry id="C2"> + <name>Second Cluster</name> + </entry> +</Cluster50> diff --git a/tests/data/xsd/spec/07_multi_root_shared_type/expected.json b/tests/data/xsd/spec/07_multi_root_shared_type/expected.json new file mode 100644 index 00000000..83219793 --- /dev/null +++ b/tests/data/xsd/spec/07_multi_root_shared_type/expected.json @@ -0,0 +1,10 @@ +[ + { + "_id": "C1", + "name": "First Cluster" + }, + { + "_id": "C2", + "name": "Second Cluster" + } +] diff --git a/tests/data/xsd/spec/07_multi_root_shared_type/expected_schema.py b/tests/data/xsd/spec/07_multi_root_shared_type/expected_schema.py new file mode 100644 index 00000000..9b5f5a7f --- /dev/null +++ b/tests/data/xsd/spec/07_multi_root_shared_type/expected_schema.py @@ -0,0 +1,12 @@ +"""Expected PyArrow schema for 07_multi_root_shared_type (xml_tag='entry').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] +) diff --git a/tests/data/xsd/spec/07_multi_root_shared_type/schema.xsd b/tests/data/xsd/spec/07_multi_root_shared_type/schema.xsd new file mode 100644 index 00000000..0f14887c --- /dev/null +++ b/tests/data/xsd/spec/07_multi_root_shared_type/schema.xsd @@ -0,0 +1,19 @@ +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:complexType name="entryType"> + <xs:sequence> + <xs:element name="name" type="xs:string"/> + </xs:sequence> + <xs:attribute name="id" type="xs:string" use="required"/> + </xs:complexType> + + <xs:complexType name="clusterSetType"> + <xs:sequence> + <xs:element name="entry" type="entryType" minOccurs="0" maxOccurs="unbounded"/> + </xs:sequence> + </xs:complexType> + + <xs:element name="Cluster100" type="clusterSetType"/> + <xs:element name="Cluster90" type="clusterSetType"/> + <xs:element name="Cluster50" type="clusterSetType"/> +</xs:schema> diff --git a/tests/data/xsd/spec/08_nullability_and_empty_tags/data.xml b/tests/data/xsd/spec/08_nullability_and_empty_tags/data.xml new file mode 100644 index 00000000..3ce65c5a --- /dev/null +++ b/tests/data/xsd/spec/08_nullability_and_empty_tags/data.xml @@ -0,0 +1,9 @@ +<?xml version="1.0" encoding="UTF-8"?> +<root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> + <entry id="e1"> + <name>BRCA1</name> + <summary/> + <geneId>672</geneId> + <comment type="function" xsi:nil="true"/> + </entry> +</root> diff --git a/tests/data/xsd/spec/08_nullability_and_empty_tags/expected.json b/tests/data/xsd/spec/08_nullability_and_empty_tags/expected.json new file mode 100644 index 00000000..7404854c --- /dev/null +++ b/tests/data/xsd/spec/08_nullability_and_empty_tags/expected.json @@ -0,0 +1,11 @@ +[ + { + "_id": "e1", + "name": "BRCA1", + "summary": "", + "geneId": 672, + "comment": { + "_type": "function" + } + } +] diff --git a/tests/data/xsd/spec/08_nullability_and_empty_tags/expected_schema.py b/tests/data/xsd/spec/08_nullability_and_empty_tags/expected_schema.py new file mode 100644 index 00000000..12b33128 --- /dev/null +++ b/tests/data/xsd/spec/08_nullability_and_empty_tags/expected_schema.py @@ -0,0 +1,27 @@ +"""Expected PyArrow schema for 08_nullability_and_empty_tags (xml_tag='entry').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("_source", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("summary", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("optionalNote", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("geneId", pa.int32(), nullable=True, metadata={"xsd_type": "xs:int"}), + pa.field("proteinId", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "comment", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/08_nullability_and_empty_tags/schema.xsd b/tests/data/xsd/spec/08_nullability_and_empty_tags/schema.xsd new file mode 100644 index 00000000..eec356df --- /dev/null +++ b/tests/data/xsd/spec/08_nullability_and_empty_tags/schema.xsd @@ -0,0 +1,32 @@ +<?xml version="1.0" encoding="UTF-8"?> +<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"> + <xs:complexType name="commentType"> + <xs:sequence> + <xs:element name="text" type="xs:string"/> + </xs:sequence> + <xs:attribute name="type" type="xs:string" use="required"/> + </xs:complexType> + + <xs:element name="root"> + <xs:complexType> + <xs:sequence> + <xs:element name="entry" maxOccurs="unbounded"> + <xs:complexType> + <xs:sequence> + <xs:element name="name" type="xs:string"/> + <xs:element name="summary" type="xs:string"/> + <xs:element name="optionalNote" type="xs:string" minOccurs="0"/> + <xs:choice> + <xs:element name="geneId" type="xs:int"/> + <xs:element name="proteinId" type="xs:string"/> + </xs:choice> + <xs:element name="comment" type="commentType" nillable="true"/> + </xs:sequence> + <xs:attribute name="id" type="xs:string" use="required"/> + <xs:attribute name="source" type="xs:string" use="optional"/> + </xs:complexType> + </xs:element> + </xs:sequence> + </xs:complexType> + </xs:element> +</xs:schema> diff --git a/tests/data/xsd/spec/09_empty_marker_elements/data.xml b/tests/data/xsd/spec/09_empty_marker_elements/data.xml new file mode 100644 index 00000000..a9f06029 --- /dev/null +++ b/tests/data/xsd/spec/09_empty_marker_elements/data.xml @@ -0,0 +1,12 @@ +<?xml version="1.0" encoding="UTF-8"?> +<books> + <book> + <title>In-Process Book + + + + + + Completed Book + + diff --git a/tests/data/xsd/spec/09_empty_marker_elements/expected.json b/tests/data/xsd/spec/09_empty_marker_elements/expected.json new file mode 100644 index 00000000..f4b88e91 --- /dev/null +++ b/tests/data/xsd/spec/09_empty_marker_elements/expected.json @@ -0,0 +1,13 @@ +[ + { + "title": "In-Process Book", + "BookInProcess": "", + "flag": [ + "", + "" + ] + }, + { + "title": "Completed Book" + } +] diff --git a/tests/data/xsd/spec/09_empty_marker_elements/expected_schema.py b/tests/data/xsd/spec/09_empty_marker_elements/expected_schema.py new file mode 100644 index 00000000..529fbc96 --- /dev/null +++ b/tests/data/xsd/spec/09_empty_marker_elements/expected_schema.py @@ -0,0 +1,19 @@ +"""Expected PyArrow schema for 09_empty_marker_elements (xml_tag='book').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("title", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("BookInProcess", pa.string(), nullable=True), + pa.field( + "flag", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/09_empty_marker_elements/schema.xsd b/tests/data/xsd/spec/09_empty_marker_elements/schema.xsd new file mode 100644 index 00000000..3ffbb59e --- /dev/null +++ b/tests/data/xsd/spec/09_empty_marker_elements/schema.xsd @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/spec/10_combinations/data.xml b/tests/data/xsd/spec/10_combinations/data.xml new file mode 100644 index 00000000..64669df1 --- /dev/null +++ b/tests/data/xsd/spec/10_combinations/data.xml @@ -0,0 +1,240 @@ + + + + + + + + a + x + only + + + + + x + + + y + b1 + b2 + + + + + + + padded + Tom & Jerry + + + + + one + two + + + + + + + solo + + + + + + + + + + + + + + + + + + + + + + + + + g1 + a1 + a2 + + + + child-name + + + + + + + + + g2 + + + 1 + + + + + + + + plain text + Hello + world again + + + x + + + A + 1B + 2C + 3 + + Intro + bold + + side text + just text + + + + + only italic + + + bare note + + + + + + a1 + b1 + a2 + b2 + + + + solo + + a + 1 + + + + + + + a + 1 + b + 2 + + + + + + Book title + + Ch1 + + + Ch2 + Ch2 alt + + + + + + + Lonely + + + + + + + l1 + + + + + r1 + r2 + + + + + + + + + + + + + r + + + + + + + + + + + + + + + + + + + + + + + + + + + + s1 + + base + x1 + + + + + s1 + s2 + + x1 + x2 + + + diff --git a/tests/data/xsd/spec/10_combinations/expected.json b/tests/data/xsd/spec/10_combinations/expected.json new file mode 100644 index 00000000..0bd55d4f --- /dev/null +++ b/tests/data/xsd/spec/10_combinations/expected.json @@ -0,0 +1,377 @@ +[ + { + "_id": "r01" + }, + { + "_id": "r02", + "plain": "a", + "plainList": [ + "x" + ], + "bounded": [ + "only" + ] + }, + { + "_id": "r03", + "plainList": [ + "x", + null, + null, + "y" + ], + "bounded": [ + "b1", + "b2" + ] + }, + { + "_id": "r04", + "plain": "a < b & c", + "plainList": [ + "padded", + "Tom & Jerry" + ] + }, + { + "_id": "r05", + "label": { + "_lang": "en", + "#text": "Hello" + }, + "labelList": [ + { + "_lang": "en", + "_source": "curated", + "#text": "one" + }, + { + "#text": "two" + }, + { + "_lang": "fr" + }, + null + ] + }, + { + "_id": "r06", + "label": { + "#text": "Hello" + }, + "labelList": [ + { + "#text": "solo" + } + ] + }, + { + "_id": "r07", + "label": { + "_lang": "de" + }, + "textNoAttr": "t", + "measure": { + "_unit": "kg", + "#text": 3.5 + } + }, + { + "_id": "r08", + "measure": { + "#text": 7.0 + } + }, + { + "_id": "r09", + "flag": { + "_on": true + }, + "flagList": [ + { + "_on": false + }, + null + ], + "marker": [ + "", + "" + ] + }, + { + "_id": "r10", + "flagList": [ + { + "_on": true + } + ], + "marker": [ + "" + ] + }, + { + "_id": "r11", + "group": { + "name": "g1", + "alias": [ + "a1", + "a2" + ] + }, + "entity": [ + { + "_type": "person", + "_name": "attr-name", + "name": "child-name", + "prop": [ + { + "_key": "k1", + "_value": "v1" + }, + { + "_key": "k2" + } + ] + }, + { + "_type": "org" + } + ] + }, + { + "_id": "r12", + "group": { + "name": "g2" + }, + "box": { + "item": [ + "1" + ] + }, + "entity": [ + { + "prop": [ + { + "_key": "only" + } + ] + } + ] + }, + { + "_id": "r13", + "para": [ + { + "#text": "plain text" + }, + { + "#text": "Hello\n world again" + }, + { + "#text": "x" + }, + null, + { + "#text": "A\n 1B\n 2C\n 3" + }, + { + "_style": "em", + "#text": "Intro\n bold" + } + ], + "note": { + "_kind": "aside", + "#text": "side text" + }, + "remark": "just text" + }, + { + "_id": "r14", + "para": [ + { + "#text": "only italic" + }, + { + "_style": "x" + } + ], + "note": { + "#text": "bare note" + } + }, + { + "_id": "r15", + "note": { + "_kind": "empty" + }, + "optA": [ + "a1", + "a2" + ], + "optB": [ + { + "_lang": "en", + "#text": "b1" + }, + { + "#text": "b2" + } + ] + }, + { + "_id": "r16", + "optA": [ + "solo" + ], + "pairs": { + "k": [ + "a" + ], + "v": [ + "1" + ] + } + }, + { + "_id": "r17", + "optB": [ + null + ], + "pairs": { + "k": [ + "a", + "b" + ], + "v": [ + "1", + "2" + ] + } + }, + { + "_id": "r18", + "book": { + "title": "Book title", + "chapter": [ + { + "title": [ + { + "#text": "Ch1" + } + ] + }, + { + "title": [ + { + "_lang": "en", + "#text": "Ch2" + }, + { + "#text": "Ch2 alt" + } + ] + } + ] + } + }, + { + "_id": "r19", + "book": { + "title": "Lonely" + } + }, + { + "_id": "r20", + "left": { + "node": { + "leaf": "l1" + } + }, + "right": { + "node": { + "leaf": [ + { + "#text": "r1" + }, + { + "_lang": "en", + "#text": "r2" + } + ] + } + } + }, + { + "_id": "r21", + "left": { + "node": { + "leaf": "" + } + }, + "right": { + "node": { + "leaf": [ + { + "#text": "r" + } + ] + } + } + }, + { + "_id": "r22", + "tree": { + "_label": "root", + "branch": [ + { + "_label": "a", + "branch": [ + "{\"_label\": \"a1\"}", + "{\"_label\": \"a2\", \"branch\": [{\"_label\": \"a2x\"}]}" + ] + }, + null + ] + } + }, + { + "_id": "r23", + "tree": { + "_label": "t", + "branch": [ + { + "branch": [ + "{\"_label\": \"deep\"}" + ] + } + ] + } + }, + { + "_id": "r24", + "shared": [ + "s1" + ], + "derived": { + "_baseAttr": "b", + "_extraAttr": "e", + "baseField": "base", + "extraField": [ + "x1" + ] + } + }, + { + "_id": "r25", + "shared": [ + "s1", + "s2" + ], + "derived": { + "extraField": [ + "x1", + "x2" + ] + } + } +] diff --git a/tests/data/xsd/spec/10_combinations/expected_schema.py b/tests/data/xsd/spec/10_combinations/expected_schema.py new file mode 100644 index 00000000..dd6c1a2f --- /dev/null +++ b/tests/data/xsd/spec/10_combinations/expected_schema.py @@ -0,0 +1,421 @@ +"""Expected PyArrow schema for 10_combinations (xml_tag='record').""" + +# ruff: noqa: INP001 + +import pyarrow as pa + +EXPECTED_SCHEMA = pa.schema( + [ + pa.field("_id", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field("plain", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "plainList", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "bounded", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "label", + pa.struct( + [ + pa.field("_lang", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_source", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field( + "labelList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_lang", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_source", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field("textNoAttr", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "measure", + pa.struct( + [ + pa.field("_unit", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.float64(), nullable=True, metadata={"xsd_type": "xs:decimal"}), + ] + ), + nullable=True, + ), + pa.field( + "flag", + pa.struct( + [ + pa.field("_on", pa.bool_(), nullable=True, metadata={"xsd_type": "xs:boolean"}), + ] + ), + nullable=True, + ), + pa.field( + "flagList", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_on", pa.bool_(), nullable=True, metadata={"xsd_type": "xs:boolean"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "marker", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + pa.field( + "group", + pa.struct( + [ + pa.field("name", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "alias", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "box", + pa.struct( + [ + pa.field( + "item", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "entity", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_type", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_name", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("name", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "prop", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_key", + pa.string(), + nullable=False, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_value", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "para", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_style", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "note", + pa.struct( + [ + pa.field("_kind", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + pa.field("remark", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "optA", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "optB", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field("_lang", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_source", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("#text", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + pa.field( + "pairs", + pa.struct( + [ + pa.field( + "k", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + ), + pa.field( + "v", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "book", + pa.struct( + [ + pa.field("title", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + pa.field( + "chapter", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "title", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_lang", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_source", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "left", + pa.struct( + [ + pa.field( + "node", + pa.struct( + [ + pa.field("leaf", pa.string(), nullable=False, metadata={"xsd_type": "xs:string"}), + ] + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "right", + pa.struct( + [ + pa.field( + "node", + pa.struct( + [ + pa.field( + "leaf", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_lang", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "_source", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + pa.field( + "#text", + pa.string(), + nullable=True, + metadata={"xsd_type": "xs:string"}, + ), + ] + ), + nullable=True, + ), + ), + nullable=False, + ), + ] + ), + nullable=False, + ), + ] + ), + nullable=True, + ), + pa.field( + "tree", + pa.struct( + [ + pa.field("_label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "branch", + pa.list_( + pa.field( + "item", + pa.struct( + [ + pa.field( + "_label", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"} + ), + pa.field( + "branch", + pa.list_( + pa.field("item", pa.string(), nullable=True), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + pa.field( + "shared", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + pa.field( + "derived", + pa.struct( + [ + pa.field("_baseAttr", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("_extraAttr", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field("baseField", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + pa.field( + "extraField", + pa.list_( + pa.field("item", pa.string(), nullable=True, metadata={"xsd_type": "xs:string"}), + ), + nullable=True, + ), + ] + ), + nullable=True, + ), + ] +) diff --git a/tests/data/xsd/spec/10_combinations/schema.xsd b/tests/data/xsd/spec/10_combinations/schema.xsd new file mode 100644 index 00000000..5a7e157a --- /dev/null +++ b/tests/data/xsd/spec/10_combinations/schema.xsd @@ -0,0 +1,235 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/abstract_element.xsd b/tests/data/xsd/spec/errors/abstract_element.xsd new file mode 100644 index 00000000..50de1258 --- /dev/null +++ b/tests/data/xsd/spec/errors/abstract_element.xsd @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/abstract_type.xsd b/tests/data/xsd/spec/errors/abstract_type.xsd new file mode 100644 index 00000000..7048d5c9 --- /dev/null +++ b/tests/data/xsd/spec/errors/abstract_type.xsd @@ -0,0 +1,15 @@ + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/ambiguous_xml_tag.xsd b/tests/data/xsd/spec/errors/ambiguous_xml_tag.xsd new file mode 100644 index 00000000..81eab948 --- /dev/null +++ b/tests/data/xsd/spec/errors/ambiguous_xml_tag.xsd @@ -0,0 +1,29 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/attr_child_name_collision.xsd b/tests/data/xsd/spec/errors/attr_child_name_collision.xsd new file mode 100644 index 00000000..ef3f2f1e --- /dev/null +++ b/tests/data/xsd/spec/errors/attr_child_name_collision.xsd @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/empty_marker_top_level.xsd b/tests/data/xsd/spec/errors/empty_marker_top_level.xsd new file mode 100644 index 00000000..9e640600 --- /dev/null +++ b/tests/data/xsd/spec/errors/empty_marker_top_level.xsd @@ -0,0 +1,6 @@ + + + + + + diff --git a/tests/data/xsd/spec/errors/scalar_top_level.xsd b/tests/data/xsd/spec/errors/scalar_top_level.xsd new file mode 100644 index 00000000..dc22ba2c --- /dev/null +++ b/tests/data/xsd/spec/errors/scalar_top_level.xsd @@ -0,0 +1,4 @@ + + + + diff --git a/tests/data/xsd/spec/errors/substitution_group.xsd b/tests/data/xsd/spec/errors/substitution_group.xsd new file mode 100644 index 00000000..638be6da --- /dev/null +++ b/tests/data/xsd/spec/errors/substitution_group.xsd @@ -0,0 +1,12 @@ + + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/wildcard_any_attribute.xsd b/tests/data/xsd/spec/errors/wildcard_any_attribute.xsd new file mode 100644 index 00000000..e319240f --- /dev/null +++ b/tests/data/xsd/spec/errors/wildcard_any_attribute.xsd @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/tests/data/xsd/spec/errors/wildcard_any_element.xsd b/tests/data/xsd/spec/errors/wildcard_any_element.xsd new file mode 100644 index 00000000..64497408 --- /dev/null +++ b/tests/data/xsd/spec/errors/wildcard_any_element.xsd @@ -0,0 +1,11 @@ + + + + + + + + + + +