feat: carry serializer metadata beside the mapping (#331)

CHANGELOG.md

@@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## \[Unreleased\]
 
+### Added
+
+- Python 3.14 is now tested and declared as supported. No source changes were needed; the full
+  suite passes on 3.14 as-is.
+- `SerializationOptions.metadata_sidecar`, which carries `__is_block__`, `__comments__` and `__inline_comments__` beside the mapping rather than among its keys. HCL reserves none of those names, so a document may declare an attribute called any of them -- and in-band one of the two has to lose: on read the marker overwrites the attribute, on write the deserializer drops it, and by then the dict holds a single value with no way to tell which happened. With the option set, `loads` returns an `HclDict`, a `dict` subclass whose `hcl_meta` holds the three, so the mapping contains attributes and nothing else. `dumps` accepts either form, including a hand-built dict using the old keys. Off by default: the keys are a documented part of the output shape, and JSON cannot carry a sidecar. `HclDict`, `HclMeta` and `meta_of` are exported from `hcl2`. The `{label: body}` levels around a labelled block are `HclDict`s as well, so a label spelled like one of the three names is not taken for metadata. The query views follow the option too: `to_dict` on a block or an attribute view returns an `HclDict`, with any adjacent comments in its metadata. Copying, merging with `|` and pickling carry the metadata; `dict(d)` and `{**d}` deliberately do not, since asking for a `dict` gives the mapping and nothing else. ([#331](https://github.com/amplify-education/python-hcl2/issues/331))
+
 ### Changed
 
 - **Breaking for direct `cli.*` imports.** The CLI modules moved from a top-level `cli` package
@@ -19,11 +25,6 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
   because a shim would still occupy the colliding name.
 - The redundant `cli/py.typed` marker is gone; `hcl2/py.typed` already covers `hcl2.cli`.
 
-### Added
-
-- Python 3.14 is now tested and declared as supported. No source changes were needed; the full
-  suite passes on 3.14 as-is.
-
 ## \[8.1.4\] - 2026-09-08
 
 ### Fixed

CLAUDE.md

@@ -25,6 +25,7 @@ The **Direct** pipeline (`parse_to_tree` → `transform` → `to_lark` → `reco
 | `hcl2/builder.py` | Programmatic HCL document construction |
 | `hcl2/walk.py` | Generic tree-walking primitives for the LarkElement IR tree |
 | `hcl2/utils.py` | `SerializationOptions`, `SerializationContext`, string helpers |
+| `hcl2/meta.py` | `HclDict`, `HclMeta`, `meta_of` — metadata carried beside a body under `metadata_sidecar` |
 | `hcl2/const.py` | Constants: `IS_BLOCK`, `COMMENTS_KEY`, `INLINE_COMMENTS_KEY` |
 | `hcl2/cli/helpers.py` | File/directory/stdin conversion helpers |
 | `hcl2/cli/hcl_to_json.py` | `hcl2tojson` entry point |
@@ -75,11 +76,12 @@ Follows the `json` module convention. All option parameters are keyword-only.
 - `dump/dumps` — Python dict → HCL2 text
 - `query` — HCL2 text/file → `DocumentView` for structured queries
 - Intermediate stages: `parse/parses`, `parse_to_tree/parses_to_tree`, `transform`, `serialize`, `from_dict`, `from_json`, `reconstruct`
+- Metadata sidecar (`hcl2/meta.py`, exported from `hcl2`): `HclDict` (a `dict` whose `hcl_meta` holds the block marker and comments), `HclMeta`, and `meta_of(value)`, which returns the metadata or `None`
 
 ### Option Dataclasses
 
 **`SerializationOptions`** (LarkElement → dict):
-`with_comments`, `with_meta`, `wrap_objects`, `wrap_tuples`, `explicit_blocks`, `preserve_heredocs`, `force_operation_parentheses`, `preserve_scientific_notation`, `strip_string_quotes`
+`with_comments`, `with_meta`, `wrap_objects`, `wrap_tuples`, `explicit_blocks`, `preserve_heredocs`, `force_operation_parentheses`, `preserve_scientific_notation`, `strip_string_quotes`, `metadata_sidecar`
 
 **`DeserializerOptions`** (dict → LarkElement):
 `heredocs_to_strings`, `strings_to_heredocs`, `object_elements_colon`, `object_elements_trailing_comma`

docs/01_getting_started.md

@@ -78,6 +78,7 @@ data = loads(text, serialization_options=SerializationOptions(
 | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations                                                                                                         |
 | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is                                                                                                                  |
 | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** |
+| `metadata_sidecar` | `bool` | `False` | Carry `__is_block__`, `__comments__` and `__inline_comments__` beside each body instead of among its keys, so a document attribute with one of those names survives. Bodies, objects and query results come back as `HclDict`, a `dict` subclass; read the metadata with `hcl2.meta_of(value)` (see [Advanced API](03_advanced_api.md#metadata-sidecar)). `dumps` accepts either form. JSON cannot carry the sidecar, so `json.dumps` of the result writes the attributes only. |
 
 ### Comment Format
 

docs/03_advanced_api.md

@@ -45,6 +45,29 @@ from hcl2 import SerializationOptions
 data = hcl2.serialize(tree, serialization_options=SerializationOptions(with_meta=True))
 ```
 
+### Metadata sidecar
+
+By default the serializer reports what it knows about a body -- that it is a
+block and its comments -- as `__is_block__`, `__comments__` and
+`__inline_comments__` keys among the attributes. HCL reserves none of those names, so a document that
+declares one loses either the attribute or the metadata. With
+`metadata_sidecar=True` each body is an `HclDict` instead: a `dict` holding the
+attributes and nothing else, with the metadata on `hcl_meta`.
+
+```python
+from hcl2 import HclDict, meta_of, SerializationOptions
+
+data = hcl2.loads(text, serialization_options=SerializationOptions(metadata_sidecar=True))
+body = data["resource"][0]['"aws_instance"']['"web"']
+meta_of(body).is_block      # True
+meta_of(body).comments      # [{"value": "..."}]
+meta_of({"plain": "dict"})  # None
+```
+
+`copy()`, `copy.copy`, `copy.deepcopy`, pickling and `|` keep the metadata;
+`dict(body)` and `{**body}` give the attributes alone. `dumps` accepts either
+form, including a hand-built dict using the in-band keys.
+
 ### from_dict / from_json — Python dict or JSON to LarkElement tree
 
 ```python

hcl2/__init__.py

@@ -24,27 +24,31 @@
 from .builder import Builder
 from .deserializer import DeserializerOptions
 from .formatter import FormatterOptions
+from .meta import HclDict, HclMeta, meta_of
 from .rules.base import StartRule
 from .utils import SerializationOptions
 
 __all__ = [
+    "Builder",
+    "DeserializerOptions",
     "dump",
     "dumps",
+    "FormatterOptions",
     "from_dict",
     "from_json",
+    "HclDict",
+    "HclMeta",
     "load",
     "loads",
+    "meta_of",
     "parse",
     "parse_to_tree",
     "parses",
     "parses_to_tree",
     "query",
     "reconstruct",
+    "SerializationOptions",
     "serialize",
-    "transform",
-    "Builder",
-    "DeserializerOptions",
-    "FormatterOptions",
     "StartRule",
-    "SerializationOptions",
+    "transform",
 ]

hcl2/deserializer.py

@@ -10,6 +10,7 @@
 from regex import regex
 
 from hcl2.const import COMMENTS_KEY, INLINE_COMMENTS_KEY, IS_BLOCK
+from hcl2.meta import meta_of
 from hcl2.parser import parser as _get_parser
 from hcl2.rules.abstract import LarkElement, LarkRule
 from hcl2.rules.base import (
@@ -144,7 +145,7 @@ def _deserialize_block_elements(self, value: dict) -> List[LarkElement]:
 
             else:
                 # otherwise it's just an attribute
-                if not self._is_reserved_key(key):
+                if not self._is_reserved_key(key, value):
                     children.append(self._deserialize_attribute(key, val))
 
         return children
@@ -294,8 +295,8 @@ def _deserialize_block(self, first_label: str, value: dict) -> BlockRule:
         body = value
 
         # Keep peeling off single-key layers until we hit the body (dict with IS_BLOCK)
-        while isinstance(body, dict) and not body.get(IS_BLOCK):
-            non_block_keys = [k for k in body.keys() if not self._is_reserved_key(k)]
+        while isinstance(body, dict) and not self._is_marked_block(body):
+            non_block_keys = [k for k in body.keys() if not self._is_reserved_key(k, body)]
             if len(non_block_keys) == 1:
                 # This is another label level
                 label = non_block_keys[0]
@@ -367,10 +368,23 @@ def _deserialize_object_elem(self, key: Any, value: Any) -> ObjectElemRule:
 
         return ObjectElemRule(result)
 
-    def _is_reserved_key(self, key: str) -> bool:
-        """Check if a key is a reserved metadata key that should be skipped during deserialization."""
+    def _is_reserved_key(self, key: str, container: Optional[dict] = None) -> bool:
+        """Whether *key* in *container* is metadata rather than an attribute.
+
+        A container carrying its metadata beside the mapping reserves nothing:
+        every key in it is an attribute the document declared, including one
+        spelled `__is_block__`. Only the in-band form has to reserve the names,
+        and only there can it lose an attribute to one.
+        """
+        if container is not None and meta_of(container) is not None:
+            return False
         return key in (IS_BLOCK, COMMENTS_KEY, INLINE_COMMENTS_KEY)
 
+    def _is_marked_block(self, body: dict) -> bool:
+        """Whether *body* is itself a block, in whichever form marks it."""
+        meta = meta_of(body)
+        return meta.is_block if meta is not None else bool(body.get(IS_BLOCK))
+
     def _is_expression(self, value: Any) -> bool:
         return isinstance(value, str) and value.startswith("${") and value.endswith("}")
 
@@ -387,8 +401,12 @@ def _is_block(self, value: Any) -> bool:
         return False
 
     def _contains_block_marker(self, obj: dict) -> bool:
-        """Recursively check if a dict contains IS_BLOCK marker anywhere"""
-        if obj.get(IS_BLOCK):
+        """Recursively check whether a dict is marked as a block, in either form"""
+        meta = meta_of(obj)
+        if meta is not None:
+            if meta.is_block:
+                return True
+        elif obj.get(IS_BLOCK):
             return True
         for value in obj.values():
             if isinstance(value, dict) and self._contains_block_marker(value):

hcl2/meta.py

@@ -0,0 +1,186 @@
+"""Out-of-band metadata for serialized bodies.
+
+The serializer has three things to say about a body that are not attributes of
+it: that it is a block, what comments surround it, and which of those were
+inline. They have always travelled as `__is_block__`, `__comments__` and
+`__inline_comments__` keys in the same dict as the attributes, which works only
+while no document declares an attribute by those names. HCL puts no such name
+out of reach, so one that does loses either the attribute or the metadata,
+silently and in both directions.
+
+`HclDict` carries them beside the mapping instead. It is a `dict`, so every
+consumer that reads attributes keeps working unchanged, and `hcl_meta` holds
+what used to sit among them.
+"""
+
+import copy as copy_module
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional, Tuple
+
+
+@dataclass
+class HclMeta:
+    """What the serializer knows about a body that is not one of its attributes."""
+
+    is_block: bool = False
+    comments: List[dict] = field(default_factory=list)
+    inline_comments: List[dict] = field(default_factory=list)
+    # The span `with_meta` reports, 1-based and inclusive. None when the
+    # option is off or the tree carries no positions -- a tree built by the
+    # deserializer has none, and that is "no span", not line zero.
+    start_line: Optional[int] = None
+    end_line: Optional[int] = None
+
+    def is_empty(self) -> bool:
+        """Whether there is nothing here worth carrying."""
+        return not (
+            self.is_block
+            or self.comments
+            or self.inline_comments
+            or self.start_line is not None
+            or self.end_line is not None
+        )
+
+    def copy(self) -> "HclMeta":
+        """A copy with lists of its own.
+
+        `copy.copy` would share them, so appending a comment to a copied
+        body's metadata would add it to the original's too. The comment dicts
+        themselves are shared, as a shallow copy of a `dict` shares its values.
+        """
+        return HclMeta(
+            self.is_block,
+            list(self.comments),
+            list(self.inline_comments),
+            self.start_line,
+            self.end_line,
+        )
+
+
+class HclDict(Dict[str, Any]):
+    """A dict whose HCL metadata lives on the object rather than among the keys.
+
+    Equality, iteration, `json.dumps` and every other mapping operation behave
+    exactly as `dict` does -- the metadata is deliberately not part of the
+    mapping, so a document declaring an attribute called `__is_block__` gets
+    that attribute back and nothing else.
+
+    JSON cannot carry the sidecar. Serializing an `HclDict` yields the
+    attributes alone, which is why the in-band keys remain the default.
+    """
+
+    __slots__ = ("hcl_meta",)
+
+    def __init__(self, *args: Any, meta: Optional[HclMeta] = None) -> None:
+        """Build from a mapping, with the metadata passed separately.
+
+        No `**kwargs`: this is the one class whose whole point is that no key
+        name is reserved, and taking keyword items would reserve `meta` --
+        `HclDict(**{"meta": "prod"})` would swallow the attribute and store a
+        string where the metadata goes. `meta` is a real name in real configs.
+        Pass the mapping positionally, as `dict` also allows.
+        """
+        super().__init__(*args)
+        if meta is not None and not isinstance(meta, HclMeta):
+            raise TypeError(
+                "HclDict(meta=...) takes an HclMeta; to store a key called "
+                f"'meta', pass the mapping positionally: HclDict({{'meta': {meta!r}}})"
+            )
+        self.hcl_meta = meta if meta is not None else HclMeta()
+
+    def __repr__(self) -> str:
+        """Show the metadata, so a debugging session does not have to guess."""
+        if self.hcl_meta.is_empty():
+            return super().__repr__()
+        return f"{super().__repr__()} + {self.hcl_meta!r}"
+
+    def copy(self) -> "HclDict":
+        """Copy the mapping and the metadata together.
+
+        `dict.copy` returns a plain `dict`, which would drop the sidecar --
+        and `document = document.copy()` is ordinary enough that losing block
+        metadata to it would be a trap. The in-band form survives a copy
+        because its metadata is among the keys; this has to say so explicitly.
+        """
+        return type(self)(self, meta=self.hcl_meta.copy())
+
+    def __copy__(self) -> "HclDict":
+        """Same for `copy.copy`."""
+        return self.copy()
+
+    def __deepcopy__(self, memo: dict) -> "HclDict":
+        """Same for `copy.deepcopy`, metadata included.
+
+        The duplicate is recorded in *memo* before anything inside it is
+        copied. A mapping may hold a reference back to itself, and copying
+        the children first means the recursion reaches this dict again with
+        nothing recorded, which does not terminate. `dict` registers its own
+        copy first for that reason; a subclass that did not would make a
+        cyclic document worse than the plain mapping it replaces.
+        """
+        duplicate = type(self)()
+        memo[id(self)] = duplicate
+        duplicate.hcl_meta = copy_module.deepcopy(self.hcl_meta, memo)
+        for key, value in self.items():
+            duplicate[copy_module.deepcopy(key, memo)] = copy_module.deepcopy(value, memo)
+        return duplicate
+
+    def __reduce__(self) -> Tuple[Any, ...]:
+        """Carry the metadata through pickling, which `dict` would not.
+
+        The items go in the reduce tuple's dict-items slot rather than as a
+        constructor argument, so the empty `HclDict` is built and memoised
+        before any of them is pickled. A mapping may hold a reference back to
+        itself, and passing `dict(self)` to the constructor pickles that
+        reference before there is anything to point it at, which does not
+        terminate -- `dict` pickles a cycle, so this has to as well. The
+        metadata is slot state, restored by the default `__setstate__`.
+        """
+        return (type(self), (), (None, {"hcl_meta": self.hcl_meta}), None, iter(self.items()))
+
+    def __or__(self, other: Any) -> "HclDict":  # type: ignore[override]
+        """Merge, keeping this side's metadata.
+
+        Narrower than `dict.__or__`, which is declared to return `dict` for any
+        mapping: this always returns an `HclDict`, so the ignore records a
+        deliberate narrowing rather than a mismatch.
+
+        `dict.__or__` returns a plain `dict`, so `body | {"size": ...}` -- the
+        idiomatic non-mutating edit -- would drop the sidecar and the block
+        would then be written as an object. `{**body, ...}` cannot be helped:
+        unpacking always builds a plain `dict`, and there is no hook for it.
+        """
+        if not isinstance(other, dict):
+            return NotImplemented
+        merged = type(self)(self, meta=self.hcl_meta.copy())
+        merged.update(other)
+        return merged
+
+    def __ror__(self, other: Any) -> "HclDict":  # type: ignore[override]
+        """Same from the left, keeping this side's metadata."""
+        if not isinstance(other, dict):
+            return NotImplemented
+        merged = type(self)(other, meta=self.hcl_meta.copy())
+        merged.update(self)
+        return merged
+
+
+def as_sidecar_dict(value: Any) -> Any:
+    """Return *value* as an `HclDict` if it is a plain dict, else unchanged.
+
+    A view's `to_dict` can return a dict the serializer never built as a body:
+    the `{label: body}` wrapper around a labelled block, or the `{name: value}`
+    of an attribute. Under `metadata_sidecar` those have to be `HclDict`s too.
+    Left plain, anything attached to them goes back in-band, and a key the
+    document spelled `__is_block__` reads back as the marker -- the collision
+    the option exists to remove.
+    """
+    if isinstance(value, dict) and meta_of(value) is None:
+        return HclDict(value)
+    return value
+
+
+def meta_of(value: Any) -> Optional[HclMeta]:
+    """Return the metadata carried beside *value*, or None if it carries none."""
+    meta = getattr(value, "hcl_meta", None)
+    return meta if isinstance(meta, HclMeta) else None