feat: carry serializer metadata beside the mapping (#331)

[livingstaccato]

Fixes #331.

What

__is_block__, __comments__ and __inline_comments__ are the serializer's, but HCL reserves none of those names. A document may declare an attribute called any of them, and in-band one of the two has to lose:

hcl2.loads('block "a" {\n  __is_block__ = 99\n  keep = 1\n}\n')
# {'block': [{'"a"': {'__is_block__': True, 'keep': 1}}]}      99 is gone

hcl2.dumps(hcl2.loads('block "a" {\n  __comments__ = 99\n  keep = 1\n}\n'))
# 'block "a" {\n  keep = 1\n}\n'                                the attribute is gone

Neither is recoverable by the caller: by the time the dict exists, it holds a single value with nothing to say which of the two it is.

The shape

SerializationOptions(metadata_sidecar=True) makes loads return an HclDict — a dict subclass whose hcl_meta carries the three. The mapping then holds attributes and nothing else, so there is nothing left to collide with.

It is a dict, deliberately: equality, iteration, len, json.dumps and every other mapping operation behave exactly as before, so anything reading attributes does not notice. That extends to the copy protocol -- copy(), copy.copy, copy.deepcopy, pickle, | and |= all carry the sidecar, and deepcopy registers its duplicate in the memo before copying into it, so a document holding a reference back to itself copies rather than recursing, as it does for a plain dict. dumps reads whichever form it is handed — a document loaded either way, or a dict built by hand with the old keys — so nothing that works today stops working.

Off by default

Two reasons, both worth stating rather than leaving to be discovered.

The keys are a documented part of the v7 and v8 output shape, and hcl2tojson --with-comments puts them in JSON that other tools read. Changing that silently is not this PR's to do.

And JSON cannot carry a sidecar: json.dumps of an HclDict yields the attributes alone. Anyone serializing to JSON wants the in-band form, and should keep getting it unless they ask otherwise.

Scope

This covers the three keys that exist in 8.1.3. #291's __start_line__ and __end_line__ are the same class of key and belong in the same sidecar; PR #333 adds them in-band; HclMeta has start_line/end_line for them, and the query views and label levels carry the sidecar too, so no mapping under this option holds an in-band key.

The test that pins the in-band loss stays, marked as the behaviour the option exists to avoid, so the difference between the two modes stays visible rather than becoming folklore.

Merging

Up to date with main as of 2f6d718, by merging rather than rebasing (the merge=union CHANGELOG duplicates entries under a rebase).

It merges cleanly on its own, and all twelve open PRs are merged together, with the full suite passing on 3.8-3.14, at livingstaccato/python-hcl2@int/pyvider-hcl-9. Landing alongside the others needs one small follow-up from whichever lands second:

• fix: emit block line numbers when with_meta is set (#291) #333 + feat: carry serializer metadata beside the mapping (#331) #345: with_meta writes its span into HclMeta when a sidecar is present (four lines in BlockRule.serialize).

---

This pull request, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Please review with that provenance in mind.

[livingstaccato]

Please hold off on merging this one for now — I want to do another review pass over it before it goes in. Opened as a draft for that reason; I will mark it ready and say so here once I am done.

[livingstaccato]

Review pass done, so the hold above no longer applies — this is ready for review now.

Rebased on current main; GitHub reports it mergeable as it stands.

_{🤖 Drafted with Claude Code.}