From 584dea38d1fa4549b41bf699aa2e6163e280eee8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Tue, 20 Oct 2026 23:16:55 +0200 Subject: [PATCH 01/30] feat(python): zig_hocon.pyhocon, pyhocon's API over zig-hocon Moving over from pyhocon is a change of import: the modules, names, arguments and output of pyhocon 0.3.63, the tree an OrderedDict with get, put, pop and with_fallback, parse_URL, from_dict, HOCONConverter and the command line. The document is read by libhocon as zig_hocon reads it, so a config means what the spec says; pyhocon's own test suite passes 167 of 263 with the import swapped, and MIGRATING.md says what the other 96 are. Closes #24 Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- CHANGELOG.md | 6 +- python/MIGRATING.md | 45 +- python/README.md | 17 +- python/pyproject.toml | 2 +- python/tests/test_pyhocon_api.py | 495 ++++++++++++++++++++++ python/zig_hocon/pyhocon/__init__.py | 43 ++ python/zig_hocon/pyhocon/config_parser.py | 255 +++++++++++ python/zig_hocon/pyhocon/config_tree.py | 314 ++++++++++++++ python/zig_hocon/pyhocon/converter.py | 172 ++++++++ python/zig_hocon/pyhocon/exceptions.py | 18 + python/zig_hocon/pyhocon/tool.py | 34 ++ 11 files changed, 1391 insertions(+), 10 deletions(-) create mode 100644 python/tests/test_pyhocon_api.py create mode 100644 python/zig_hocon/pyhocon/__init__.py create mode 100644 python/zig_hocon/pyhocon/config_parser.py create mode 100644 python/zig_hocon/pyhocon/config_tree.py create mode 100644 python/zig_hocon/pyhocon/converter.py create mode 100644 python/zig_hocon/pyhocon/exceptions.py create mode 100644 python/zig_hocon/pyhocon/tool.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 7de7412c..ed2bab29 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,10 +4,14 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ## [Unreleased] +### Added + +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. A config still reads by the spec; pyhocon's own test suite passes 167 of 263 with the import swapped, and `python/MIGRATING.md` says what the rest are. + ### Fixed + - An include whose file cannot be read reports why: `include url("http://…")` with a host past 255 bytes no longer aborts the process (`NameTooLong`), `HTTP://` is fetched like `http://`, a path through a file (`x.conf/y.conf`) is a missing include and so ignored, and a symlink to itself is `SymLinkLoop`, not an include cycle that is not there. - `d.z += 5` reached again through a field that copies `d`, `d { z = ${?a.k} }`, `a += ${d}`, appended 5 twice, `[5, 5]`, when `d` was written before `a`, while the copy in `a` held `[5]`; it is `[5]` in every order, as the spec has it. - - Optional substitutions on a cycle resolve the same whichever field comes first, as the spec has it, where v0.5.3 dropped or kept them by the order of the lines, and in two cases lost a value: `b.c += ${y}`, `b.c += 2`, `y = [1, ${?b}]` is `b.c = [[1], 2]` again, and `f = x${?a}`, `f = { z = 3 }`, `a = ${?f}` keeps `a`, since text under an object is hidden and never evaluated, as a list is. An include inside an object concatenated with an optional, `k = { include "x" } ${?y}`, no longer reports a cycle. Found by a bug hunt on v0.5.3. - Four regressions in that change, found by a bug hunt on it, are fixed. A field written twice and reached again from inside a look-back, `d { z = ${?a.k} }`, `d.z += ${?c}`, `a : []`, `a += ${f}`, `f = ${d}`, crashed with `reached unreachable code` and resolves again, as typesafe/config has it; an undefined `${b}` in such a field, a crash already in v0.5.3, is the `Unresolved` error. `f += ${?b}`, `b = x${?f}`, `b += 1` dropped b's text and gave `b = [1]`, and is the error `+=` onto text is, whichever line comes first. `e = [${?f.k}y]`, `f = ${?e}${e}` and `e.k : ${?b}`, `b.k { k = ${?e}, k += ${?d} }` resolve the same in either order: an optional found to lead back to its own field stays nonexistent when it is met again. Ten new conformance cases, all 583 pass. - A value concatenating many substitutions, `a = ${x} ${x} …`, resolves in memory linear in its length: 16 000 one-element lists took 3.9 GB and now take 18 MB, 32 000 strings 2.95 GB and now 24 MB. diff --git a/python/MIGRATING.md b/python/MIGRATING.md index 7ac4cdac..a2c63f0b 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -1,6 +1,11 @@ # Migrating from pyhocon -This guide moves a project from pyhocon to zig_hocon. For most code it is a change of imports: `ConfigFactory`, the typed getters and the exception names are the same. Code that uses the tree as a dict needs a few lines more, and a config with `include package(...)` needs a change. Steps 1 to 3 cover all of it, and the checklist at the end finds every place to look at. +This guide moves a project from pyhocon to zig_hocon. There are two ways: + +- **`zig_hocon.pyhocon`, a change of import.** pyhocon's API as pyhocon 0.3.63 has it: the same modules, names, arguments and output, the tree an `OrderedDict` you can `put` into and `pop` from. The section below says what it covers and what it does not. +- **`zig_hocon`, the package's own API.** Typed getters on a document that stays in the library until one asks, and no tree in Python at all: Steps 1 and 2. + +Either way a config reads by the spec, which in a few places is not how pyhocon reads it: Step 3. The checklist at the end finds every place to look at. ## Why move @@ -31,6 +36,39 @@ This guide moves a project from pyhocon to zig_hocon. For most code it is a chan - **One exception to catch.** Every error is a `ConfigException`, with the file, line and column. A syntax error in pyhocon is pyparsing's `ParseSyntaxException`, which `except ConfigException` misses. +## The short way: zig_hocon.pyhocon + +Replace `pyhocon` with `zig_hocon.pyhocon` in the imports and leave the rest: + +```python +from zig_hocon.pyhocon import ConfigFactory, ConfigTree, HOCONConverter +from zig_hocon.pyhocon.exceptions import ConfigMissingException +``` + +Everything of pyhocon's public API is there: `ConfigFactory.parse_file` (with `encoding` and `required`), `parse_string`, `parse_URL`, `from_dict`; the `ConfigTree` with `get`, the typed getters, `get_list`, `get_config`, `put`, `pop`, `with_fallback`, `merge_configs`, `as_plain_ordered_dict`, `conf["db.host"]` and `conf.db.host`; `ConfigParser.parse`, `resolve_substitutions` and `resolve_package_path`; `HOCONConverter` with its JSON, HOCON, YAML and .properties output to the character; and pyhocon's command line as `python -m zig_hocon.pyhocon.tool`. The exceptions are zig_hocon's, so one `except` catches either module's. + +The measure is pyhocon's own test suite with the import swapped: 167 of its 263 tests pass. Of the 96 that do not, 95 are a config that reads differently by the spec (Step 3) or the class of a syntax error: + +| tests | what pyhocon does | +|---|---| +| 51 | turns `30s` into a `timedelta` while it parses | +| 12 | raises pyparsing's exceptions for a syntax error; here it is a `ConfigParseException` | +| 9 | flattens `x += [3, 4]` into `x`; by the spec it appends the list as one element | +| 6 | reads `-Xmx1g` unquoted; the spec does not let an unquoted string start with `-` | +| 6 | `include` inside a list, `include glob(…)` and `include package(…)`, its own extensions | +| 5 | resolves a self-reference typesafe/config refuses too: `x = ${x}` with `x` only in the environment, `x = [${x}, 2]`, an object that refers into itself | +| 3 | accepts `.54`, and a backslash escape in an unquoted string: `abc\"test`, `abc\n` | +| 2 | includes a file named without an extension; the spec tries `.conf`, `.json` and `.properties` after the name | +| 1 | reads `${?nope} 4` as the number 4; by the spec it is the text `" 4"` | + +The one left is `ConfigTree.resolve()`, which is not there. Nor are a few things pyhocon has inside its parser: + +- **A value with a `${…}` is no `ConfigValues`.** A document read with `resolve=False` has its values where it resolves on its own; where it does not, it has none until `with_fallback()` or `ConfigParser.resolve_substitutions()` gives it what it waits for, and reading it before is a `ConfigSubstitutionException`. +- **`unresolved_value`** takes `DEFAULT_SUBSTITUTION`, `MANDATORY_SUBSTITUTION` and `NO_SUBSTITUTION`; `STR_SUBSTITUTION` and a value of your own are a `NotImplementedError`. +- **`parse_URL`** reads a document with no `include` in it: one is an error, as it has no directory to be found in. + +The whole document is in Python, as in pyhocon, so it is a little slower than `zig_hocon`, still far from pyhocon: a 5 KB config in 0.24 ms (pyhocon 29 ms), 10 MB in 0.32 s (pyhocon 68 s). + ## Step 1: imports | pyhocon | zig_hocon | @@ -91,13 +129,14 @@ settings["db"]["host"] - **A concatenation is a string.** `a = ${?nope}42` is `"42"` by the spec; pyhocon gives the number `42`. A model that wants an int converts it. - **An `include url(…)` gives up after 30 s.** pyhocon waits as long as `socket.setdefaulttimeout` says, by default for ever. Here `http_timeout=` on `parse_file` and `parse_string` sets it, and a server that does not answer in time is a `ConfigParseException` named `IncludeTimeout`. - **A leading zero keeps the text.** A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. +- **`+=` appends one element.** `x += [3, 4]` after `x = [1, 2]` is `[1, 2, [3, 4]]`, as `x = ${?x} [[3, 4]]` is; pyhocon gives `[1, 2, 3, 4]`. To extend a list with another, `x = ${x} [3, 4]`. - **Strict syntax.** The other inputs in the table under *No silent guesses* are errors now. They are rare, and each error says where to look. -Everything else (variables, `+=`, includes, merging) reads the same, except where pyhocon breaks the spec; the conformance report lists those cases. +Everything else (variables, includes, merging) reads the same, except where pyhocon breaks the spec; the conformance report lists those cases. ## Checklist -Run this over the code that uses pyhocon; each hit is a row in Step 2 or 3: +With `zig_hocon.pyhocon`, only Step 3 applies. With `zig_hocon`, run this over the code that uses pyhocon; each hit is a row in Step 2 or 3: ```sh grep -rnE "pyhocon|include package|as_plain_ordered_dict|get_list|with_fallback|from_dict|parse_URL|HOCONConverter|\.put\(|\.pop\(|conf\[" . diff --git a/python/README.md b/python/README.md index 8e30077d..96867ba0 100644 --- a/python/README.md +++ b/python/README.md @@ -106,12 +106,19 @@ Path("deploy.conf").write_text(ConfigFactory.parse_file("app.conf", env=KEEP).to ## Coming from pyhocon -Change the import: `ConfigFactory`, the getters and the exception names are the same. What reads differently: +**The short way** keeps every line of pyhocon code: `zig_hocon.pyhocon` is pyhocon's API, its modules, names and arguments, the tree an `OrderedDict` with `get`, `put`, `pop` and `with_fallback`, and `HOCONConverter`: -- **The tree is not a dict.** `conf["db"]["host"]` and `json.dumps(conf)` fail; take `conf.as_dict()` first. `as_plain_ordered_dict()` is `as_dict()`. -- **Fewer methods.** `get`, `get_list`, `with_fallback` and `put` are not there: a typed getter, an `include`, or the dict does the job. -- **Durations stay text.** `30s` is `"30s"` in `as_dict()` until a getter or a model reads it. -- **Strict syntax.** Input pyhocon quietly accepted, such as `a..b = 1` or `a = b:c`, is an error with its line and column. +```python +from zig_hocon.pyhocon import ConfigFactory # was: from pyhocon import ConfigFactory +``` + +**The package's own API** has the same `ConfigFactory`, getters and exception names, without a tree in Python: `conf.as_dict()` when you need one, a typed getter or an `include` where pyhocon has `get`, `with_fallback` or `put`. + +Either way, a config reads by the spec, and in a few places pyhocon reads it otherwise: + +- **Durations stay text.** `30s` is `"30s"` until a getter or a model reads it. +- **`+=` appends one element.** `x += [3, 4]` adds the list `[3, 4]` to `x`; `x = ${x} [3, 4]` extends it. +- **Strict syntax.** Input pyhocon quietly accepted, such as `a..b = 1`, `a = b:c` or an unquoted `-Xmx1g`, is an error with its line and column. - **No `include package(...)`.** It is pyhocon's own; include by path. The [migration guide](https://github.com/lejmr/zig-hocon/blob/main/python/MIGRATING.md) has every difference, with what to write instead. diff --git a/python/pyproject.toml b/python/pyproject.toml index db4c442d..9fc5108c 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -39,7 +39,7 @@ Conformance = "https://htmlpreview.github.io/?https://github.com/lejmr/zig-hocon Migrating = "https://github.com/lejmr/zig-hocon/blob/main/python/MIGRATING.md" [tool.setuptools] -packages = ["zig_hocon"] +packages = ["zig_hocon", "zig_hocon.pyhocon"] [tool.setuptools.package-data] # The library is put there by `zig build python`; py.typed says the package diff --git a/python/tests/test_pyhocon_api.py b/python/tests/test_pyhocon_api.py new file mode 100644 index 00000000..fe18e3cd --- /dev/null +++ b/python/tests/test_pyhocon_api.py @@ -0,0 +1,495 @@ +"""zig_hocon.pyhocon: pyhocon 0.3.63's API over libhocon. Each group says +where it differs from pyhocon; pyhocon ✓ is the same result, checked +against pyhocon 0.3.63 itself.""" + +import copy +import datetime +import logging +import pathlib +import pickle +import sys +from collections import OrderedDict + +import pytest + +import zig_hocon.pyhocon as pyhocon +from zig_hocon.pyhocon import ( + ConfigException, + ConfigFactory, + ConfigList, + ConfigMissingException, + ConfigParser, + ConfigSubstitutionException, + ConfigTree, + ConfigWrongTypeException, + HOCONConverter, + UndefinedKey, +) +from zig_hocon.pyhocon import config_parser, config_tree, converter, exceptions, tool +from zig_hocon.pyhocon.config_parser import ( + MANDATORY_SUBSTITUTION, + NO_SUBSTITUTION, + STR_SUBSTITUTION, + IncludeNotFoundError, +) +from zig_hocon.pyhocon.config_tree import NoneValue + +DOC = """ +db { host = burrow.example.org, port = 5432, "pool.size" = 4 } +owls = [hoot, screech] +night = true +quiet = null +ratio = 0.75 +""" + + +def conf(text=DOC, **kw): + return ConfigFactory.parse_string(text, **kw) + + +# pyhocon ✓: the same names in the same modules +def test_the_modules_are_pyhocon_s(): + for name in ["ConfigFactory", "ConfigParser", "ConfigTree", "ConfigList", "UndefinedKey", "HOCONConverter", "ConfigInclude", + "ConfigSubstitution", "ConfigUnquotedString", "ConfigValues", "ConfigException", "ConfigMissingException", + "ConfigWrongTypeException", "ConfigSubstitutionException"]: + assert hasattr(pyhocon, name) + assert config_parser.ConfigFactory is ConfigFactory + assert config_tree.ConfigTree is ConfigTree + assert converter.HOCONConverter is HOCONConverter is tool.HOCONConverter + # one except catches either module's + import zig_hocon + + assert exceptions.ConfigException is zig_hocon.ConfigException + + +# pyhocon ✓ +def test_a_document_is_an_ordered_dict(): + c = conf() + assert isinstance(c, OrderedDict) and c.root + assert list(c) == ["db", "owls", "night", "quiet", "ratio"] + assert c["db"] == {"host": "burrow.example.org", "port": 5432, '"pool.size"': 4} + assert isinstance(c["db"], ConfigTree) and not c["db"].root + + +# pyhocon ✓ +def test_get_by_path(): + c = conf() + assert c.get("db.host") == "burrow.example.org" + assert c["db.port"] == 5432 + assert c.db.port == 5432 + assert c.get('db."pool.size"') == 4 + assert c.get("quiet") is None + assert c.get("nope", "a default owl") == "a default owl" + # a list is a copy + c.get("owls").append("squawk") + assert c.get("owls") == ["hoot", "screech"] + + +# pyhocon ✓: the messages too +def test_get_what_is_not_there(): + c = conf() + with pytest.raises(ConfigMissingException, match="No configuration setting found for key db.user"): + c.get("db.user") + with pytest.raises(ConfigWrongTypeException, match="night has type bool rather than dict"): + c.get("night.owl") + assert c.get("night.owl", 7) == 7 + with pytest.raises(AttributeError): + c.nope + with pytest.raises(AttributeError): + c.__nope__ + + +# pyhocon ✓ +def test_contains(): + c = conf() + assert "db.host" in c and "quiet" in c + assert "db.user" not in c and "night.owl" not in c and 5 not in c + + +# pyhocon ✓ +@pytest.mark.parametrize( + "path, want", + [("a.b.c", ["a", "b", "c"]), ('a."b.c"', ["a", '"b.c"']), ('"a"', ["a"]), ('a.b."c"', ["a", "b", "c"]), ("a b.c", ["a b", "c"])], +) +def test_parse_key(path, want): + assert ConfigTree.parse_key(path) == want + + +# pyhocon ✓ (get_int truncates 1.5, a default goes through the conversion) +@pytest.mark.parametrize( + "method, path, want", + [ + ("get_string", "night", "true"), + ("get_string", "db.port", "5432"), + ("get_string", "quiet", None), + ("get_int", "db.port", 5432), + ("get_int", "ratio", 0), + ("get_int", "quiet", None), + ("get_float", "ratio", 0.75), + ("get_float", "quiet", None), + ("get_bool", "night", True), + ("get_bool", "quiet", None), + ("get_list", "owls", ["hoot", "screech"]), + ("get_list", "quiet", None), + ("get_config", "db", {"host": "burrow.example.org", "port": 5432, '"pool.size"': 4}), + ("get_config", "quiet", None), + ], +) +def test_getters(method, path, want): + assert getattr(conf(), method)(path) == want + + +def test_getters_from_text_and_defaults(): + c = conf('a = "5"\nb = "1.5"\nc = yes\nd = OFF\ne = maybe') + assert c.get_int("a") == 5 and c.get_float("b") == 1.5 + assert c.get_bool("c") is True and c.get_bool("d") is False + assert c.get_string("nope", 5) == "5" + assert c.get_bool("nope", False) is False + + +@pytest.mark.parametrize( + "method, path, message", + [ + ("get_int", "e", "e has type 'str' rather than 'int'"), + ("get_int", "f", "f has type 'list' rather than 'int'"), + ("get_float", "e", "e has type 'str' rather than 'float'"), + ("get_bool", "e", "e does not translate to a Boolean value"), + ("get_list", "e", "e has type 'str' rather than 'list'"), + ("get_config", "e", "e has type 'str' rather than 'config'"), + ], +) +def test_getters_refuse(method, path, message): + with pytest.raises(ConfigException, match=message): + getattr(conf("e = maybe\nf = [1]"), method)(path) + + +# pyhocon ✗: an object with whole-number keys is a list by the spec, in the +# order of the numbers and the other keys left out; pyhocon sorts them as +# text and refuses any other key +def test_get_list_of_numbered_keys(): + c = conf('a { "10" = ten, "2" = two, x = skipped }\nb {}\nc { x = 1 }') + assert c.get_list("a") == ["two", "ten"] + for path in ("b", "c"): + with pytest.raises(ConfigException, match="does not translate to a list"): + c.get_list(path) + + +# pyhocon ✓ +def test_put(): + c = conf() + c.put("db.user", "night shift") + c.put("night.owl", "hoot") # through a scalar: an object in its place + c.put("owls", ["squawk"], append=True) + c.put("new", ["first"], append=True) + c.put("quiet", NoneValue()) + assert c.get("db.user") == "night shift" + assert c.get("night") == {"owl": "hoot"} + assert c.get("owls") == ["hoot", "screech", "squawk"] + assert c.get("new") == ["first"] + assert list(c.items())[3] == ("quiet", None) + with pytest.raises(ConfigWrongTypeException, match="Cannot concatenate the list ratio"): + c.put("ratio", [1], append=True) + + +# pyhocon ✓: an object put on an object merges; at the root into a new one, +# so a tree taken before stays as it was +def test_put_merges_objects(): + c = conf() + before = c["db"] + c.put("db", ConfigFactory.from_dict({"port": 6543, "tls": True})) + assert c["db"] == {"host": "burrow.example.org", "port": 6543, '"pool.size"': 4, "tls": True} + assert before["port"] == 5432 + inner = conf("a { b { c = 1 } }") + kept = inner.get("a.b") + inner.put("a.b", ConfigFactory.from_dict({"d": 2})) + assert kept == {"c": 1, "d": 2} + + +# pyhocon ✓ +def test_pop(): + c = conf() + assert c.pop("db.port") == 5432 + assert "db.port" not in c + assert c.pop("owls") == ["hoot", "screech"] + assert c.pop("nope", "a default owl") == "a default owl" + with pytest.raises(ConfigMissingException): + c.pop("nope") + + +# pyhocon ✓ +@pytest.mark.parametrize("copy_trees", [False, True]) +def test_merge_configs(copy_trees): + a = conf("x { y = 1, z = 2 }\nw = 1") + inner = a["x"] + merged = ConfigTree.merge_configs(a, conf("x { z = 3 }\nw { v = 1 }"), copy_trees=copy_trees) + assert merged is a + assert a == {"x": {"y": 1, "z": 3}, "w": {"v": 1}} + assert (inner["z"] == 2) is copy_trees + + +# pyhocon ✓ +def test_as_plain_ordered_dict(): + plain = conf('a { "b.c" = [1, { d = 2 }] }').as_plain_ordered_dict() + assert plain == OrderedDict([("a", OrderedDict([("b.c", [1, OrderedDict([("d", 2)])])]))]) + assert type(plain["a"]["b.c"][1]) is OrderedDict + + +# pyhocon ✓ +def test_copies(): + c = conf() + for other in (copy.deepcopy(c), c.copy(), pickle.loads(pickle.dumps(c))): + assert other == c + + +# pyhocon ✓: the fallback's keys first, as two files one after the other +def test_with_fallback(): + c = conf("db.port = 6543\nurl = ${db.host}\":\"${db.port}", resolve=False) + merged = c.with_fallback(conf()) + assert merged.get("url") == "burrow.example.org:6543" + assert list(merged)[0] == "db" + assert merged.get("db") == {"host": "burrow.example.org", "port": 6543, '"pool.size"': 4} + + +def test_with_fallback_a_file(tmp_path): + (tmp_path / "defaults.conf").write_text("night = false\nmoon = full") + merged = conf().with_fallback(str(tmp_path / "defaults.conf")) + assert merged["night"] is True and merged["moon"] == "full" + + +def test_with_fallback_unresolved(): + merged = conf("a = ${b}", resolve=False).with_fallback(conf("c = 1"), resolve=False) + with pytest.raises(ConfigSubstitutionException): + merged.get("a") + assert ConfigParser.resolve_substitutions(merged, accept_unresolved=True) is True + + +def test_with_fallback_of_put_values(): + c = ConfigTree() + c.put("wait", datetime.timedelta(seconds=2)) + assert c.with_fallback(ConfigTree()).get("wait") == 2000 + c.put("owl", object()) + with pytest.raises(TypeError, match="object is not a HOCON value"): + c.with_fallback(ConfigTree()) + + +# pyhocon ⚠️: a value with a ${…} is no ConfigValues here; a document that +# resolves alone has its values, one that does not has none until resolved +def test_resolve_false(): + alone = conf("a = 1\nb = ${a}", resolve=False) + assert alone == {"a": 1, "b": 1} + waiting = conf("a = ${nope}", resolve=False) + assert waiting == {} + for read in (lambda: waiting.get("a"), lambda: waiting.a, lambda: "a" in waiting, waiting.as_plain_ordered_dict): + with pytest.raises(ConfigSubstitutionException, match="read with resolve=False"): + read() + + +def test_resolve_substitutions(): + c = conf("a = ${b}", resolve=False) + with pytest.raises(ConfigSubstitutionException): + ConfigParser.resolve_substitutions(c) + assert ConfigParser.resolve_substitutions(c, accept_unresolved=True) is True + merged = ConfigTree.merge_configs(c, conf("b = owl", resolve=False)) + assert merged is c + assert ConfigParser.resolve_substitutions(c) is False + assert c == {"a": "owl", "b": "owl"} + assert ConfigParser.resolve_substitutions(c) is False + + +def test_unresolved_value(): + assert conf("a = ${nope}", unresolved_value=NO_SUBSTITUTION) == {} + assert conf("a = 1", unresolved_value=MANDATORY_SUBSTITUTION) == {"a": 1} + with pytest.raises(NotImplementedError): + conf("a = ${nope}", unresolved_value=STR_SUBSTITUTION) + + +# pyhocon ✓: positional arguments, includes from the working directory +def test_parse_string_includes_from_the_working_directory(tmp_path, monkeypatch): + (tmp_path / "inc.conf").write_text("moon = full") + monkeypatch.chdir(tmp_path) + assert ConfigFactory.parse_string('include "inc.conf"\nnight = true') == {"moon": "full", "night": True} + sub = tmp_path / "sub" + sub.mkdir() + (sub / "inc.conf").write_text("moon = new") + assert ConfigFactory.parse_string('include "inc.conf"', str(sub), True) == {"moon": "new"} + assert ConfigParser.parse('include "inc.conf"', str(sub)) == {"moon": "new"} + + +# pyhocon ✓ +def test_a_document_that_is_a_list(): + c = conf("[1, 2]") + assert isinstance(c, ConfigList) and c == [1, 2] + + +# pyhocon ✓: the classes pyhocon raises +def test_errors_in_pyhocon_s_classes(): + with pytest.raises(ConfigWrongTypeException) as e: + conf("a = [1] owl") + assert e.value.name == "UnsupportedConcatenation" and e.value.line == 1 + with pytest.raises(IncludeNotFoundError) as e: + conf('include required("/nowhere/owl.conf")') + assert isinstance(e.value, FileNotFoundError) and isinstance(e.value, ConfigException) + assert e.value.name == "IncludeNotFound" and "IncludeNotFound" in str(e.value) + # the others as zig_hocon raises them + with pytest.raises(exceptions.ConfigParseException) as e: + conf("a = {") + assert type(e.value) is exceptions.ConfigParseException + + +def test_parse_file(tmp_path, caplog): + path = tmp_path / "app.conf" + path.write_text('include "inc.conf"\nowl = "hoot"') + (tmp_path / "inc.conf").write_text("moon = full") + assert ConfigFactory.parse_file(path) == {"moon": "full", "owl": "hoot"} + (tmp_path / "bad.conf").write_text("a = {") + with pytest.raises(exceptions.ConfigParseException, match="bad.conf:1"): + ConfigFactory.parse_file(str(tmp_path / "bad.conf")) + # another encoding, its includes still next to it + latin = tmp_path / "latin.conf" + latin.write_bytes('include "inc.conf"\nowl = "hibou café"'.encode("latin-1")) + assert ConfigFactory.parse_file(latin, "latin-1") == {"moon": "full", "owl": "hibou café"} + with pytest.raises(FileNotFoundError): + ConfigFactory.parse_file(tmp_path / "nope.conf") + with caplog.at_level(logging.WARNING): + assert ConfigFactory.parse_file(tmp_path / "nope.conf", required=False) == [] + assert "Cannot include file" in caplog.text + + +def test_parse_file_resolve_false(tmp_path): + path = tmp_path / "app.conf" + path.write_text("a = ${b}") + c = ConfigFactory.parse_file(path, resolve=False) + assert c.with_fallback(conf("b = 1")) == {"b": 1, "a": 1} + + +def test_parse_url(tmp_path, caplog): + path = tmp_path / "remote.conf" + path.write_text("owl = hoot") + assert ConfigFactory.parse_URL(path.as_uri()) == {"owl": "hoot"} + assert ConfigFactory.parse_URL(path.as_uri(), 5, False, True) == {"owl": "hoot"} + with caplog.at_level(logging.WARNING): + assert ConfigFactory.parse_URL("http://127.0.0.1:1/owl.conf", timeout=1) == [] + assert "Cannot include url" in caplog.text + with pytest.raises(OSError): + ConfigFactory.parse_URL("http://127.0.0.1:1/owl.conf", timeout=1, required=True) + + +# pyhocon ✓ +def test_from_dict(): + c = ConfigFactory.from_dict({"a.b": 1, "c": [{"d": 2}, 3]}, root=True) + assert c == {"a": {"b": 1}, "c": [{"d": 2}, 3]} + assert c.root and isinstance(c["c"][0], ConfigTree) + assert c.get("a.b") == 1 + + +def test_resolve_package_path(): + assert pathlib.Path(ConfigParser.resolve_package_path("zig_hocon:pyhocon/__init__.py")) == pathlib.Path(pyhocon.__file__) + with pytest.raises(ValueError, match="PACKAGE:PATH"): + ConfigParser.resolve_package_path("zig_hocon") + for missing in ("no_such_owl_package:x.conf", "zig_hocon.units:x.conf"): + with pytest.raises(ImportError): + ConfigParser.resolve_package_path(missing) + + +def test_undefined_key_is_the_default(): + assert ConfigTree.get.__defaults__ == (UndefinedKey,) + + +CONVERT = conf(''' +a = {b: 1} +b = [1, 2, {x = []}, null] +d = placeholder +e = """1 + 2""" +f1 = true +f2 = false +h = null +i = {} +"a.b" = 2 +j { k { l = 1 } } +m = "x=y!#" +''') +CONVERT.put("d", 'a\tq"x\x01') + + +# pyhocon ✓: each checked against pyhocon 0.3.63's output to the character +def test_to_json(): + assert HOCONConverter.to_json(CONVERT) == ( + '{\n "a": {\n "b": 1\n },\n "b": [\n 1,\n 2,\n {\n "x": []\n },\n null\n ],\n' + ' "d": "a\\tq\\"x\\u0001",\n "e": "1\\n 2",\n "f1": true,\n "f2": false,\n "h": null,\n "i": {},\n' + ' "a.b": 2,\n "j": {\n "k": {\n "l": 1\n }\n },\n "m": "x=y!#"\n}' + ) + assert HOCONConverter.to_json(conf("a = []"), indent=4) == '{\n "a": []\n}' + + +def test_to_hocon(): + assert HOCONConverter.to_hocon(CONVERT) == ( + 'a {\n b = 1\n}\nb = [\n 1\n 2\n {\n x = []\n }\n null\n]\nd = "a\\tq\\"x\\u0001"\ne = """1\n 2"""\n' + 'f1 = true\nf2 = false\nh = null\ni {}\n"a.b" = 2\nj {\n k {\n l = 1\n }\n}\nm = "x=y!#"' + ) + assert HOCONConverter.to_hocon(conf("j { k { l = 1 } }\nm = 1"), compact=True) == "j.k.l = 1\nm = 1" + assert HOCONConverter.to_hocon(ConfigTree()) == "{}" + + +def test_to_yaml(): + assert HOCONConverter.to_yaml(CONVERT) == ( + 'a: \n b: 1\nb: \n - 1\n - 2\n - \n x: []\nd: a\tq"x\x01\ne: |\n 1\n 2\nf1: true\nf2: false\nh: null\n' + "i: \na.b: 2\nj: \n k: \n l: 1\nm: x=y!#" + ) + + +def test_to_properties(): + assert HOCONConverter.to_properties(CONVERT) == ( + 'a.b = 1\nb.0 = 1\nb.1 = 2\nd = a\tq"x\x01\ne = 1\\\n 2\nf1 = true\nf2 = false\na.b = 2\nj.k.l = 1\nm = x\\=y\\!\\#' + ) + assert HOCONConverter.to_properties(None) == "" + + +@pytest.mark.parametrize( + "td, hocon", + [ + (datetime.timedelta(days=2), "2 days"), + (datetime.timedelta(seconds=3), "3 seconds"), + (datetime.timedelta(microseconds=4), "4 microseconds"), + (datetime.timedelta(0), "0 seconds"), + ], +) +def test_timedeltas_put_in_a_tree(td, hocon): + c = ConfigTree() + c.put("wait", td) + ms = str(int(td.total_seconds() * 1000)) + assert HOCONConverter.to_hocon(c) == f"wait = {hocon}" + assert HOCONConverter.to_json(c) == f'{{\n "wait": {ms}\n}}' + assert HOCONConverter.to_yaml(c) == f"wait: {ms}" + assert HOCONConverter.to_properties(c) == f"wait = {ms}" + + +def test_convert(): + c = conf("a = 1") + assert HOCONConverter.convert(c, "json") == '{\n "a": 1\n}' + assert HOCONConverter.convert(c, "properties") == "a = 1" + assert HOCONConverter.convert(c, "yaml") == "a: 1" + assert HOCONConverter.convert(c, "hocon") == "a = 1" + with pytest.raises(Exception, match="Invalid format 'toml'"): + HOCONConverter.convert(c, "toml") + + +def test_convert_from_file(tmp_path, monkeypatch, capsys): + (tmp_path / "in.conf").write_text("owl = hoot") + HOCONConverter.convert_from_file(str(tmp_path / "in.conf"), str(tmp_path / "out.yaml"), "yaml") + assert (tmp_path / "out.yaml").read_text() == "owl: hoot" + monkeypatch.setattr(sys, "stdin", __import__("io").StringIO("owl = screech")) + HOCONConverter.convert_from_file() + assert capsys.readouterr().out == '{\n "owl": "screech"\n}\n' + + +@pytest.mark.parametrize("verbosity", [[], ["-v"], ["-vv"], ["-vvv"]]) +def test_tool(tmp_path, verbosity): + (tmp_path / "in.conf").write_text("a { b = 1 }") + root = logging.getLogger() + handlers, level = list(root.handlers), root.level + try: + tool.main(["-i", str(tmp_path / "in.conf"), "-o", str(tmp_path / "out"), "-f", "HOCON", "-c", "-n", "4", *verbosity]) + finally: + root.handlers[:], root.level = handlers, level + assert (tmp_path / "out").read_text() == "a.b = 1" diff --git a/python/zig_hocon/pyhocon/__init__.py b/python/zig_hocon/pyhocon/__init__.py new file mode 100644 index 00000000..8d0fd9f7 --- /dev/null +++ b/python/zig_hocon/pyhocon/__init__.py @@ -0,0 +1,43 @@ +"""pyhocon's API, by chimpler and its contributors, over zig-hocon: its +modules, names and arguments as pyhocon 0.3.63 has them, so that moving over +is a change of import:: + + from zig_hocon.pyhocon import ConfigFactory + + conf = ConfigFactory.parse_file("app.conf") + conf.get("db.pool") + conf.put("db.host", "db.example.org") + +What a document means is the spec's, as ``zig_hocon`` reads it: where +pyhocon reads it otherwise, python/MIGRATING.md says how. +""" + +from .config_parser import ConfigFactory, ConfigParser, ConfigSubstitutionException +from .config_tree import ( + ConfigInclude, + ConfigList, + ConfigSubstitution, + ConfigTree, + ConfigUnquotedString, + ConfigValues, + UndefinedKey, +) +from .converter import HOCONConverter +from .exceptions import ConfigException, ConfigMissingException, ConfigWrongTypeException + +__all__ = [ + "ConfigException", + "ConfigFactory", + "ConfigInclude", + "ConfigList", + "ConfigMissingException", + "ConfigParser", + "ConfigSubstitution", + "ConfigSubstitutionException", + "ConfigTree", + "ConfigUnquotedString", + "ConfigValues", + "ConfigWrongTypeException", + "HOCONConverter", + "UndefinedKey", +] diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py new file mode 100644 index 00000000..348d4585 --- /dev/null +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -0,0 +1,255 @@ +"""pyhocon's ``ConfigFactory`` and ``ConfigParser``, with its arguments. The +document is read by libhocon, as ``zig_hocon.ConfigFactory`` reads it, and +handed over as a ConfigTree of Python's own values.""" + +import codecs +import contextlib +import importlib.util +import json +import logging +import os +import socket +import urllib.error +import urllib.request +from collections.abc import Callable, Mapping +from typing import Any, Literal, TypeVar, cast, overload + +from .. import config_parser as _core +from .. import config_tree as _core_tree +from ..exceptions import ConfigException, ConfigParseException, ConfigSubstitutionException, ConfigWrongTypeException +from .config_tree import ConfigList, ConfigTree, quoted + +__all__ = [ + "DEFAULT_SUBSTITUTION", + "MANDATORY_SUBSTITUTION", + "NO_SUBSTITUTION", + "STR_SUBSTITUTION", + "ConfigFactory", + "ConfigParser", + "ConfigSubstitutionException", +] + +logger = logging.getLogger(__name__) + +Path = str | os.PathLike[str] +"""A file name.""" + + +# pyhocon's `unresolved_value`: what a ${…} with no value becomes. +class DEFAULT_SUBSTITUTION: + """An error (the default).""" + + +class MANDATORY_SUBSTITUTION: + """An error.""" + + +class NO_SUBSTITUTION: + """Nothing yet: the document stays unresolved, as with resolve=False.""" + + +class STR_SUBSTITUTION: + """Its own text, ``"${x}"``. Not supported here.""" + + +class IncludeNotFoundError(ConfigParseException, FileNotFoundError): + """A ``required(…)`` include that is not there: an OSError too, as + pyhocon raises it.""" + + +# libhocon's errors that pyhocon raises as another class +_AS_PYHOCON: dict[str, type[ConfigException]] = { + "UnsupportedConcatenation": ConfigWrongTypeException, + "IncludeNotFound": IncludeNotFoundError, +} + +_T = TypeVar("_T") + + +def _call(read: Callable[[], _T]) -> _T: + """``read()``, its error in pyhocon's class.""" + try: + return read() + except ConfigException as e: + cls = _AS_PYHOCON.get(e.name or "") + if cls is None: + raise + # the message alone: OSError takes more arguments as errno and strerror + err = cls(str(e)) + err.name, err.file, err.line, err.column, err.source_line = e.name, e.file, e.line, e.column, e.source_line + raise err from None + + +def _tree(pairs: list[tuple[str, Any]]) -> ConfigTree: + return ConfigTree((quoted(k), v) for k, v in pairs) + + +def _resolved(doc: _core_tree.ConfigTree) -> ConfigTree: + value = json.loads(doc.to_json(), object_pairs_hook=_tree) + if isinstance(value, list): + # a document that is a list, as pyhocon gives it; typed as the + # object a document nearly always is + return cast(ConfigTree, ConfigList(value)) + value.root = True + return cast(ConfigTree, value) + + +def as_written(text: str) -> ConfigTree: + """A document read with resolve=False, from its text: its values when + it resolves alone, else empty and unresolved; the text kept for + with_fallback() and resolve_substitutions() to finish.""" + try: + tree = _call(lambda: _resolved(_core.ConfigFactory.parse_string(text))) + except ConfigSubstitutionException: + tree = ConfigTree(root=True) + tree._unresolved = True + tree._as_written = text + return tree + + +def _read(parse: Callable[[bool], _core_tree.ConfigTree], resolve: bool, unresolved_value: Any) -> ConfigTree: + """The document ``parse`` reads, resolved or not, as a ConfigTree.""" + if unresolved_value is NO_SUBSTITUTION: + resolve = False + elif unresolved_value is not DEFAULT_SUBSTITUTION and unresolved_value is not MANDATORY_SUBSTITUTION: + raise NotImplementedError("unresolved_value: only DEFAULT_SUBSTITUTION, MANDATORY_SUBSTITUTION and NO_SUBSTITUTION") + if resolve: + return _call(lambda: _resolved(parse(True))) + return as_written(_call(lambda: parse(False).to_hocon())) + + +class ConfigFactory: + """Reads a document, as pyhocon's ``ConfigFactory``. ``${NAME}`` that is + not in the document is looked up in ``os.environ``.""" + + @overload + @classmethod + def parse_file( + cls, filename: Path, encoding: str = ..., required: Literal[True] = ..., resolve: bool = ..., unresolved_value: Any = ... + ) -> ConfigTree: ... + @overload + @classmethod + def parse_file(cls, filename: Path, encoding: str, required: bool, resolve: bool = ..., unresolved_value: Any = ...) -> ConfigTree | list[Any]: ... + @overload + @classmethod + def parse_file( + cls, filename: Path, encoding: str = ..., *, required: bool, resolve: bool = ..., unresolved_value: Any = ... + ) -> ConfigTree | list[Any]: ... + @classmethod + def parse_file( + cls, filename: Path, encoding: str = "utf-8", required: bool = True, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION + ) -> ConfigTree | list[Any]: + """The file at ``filename``, in ``encoding``; its includes are found + next to it. A file that cannot be read is an OSError, or with + ``required=False`` a warning and an empty list.""" + try: + with open(filename, encoding=encoding) as fd: + content = fd.read() + except OSError: + if required: + raise + logger.warning("Cannot include file %s. File does not exist or cannot be read.", filename) + return [] + # A UTF-8 file is read again by libhocon, for its name in an error. + if codecs.lookup(encoding).name == "utf-8": + return _read(lambda r: _core.ConfigFactory.parse_file(filename, resolve=r), resolve, unresolved_value) + return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value) + + @overload + @classmethod + def parse_URL( + cls, url: str, timeout: float | None = ..., resolve: bool = ..., required: Literal[False] = ..., unresolved_value: Any = ... + ) -> ConfigTree | list[Any]: ... + @overload + @classmethod + def parse_URL(cls, url: str, timeout: float | None, resolve: bool, required: Literal[True], unresolved_value: Any = ...) -> ConfigTree: ... + @overload + @classmethod + def parse_URL( + cls, url: str, timeout: float | None = ..., resolve: bool = ..., *, required: Literal[True], unresolved_value: Any = ... + ) -> ConfigTree: ... + @classmethod + def parse_URL( + cls, url: str, timeout: float | None = None, resolve: bool = True, required: bool = False, unresolved_value: Any = DEFAULT_SUBSTITUTION + ) -> ConfigTree | list[Any]: + """The document at ``url``. ``timeout`` in seconds, by default + ``socket.getdefaulttimeout()``. One that cannot be fetched is a + warning and an empty list, or with ``required=True`` the urllib + error. An include in it is an error: it has no directory.""" + try: + with contextlib.closing(urllib.request.urlopen(url, timeout=socket.getdefaulttimeout() if timeout is None else timeout)) as fd: + content = fd.read().decode("utf-8") + except urllib.error.URLError: + logger.warning("Cannot include url %s. Resource is inaccessible.", url) + if required: + raise + return [] + return _read(lambda r: _core.ConfigFactory.parse_string(content, resolve=r), resolve, unresolved_value) + + @classmethod + def parse_string( + cls, content: str, basedir: Path | None = None, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION + ) -> ConfigTree: + """HOCON text; an include in it is found in ``basedir``, by default + the working directory, as pyhocon finds it.""" + where = os.getcwd() if basedir is None else basedir + return _read(lambda r: _core.ConfigFactory.parse_string(content, basedir=where, resolve=r), resolve, unresolved_value) + + @classmethod + def from_dict(cls, dictionary: Mapping[str, Any], root: bool = False) -> Any: + """``dictionary`` as a ConfigTree, its dicts too; a key is a path, + so ``{"a.b": 1}`` is ``a { b = 1 }``.""" + + def tree(value: Any) -> Any: + if isinstance(value, Mapping): + res = ConfigTree(root=root) + for key, child in value.items(): + res.put(key, tree(child)) + return res + if isinstance(value, list): + return [tree(v) for v in value] + return value + + return tree(dictionary) + + +class ConfigParser: + """pyhocon's parser, which ConfigFactory calls.""" + + @classmethod + def parse(cls, content: str, basedir: Path | None = None, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION) -> ConfigTree: + """``ConfigFactory.parse_string``.""" + return ConfigFactory.parse_string(content, basedir, resolve, unresolved_value) + + @classmethod + def resolve_package_path(cls, package_path: str) -> str: + """``"my_package:foo/bar.conf"`` as the path of ``foo/bar.conf`` in + the installed package ``my_package``. ValueError without the + ``:``, ImportError for a package that is not there.""" + if ":" not in package_path: + raise ValueError("Expected format is 'PACKAGE:PATH'") + package, path = package_path.split(":", 1) + spec = importlib.util.find_spec(package) + if spec is None or not spec.submodule_search_locations: + raise ImportError(f"No module named {package!r}") + return os.path.join(spec.submodule_search_locations[0], path) + + @classmethod + def resolve_substitutions(cls, config: ConfigTree, accept_unresolved: bool = False) -> bool: + """Resolves a tree read with ``resolve=False``, in place; a ${…} + with no value is a ConfigSubstitutionException, or with + ``accept_unresolved=True`` stays. True when one stays.""" + if config._as_written is None: + return False + try: + text = config._as_written + tree = _call(lambda: _resolved(_core.ConfigFactory.parse_string(text))) + except ConfigSubstitutionException: + if accept_unresolved: + return True + raise + config.clear() + config.update(tree.items()) + config._as_written = None + config._unresolved = False + return False diff --git a/python/zig_hocon/pyhocon/config_tree.py b/python/zig_hocon/pyhocon/config_tree.py new file mode 100644 index 00000000..461e8fcc --- /dev/null +++ b/python/zig_hocon/pyhocon/config_tree.py @@ -0,0 +1,314 @@ +"""pyhocon's ``ConfigTree``: the document as an OrderedDict of Python's own +values, read and changed by dotted paths, as pyhocon has it.""" + +import datetime +import json +import re +from collections import OrderedDict +from typing import TYPE_CHECKING, Any + +from ..exceptions import ( + ConfigException, + ConfigMissingException, + ConfigSubstitutionException, + ConfigWrongTypeException, +) + +if TYPE_CHECKING: + from .config_parser import Path + +__all__ = [ + "ConfigInclude", + "ConfigList", + "ConfigQuotedString", + "ConfigSubstitution", + "ConfigTree", + "ConfigUnquotedString", + "ConfigValues", + "NonExistentKey", + "NoneValue", + "UndefinedKey", +] + + +class UndefinedKey: + """A getter's ``default`` when none is given.""" + + +class NonExistentKey: + """What ``get`` gives ``__getattr__`` for a name that is no key.""" + + +class NoneValue: + """``null`` inside pyhocon's parser. Here a ``null`` is None.""" + + +# pyhocon's parser objects, for code that imports them or checks a value +# against them. No value here is one: libhocon reads the document. +class ConfigInclude: + """pyhocon's ``include`` while it parses.""" + + +class ConfigValues: + """pyhocon's value that waits for a ``${…}``.""" + + +class ConfigSubstitution: + """pyhocon's ``${…}`` while it parses.""" + + +class ConfigQuotedString: + """pyhocon's quoted string while it parses.""" + + +class ConfigUnquotedString(str): + """pyhocon's unquoted string while it parses.""" + + +class ConfigList(list[Any]): + """pyhocon's list. A list here is a plain list.""" + + +# A path is cut at these; a key with one in it is kept in quotes, '"a.b"', +# so that a path can name it. +_SPECIAL = "$}[]:=+#`^?!@*&." +_PATH_KEY = re.compile('"[^"]+"|[^' + re.escape(_SPECIAL) + "]+") +_HAS_SPECIAL = re.compile("[" + re.escape(_SPECIAL) + "]").search + + +def quoted(key: str) -> str: + """``key`` as a ConfigTree keeps it.""" + return f'"{key}"' if _HAS_SPECIAL(key) else key + + +def _json_value(value: Any) -> Any: + """A timedelta put in a tree, as pyhocon writes it: milliseconds.""" + if isinstance(value, datetime.timedelta): + return int(value.total_seconds() * 1000) + raise TypeError(f"{type(value).__name__} is not a HOCON value") + + +class ConfigTree(OrderedDict[str, Any]): + """A document, or an object in one: an OrderedDict of its keys. ``get`` + and the getters take a path, ``db.host`` or ``a."b.c"``; ``conf.db.host`` + and ``conf["db.host"]`` are ``get`` too. A key with a path character in + it is kept in quotes: ``"b.c"``. A value is a ConfigTree, a list, text, + a number, a bool or None; durations and sizes are the text they are + written as.""" + + KEY_SEP = "." + + def __init__(self, *args: Any, **kwds: Any) -> None: + self.root = bool(kwds.pop("root", False)) + # A document read with resolve=False: its text, with every ${…} as + # written, for with_fallback() and resolve_substitutions() to finish; + # `_unresolved` when it could not be read without them. Private, as a + # name here hides a key of that name from `conf.name`. + self._as_written: str | None = None + self._unresolved = False + super().__init__(*args, **kwds) + + def _check_resolved(self) -> None: + if self._unresolved: + raise ConfigSubstitutionException( + "the document was read with resolve=False and a ${…} in it has no value yet: " + "with_fallback() or ConfigParser.resolve_substitutions() gives it one", + name="Unresolved", + ) + + def _hocon_text(self) -> str: + """This tree as HOCON text that merges after another's.""" + if self._as_written is not None: + return self._as_written + # JSON is HOCON; without its braces it is the fields of the root. + return json.dumps(self.as_plain_ordered_dict(), default=_json_value)[1:-1] + + @staticmethod + def merge_configs(a: "ConfigTree", b: "ConfigTree", copy_trees: bool = False) -> "ConfigTree": + """``b`` merged into ``a``: an object into an object key by key, + anything else in place of what was there. ``copy_trees`` merges into + copies of ``a``'s objects. Gives ``a``.""" + if a._as_written is not None or b._as_written is not None: + # a document read with resolve=False is its text: the two texts, + # one after the other, merge as two files do + from .config_parser import as_written + + merged = as_written(a._hocon_text() + "\n" + b._hocon_text()) + a.clear() + a.update(merged.items()) + a._as_written, a._unresolved = merged._as_written, merged._unresolved + return a + for key, value in b.items(): + mine = OrderedDict.get(a, key) + if isinstance(mine, ConfigTree) and isinstance(value, ConfigTree): + if copy_trees: + mine = a[key] = mine.copy() + ConfigTree.merge_configs(mine, value, copy_trees) + else: + a[key] = value + return a + + @staticmethod + def parse_key(string: str) -> list[str]: + """``a.b."c.d"`` as ``["a", "b", '"c.d"']``: a key keeps its quotes + when it needs them.""" + return [key if _HAS_SPECIAL(key) else key.strip('"') for key in _PATH_KEY.findall(string)] + + def _get(self, path: list[str], default: Any = UndefinedKey) -> Any: + self._check_resolved() + node: Any = self + for i, key in enumerate(path): + if not isinstance(node, ConfigTree): + if default is UndefinedKey: + raise ConfigWrongTypeException(f"{'.'.join(path[:i])} has type {type(node).__name__} rather than dict", name="WrongType") + return default + node = OrderedDict.get(node, key, UndefinedKey) + if node is UndefinedKey: + if default is UndefinedKey: + raise ConfigMissingException(f"No configuration setting found for key {'.'.join(path[: i + 1])}", name="Missing") + return default + # a copy, as pyhocon gives it + return list(node) if isinstance(node, list) else node + + def put(self, key: str, value: Any, append: bool = False) -> None: + """``value`` at ``key``, the objects on the way made where there are + none. An object put on an object is merged into it; ``append=True`` + adds a list to the list there.""" + if isinstance(value, NoneValue): + value = None + path = self.parse_key(key) + node = self + for k in path[:-1]: + child = OrderedDict.get(node, k) + if not isinstance(child, ConfigTree): + child = node[k] = ConfigTree() + node = child + last = path[-1] + before = OrderedDict.get(node, last) + if isinstance(before, ConfigTree) and isinstance(value, ConfigTree): + if node.root: + # the root's object is replaced, not changed: a tree that + # was taken from it before stays as it was + node[last] = ConfigTree.merge_configs(ConfigTree.merge_configs(ConfigTree(), before, True), value, True) + else: + ConfigTree.merge_configs(before, value) + elif append and isinstance(before, list): + node[last] = before + value + elif append and before is not None: + raise ConfigWrongTypeException( + f"Cannot concatenate the list {key}: {value} to {before} of {type(before).__name__}", name="WrongType" + ) + else: + node[last] = value + + def get(self, key: str, default: Any = UndefinedKey) -> Any: + """The value at ``key``; ``default`` when there is none, else + ConfigMissingException. A list is a copy.""" + return self._get(self.parse_key(key), default) + + def pop(self, key: str, default: Any = UndefinedKey) -> Any: + """The value at ``key``, taken out of its object; ``default`` when + there is none, else ConfigMissingException.""" + if default is not UndefinedKey and key not in self: + return default + path = self.parse_key(key) + value = self._get(path) + del self._get(path[:-1])[path[-1]] + return value + + def get_string(self, key: str, default: Any = UndefinedKey) -> str | None: + """``str()`` of the value, a bool as ``true``/``false``; None for + ``null``.""" + value = self.get(key, default) + if value is None: + return None + return str(value).lower() if isinstance(value, bool) else str(value) + + def get_int(self, key: str, default: Any = UndefinedKey) -> int | None: + """``int()`` of the value: ``"5"`` is 5 and ``1.5`` is 1. None for + ``null``.""" + value = self.get(key, default) + try: + return int(value) if value is not None else None + except (TypeError, ValueError): + raise ConfigException(f"{key} has type '{type(value).__name__}' rather than 'int'", name="WrongType") from None + + def get_float(self, key: str, default: Any = UndefinedKey) -> float | None: + """``float()`` of the value. None for ``null``.""" + value = self.get(key, default) + try: + return float(value) if value is not None else None + except (TypeError, ValueError): + raise ConfigException(f"{key} has type '{type(value).__name__}' rather than 'float'", name="WrongType") from None + + def get_bool(self, key: str, default: Any = UndefinedKey) -> bool | None: + """``true``/``yes``/``on`` and ``false``/``no``/``off``, in any case. + None for ``null``.""" + text = self.get_string(key, default) + bools = {"true": True, "yes": True, "on": True, "false": False, "no": False, "off": False} + if text is None: + return None + if text.lower() not in bools: + raise ConfigException(f"{key} does not translate to a Boolean value", name="WrongType") + return bools[text.lower()] + + def get_list(self, key: str, default: Any = UndefinedKey) -> list[Any] | None: + """The list at ``key``. An object with whole-number keys is a list + too, as the spec has it: in the order of the numbers, the other keys + left out. None for ``null``.""" + value = self.get(key, default) + if value is None or isinstance(value, list): + return value + if isinstance(value, ConfigTree): + indexed = sorted(((int(k), v) for k, v in value.items() if k.isascii() and k.isdigit()), key=lambda kv: kv[0]) + if not indexed: + raise ConfigException(f"{key} does not translate to a list", name="WrongType") + return [v for _, v in indexed] + raise ConfigException(f"{key} has type '{type(value).__name__}' rather than 'list'", name="WrongType") + + def get_config(self, key: str, default: Any = UndefinedKey) -> Any: + """The object at ``key``, the tree itself and not a copy. None for + ``null``.""" + value = self.get(key, default) + if value is None or isinstance(value, dict): + return value + raise ConfigException(f"{key} has type '{type(value).__name__}' rather than 'config'", name="WrongType") + + def __getitem__(self, item: str) -> Any: + return self.get(item) + + def __getattr__(self, item: str) -> Any: + # copy and pickle look for their methods here + if item.startswith("__"): + raise AttributeError(item) + value = self.get(item, NonExistentKey) + if value is NonExistentKey: + raise AttributeError(item) + return value + + def __contains__(self, item: object) -> bool: + return isinstance(item, str) and self._get(self.parse_key(item), NoneValue) is not NoneValue + + def with_fallback(self, config: "ConfigTree | Path", resolve: bool = True) -> "ConfigTree": + """A new tree: ``config`` (a tree, or the name of a file) with this + one over it, merged as two files one after the other, and resolved + unless ``resolve=False``: a ``${…}`` here can take its value from + ``config``.""" + from .config_parser import ConfigFactory + + fallback = config if isinstance(config, ConfigTree) else ConfigFactory.parse_file(config, resolve=False) + return ConfigFactory.parse_string(fallback._hocon_text() + "\n" + self._hocon_text(), None, resolve) + + def as_plain_ordered_dict(self) -> "OrderedDict[str, Any]": + """A deep copy as OrderedDicts and lists, keys without their quotes.""" + self._check_resolved() + + def plain(value: Any) -> Any: + if isinstance(value, ConfigTree): + return value.as_plain_ordered_dict() + if isinstance(value, list): + return [plain(v) for v in value] + return value + + return OrderedDict((key.strip('"'), plain(value)) for key, value in self.items()) + diff --git a/python/zig_hocon/pyhocon/converter.py b/python/zig_hocon/pyhocon/converter.py new file mode 100644 index 00000000..a7a48635 --- /dev/null +++ b/python/zig_hocon/pyhocon/converter.py @@ -0,0 +1,172 @@ +"""pyhocon's ``HOCONConverter``: a ConfigTree as JSON, HOCON, YAML or a +.properties file, in pyhocon's layout to the character.""" + +import datetime +import json +import re +import sys +from typing import Any + +from .config_parser import ConfigFactory, Path +from .config_tree import ConfigTree + +__all__ = ["HOCONConverter"] + + +def _ms(value: datetime.timedelta) -> str: + """A timedelta put in a tree, as pyhocon writes it: milliseconds.""" + return str(int(value.total_seconds() * 1000)) + + +def _hocon_duration(value: datetime.timedelta) -> str: + if value.days > 0: + return f"{value.days} days" + if value.seconds > 0: + return f"{value.seconds} seconds" + if value.microseconds > 0: + return f"{value.microseconds} microseconds" + return "0 seconds" + + +def _scalar(value: Any) -> str: + if value is None: + return "null" + if value is True: + return "true" + if value is False: + return "false" + return str(value) + + +class HOCONConverter: + """Writes a ConfigTree out, as pyhocon's converter.""" + + _ESCAPES = {"\b": r"\b", "\t": r"\t", "\n": r"\n", "\f": r"\f", "\r": r"\r", '"': r"\"", "\\": r"\\"} + + @classmethod + def to_json(cls, config: Any, compact: bool = False, indent: int = 2, level: int = 0) -> str: + """JSON, ``indent`` spaces a level. ``compact`` is pyhocon's and + changes nothing.""" + pad, inner = " " * (level * indent), " " * ((level + 1) * indent) + if isinstance(config, ConfigTree): + if not config: + return "{}" + fields = (f'{inner}"{key.strip(chr(34))}": {cls.to_json(item, compact, indent, level + 1)}' for key, item in config.items()) + return "{\n" + ",\n".join(fields) + f"\n{pad}}}" + if isinstance(config, list): + if not config: + return "[]" + return "[\n" + ",\n".join(f"{inner}{cls.to_json(item, compact, indent, level + 1)}" for item in config) + f"\n{pad}]" + if isinstance(config, datetime.timedelta): + return _ms(config) + if isinstance(config, str): + return json.dumps(config, ensure_ascii=False) + return _scalar(config) + + @classmethod + def to_hocon(cls, config: Any, compact: bool = False, indent: int = 2, level: int = 0) -> str: + """HOCON, ``indent`` spaces a level; ``compact`` writes an object of + one key as a path, ``a.b.c = 1``.""" + if isinstance(config, ConfigTree): + if not config: + return "{}" + fields = [] + for key, item in config.items(): + full_key = key + while compact and isinstance(item, ConfigTree) and len(item) == 1: + key, item = next(iter(item.items())) + full_key += "." + key + sign = "" if isinstance(item, dict) else " =" + fields.append(f"{' ' * (level * indent)}{full_key}{sign} {cls.to_hocon(item, compact, indent, level + 1)}") + body = "\n".join(fields) + # the root has no braces + return body if level == 0 else "{\n" + body + f"\n{' ' * ((level - 1) * indent)}}}" + if isinstance(config, list): + if not config: + return "[]" + items = "\n".join(f"{' ' * (level * indent)}{cls.to_hocon(item, compact, indent, level + 1)}" for item in config) + return "[\n" + items + f"\n{' ' * ((level - 1) * indent)}]" + if isinstance(config, str): + if "\n" in config and len(config) > 1: + return f'"""{config}"""' + return '"' + cls._escape_string(config) + '"' + if isinstance(config, datetime.timedelta): + return _hocon_duration(config) + return _scalar(config) + + @classmethod + def to_yaml(cls, config: Any, compact: bool = False, indent: int = 2, level: int = 0) -> str: + """YAML, ``indent`` spaces a level; a ``null`` in a list is left out. + ``compact`` is pyhocon's and changes nothing.""" + pad = " " * (level * indent) + if isinstance(config, ConfigTree): + if not config: + return "" + fields = "\n".join(f"{pad}{key.strip(chr(34))}: {cls.to_yaml(item, compact, indent, level + 1)}" for key, item in config.items()) + return ("\n" if level > 0 else "") + fields + if isinstance(config, list): + items = [item for item in config if item is not None] + if not items: + return "[]" + return "\n" + "\n".join(f"{pad}- {cls.to_yaml(item, compact, indent, level + 1)}" for item in items) + if isinstance(config, datetime.timedelta): + return _ms(config) + if isinstance(config, str): + lines = config.split("\n") + if len(lines) == 1: + return config + return "|\n" + "\n".join(line.rjust(level * indent, " ") for line in lines) + return _scalar(config) + + @classmethod + def to_properties(cls, config: Any, compact: bool = False, indent: int = 2, key_stack: list[str] | None = None) -> str: + """A .properties file: ``a.b = 1``, a list's items as ``a.0``; + ``null`` left out. ``compact`` and ``indent`` are pyhocon's and + change nothing.""" + keys = [key.strip('"') for key in key_stack or []] + if isinstance(config, ConfigTree): + lines = [cls.to_properties(item, compact, indent, keys + [key]) for key, item in config.items() if item is not None] + elif isinstance(config, list): + lines = [cls.to_properties(item, compact, indent, keys + [str(i)]) for i, item in enumerate(config) if item is not None] + elif config is None: + lines = [] + elif isinstance(config, datetime.timedelta): + lines = [".".join(keys) + " = " + _ms(config)] + elif isinstance(config, str): + escaped = config.replace("=", "\\=").replace("!", "\\!").replace("#", "\\#").replace("\n", "\\\n") + lines = [".".join(keys) + " = " + escaped] + else: + lines = [".".join(keys) + " = " + _scalar(config)] + return "\n".join(line for line in lines if line) + + @classmethod + def convert(cls, config: Any, output_format: str = "json", indent: int = 2, compact: bool = False) -> str: + """``config`` in ``output_format``: ``json``, ``properties``, + ``yaml`` or ``hocon``.""" + if output_format == "json": + return cls.to_json(config, compact, indent) + if output_format == "properties": + return cls.to_properties(config, compact, indent) + if output_format == "yaml": + return cls.to_yaml(config, compact, indent) + if output_format == "hocon": + return cls.to_hocon(config, compact, indent) + raise Exception(f"Invalid format '{output_format}'. Format must be 'json', 'properties', 'yaml' or 'hocon'") + + @classmethod + def convert_from_file( + cls, input_file: Path | None = None, output_file: Path | None = None, output_format: str = "json", indent: int = 2, compact: bool = False + ) -> None: + """Reads ``input_file`` (stdin without one) and writes it in + ``output_format`` to ``output_file`` (stdout without one).""" + config = ConfigFactory.parse_string(sys.stdin.read()) if input_file is None else ConfigFactory.parse_file(input_file) + res = cls.convert(config, output_format, indent, compact) + if output_file is None: + print(res) + else: + with open(output_file, "w") as fd: + fd.write(res) + + @classmethod + def _escape_string(cls, string: str) -> str: + return re.sub(r'[\x00-\x1F"\\]', lambda m: cls._ESCAPES.get(m.group(0)) or rf"\u{ord(m.group(0)):04x}", string) diff --git a/python/zig_hocon/pyhocon/exceptions.py b/python/zig_hocon/pyhocon/exceptions.py new file mode 100644 index 00000000..ba712153 --- /dev/null +++ b/python/zig_hocon/pyhocon/exceptions.py @@ -0,0 +1,18 @@ +"""pyhocon's exceptions, which are zig_hocon's: one ``except`` catches +either module's.""" + +from ..exceptions import ( + ConfigException, + ConfigMissingException, + ConfigParseException, + ConfigSubstitutionException, + ConfigWrongTypeException, +) + +__all__ = [ + "ConfigException", + "ConfigMissingException", + "ConfigParseException", + "ConfigSubstitutionException", + "ConfigWrongTypeException", +] diff --git a/python/zig_hocon/pyhocon/tool.py b/python/zig_hocon/pyhocon/tool.py new file mode 100644 index 00000000..a1934a9f --- /dev/null +++ b/python/zig_hocon/pyhocon/tool.py @@ -0,0 +1,34 @@ +"""pyhocon's command line, ``python -m zig_hocon.pyhocon.tool``: a file, or +stdin, as JSON, HOCON, YAML or a .properties file. zig-hocon's own is +``python -m zig_hocon``.""" + +import argparse +import logging +from collections.abc import Sequence + +from .converter import HOCONConverter + +__all__ = ["HOCONConverter", "main"] + +LOG_FORMAT = "%(asctime)s %(levelname)s: %(message)s" + + +def main(argv: Sequence[str] | None = None) -> None: + parser = argparse.ArgumentParser(description="pyhocon tool") + parser.add_argument("-i", "--input", help="input file") + parser.add_argument("-o", "--output", help="output file") + parser.add_argument("-c", "--compact", action="store_true", default=False, help="compact format") + parser.add_argument("-f", "--format", help="output format: json, properties, yaml or hocon", default="json") + parser.add_argument("-n", "--indent", help="indentation step (default is 2)", default=2, type=int) + parser.add_argument("-v", "--verbosity", action="count", default=0, help="increase output verbosity") + args = parser.parse_args(argv) + if args.verbosity: + handler = logging.StreamHandler() + handler.setFormatter(logging.Formatter(LOG_FORMAT)) + logging.getLogger().addHandler(handler) + logging.getLogger().setLevel({1: logging.ERROR, 2: logging.INFO}.get(args.verbosity, logging.DEBUG)) + HOCONConverter.convert_from_file(args.input, args.output, args.format.lower(), args.indent, args.compact) + + +if __name__ == "__main__": # pragma: no cover + main() From f8b7eff2b5e9c1f40be0689962bcd68d2cea4d78 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Wed, 21 Oct 2026 06:56:38 +0200 Subject: [PATCH 02/30] fix(python): zig_hocon.pyhocon gives durations pyhocon's types `30s` is a timedelta, a relativedelta with python-dateutil, in objects and lists too, as pyhocon parses it; they stay durations through with_fallback, and HOCONConverter writes a relativedelta as pyhocon does. pyhocon's own suite goes from 167 to 218 of 263. A quoted "30s" is a duration too: libhocon's JSON has no quotes left. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- CHANGELOG.md | 2 +- build.zig | 5 +- python/MIGRATING.md | 9 +-- python/README.md | 2 +- python/tests/test_pyhocon_api.py | 60 ++++++++++++++++- python/zig_hocon/pyhocon/config_parser.py | 15 ++++- python/zig_hocon/pyhocon/config_tree.py | 79 +++++++++++++++++++++-- python/zig_hocon/pyhocon/converter.py | 24 ++++--- 8 files changed, 171 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed2bab29..e442ad53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ### Added -- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. A config still reads by the spec; pyhocon's own test suite passes 167 of 263 with the import swapped, and `python/MIGRATING.md` says what the rest are. +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 218 of 263 with the import swapped, and `python/MIGRATING.md` says what the rest are. ### Fixed diff --git a/build.zig b/build.zig index d98bfecd..4eab3ff6 100644 --- a/build.zig +++ b/build.zig @@ -79,10 +79,11 @@ pub fn build(b: *std.Build) void { // Every line and branch of the package runs, or it fails (`pragma: no // cover` for what no test can reach). `python -m zig_hocon` is left out: // test_main.py runs it as a process, as the conformance adapter does, - // where coverage does not follow. + // where coverage does not follow. dateutil is there for + // zig_hocon.pyhocon's months and years, as pyhocon has them with it. const python_version = b.option([]const u8, "python", "The Python test-python runs on (default 3.12)") orelse "3.12"; const coverage_file = b.pathFromRoot(".zig-cache/python.coverage"); - const pytest = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--python", python_version, "--with", "pytest", "--with", "coverage", "python", "-m", "coverage", "run", "--branch", "--source=python/zig_hocon", "--omit=python/zig_hocon/__main__.py", "-m", "pytest", "-q", "python/tests" }); + const pytest = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--python", python_version, "--with", "pytest", "--with", "coverage", "--with", "python-dateutil", "python", "-m", "coverage", "run", "--branch", "--source=python/zig_hocon", "--omit=python/zig_hocon/__main__.py", "-m", "pytest", "-q", "python/tests" }); pytest.setCwd(b.path(".")); pytest.setEnvironmentVariable("PYTHONPATH", "python"); pytest.setEnvironmentVariable("COVERAGE_FILE", coverage_file); diff --git a/python/MIGRATING.md b/python/MIGRATING.md index a2c63f0b..6231f3b1 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -47,11 +47,12 @@ from zig_hocon.pyhocon.exceptions import ConfigMissingException Everything of pyhocon's public API is there: `ConfigFactory.parse_file` (with `encoding` and `required`), `parse_string`, `parse_URL`, `from_dict`; the `ConfigTree` with `get`, the typed getters, `get_list`, `get_config`, `put`, `pop`, `with_fallback`, `merge_configs`, `as_plain_ordered_dict`, `conf["db.host"]` and `conf.db.host`; `ConfigParser.parse`, `resolve_substitutions` and `resolve_package_path`; `HOCONConverter` with its JSON, HOCON, YAML and .properties output to the character; and pyhocon's command line as `python -m zig_hocon.pyhocon.tool`. The exceptions are zig_hocon's, so one `except` catches either module's. -The measure is pyhocon's own test suite with the import swapped: 167 of its 263 tests pass. Of the 96 that do not, 95 are a config that reads differently by the spec (Step 3) or the class of a syntax error: +Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. One difference: a quoted `"30s"` is a duration as well, where pyhocon keeps it text. + +The measure is pyhocon's own test suite with the import swapped: 218 of its 263 tests pass. Of the 45 that do not, 44 are a config that reads differently by the spec (Step 3) or the class of a syntax error: | tests | what pyhocon does | |---|---| -| 51 | turns `30s` into a `timedelta` while it parses | | 12 | raises pyparsing's exceptions for a syntax error; here it is a `ConfigParseException` | | 9 | flattens `x += [3, 4]` into `x`; by the spec it appends the list as one element | | 6 | reads `-Xmx1g` unquoted; the spec does not let an unquoted string start with `-` | @@ -67,7 +68,7 @@ The one left is `ConfigTree.resolve()`, which is not there. Nor are a few things - **`unresolved_value`** takes `DEFAULT_SUBSTITUTION`, `MANDATORY_SUBSTITUTION` and `NO_SUBSTITUTION`; `STR_SUBSTITUTION` and a value of your own are a `NotImplementedError`. - **`parse_URL`** reads a document with no `include` in it: one is an error, as it has no directory to be found in. -The whole document is in Python, as in pyhocon, so it is a little slower than `zig_hocon`, still far from pyhocon: a 5 KB config in 0.24 ms (pyhocon 29 ms), 10 MB in 0.32 s (pyhocon 68 s). +The whole document is in Python, as in pyhocon, so it is a little slower than `zig_hocon`, still far from pyhocon: a 5 KB config in 0.3 ms (pyhocon 29 ms), 10 MB in 0.38 s (pyhocon 68 s), against 0.2 ms and 0.13 s for `zig_hocon`. ## Step 1: imports @@ -125,7 +126,7 @@ settings["db"]["host"] conf = ConfigFactory.parse_file(str(files("mypackage") / "defaults.conf")) ``` -- **Durations and sizes are text until something asks.** pyhocon turns `30s` into a `timedelta` while it parses; here `as_dict()` has `"30s"`, and `get_duration` or the model converts it. With pydantic, a `BeforeValidator(zig_hocon.parse_duration)` on the field; [examples/durations-and-sizes](examples/durations-and-sizes) has it for pydantic and msgspec. +- **Durations and sizes are text until something asks** (not in `zig_hocon.pyhocon`, which has pyhocon's `timedelta`). pyhocon turns `30s` into a `timedelta` while it parses; here `as_dict()` has `"30s"`, and `get_duration` or the model converts it. With pydantic, a `BeforeValidator(zig_hocon.parse_duration)` on the field; [examples/durations-and-sizes](examples/durations-and-sizes) has it for pydantic and msgspec. - **A concatenation is a string.** `a = ${?nope}42` is `"42"` by the spec; pyhocon gives the number `42`. A model that wants an int converts it. - **An `include url(…)` gives up after 30 s.** pyhocon waits as long as `socket.setdefaulttimeout` says, by default for ever. Here `http_timeout=` on `parse_file` and `parse_string` sets it, and a server that does not answer in time is a `ConfigParseException` named `IncludeTimeout`. - **A leading zero keeps the text.** A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. diff --git a/python/README.md b/python/README.md index 96867ba0..15fabb1c 100644 --- a/python/README.md +++ b/python/README.md @@ -116,7 +116,7 @@ from zig_hocon.pyhocon import ConfigFactory # was: from pyhocon import ConfigF Either way, a config reads by the spec, and in a few places pyhocon reads it otherwise: -- **Durations stay text.** `30s` is `"30s"` until a getter or a model reads it. +- **Durations are text in the package's own API.** `30s` is `"30s"` until a getter or a model reads it; `zig_hocon.pyhocon` makes it a `timedelta`, as pyhocon does. - **`+=` appends one element.** `x += [3, 4]` adds the list `[3, 4]` to `x`; `x = ${x} [3, 4]` extends it. - **Strict syntax.** Input pyhocon quietly accepted, such as `a..b = 1`, `a = b:c` or an unquoted `-Xmx1g`, is an error with its line and column. - **No `include package(...)`.** It is pyhocon's own; include by path. diff --git a/python/tests/test_pyhocon_api.py b/python/tests/test_pyhocon_api.py index fe18e3cd..8f5ae1be 100644 --- a/python/tests/test_pyhocon_api.py +++ b/python/tests/test_pyhocon_api.py @@ -11,6 +11,7 @@ from collections import OrderedDict import pytest +from dateutil.relativedelta import relativedelta import zig_hocon.pyhocon as pyhocon from zig_hocon.pyhocon import ( @@ -266,12 +267,69 @@ def test_with_fallback_unresolved(): def test_with_fallback_of_put_values(): c = ConfigTree() c.put("wait", datetime.timedelta(seconds=2)) - assert c.with_fallback(ConfigTree()).get("wait") == 2000 + assert c.with_fallback(ConfigTree()).get("wait") == datetime.timedelta(seconds=2) c.put("owl", object()) with pytest.raises(TypeError, match="object is not a HOCON value"): c.with_fallback(ConfigTree()) +# pyhocon ✓: a whole number and a unit is a duration, a relativedelta with +# dateutil (milliseconds a timedelta), in objects and lists too +@pytest.mark.parametrize( + "text, value", + [ + ("30s", relativedelta(seconds=30)), + ("10 weeks", relativedelta(days=70)), + ("3m", relativedelta(minutes=3)), + ("3mo", relativedelta(months=3)), + ("2 year", relativedelta(years=2)), + ("1140000 ns", relativedelta(microseconds=1140)), + ("114 ms", datetime.timedelta(milliseconds=114)), + ("5 seconds", relativedelta(seconds=5)), + ("[4 h, 5 ms]", [relativedelta(hours=4), datetime.timedelta(milliseconds=5)]), + ("{ x = 2 h }", {"x": relativedelta(hours=2)}), + ("1.5 s", "1.5 s"), + ("-3 s", "-3 s"), + ("3 monthsx", "3 monthsx"), + # pyhocon ✗: it makes '0:00:00.005000foo' of it + ("5 ms foo", "5 ms foo"), + # pyhocon ✗: quoted is text there; libhocon's JSON has no quotes left + ('"30s"', relativedelta(seconds=30)), + ], +) +def test_durations(text, value): + assert conf("a = " + text)["a"] == value + + +# pyhocon ✓: without dateutil a timedelta, and months and years are text +def test_durations_without_dateutil(monkeypatch): + monkeypatch.setattr(config_tree, "relativedelta", None) + c = conf("a = 30s\nb = 3mo\nc = [1 d]") + assert c["a"] == datetime.timedelta(seconds=30) and type(c["a"]) is datetime.timedelta + assert c["b"] == "3mo" and c["c"] == [datetime.timedelta(days=1)] + + +# pyhocon ✓: a duration stays one through with_fallback, as pyhocon merges +# trees; a relativedelta of months and days is no one HOCON duration. +# pyhocon ⚠️: a timedelta below a millisecond comes back a relativedelta +def test_durations_through_with_fallback(): + c = conf("a = 3mo\nb = 30s\nc = 114 ms\nd = 2 y") + c.put("e", datetime.timedelta(microseconds=7)) + merged = c.with_fallback(ConfigTree()) + assert merged == {"a": relativedelta(months=3), "b": relativedelta(seconds=30), "c": datetime.timedelta(milliseconds=114), + "d": relativedelta(years=2), "e": relativedelta(microseconds=7)} + c.put("f", relativedelta(months=1, days=2)) + with pytest.raises(TypeError, match="is not one HOCON duration"): + c.with_fallback(ConfigTree()) + + +# pyhocon ✓: checked against pyhocon 0.3.63's output +def test_relativedeltas_converted(): + c = conf("a = 7 h\nb = 3 m\nc = 12 d\nd = 3mo\ne = 112 micros") + assert HOCONConverter.to_hocon(c) == "a = 7 hours\nb = 3 minutes\nc = 12 days\nd = 0 seconds\ne = 112 microseconds" + assert HOCONConverter.to_json(c) == '{\n "a": 25200000,\n "b": 180000,\n "c": 1036800000,\n "d": 0,\n "e": 0\n}' + + # pyhocon ⚠️: a value with a ${…} is no ConfigValues here; a document that # resolves alone has its values, one that does not has none until resolved def test_resolve_false(): diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index 348d4585..d2d8cb41 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -17,7 +17,7 @@ from .. import config_parser as _core from .. import config_tree as _core_tree from ..exceptions import ConfigException, ConfigParseException, ConfigSubstitutionException, ConfigWrongTypeException -from .config_tree import ConfigList, ConfigTree, quoted +from .config_tree import ConfigList, ConfigTree, duration, quoted __all__ = [ "DEFAULT_SUBSTITUTION", @@ -80,8 +80,17 @@ def _call(read: Callable[[], _T]) -> _T: raise err from None +def _typed(value: Any) -> Any: + """A value from libhocon's JSON as pyhocon has it: ``30s`` a duration.""" + if isinstance(value, str): + return duration(value) + if isinstance(value, list): + return [_typed(v) for v in value] + return value + + def _tree(pairs: list[tuple[str, Any]]) -> ConfigTree: - return ConfigTree((quoted(k), v) for k, v in pairs) + return ConfigTree((quoted(k), _typed(v)) for k, v in pairs) def _resolved(doc: _core_tree.ConfigTree) -> ConfigTree: @@ -89,7 +98,7 @@ def _resolved(doc: _core_tree.ConfigTree) -> ConfigTree: if isinstance(value, list): # a document that is a list, as pyhocon gives it; typed as the # object a document nearly always is - return cast(ConfigTree, ConfigList(value)) + return cast(ConfigTree, ConfigList(_typed(value))) value.root = True return cast(ConfigTree, value) diff --git a/python/zig_hocon/pyhocon/config_tree.py b/python/zig_hocon/pyhocon/config_tree.py index 461e8fcc..49ff1334 100644 --- a/python/zig_hocon/pyhocon/config_tree.py +++ b/python/zig_hocon/pyhocon/config_tree.py @@ -2,6 +2,7 @@ values, read and changed by dotted paths, as pyhocon has it.""" import datetime +import importlib import json import re from collections import OrderedDict @@ -81,10 +82,78 @@ def quoted(key: str) -> str: return f'"{key}"' if _HAS_SPECIAL(key) else key -def _json_value(value: Any) -> Any: - """A timedelta put in a tree, as pyhocon writes it: milliseconds.""" +# pyhocon's durations: an unquoted `30s` or `10 weeks` is a timedelta, or a +# dateutil relativedelta when dateutil is installed, which months and years +# need. Milliseconds are always a timedelta. +try: + relativedelta: Any = importlib.import_module("dateutil.relativedelta").relativedelta +except ImportError: # pragma: no cover - the tests have dateutil and patch it away + relativedelta = None + +_UNITS = { + "nanoseconds": ["ns", "nano", "nanos", "nanosecond", "nanoseconds"], + "microseconds": ["us", "micro", "micros", "microsecond", "microseconds"], + "milliseconds": ["ms", "milli", "millis", "millisecond", "milliseconds"], + "seconds": ["s", "second", "seconds"], + "minutes": ["m", "minute", "minutes"], + "hours": ["h", "hour", "hours"], + "weeks": ["w", "week", "weeks"], + "days": ["d", "day", "days"], +} +_DATEUTIL_UNITS = {"months": ["mo", "month", "months"], "years": ["y", "year", "years"]} + + +def _duration_re(units: dict[str, list[str]]) -> "re.Pattern[str]": + # the longest spelling first: `ms` before `m`, `mo` before `m` + names = sorted((n for spellings in units.values() for n in spellings), key=len, reverse=True) + return re.compile(r"(\d+) *(" + "|".join(names) + ")") + + +_DURATION = _duration_re(_UNITS) +_DURATION_DATEUTIL = _duration_re(_UNITS | _DATEUTIL_UNITS) +_UNIT_OF = {n: unit for unit, spellings in (_UNITS | _DATEUTIL_UNITS).items() for n in spellings} + + +def duration(text: str) -> Any: + """``text`` as pyhocon reads a value: a duration when it is a whole + number and a unit, else ``text``. + + ponytail: libhocon's JSON has no quotes left, so a quoted "30s" is a + duration too, where pyhocon keeps it text; hocon.h telling quoted from + unquoted would make it exact.""" + m = (_DURATION if relativedelta is None else _DURATION_DATEUTIL).fullmatch(text) + if m is None: + return text + value, unit = int(m[1]), _UNIT_OF[m[2]] + if unit == "nanoseconds": + value, unit = value // 1000, "microseconds" + if unit == "milliseconds" or relativedelta is None: + return datetime.timedelta(**{unit: value}) + return relativedelta(**{unit: value}) + + +def _duration_text(value: Any) -> str: + """A duration as HOCON that reads back as the same one: milliseconds, + or microseconds below them, for a timedelta; a relativedelta in months or + microseconds, the units it keeps.""" if isinstance(value, datetime.timedelta): - return int(value.total_seconds() * 1000) + us = value // datetime.timedelta(microseconds=1) + return f"{us // 1000} ms" if us % 1000 == 0 else f"{us} us" + months = value.years * 12 + value.months + us = (((value.days * 24 + value.hours) * 60 + value.minutes) * 60 + value.seconds) * 1_000_000 + value.microseconds + if months and us: + raise TypeError(f"{value!r} is not one HOCON duration") + return f"{months} months" if months else f"{us} us" + + +def is_duration(value: Any) -> bool: + return isinstance(value, datetime.timedelta) or relativedelta is not None and isinstance(value, relativedelta) + + +def _json_value(value: Any) -> Any: + """A duration in a tree, as HOCON text that reads back as it.""" + if is_duration(value): + return _duration_text(value) raise TypeError(f"{type(value).__name__} is not a HOCON value") @@ -93,8 +162,8 @@ class ConfigTree(OrderedDict[str, Any]): and the getters take a path, ``db.host`` or ``a."b.c"``; ``conf.db.host`` and ``conf["db.host"]`` are ``get`` too. A key with a path character in it is kept in quotes: ``"b.c"``. A value is a ConfigTree, a list, text, - a number, a bool or None; durations and sizes are the text they are - written as.""" + a number, a bool or None, or a duration as pyhocon has it (``30s`` is + a timedelta); sizes are the text they are written as.""" KEY_SEP = "." diff --git a/python/zig_hocon/pyhocon/converter.py b/python/zig_hocon/pyhocon/converter.py index a7a48635..a021a68a 100644 --- a/python/zig_hocon/pyhocon/converter.py +++ b/python/zig_hocon/pyhocon/converter.py @@ -8,17 +8,25 @@ from typing import Any from .config_parser import ConfigFactory, Path -from .config_tree import ConfigTree +from .config_tree import ConfigTree, is_duration __all__ = ["HOCONConverter"] -def _ms(value: datetime.timedelta) -> str: - """A timedelta put in a tree, as pyhocon writes it: milliseconds.""" +def _ms(value: Any) -> str: + """A duration, as pyhocon writes it: milliseconds, a relativedelta's + months and years left out.""" + if not isinstance(value, datetime.timedelta): + value = datetime.timedelta(days=value.days, hours=value.hours, minutes=value.minutes, seconds=value.seconds, microseconds=value.microseconds) return str(int(value.total_seconds() * 1000)) -def _hocon_duration(value: datetime.timedelta) -> str: +def _hocon_duration(value: Any) -> str: + if not isinstance(value, datetime.timedelta): # a relativedelta + if value.hours > 0: + return f"{value.hours} hours" + if value.minutes > 0: + return f"{value.minutes} minutes" if value.days > 0: return f"{value.days} days" if value.seconds > 0: @@ -57,7 +65,7 @@ def to_json(cls, config: Any, compact: bool = False, indent: int = 2, level: int if not config: return "[]" return "[\n" + ",\n".join(f"{inner}{cls.to_json(item, compact, indent, level + 1)}" for item in config) + f"\n{pad}]" - if isinstance(config, datetime.timedelta): + if is_duration(config): return _ms(config) if isinstance(config, str): return json.dumps(config, ensure_ascii=False) @@ -90,7 +98,7 @@ def to_hocon(cls, config: Any, compact: bool = False, indent: int = 2, level: in if "\n" in config and len(config) > 1: return f'"""{config}"""' return '"' + cls._escape_string(config) + '"' - if isinstance(config, datetime.timedelta): + if is_duration(config): return _hocon_duration(config) return _scalar(config) @@ -109,7 +117,7 @@ def to_yaml(cls, config: Any, compact: bool = False, indent: int = 2, level: int if not items: return "[]" return "\n" + "\n".join(f"{pad}- {cls.to_yaml(item, compact, indent, level + 1)}" for item in items) - if isinstance(config, datetime.timedelta): + if is_duration(config): return _ms(config) if isinstance(config, str): lines = config.split("\n") @@ -130,7 +138,7 @@ def to_properties(cls, config: Any, compact: bool = False, indent: int = 2, key_ lines = [cls.to_properties(item, compact, indent, keys + [str(i)]) for i, item in enumerate(config) if item is not None] elif config is None: lines = [] - elif isinstance(config, datetime.timedelta): + elif is_duration(config): lines = [".".join(keys) + " = " + _ms(config)] elif isinstance(config, str): escaped = config.replace("=", "\\=").replace("!", "\\!").replace("#", "\\#").replace("\n", "\\\n") From f83822d4209d70a8c0a018f62a895460a5fbd1ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Wed, 21 Oct 2026 07:00:27 +0200 Subject: [PATCH 03/30] build(python): ruff and ty in zig build test-python ruff check, ruff format --check and ty check on the package, pinned, with their settings in python/pyproject.toml. The tests and examples stay out: they are the users' side, as they are written. mypy --strict stays. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- build.zig | 12 +++++++++++- python/pyproject.toml | 17 +++++++++++++++++ python/zig_hocon/config_parser.py | 8 ++++++-- python/zig_hocon/pyhocon/config_parser.py | 16 ++++------------ python/zig_hocon/pyhocon/config_tree.py | 7 ++----- python/zig_hocon/pyhocon/converter.py | 6 +++--- 6 files changed, 43 insertions(+), 23 deletions(-) diff --git a/build.zig b/build.zig index 4eab3ff6..8d8035aa 100644 --- a/build.zig +++ b/build.zig @@ -101,9 +101,19 @@ pub fn build(b: *std.Build) void { mypy.setEnvironmentVariable("PYTHONPATH", "python"); mypy.has_side_effects = true; mypy.step.dependOn(&py_lib.step); - const test_python_step = b.step("test-python", "Run the Python package's tests with coverage, and type check (needs uv)"); + // ruff and ty on the package, as python/pyproject.toml sets them; pinned, + // as a new ruff formats differently. + const ruff_check = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--with", "ruff==0.17.1", "ruff", "check", "python" }); + const ruff_format = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--with", "ruff==0.17.1", "ruff", "format", "--check", "python" }); + const ty = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--with", "ty==0.0.90", "ty", "check", "--project", "python" }); + const test_python_step = b.step("test-python", "Run the Python package's tests with coverage, lint and type check (needs uv)"); test_python_step.dependOn(&coverage.step); test_python_step.dependOn(&mypy.step); + for ([_]*std.Build.Step.Run{ ruff_check, ruff_format, ty }) |run| { + run.setCwd(b.path(".")); + run.has_side_effects = true; + test_python_step.dependOn(&run.step); + } // `zig build wheels`: a wheel for every platform into zig-out/wheels/, // each with libhocon cross-compiled for it, packed by tools/wheel.py. diff --git a/python/pyproject.toml b/python/pyproject.toml index 9fc5108c..bcbef562 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -49,3 +49,20 @@ zig_hocon = ["libhocon.so", "libhocon.dylib", "hocon.dll", "py.typed"] [tool.uv] # uv rebuilds when these change; the library is no file it would look at. cache-keys = [{ file = "pyproject.toml" }, { file = "setup.py" }, { file = "zig_hocon/*" }] + +[tool.ruff] +line-length = 160 +target-version = "py310" +# The tests and examples are the users' side and stay as they are written; +# the docs line their comments up by hand. +extend-exclude = ["tests", "examples", "*.md"] + +[tool.ruff.lint] +# `int | float` says an int is welcome, where `float` alone reads as if not. +ignore = ["PYI041"] + +[tool.ty.environment] +python-version = "3.10" + +[tool.ty.src] +include = ["zig_hocon"] diff --git a/python/zig_hocon/config_parser.py b/python/zig_hocon/config_parser.py index 494003d6..30b76853 100644 --- a/python/zig_hocon/config_parser.py +++ b/python/zig_hocon/config_parser.py @@ -83,7 +83,9 @@ def _options(env: Env, resolve: bool, include_dir: Path | None = None, http_time return opts, entries -def _parse(parse: Callable[..., int | None], args: Sequence[Any], env: Env, resolve: bool, include_dir: Path | None, http_timeout: Timeout, given: str | None) -> ConfigTree: +def _parse( + parse: Callable[..., int | None], args: Sequence[Any], env: Env, resolve: bool, include_dir: Path | None, http_timeout: Timeout, given: str | None +) -> ConfigTree: opts, _entries = _options(env, resolve, include_dir, http_timeout) err = _capi.Error() handle = parse(*args, ctypes.byref(opts), ctypes.byref(err)) @@ -120,7 +122,9 @@ def parse_file(path: Path, *, env: Env = os.environ, resolve: bool = True, http_ return _parse(lib.hocon_parse_file, [_capi.c_string(os.fsencode(path))], env, resolve, None, http_timeout, os.fsdecode(path)) @staticmethod - def parse_string(content: str | bytes, *, basedir: Path | None = None, env: Env = os.environ, resolve: bool = True, http_timeout: Timeout = None) -> ConfigTree: + def parse_string( + content: str | bytes, *, basedir: Path | None = None, env: Env = os.environ, resolve: bool = True, http_timeout: Timeout = None + ) -> ConfigTree: """HOCON text; an include in it is found in ``basedir``, and without one is an error.""" data = content.encode("utf-8", "surrogatepass") if isinstance(content, str) else bytes(content) diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index d2d8cb41..9af99b8c 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -133,17 +133,13 @@ class ConfigFactory: @overload @classmethod - def parse_file( - cls, filename: Path, encoding: str = ..., required: Literal[True] = ..., resolve: bool = ..., unresolved_value: Any = ... - ) -> ConfigTree: ... + def parse_file(cls, filename: Path, encoding: str = ..., required: Literal[True] = ..., resolve: bool = ..., unresolved_value: Any = ...) -> ConfigTree: ... @overload @classmethod def parse_file(cls, filename: Path, encoding: str, required: bool, resolve: bool = ..., unresolved_value: Any = ...) -> ConfigTree | list[Any]: ... @overload @classmethod - def parse_file( - cls, filename: Path, encoding: str = ..., *, required: bool, resolve: bool = ..., unresolved_value: Any = ... - ) -> ConfigTree | list[Any]: ... + def parse_file(cls, filename: Path, encoding: str = ..., *, required: bool, resolve: bool = ..., unresolved_value: Any = ...) -> ConfigTree | list[Any]: ... @classmethod def parse_file( cls, filename: Path, encoding: str = "utf-8", required: bool = True, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION @@ -174,9 +170,7 @@ def parse_URL( def parse_URL(cls, url: str, timeout: float | None, resolve: bool, required: Literal[True], unresolved_value: Any = ...) -> ConfigTree: ... @overload @classmethod - def parse_URL( - cls, url: str, timeout: float | None = ..., resolve: bool = ..., *, required: Literal[True], unresolved_value: Any = ... - ) -> ConfigTree: ... + def parse_URL(cls, url: str, timeout: float | None = ..., resolve: bool = ..., *, required: Literal[True], unresolved_value: Any = ...) -> ConfigTree: ... @classmethod def parse_URL( cls, url: str, timeout: float | None = None, resolve: bool = True, required: bool = False, unresolved_value: Any = DEFAULT_SUBSTITUTION @@ -196,9 +190,7 @@ def parse_URL( return _read(lambda r: _core.ConfigFactory.parse_string(content, resolve=r), resolve, unresolved_value) @classmethod - def parse_string( - cls, content: str, basedir: Path | None = None, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION - ) -> ConfigTree: + def parse_string(cls, content: str, basedir: Path | None = None, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION) -> ConfigTree: """HOCON text; an include in it is found in ``basedir``, by default the working directory, as pyhocon finds it.""" where = os.getcwd() if basedir is None else basedir diff --git a/python/zig_hocon/pyhocon/config_tree.py b/python/zig_hocon/pyhocon/config_tree.py index 49ff1334..6165af91 100644 --- a/python/zig_hocon/pyhocon/config_tree.py +++ b/python/zig_hocon/pyhocon/config_tree.py @@ -264,13 +264,11 @@ def put(self, key: str, value: Any, append: bool = False) -> None: elif append and isinstance(before, list): node[last] = before + value elif append and before is not None: - raise ConfigWrongTypeException( - f"Cannot concatenate the list {key}: {value} to {before} of {type(before).__name__}", name="WrongType" - ) + raise ConfigWrongTypeException(f"Cannot concatenate the list {key}: {value} to {before} of {type(before).__name__}", name="WrongType") else: node[last] = value - def get(self, key: str, default: Any = UndefinedKey) -> Any: + def get(self, key: str, default: Any = UndefinedKey) -> Any: # ty: ignore[invalid-method-override] - a path, as pyhocon has it """The value at ``key``; ``default`` when there is none, else ConfigMissingException. A list is a copy.""" return self._get(self.parse_key(key), default) @@ -380,4 +378,3 @@ def plain(value: Any) -> Any: return value return OrderedDict((key.strip('"'), plain(value)) for key, value in self.items()) - diff --git a/python/zig_hocon/pyhocon/converter.py b/python/zig_hocon/pyhocon/converter.py index a021a68a..d50fe49f 100644 --- a/python/zig_hocon/pyhocon/converter.py +++ b/python/zig_hocon/pyhocon/converter.py @@ -5,7 +5,7 @@ import json import re import sys -from typing import Any +from typing import Any, ClassVar from .config_parser import ConfigFactory, Path from .config_tree import ConfigTree, is_duration @@ -49,7 +49,7 @@ def _scalar(value: Any) -> str: class HOCONConverter: """Writes a ConfigTree out, as pyhocon's converter.""" - _ESCAPES = {"\b": r"\b", "\t": r"\t", "\n": r"\n", "\f": r"\f", "\r": r"\r", '"': r"\"", "\\": r"\\"} + _ESCAPES: ClassVar[dict[str, str]] = {"\b": r"\b", "\t": r"\t", "\n": r"\n", "\f": r"\f", "\r": r"\r", '"': r"\"", "\\": r"\\"} @classmethod def to_json(cls, config: Any, compact: bool = False, indent: int = 2, level: int = 0) -> str: @@ -159,7 +159,7 @@ def convert(cls, config: Any, output_format: str = "json", indent: int = 2, comp return cls.to_yaml(config, compact, indent) if output_format == "hocon": return cls.to_hocon(config, compact, indent) - raise Exception(f"Invalid format '{output_format}'. Format must be 'json', 'properties', 'yaml' or 'hocon'") + raise Exception(f"Invalid format '{output_format}'. Format must be 'json', 'properties', 'yaml' or 'hocon'") # noqa: TRY002 - pyhocon raises a bare Exception @classmethod def convert_from_file( From 066fbe99c306c7c8d18e2e90eb166f082a3722c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Wed, 21 Oct 2026 07:05:20 +0200 Subject: [PATCH 04/30] test(python): zig_hocon.pyhocon held against pyhocon 0.3.63 One parametrized test reads each document with both: the values with their types, and HOCONConverter's four outputs to the character. A difference by design is a strict xfail with its reason, so one that goes away shows. pyhocon joins the test environment for it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- build.zig | 5 +- python/tests/test_pyhocon_compat.py | 101 ++++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 2 deletions(-) create mode 100644 python/tests/test_pyhocon_compat.py diff --git a/build.zig b/build.zig index 8d8035aa..6e8b27ac 100644 --- a/build.zig +++ b/build.zig @@ -80,10 +80,11 @@ pub fn build(b: *std.Build) void { // cover` for what no test can reach). `python -m zig_hocon` is left out: // test_main.py runs it as a process, as the conformance adapter does, // where coverage does not follow. dateutil is there for - // zig_hocon.pyhocon's months and years, as pyhocon has them with it. + // zig_hocon.pyhocon's months and years, as pyhocon has them with it, and + // pyhocon itself for test_pyhocon_compat.py to hold it against. const python_version = b.option([]const u8, "python", "The Python test-python runs on (default 3.12)") orelse "3.12"; const coverage_file = b.pathFromRoot(".zig-cache/python.coverage"); - const pytest = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--python", python_version, "--with", "pytest", "--with", "coverage", "--with", "python-dateutil", "python", "-m", "coverage", "run", "--branch", "--source=python/zig_hocon", "--omit=python/zig_hocon/__main__.py", "-m", "pytest", "-q", "python/tests" }); + const pytest = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--python", python_version, "--with", "pytest", "--with", "coverage", "--with", "python-dateutil", "--with", "pyhocon==0.3.63", "python", "-m", "coverage", "run", "--branch", "--source=python/zig_hocon", "--omit=python/zig_hocon/__main__.py", "-m", "pytest", "-q", "python/tests" }); pytest.setCwd(b.path(".")); pytest.setEnvironmentVariable("PYTHONPATH", "python"); pytest.setEnvironmentVariable("COVERAGE_FILE", coverage_file); diff --git a/python/tests/test_pyhocon_compat.py b/python/tests/test_pyhocon_compat.py new file mode 100644 index 00000000..f9b4e22b --- /dev/null +++ b/python/tests/test_pyhocon_compat.py @@ -0,0 +1,101 @@ +"""zig_hocon.pyhocon against pyhocon 0.3.63 itself: each document read by +both, the values and their types the same, and HOCONConverter's output to the +character. Where they differ by design the case is an xfail that says why, +strict, so a difference that goes away shows.""" + +import datetime + +import pytest + +pyhocon = pytest.importorskip("pyhocon") +relativedelta = pytest.importorskip("dateutil.relativedelta").relativedelta + +import zig_hocon.pyhocon as compat # noqa: E402 + + +def differs(text, why): + return pytest.param(text, marks=pytest.mark.xfail(strict=True, reason=why)) + + +CASES = { + # scalars + "numbers": "a = 1\nb = -2\nc = 2.5\nd = 1e3\ne = 0.75", + "bools and null": "a = true\nb = false\nc = null\nd = yes", + "unquoted text": "owl = the owl hoots at midnight", + "quoted text": 'owl = "hoot, hoot"\nempty = ""', + "escapes": r'a = "tab\there"' + "\n" + r'b = "quote \" and \\ slash"', + "multiline text": 'poem = """the owl\nflew home"""', + "unicode": "sova = \"hú hú\"\n\"klíč\" = ✓", + # objects and paths + "nested objects": "burrow { depth = 3, owls { count = 2 } }", + "dotted paths": "burrow.depth = 3\nburrow.owls.count = 2", + "quoted keys": '"a.b" = 1\nc { "d.e" = 2 }', + "objects merge": "a { x = 1 }\na { y = 2 }", + "later wins": "a = 1\na = 2", + "object replaced": "a { x = 1 }\na = 2", + "colon and no separator": "a : 1\nb { c : 2 }", + "json": '{"owls": [1, 2], "night": {"dark": true}}', + "comments": "# a comment\na = 1 // another\nb = 2", + # lists + "lists": "owls = [hoot, screech, 3, 4.5, true, null]", + "nested lists": "a = [[1, 2], [3], []]", + "objects in lists": "a = [{ x = 1 }, { y = 2 }]", + "lists on lines": "a = [\n 1\n 2\n]", + "list concatenation": "a = [1, 2] [3]", + # substitutions + "substitution": "a = 1\nb = ${a}", + "substitution of an object": "a { x = 1 }\nb = ${a}", + "substitution in text": 'name = owl\ngreeting = "hello "${name}" friend"', + "optional substitution": "a = ${?nope}\nb = 1", + "self reference append": "a = [1]\na = ${a} [2]", + "object merge by substitution": "base { x = 1 }\nfull = ${base} { y = 2 }", + "forward reference": "b = ${a}\na = 7", + "text concatenation": "a = owl\nb = ${a} hoots", + "number then text": "a = 1 owl\nb = 2.5 owls fly", + # durations + "durations": "a = 30s\nb = 10 weeks\nc = 3m\nd = 114 ms\ne = 1140000 ns", + "months and years": "a = 3mo\nb = 2 year", + "durations in lists": "a = [4 h, 5 ms]", + "not durations": "a = 1.5 s\nb = -3 s\nc = 3 monthsx\nd = 5x", + # errors in both, of their own classes + "unclosed list": "a = [1, 2", + "unclosed object": "a { b = 1", + "missing substitution": "a = ${nope}", + "substitution cycle": "a = ${b}\nb = ${a}", + # what the spec reads otherwise + "quoted duration": differs('a = "30s"', 'a quoted "30s" is a duration here too: libhocon\'s JSON has no quotes left'), + "+= appends one element": differs("a = [1]\na += [2, 3]", "by the spec += appends the list as one element"), + "-Xmx unquoted": differs("opts = [-Xmx1g]", "by the spec an unquoted string does not start with -"), + "optional substitution then text": differs("a = ${?nope} 4", 'by the spec it is the text " 4", pyhocon the number 4'), + "duration then text": differs("a = 5 ms foo", "pyhocon makes '0:00:00.005000foo' of it"), + "bool then text": differs("a = true love", "pyhocon makes 'Truelove' of it; by the spec it is 'true love'"), +} + + +def _parse(module, text): + try: + return module.ConfigFactory.parse_string(text) + except Exception: + return "error" # pyhocon raises pyparsing's classes, MIGRATING.md has it + + +def _typed(value): + """``value`` with the type of each part, as == alone lets 1 == 1.0 == True.""" + if isinstance(value, dict): + return {k: _typed(v) for k, v in value.items()} + if isinstance(value, list): + return [_typed(v) for v in value] + for t in (bool, int, float, str, datetime.timedelta, relativedelta, type(None)): + if isinstance(value, t): + return (t.__name__, value) + return (type(value).__name__, value) + + +@pytest.mark.parametrize("text", CASES.values(), ids=CASES.keys()) +def test_same_as_pyhocon(text): + want, got = _parse(pyhocon, text), _parse(compat, text) + assert _typed(got) == _typed(want) + if isinstance(want, str): + return + for fmt in ["json", "hocon", "yaml", "properties"]: + assert compat.HOCONConverter.convert(got, fmt) == pyhocon.HOCONConverter.convert(want, fmt), fmt From 368b225f3fdd869f807fa2988e7d14f527b12639 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Wed, 21 Oct 2026 07:16:26 +0200 Subject: [PATCH 05/30] test(python): zig_hocon.pyhocon's interface held against pyhocon's test_pyhocon_signatures.py reads every public name of the installed pyhocon's modules and checks each is here, in the same module, a function with pyhocon's parameters and at most more with a default after them, a class with pyhocon's bases. A difference by design is a strict xfail with its reason. The test environment takes the newest pyhocon, so its next release shows. What it found is added: period_parser and period_serializer, the duration code moved there, and the names pyhocon's config_parser and converter have from where they import them. pyhocon's own suite, test_periods.py now included: 263 of 308. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- CHANGELOG.md | 2 +- build.zig | 5 +- python/MIGRATING.md | 6 +- python/tests/test_pyhocon_api.py | 16 ++- python/tests/test_pyhocon_compat.py | 2 +- python/tests/test_pyhocon_signatures.py | 114 ++++++++++++++++++ python/zig_hocon/pyhocon/config_parser.py | 13 +- python/zig_hocon/pyhocon/config_tree.py | 58 +-------- python/zig_hocon/pyhocon/converter.py | 52 +++----- python/zig_hocon/pyhocon/period_parser.py | 74 ++++++++++++ python/zig_hocon/pyhocon/period_serializer.py | 45 +++++++ 11 files changed, 287 insertions(+), 100 deletions(-) create mode 100644 python/tests/test_pyhocon_signatures.py create mode 100644 python/zig_hocon/pyhocon/period_parser.py create mode 100644 python/zig_hocon/pyhocon/period_serializer.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e442ad53..ab2cf050 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ### Added -- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 218 of 263 with the import swapped, and `python/MIGRATING.md` says what the rest are. +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 263 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. ### Fixed diff --git a/build.zig b/build.zig index 6e8b27ac..d3c60b01 100644 --- a/build.zig +++ b/build.zig @@ -81,10 +81,11 @@ pub fn build(b: *std.Build) void { // test_main.py runs it as a process, as the conformance adapter does, // where coverage does not follow. dateutil is there for // zig_hocon.pyhocon's months and years, as pyhocon has them with it, and - // pyhocon itself for test_pyhocon_compat.py to hold it against. + // the newest pyhocon for test_pyhocon_compat.py and + // test_pyhocon_signatures.py to hold it against: its next release shows. const python_version = b.option([]const u8, "python", "The Python test-python runs on (default 3.12)") orelse "3.12"; const coverage_file = b.pathFromRoot(".zig-cache/python.coverage"); - const pytest = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--python", python_version, "--with", "pytest", "--with", "coverage", "--with", "python-dateutil", "--with", "pyhocon==0.3.63", "python", "-m", "coverage", "run", "--branch", "--source=python/zig_hocon", "--omit=python/zig_hocon/__main__.py", "-m", "pytest", "-q", "python/tests" }); + const pytest = b.addSystemCommand(&.{ "uv", "run", "--no-project", "--python", python_version, "--with", "pytest", "--with", "coverage", "--with", "python-dateutil", "--with", "pyhocon", "python", "-m", "coverage", "run", "--branch", "--source=python/zig_hocon", "--omit=python/zig_hocon/__main__.py", "-m", "pytest", "-q", "python/tests" }); pytest.setCwd(b.path(".")); pytest.setEnvironmentVariable("PYTHONPATH", "python"); pytest.setEnvironmentVariable("COVERAGE_FILE", coverage_file); diff --git a/python/MIGRATING.md b/python/MIGRATING.md index 6231f3b1..e4672376 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -45,11 +45,13 @@ from zig_hocon.pyhocon import ConfigFactory, ConfigTree, HOCONConverter from zig_hocon.pyhocon.exceptions import ConfigMissingException ``` -Everything of pyhocon's public API is there: `ConfigFactory.parse_file` (with `encoding` and `required`), `parse_string`, `parse_URL`, `from_dict`; the `ConfigTree` with `get`, the typed getters, `get_list`, `get_config`, `put`, `pop`, `with_fallback`, `merge_configs`, `as_plain_ordered_dict`, `conf["db.host"]` and `conf.db.host`; `ConfigParser.parse`, `resolve_substitutions` and `resolve_package_path`; `HOCONConverter` with its JSON, HOCON, YAML and .properties output to the character; and pyhocon's command line as `python -m zig_hocon.pyhocon.tool`. The exceptions are zig_hocon's, so one `except` catches either module's. +Everything of pyhocon's public API is there: `ConfigFactory.parse_file` (with `encoding` and `required`), `parse_string`, `parse_URL`, `from_dict`; the `ConfigTree` with `get`, the typed getters, `get_list`, `get_config`, `put`, `pop`, `with_fallback`, `merge_configs`, `as_plain_ordered_dict`, `conf["db.host"]` and `conf.db.host`; `ConfigParser.parse`, `resolve_substitutions` and `resolve_package_path`; `HOCONConverter` with its JSON, HOCON, YAML and .properties output to the character; `period_parser` and `period_serializer`; and pyhocon's command line as `python -m zig_hocon.pyhocon.tool`. The exceptions are zig_hocon's, so one `except` catches either module's. + +Two tests hold it against the newest pyhocon on every change: [test_pyhocon_signatures.py](tests/test_pyhocon_signatures.py) has every public name of pyhocon's modules, with the same parameters, and [test_pyhocon_compat.py](tests/test_pyhocon_compat.py) reads documents with both and compares the values, their types and the converter's output. Each difference by design is in them with its reason. Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. One difference: a quoted `"30s"` is a duration as well, where pyhocon keeps it text. -The measure is pyhocon's own test suite with the import swapped: 218 of its 263 tests pass. Of the 45 that do not, 44 are a config that reads differently by the spec (Step 3) or the class of a syntax error: +The measure is pyhocon's own test suite with the import swapped: 263 of its 308 tests pass. Of the 45 that do not, 44 are a config that reads differently by the spec (Step 3) or the class of a syntax error: | tests | what pyhocon does | |---|---| diff --git a/python/tests/test_pyhocon_api.py b/python/tests/test_pyhocon_api.py index 8f5ae1be..e3bfc0a0 100644 --- a/python/tests/test_pyhocon_api.py +++ b/python/tests/test_pyhocon_api.py @@ -26,7 +26,7 @@ HOCONConverter, UndefinedKey, ) -from zig_hocon.pyhocon import config_parser, config_tree, converter, exceptions, tool +from zig_hocon.pyhocon import config_parser, config_tree, converter, exceptions, period_parser, period_serializer, tool from zig_hocon.pyhocon.config_parser import ( MANDATORY_SUBSTITUTION, NO_SUBSTITUTION, @@ -303,7 +303,7 @@ def test_durations(text, value): # pyhocon ✓: without dateutil a timedelta, and months and years are text def test_durations_without_dateutil(monkeypatch): - monkeypatch.setattr(config_tree, "relativedelta", None) + monkeypatch.setattr(period_parser, "relativedelta", None) c = conf("a = 30s\nb = 3mo\nc = [1 d]") assert c["a"] == datetime.timedelta(seconds=30) and type(c["a"]) is datetime.timedelta assert c["b"] == "3mo" and c["c"] == [datetime.timedelta(days=1)] @@ -323,6 +323,18 @@ def test_durations_through_with_fallback(): c.with_fallback(ConfigTree()) +# pyhocon ✓: period_parser and period_serializer, as pyhocon has them; +# pyhocon ⚠️: parse_period of no duration is a ValueError, not pyparsing's +def test_period_modules(): + assert period_parser.period(3, "minutes") == relativedelta(minutes=3) + assert period_parser.period(5, "milliseconds") == datetime.timedelta(milliseconds=5) + assert period_parser.parse_period("10 weeks") == relativedelta(days=70) + with pytest.raises(ValueError, match="'owl' is not a duration"): + period_parser.parse_period("owl") + assert period_serializer.relative_delta_to_timedelta(relativedelta(hours=1, months=2)) == datetime.timedelta(hours=1) + assert period_serializer.is_timedelta_like(relativedelta(days=1)) and not period_serializer.is_timedelta_like(1) + + # pyhocon ✓: checked against pyhocon 0.3.63's output def test_relativedeltas_converted(): c = conf("a = 7 h\nb = 3 m\nc = 12 d\nd = 3mo\ne = 112 micros") diff --git a/python/tests/test_pyhocon_compat.py b/python/tests/test_pyhocon_compat.py index f9b4e22b..8361e327 100644 --- a/python/tests/test_pyhocon_compat.py +++ b/python/tests/test_pyhocon_compat.py @@ -1,4 +1,4 @@ -"""zig_hocon.pyhocon against pyhocon 0.3.63 itself: each document read by +"""zig_hocon.pyhocon against pyhocon itself, the newest release: each document read by both, the values and their types the same, and HOCONConverter's output to the character. Where they differ by design the case is an xfail that says why, strict, so a difference that goes away shows.""" diff --git a/python/tests/test_pyhocon_signatures.py b/python/tests/test_pyhocon_signatures.py new file mode 100644 index 00000000..b2e157b7 --- /dev/null +++ b/python/tests/test_pyhocon_signatures.py @@ -0,0 +1,114 @@ +"""zig_hocon.pyhocon has pyhocon's interface: every public name of the +installed pyhocon, in the same module, a function or method with pyhocon's +parameters (names, kinds, defaults) and at most more of its own after them +that have a default, a class with pyhocon's bases among its own. The modules +and names are read from pyhocon when the tests are collected, so a pyhocon +release with a new module, name or argument is a new case that fails, and so +is a change here that drops one. Where they differ by design the case is an +xfail that says why, strict, so a difference that goes away shows.""" + +import importlib +import inspect +import pkgutil + +import pytest + +pyhocon = pytest.importorskip("pyhocon") + +MODULES = [""] + ["." + m.name for m in pkgutil.iter_modules(pyhocon.__path__)] + +# pyhocon's parser objects, made while it parses; here a placeholder of the +# name, for code that imports one or checks a value against it +_PARSER_OBJECT = "a parser object pyhocon makes while it parses; a placeholder here" + +_GRAMMAR = "pyhocon's pyparsing grammar; libhocon reads the document" + +# The differences by design: a name, `Class.member`, or a class for all its +# members. +DIFFERS = { + "ConfigValues.*": _PARSER_OBJECT, + "ConfigSubstitution.*": _PARSER_OBJECT, + "ConfigQuotedString.*": _PARSER_OBJECT, + "ConfigInclude.*": _PARSER_OBJECT, + "ConfigList.__init__": "a plain list subclass, constructed as list is", + "ConfigTree.items": "OrderedDict's own, which pyhocon overrides for Python 2", + "ConfigTree.resolve": "not there yet, MIGRATING.md", + "ConfigParser.unresolve_substitutions_to_value": "pyhocon's STR_SUBSTITUTION, which is not supported, MIGRATING.md", + "ConfigException.__init__": "zig_hocon's exceptions: (message, name, file, line, column, source_line), not (message, ex)", + "get_period_expr": _GRAMMAR, + "convert_period": _GRAMMAR, + "fixed_get_attr": _GRAMMAR, + "ListParser": _GRAMMAR, + "ListParser.*": _GRAMMAR, + "ConcatenatedValueParser": _GRAMMAR, + "ConcatenatedValueParser.*": _GRAMMAR, + "ConfigTreeParser": _GRAMMAR, + "ConfigTreeParser.*": _GRAMMAR, + "find_package_dirs": "pyhocon's include package(…), which the spec does not have", +} + + +def _ours(module): + return importlib.import_module("zig_hocon.pyhocon" + module) + + +def _theirs(module): + return importlib.import_module("pyhocon" + module) + + +def _public(module): + """pyhocon's public names in ``module``: (case id, name, its object).""" + short = module.lstrip(".") or "pyhocon" + for name, obj in vars(_theirs(module)).items(): + if not name.startswith("_") and getattr(obj, "__module__", "").split(".")[0] == "pyhocon": + yield f"{short}.{name}", name, obj + + +def _cases(): + seen = set() + for module in MODULES: + for case, name, obj in _public(module): + yield case, module, name, None, DIFFERS.get(name) + if inspect.isclass(obj) and obj not in seen: + seen.add(obj) + for member, value in vars(obj).items(): + if callable(value) or isinstance(value, (classmethod, staticmethod)): + if not member.startswith("_") or member == "__init__": + why = DIFFERS.get(f"{name}.{member}") or DIFFERS.get(f"{name}.*") + yield f"{name}.{member}", module, name, member, why + + +def _param(case, module, name, member, why): + marks = [pytest.mark.xfail(strict=True, reason=why)] if why else [] + return pytest.param(module, name, member, id=case, marks=marks) + + +def _signature(f): + """The parameters of ``f``: name, kind and default, a class default by + its name (pyhocon's UndefinedKey is ours by name, not by identity).""" + params = [] + for p in inspect.signature(f).parameters.values(): + default = p.default + if default is not p.empty: + default = default.__name__ if inspect.isclass(default) else repr(default) + params.append((p.name, p.kind.name, default)) + return params + + +def _mro(cls): + return [c.__name__ for c in cls.__mro__ if c is not object] + + +@pytest.mark.parametrize("module, name, member", [_param(*c) for c in _cases()]) +def test_same_interface_as_pyhocon(module, name, member): + theirs, ours = getattr(_theirs(module), name), getattr(_ours(module), name, None) + assert ours is not None, f"no {name} in zig_hocon.pyhocon{module}" + if member is not None: + theirs, ours = getattr(theirs, member), getattr(ours, member, None) + assert ours is not None, f"no {name}.{member}" + if inspect.isclass(theirs): + assert set(_mro(theirs)) <= set(_mro(ours)) + elif callable(theirs): + mine, pyhocons = _signature(ours), _signature(theirs) + assert mine[: len(pyhocons)] == pyhocons + assert all(default is not None for _, _, default in mine[len(pyhocons) :]), "a parameter pyhocon has not, with no default" diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index 9af99b8c..18757d03 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -16,17 +16,26 @@ from .. import config_parser as _core from .. import config_tree as _core_tree -from ..exceptions import ConfigException, ConfigParseException, ConfigSubstitutionException, ConfigWrongTypeException -from .config_tree import ConfigList, ConfigTree, duration, quoted +from ..exceptions import ConfigException, ConfigMissingException, ConfigParseException, ConfigSubstitutionException, ConfigWrongTypeException +from .config_tree import ConfigInclude, ConfigList, ConfigQuotedString, ConfigSubstitution, ConfigTree, ConfigUnquotedString, ConfigValues, NoneValue, quoted +from .period_parser import duration +# with the names pyhocon's module has from where it imports them __all__ = [ "DEFAULT_SUBSTITUTION", "MANDATORY_SUBSTITUTION", "NO_SUBSTITUTION", "STR_SUBSTITUTION", "ConfigFactory", + "ConfigInclude", + "ConfigMissingException", "ConfigParser", + "ConfigQuotedString", + "ConfigSubstitution", "ConfigSubstitutionException", + "ConfigUnquotedString", + "ConfigValues", + "NoneValue", ] logger = logging.getLogger(__name__) diff --git a/python/zig_hocon/pyhocon/config_tree.py b/python/zig_hocon/pyhocon/config_tree.py index 6165af91..cccaeb53 100644 --- a/python/zig_hocon/pyhocon/config_tree.py +++ b/python/zig_hocon/pyhocon/config_tree.py @@ -2,7 +2,6 @@ values, read and changed by dotted paths, as pyhocon has it.""" import datetime -import importlib import json import re from collections import OrderedDict @@ -14,6 +13,7 @@ ConfigSubstitutionException, ConfigWrongTypeException, ) +from .period_serializer import is_timedelta_like if TYPE_CHECKING: from .config_parser import Path @@ -82,56 +82,6 @@ def quoted(key: str) -> str: return f'"{key}"' if _HAS_SPECIAL(key) else key -# pyhocon's durations: an unquoted `30s` or `10 weeks` is a timedelta, or a -# dateutil relativedelta when dateutil is installed, which months and years -# need. Milliseconds are always a timedelta. -try: - relativedelta: Any = importlib.import_module("dateutil.relativedelta").relativedelta -except ImportError: # pragma: no cover - the tests have dateutil and patch it away - relativedelta = None - -_UNITS = { - "nanoseconds": ["ns", "nano", "nanos", "nanosecond", "nanoseconds"], - "microseconds": ["us", "micro", "micros", "microsecond", "microseconds"], - "milliseconds": ["ms", "milli", "millis", "millisecond", "milliseconds"], - "seconds": ["s", "second", "seconds"], - "minutes": ["m", "minute", "minutes"], - "hours": ["h", "hour", "hours"], - "weeks": ["w", "week", "weeks"], - "days": ["d", "day", "days"], -} -_DATEUTIL_UNITS = {"months": ["mo", "month", "months"], "years": ["y", "year", "years"]} - - -def _duration_re(units: dict[str, list[str]]) -> "re.Pattern[str]": - # the longest spelling first: `ms` before `m`, `mo` before `m` - names = sorted((n for spellings in units.values() for n in spellings), key=len, reverse=True) - return re.compile(r"(\d+) *(" + "|".join(names) + ")") - - -_DURATION = _duration_re(_UNITS) -_DURATION_DATEUTIL = _duration_re(_UNITS | _DATEUTIL_UNITS) -_UNIT_OF = {n: unit for unit, spellings in (_UNITS | _DATEUTIL_UNITS).items() for n in spellings} - - -def duration(text: str) -> Any: - """``text`` as pyhocon reads a value: a duration when it is a whole - number and a unit, else ``text``. - - ponytail: libhocon's JSON has no quotes left, so a quoted "30s" is a - duration too, where pyhocon keeps it text; hocon.h telling quoted from - unquoted would make it exact.""" - m = (_DURATION if relativedelta is None else _DURATION_DATEUTIL).fullmatch(text) - if m is None: - return text - value, unit = int(m[1]), _UNIT_OF[m[2]] - if unit == "nanoseconds": - value, unit = value // 1000, "microseconds" - if unit == "milliseconds" or relativedelta is None: - return datetime.timedelta(**{unit: value}) - return relativedelta(**{unit: value}) - - def _duration_text(value: Any) -> str: """A duration as HOCON that reads back as the same one: milliseconds, or microseconds below them, for a timedelta; a relativedelta in months or @@ -146,13 +96,9 @@ def _duration_text(value: Any) -> str: return f"{months} months" if months else f"{us} us" -def is_duration(value: Any) -> bool: - return isinstance(value, datetime.timedelta) or relativedelta is not None and isinstance(value, relativedelta) - - def _json_value(value: Any) -> Any: """A duration in a tree, as HOCON text that reads back as it.""" - if is_duration(value): + if is_timedelta_like(value): return _duration_text(value) raise TypeError(f"{type(value).__name__} is not a HOCON value") diff --git a/python/zig_hocon/pyhocon/converter.py b/python/zig_hocon/pyhocon/converter.py index d50fe49f..b672affa 100644 --- a/python/zig_hocon/pyhocon/converter.py +++ b/python/zig_hocon/pyhocon/converter.py @@ -1,39 +1,23 @@ """pyhocon's ``HOCONConverter``: a ConfigTree as JSON, HOCON, YAML or a .properties file, in pyhocon's layout to the character.""" -import datetime import json import re import sys from typing import Any, ClassVar from .config_parser import ConfigFactory, Path -from .config_tree import ConfigTree, is_duration +from .config_tree import ConfigQuotedString, ConfigSubstitution, ConfigTree, ConfigValues, NoneValue +from .period_serializer import is_timedelta_like, timedelta_to_hocon, timedelta_to_str -__all__ = ["HOCONConverter"] - - -def _ms(value: Any) -> str: - """A duration, as pyhocon writes it: milliseconds, a relativedelta's - months and years left out.""" - if not isinstance(value, datetime.timedelta): - value = datetime.timedelta(days=value.days, hours=value.hours, minutes=value.minutes, seconds=value.seconds, microseconds=value.microseconds) - return str(int(value.total_seconds() * 1000)) - - -def _hocon_duration(value: Any) -> str: - if not isinstance(value, datetime.timedelta): # a relativedelta - if value.hours > 0: - return f"{value.hours} hours" - if value.minutes > 0: - return f"{value.minutes} minutes" - if value.days > 0: - return f"{value.days} days" - if value.seconds > 0: - return f"{value.seconds} seconds" - if value.microseconds > 0: - return f"{value.microseconds} microseconds" - return "0 seconds" +# with the names pyhocon's module has from where it imports them +__all__ = [ + "ConfigQuotedString", + "ConfigSubstitution", + "ConfigValues", + "HOCONConverter", + "NoneValue", +] def _scalar(value: Any) -> str: @@ -65,8 +49,8 @@ def to_json(cls, config: Any, compact: bool = False, indent: int = 2, level: int if not config: return "[]" return "[\n" + ",\n".join(f"{inner}{cls.to_json(item, compact, indent, level + 1)}" for item in config) + f"\n{pad}]" - if is_duration(config): - return _ms(config) + if is_timedelta_like(config): + return timedelta_to_str(config) if isinstance(config, str): return json.dumps(config, ensure_ascii=False) return _scalar(config) @@ -98,8 +82,8 @@ def to_hocon(cls, config: Any, compact: bool = False, indent: int = 2, level: in if "\n" in config and len(config) > 1: return f'"""{config}"""' return '"' + cls._escape_string(config) + '"' - if is_duration(config): - return _hocon_duration(config) + if is_timedelta_like(config): + return timedelta_to_hocon(config) return _scalar(config) @classmethod @@ -117,8 +101,8 @@ def to_yaml(cls, config: Any, compact: bool = False, indent: int = 2, level: int if not items: return "[]" return "\n" + "\n".join(f"{pad}- {cls.to_yaml(item, compact, indent, level + 1)}" for item in items) - if is_duration(config): - return _ms(config) + if is_timedelta_like(config): + return timedelta_to_str(config) if isinstance(config, str): lines = config.split("\n") if len(lines) == 1: @@ -138,8 +122,8 @@ def to_properties(cls, config: Any, compact: bool = False, indent: int = 2, key_ lines = [cls.to_properties(item, compact, indent, keys + [str(i)]) for i, item in enumerate(config) if item is not None] elif config is None: lines = [] - elif is_duration(config): - lines = [".".join(keys) + " = " + _ms(config)] + elif is_timedelta_like(config): + lines = [".".join(keys) + " = " + timedelta_to_str(config)] elif isinstance(config, str): escaped = config.replace("=", "\\=").replace("!", "\\!").replace("#", "\\#").replace("\n", "\\\n") lines = [".".join(keys) + " = " + escaped] diff --git a/python/zig_hocon/pyhocon/period_parser.py b/python/zig_hocon/pyhocon/period_parser.py new file mode 100644 index 00000000..94e836c1 --- /dev/null +++ b/python/zig_hocon/pyhocon/period_parser.py @@ -0,0 +1,74 @@ +"""pyhocon's ``period_parser``: a duration from its text, ``30s`` or +``10 weeks``, as pyhocon reads an unquoted value.""" + +import datetime +import importlib +import re +from typing import Any + +__all__ = ["parse_period", "period"] + +# pyhocon's durations: an unquoted `30s` or `10 weeks` is a timedelta, or a +# dateutil relativedelta when dateutil is installed, which months and years +# need. Milliseconds are always a timedelta. +try: + relativedelta: Any = importlib.import_module("dateutil.relativedelta").relativedelta +except ImportError: # pragma: no cover - the tests have dateutil and patch it away + relativedelta = None + +_UNITS = { + "nanoseconds": ["ns", "nano", "nanos", "nanosecond", "nanoseconds"], + "microseconds": ["us", "micro", "micros", "microsecond", "microseconds"], + "milliseconds": ["ms", "milli", "millis", "millisecond", "milliseconds"], + "seconds": ["s", "second", "seconds"], + "minutes": ["m", "minute", "minutes"], + "hours": ["h", "hour", "hours"], + "weeks": ["w", "week", "weeks"], + "days": ["d", "day", "days"], +} +_DATEUTIL_UNITS = {"months": ["mo", "month", "months"], "years": ["y", "year", "years"]} + + +def _duration_re(units: dict[str, list[str]]) -> "re.Pattern[str]": + # the longest spelling first: `ms` before `m`, `mo` before `m` + names = sorted((n for spellings in units.values() for n in spellings), key=len, reverse=True) + return re.compile(r"(\d+) *(" + "|".join(names) + ")") + + +_DURATION = _duration_re(_UNITS) +_DURATION_DATEUTIL = _duration_re(_UNITS | _DATEUTIL_UNITS) +_UNIT_OF = {n: unit for unit, spellings in (_UNITS | _DATEUTIL_UNITS).items() for n in spellings} + + +def duration(text: str) -> Any: + """``text`` as pyhocon reads a value: a duration when it is a whole + number and a unit, else ``text``. + + ponytail: libhocon's JSON has no quotes left, so a quoted "30s" is a + duration too, where pyhocon keeps it text; hocon.h telling quoted from + unquoted would make it exact.""" + m = (_DURATION if relativedelta is None else _DURATION_DATEUTIL).fullmatch(text) + if m is None: + return text + value, unit = int(m[1]), _UNIT_OF[m[2]] + if unit == "nanoseconds": + value, unit = value // 1000, "microseconds" + if unit == "milliseconds" or relativedelta is None: + return datetime.timedelta(**{unit: value}) + return relativedelta(**{unit: value}) + + +def period(period_value: int, period_unit: str) -> Any: + """``period_value`` of ``period_unit``, ``"seconds"`` or another + timedelta argument, ``"months"`` and ``"years"`` with dateutil, as + pyhocon makes it.""" + return duration(f"{period_value} {period_unit}") + + +def parse_period(content: str) -> Any: + """``content``, a whole number and a unit, as a duration; ValueError + for anything else.""" + value = duration(content) + if value is content: + raise ValueError(f"{content!r} is not a duration") + return value diff --git a/python/zig_hocon/pyhocon/period_serializer.py b/python/zig_hocon/pyhocon/period_serializer.py new file mode 100644 index 00000000..5cf5557a --- /dev/null +++ b/python/zig_hocon/pyhocon/period_serializer.py @@ -0,0 +1,45 @@ +"""pyhocon's ``period_serializer``: a duration written out, as pyhocon's +HOCONConverter writes it.""" + +import datetime +from typing import Any + +from . import period_parser + +__all__ = ["is_timedelta_like", "relative_delta_to_timedelta", "timedelta_to_hocon", "timedelta_to_str"] + + +def is_timedelta_like(config: Any) -> bool: + """A timedelta, or a dateutil relativedelta.""" + relativedelta = period_parser.relativedelta + return isinstance(config, datetime.timedelta) or relativedelta is not None and isinstance(config, relativedelta) + + +def relative_delta_to_timedelta(relative_delta: Any) -> datetime.timedelta: + """A relativedelta as a timedelta, its months and years left out.""" + r = relative_delta + return datetime.timedelta(days=r.days, hours=r.hours, minutes=r.minutes, seconds=r.seconds, microseconds=r.microseconds) + + +def timedelta_to_hocon(config: Any) -> str: + """A duration in its largest unit, as pyhocon's to_hocon writes it.""" + if not isinstance(config, datetime.timedelta): # a relativedelta + if config.hours > 0: + return f"{config.hours} hours" + if config.minutes > 0: + return f"{config.minutes} minutes" + if config.days > 0: + return f"{config.days} days" + if config.seconds > 0: + return f"{config.seconds} seconds" + if config.microseconds > 0: + return f"{config.microseconds} microseconds" + return "0 seconds" + + +def timedelta_to_str(config: Any) -> str: + """A duration in milliseconds, as pyhocon's JSON, YAML and .properties + write it.""" + if not isinstance(config, datetime.timedelta): + config = relative_delta_to_timedelta(config) + return str(int(config.total_seconds() * 1000)) From 5969ac5d76fd38fb162df617a9558a74b45eae08 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Wed, 21 Oct 2026 07:36:45 +0200 Subject: [PATCH 06/30] fix(python): pyhocon's Apache-2.0 for zig_hocon.pyhocon, and whole wheels zig_hocon.pyhocon is changed in parts from pyhocon's code, which is Apache-2.0: pyhocon's LICENSE goes next to it, each changed file says what it comes from, and the package's license is BSD-3-Clause AND Apache-2.0 with both texts in license-files. tools/wheel.py packed only the .py files at the top of zig_hocon/: the release wheels would have had no zig_hocon.pyhocon, and have never had py.typed. It takes the whole package now, and both licenses into .dist-info/licenses/. tools/test-wheel.sh installs python-dateutil and pyhocon for the tests that hold the module against it. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- CHANGELOG.md | 6 +- README.md | 2 +- python/README.md | 4 +- python/pyproject.toml | 4 +- python/tests/test_pyhocon_signatures.py | 19 +- python/zig_hocon/pyhocon/LICENSE | 202 ++++++++++++++++++ python/zig_hocon/pyhocon/__init__.py | 10 +- python/zig_hocon/pyhocon/config_parser.py | 5 +- python/zig_hocon/pyhocon/config_tree.py | 5 +- python/zig_hocon/pyhocon/converter.py | 5 +- python/zig_hocon/pyhocon/period_parser.py | 5 +- python/zig_hocon/pyhocon/period_serializer.py | 5 +- python/zig_hocon/pyhocon/tool.py | 5 +- tools/test-wheel.sh | 7 +- tools/wheel.py | 22 +- 15 files changed, 278 insertions(+), 28 deletions(-) create mode 100644 python/zig_hocon/pyhocon/LICENSE diff --git a/CHANGELOG.md b/CHANGELOG.md index ab2cf050..59bebd29 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,11 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ### Added -- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 263 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 263 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. + +### Fixed + +- Python: the wheels have `py.typed`, which `pip install python/` had and the release wheels did not, so a type checker reads the package's types. ### Fixed diff --git a/README.md b/README.md index 8948023d..dd7aac0a 100644 --- a/README.md +++ b/README.md @@ -253,4 +253,4 @@ If you need a HOCON feature that is not there, open a pull request; a failing te ## License -BSD 3-Clause, see [LICENSE](LICENSE). +BSD 3-Clause, see [LICENSE](LICENSE). The Python module `zig_hocon.pyhocon` is changed in parts from [pyhocon](https://github.com/chimpler/pyhocon)'s code and those parts are under pyhocon's Apache License 2.0, in [python/zig_hocon/pyhocon/LICENSE](python/zig_hocon/pyhocon/LICENSE). diff --git a/python/README.md b/python/README.md index 15fabb1c..46a6a809 100644 --- a/python/README.md +++ b/python/README.md @@ -51,7 +51,7 @@ pip install zig-hocon One wheel per platform, for every Python from 3.10 on, free-threaded builds and PyPy included: macOS (arm64, x86_64), Linux (x86_64, aarch64; glibc 2.17+ and musl) and Windows (x86_64). The import name is `zig_hocon`. Anywhere else, build it from source (below). -Beta: the API can still change before 1.0. BSD-3-Clause. +Beta: the API can still change before 1.0. BSD-3-Clause; `zig_hocon.pyhocon` in parts Apache-2.0, from pyhocon. ## Use @@ -211,4 +211,4 @@ zig build test-python # the package's tests, with uv zig build wheels # a wheel for every platform, into zig-out/wheels/ ``` -Needs Zig 0.16. Source, issues and the Zig and C libraries: [github.com/lejmr/zig-hocon](https://github.com/lejmr/zig-hocon). BSD-3-Clause. +Needs Zig 0.16. Source, issues and the Zig and C libraries: [github.com/lejmr/zig-hocon](https://github.com/lejmr/zig-hocon). BSD-3-Clause; `zig_hocon.pyhocon` in parts Apache-2.0, from pyhocon. diff --git a/python/pyproject.toml b/python/pyproject.toml index bcbef562..052b1115 100644 --- a/python/pyproject.toml +++ b/python/pyproject.toml @@ -8,7 +8,9 @@ name = "zig-hocon" version = "0.5.3" description = "HOCON configuration for Python: the whole spec, fast, with pyhocon's API" readme = "README.md" -license = "BSD-3-Clause" +# zig_hocon.pyhocon is changed from pyhocon's code, which is Apache-2.0. +license = "BSD-3-Clause AND Apache-2.0" +license-files = ["LICENSE", "zig_hocon/pyhocon/LICENSE"] requires-python = ">=3.10" dependencies = [] classifiers = [ diff --git a/python/tests/test_pyhocon_signatures.py b/python/tests/test_pyhocon_signatures.py index b2e157b7..96afb9da 100644 --- a/python/tests/test_pyhocon_signatures.py +++ b/python/tests/test_pyhocon_signatures.py @@ -31,7 +31,6 @@ "ConfigQuotedString.*": _PARSER_OBJECT, "ConfigInclude.*": _PARSER_OBJECT, "ConfigList.__init__": "a plain list subclass, constructed as list is", - "ConfigTree.items": "OrderedDict's own, which pyhocon overrides for Python 2", "ConfigTree.resolve": "not there yet, MIGRATING.md", "ConfigParser.unresolve_substitutions_to_value": "pyhocon's STR_SUBSTITUTION, which is not supported, MIGRATING.md", "ConfigException.__init__": "zig_hocon's exceptions: (message, name, file, line, column, source_line), not (message, ex)", @@ -85,9 +84,14 @@ def _param(case, module, name, member, why): def _signature(f): """The parameters of ``f``: name, kind and default, a class default by - its name (pyhocon's UndefinedKey is ours by name, not by identity).""" + its name (pyhocon's UndefinedKey is ours by name, not by identity). None + for a builtin that tells none: OrderedDict.items on CPython.""" + try: + signature = inspect.signature(f) + except ValueError: + return None params = [] - for p in inspect.signature(f).parameters.values(): + for p in signature.parameters.values(): default = p.default if default is not p.empty: default = default.__name__ if inspect.isclass(default) else repr(default) @@ -95,6 +99,11 @@ def _signature(f): return params +def _builtin_base(module, name): + """The builtin base of our class ``name``, where a member comes from.""" + return next(c for c in getattr(_ours(module), name).__mro__ if c.__module__ in ("builtins", "collections")) + + def _mro(cls): return [c.__name__ for c in cls.__mro__ if c is not object] @@ -110,5 +119,9 @@ def test_same_interface_as_pyhocon(module, name, member): assert set(_mro(theirs)) <= set(_mro(ours)) elif callable(theirs): mine, pyhocons = _signature(ours), _signature(theirs) + if mine is None: + # OrderedDict's own, which pyhocon overrides for Python 2's sake + assert ours is getattr(_builtin_base(module, name), member, None) + return assert mine[: len(pyhocons)] == pyhocons assert all(default is not None for _, _, default in mine[len(pyhocons) :]), "a parameter pyhocon has not, with no default" diff --git a/python/zig_hocon/pyhocon/LICENSE b/python/zig_hocon/pyhocon/LICENSE new file mode 100644 index 00000000..f6c8a5c7 --- /dev/null +++ b/python/zig_hocon/pyhocon/LICENSE @@ -0,0 +1,202 @@ +Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2014 Francois Dang Ngoc + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + diff --git a/python/zig_hocon/pyhocon/__init__.py b/python/zig_hocon/pyhocon/__init__.py index 8d0fd9f7..1c82f025 100644 --- a/python/zig_hocon/pyhocon/__init__.py +++ b/python/zig_hocon/pyhocon/__init__.py @@ -1,6 +1,5 @@ -"""pyhocon's API, by chimpler and its contributors, over zig-hocon: its -modules, names and arguments as pyhocon 0.3.63 has them, so that moving over -is a change of import:: +"""pyhocon's API over zig-hocon: its modules, names and arguments as +pyhocon 0.3.63 has them, so that moving over is a change of import:: from zig_hocon.pyhocon import ConfigFactory @@ -10,6 +9,11 @@ What a document means is the spec's, as ``zig_hocon`` reads it: where pyhocon reads it otherwise, python/MIGRATING.md says how. + +This module follows pyhocon, Copyright 2014 Francois Dang Ngoc, under the +Apache License 2.0: the files that are changed from its code say so, and +LICENSE in this directory is pyhocon's. zig-hocon is not part of the pyhocon +project; the rest of zig-hocon is BSD-3-Clause. """ from .config_parser import ConfigFactory, ConfigParser, ConfigSubstitutionException diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index 18757d03..c8091a7b 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -1,6 +1,9 @@ """pyhocon's ``ConfigFactory`` and ``ConfigParser``, with its arguments. The document is read by libhocon, as ``zig_hocon.ConfigFactory`` reads it, and -handed over as a ConfigTree of Python's own values.""" +handed over as a ConfigTree of Python's own values. + +Changed from pyhocon's config_parser.py (Apache-2.0, LICENSE in this +directory): its arguments and messages over libhocon's parser.""" import codecs import contextlib diff --git a/python/zig_hocon/pyhocon/config_tree.py b/python/zig_hocon/pyhocon/config_tree.py index cccaeb53..aec8a646 100644 --- a/python/zig_hocon/pyhocon/config_tree.py +++ b/python/zig_hocon/pyhocon/config_tree.py @@ -1,5 +1,8 @@ """pyhocon's ``ConfigTree``: the document as an OrderedDict of Python's own -values, read and changed by dotted paths, as pyhocon has it.""" +values, read and changed by dotted paths, as pyhocon has it. + +Changed from pyhocon's config_tree.py (Apache-2.0, LICENSE in this +directory): its methods, their conversions and messages.""" import datetime import json diff --git a/python/zig_hocon/pyhocon/converter.py b/python/zig_hocon/pyhocon/converter.py index b672affa..f69b4b2a 100644 --- a/python/zig_hocon/pyhocon/converter.py +++ b/python/zig_hocon/pyhocon/converter.py @@ -1,5 +1,8 @@ """pyhocon's ``HOCONConverter``: a ConfigTree as JSON, HOCON, YAML or a -.properties file, in pyhocon's layout to the character.""" +.properties file, in pyhocon's layout to the character. + +Changed from pyhocon's converter.py (Apache-2.0, LICENSE in this +directory): its writers, typed, over this module's ConfigTree.""" import json import re diff --git a/python/zig_hocon/pyhocon/period_parser.py b/python/zig_hocon/pyhocon/period_parser.py index 94e836c1..c0b68620 100644 --- a/python/zig_hocon/pyhocon/period_parser.py +++ b/python/zig_hocon/pyhocon/period_parser.py @@ -1,5 +1,8 @@ """pyhocon's ``period_parser``: a duration from its text, ``30s`` or -``10 weeks``, as pyhocon reads an unquoted value.""" +``10 weeks``, as pyhocon reads an unquoted value. + +Changed from pyhocon's period_parser.py (Apache-2.0, LICENSE in this +directory): its units, read with a regular expression for pyparsing.""" import datetime import importlib diff --git a/python/zig_hocon/pyhocon/period_serializer.py b/python/zig_hocon/pyhocon/period_serializer.py index 5cf5557a..b369063f 100644 --- a/python/zig_hocon/pyhocon/period_serializer.py +++ b/python/zig_hocon/pyhocon/period_serializer.py @@ -1,5 +1,8 @@ """pyhocon's ``period_serializer``: a duration written out, as pyhocon's -HOCONConverter writes it.""" +HOCONConverter writes it. + +Changed from pyhocon's period_serializer.py (Apache-2.0, LICENSE in this +directory): typed.""" import datetime from typing import Any diff --git a/python/zig_hocon/pyhocon/tool.py b/python/zig_hocon/pyhocon/tool.py index a1934a9f..580497b0 100644 --- a/python/zig_hocon/pyhocon/tool.py +++ b/python/zig_hocon/pyhocon/tool.py @@ -1,6 +1,9 @@ """pyhocon's command line, ``python -m zig_hocon.pyhocon.tool``: a file, or stdin, as JSON, HOCON, YAML or a .properties file. zig-hocon's own is -``python -m zig_hocon``.""" +``python -m zig_hocon``. + +Changed from pyhocon's tool.py (Apache-2.0, LICENSE in this directory): +its options, over this module's converter.""" import argparse import logging diff --git a/tools/test-wheel.sh b/tools/test-wheel.sh index 6445de1d..fec6a577 100755 --- a/tools/test-wheel.sh +++ b/tools/test-wheel.sh @@ -15,10 +15,11 @@ uv venv -q --python "$version" "$tmp/venv" py=$tmp/venv/bin/python [ -x "$py" ] || py=$tmp/venv/Scripts/python.exe # The examples read into pydantic and msgspec, and msgspec has no build for -# PyPy or a free-threaded Python: there only the package's tests run. +# PyPy or a free-threaded Python: there only the package's tests run. Those +# hold zig_hocon.pyhocon against pyhocon, with dateutil for its months. case $version in - pypy* | *t) examples=no; uv pip install -q --python "$py" "$wheel" pytest ;; - *) examples=yes; uv pip install -q --python "$py" "$wheel" pytest pydantic msgspec ;; + pypy* | *t) examples=no; uv pip install -q --python "$py" "$wheel" pytest python-dateutil pyhocon ;; + *) examples=yes; uv pip install -q --python "$py" "$wheel" pytest python-dateutil pyhocon pydantic msgspec ;; esac # From elsewhere, so nothing imports python/zig_hocon by accident. diff --git a/tools/wheel.py b/tools/wheel.py index 47f3379e..b3002e67 100644 --- a/tools/wheel.py +++ b/tools/wheel.py @@ -41,7 +41,7 @@ f"Version: {version}", f"Summary: {project['description']}", f"License-Expression: {project['license']}", - "License-File: LICENSE", + *[f"License-File: {path}" for path in project["license-files"]], f"Requires-Python: {project['requires-python']}", *[f"Classifier: {c}" for c in project["classifiers"]], *[f"Project-URL: {k}, {v}" for k, v in project["urls"].items()], @@ -54,15 +54,21 @@ wheel += [f"Tag: py3-none-{tag}" for tag in platform.split(".")] files = [] # (path in the wheel, bytes, executable) -package = os.path.join(python, name) -for file in sorted(os.listdir(package)): - if file.endswith(".py") and not file.startswith("."): # not an editor's .#lock.py - with open(os.path.join(package, file), "rb") as f: - files.append((f"{name}/{file}", f.read(), False)) +for dirpath, dirnames, filenames in os.walk(os.path.join(python, name)): + dirnames[:] = sorted(d for d in dirnames if d != "__pycache__") + for file in sorted(filenames): + # not an editor's .#lock.py; py.typed says the package has its types + if (file.endswith(".py") or file == "py.typed") and not file.startswith("."): + path = os.path.join(dirpath, file) + with open(path, "rb") as f: + files.append((os.path.relpath(path, python).replace(os.sep, "/"), f.read(), False)) with open(library, "rb") as f: files.append((f"{name}/{lib_name}", f.read(), True)) -with open(os.path.join(root, "LICENSE"), "rb") as f: - files.append((f"{dist_info}/licenses/LICENSE", f.read(), False)) +# pyproject's license-files, the top one from the repository as python/ has +# it only once `zig build python` has copied it +for path in project["license-files"]: + with open(os.path.join(root if path == "LICENSE" else python, path), "rb") as f: + files.append((f"{dist_info}/licenses/{path}", f.read(), False)) files.append((f"{dist_info}/METADATA", "\n".join(metadata).encode(), False)) files.append((f"{dist_info}/WHEEL", ("\n".join(wheel) + "\n").encode(), False)) From ecbcef3d4424d4150b309ae5afe3088cdb199393 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Thu, 22 Oct 2026 21:25:23 +0200 Subject: [PATCH 07/30] test(python): self of any kind in the signature test, as CPython 3.13 makes OrderedDict.items's positional-only Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CbBo1aotFTDLY2gery7qYD --- python/tests/test_pyhocon_signatures.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/python/tests/test_pyhocon_signatures.py b/python/tests/test_pyhocon_signatures.py index 96afb9da..341b74e0 100644 --- a/python/tests/test_pyhocon_signatures.py +++ b/python/tests/test_pyhocon_signatures.py @@ -84,8 +84,10 @@ def _param(case, module, name, member, why): def _signature(f): """The parameters of ``f``: name, kind and default, a class default by - its name (pyhocon's UndefinedKey is ours by name, not by identity). None - for a builtin that tells none: OrderedDict.items on CPython.""" + its name (pyhocon's UndefinedKey is ours by name, not by identity), and + ``self`` of any kind, as no one passes it by name: a builtin's is + positional-only. None for a builtin that tells none: OrderedDict.items + before CPython 3.13.""" try: signature = inspect.signature(f) except ValueError: @@ -95,7 +97,7 @@ def _signature(f): default = p.default if default is not p.empty: default = default.__name__ if inspect.isclass(default) else repr(default) - params.append((p.name, p.kind.name, default)) + params.append((p.name, "POSITIONAL_OR_KEYWORD" if p.name == "self" else p.kind.name, default)) return params From 721e0203f1ddfd9f32e124c00875e71eef8a9336 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 17:43:10 +0100 Subject: [PATCH 08/30] test(python): -Xmx1g unquoted reads as pyhocon reads it, since 0.5.3 On main a value that begins with - and no number is text, so the xfail in test_pyhocon_compat.py passed and is a plain case now. The docs no longer name -Xmx1g a syntax error, and MIGRATING.md says the leading-zero getters raise only outside zig_hocon.pyhocon. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- python/MIGRATING.md | 2 +- python/README.md | 2 +- python/tests/test_pyhocon_compat.py | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/python/MIGRATING.md b/python/MIGRATING.md index e4672376..2d5c7b8c 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -131,7 +131,7 @@ settings["db"]["host"] - **Durations and sizes are text until something asks** (not in `zig_hocon.pyhocon`, which has pyhocon's `timedelta`). pyhocon turns `30s` into a `timedelta` while it parses; here `as_dict()` has `"30s"`, and `get_duration` or the model converts it. With pydantic, a `BeforeValidator(zig_hocon.parse_duration)` on the field; [examples/durations-and-sizes](examples/durations-and-sizes) has it for pydantic and msgspec. - **A concatenation is a string.** `a = ${?nope}42` is `"42"` by the spec; pyhocon gives the number `42`. A model that wants an int converts it. - **An `include url(…)` gives up after 30 s.** pyhocon waits as long as `socket.setdefaulttimeout` says, by default for ever. Here `http_timeout=` on `parse_file` and `parse_string` sets it, and a server that does not answer in time is a `ConfigParseException` named `IncludeTimeout`. -- **A leading zero keeps the text.** A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. +- **A leading zero keeps the text** (the getters not in `zig_hocon.pyhocon`, whose `get_int` and `get_float` read the text as pyhocon does). A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. - **`+=` appends one element.** `x += [3, 4]` after `x = [1, 2]` is `[1, 2, [3, 4]]`, as `x = ${?x} [[3, 4]]` is; pyhocon gives `[1, 2, 3, 4]`. To extend a list with another, `x = ${x} [3, 4]`. - **Strict syntax.** The other inputs in the table under *No silent guesses* are errors now. They are rare, and each error says where to look. diff --git a/python/README.md b/python/README.md index 46a6a809..22098b3a 100644 --- a/python/README.md +++ b/python/README.md @@ -118,7 +118,7 @@ Either way, a config reads by the spec, and in a few places pyhocon reads it oth - **Durations are text in the package's own API.** `30s` is `"30s"` until a getter or a model reads it; `zig_hocon.pyhocon` makes it a `timedelta`, as pyhocon does. - **`+=` appends one element.** `x += [3, 4]` adds the list `[3, 4]` to `x`; `x = ${x} [3, 4]` extends it. -- **Strict syntax.** Input pyhocon quietly accepted, such as `a..b = 1`, `a = b:c` or an unquoted `-Xmx1g`, is an error with its line and column. +- **Strict syntax.** Input pyhocon quietly accepted, such as `a..b = 1` or `a = b:c`, is an error with its line and column. - **No `include package(...)`.** It is pyhocon's own; include by path. The [migration guide](https://github.com/lejmr/zig-hocon/blob/main/python/MIGRATING.md) has every difference, with what to write instead. diff --git a/python/tests/test_pyhocon_compat.py b/python/tests/test_pyhocon_compat.py index 8361e327..db40e3d7 100644 --- a/python/tests/test_pyhocon_compat.py +++ b/python/tests/test_pyhocon_compat.py @@ -22,6 +22,8 @@ def differs(text, why): "numbers": "a = 1\nb = -2\nc = 2.5\nd = 1e3\ne = 0.75", "bools and null": "a = true\nb = false\nc = null\nd = yes", "unquoted text": "owl = the owl hoots at midnight", + # text since 0.5.3, as the spec has it; it was an error and an xfail here + "-Xmx unquoted": "opts = [-Xmx1g]", "quoted text": 'owl = "hoot, hoot"\nempty = ""', "escapes": r'a = "tab\there"' + "\n" + r'b = "quote \" and \\ slash"', "multiline text": 'poem = """the owl\nflew home"""', @@ -65,7 +67,6 @@ def differs(text, why): # what the spec reads otherwise "quoted duration": differs('a = "30s"', 'a quoted "30s" is a duration here too: libhocon\'s JSON has no quotes left'), "+= appends one element": differs("a = [1]\na += [2, 3]", "by the spec += appends the list as one element"), - "-Xmx unquoted": differs("opts = [-Xmx1g]", "by the spec an unquoted string does not start with -"), "optional substitution then text": differs("a = ${?nope} 4", 'by the spec it is the text " 4", pyhocon the number 4'), "duration then text": differs("a = 5 ms foo", "pyhocon makes '0:00:00.005000foo' of it"), "bool then text": differs("a = true love", "pyhocon makes 'Truelove' of it; by the spec it is 'true love'"), From cfefe02b360b2587c79381e4c3b04fff8f7f6b7c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 17:48:42 +0100 Subject: [PATCH 09/30] docs(python): pyhocon's own suite passes 272 of 308 on 0.5.3 Measured again on main's parser: the six -Xmx tests pass, and so do three self-references (two from the environment, one object that refers into itself), so the table loses its -Xmx row and the self-reference row keeps two. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- CHANGELOG.md | 2 +- python/MIGRATING.md | 5 ++--- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 59bebd29..420322a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ### Added -- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 263 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 272 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. ### Fixed diff --git a/python/MIGRATING.md b/python/MIGRATING.md index 2d5c7b8c..8f165b3f 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -51,15 +51,14 @@ Two tests hold it against the newest pyhocon on every change: [test_pyhocon_sign Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. One difference: a quoted `"30s"` is a duration as well, where pyhocon keeps it text. -The measure is pyhocon's own test suite with the import swapped: 263 of its 308 tests pass. Of the 45 that do not, 44 are a config that reads differently by the spec (Step 3) or the class of a syntax error: +The measure is pyhocon's own test suite with the import swapped: 272 of its 308 tests pass. Of the 36 that do not, 35 are a config that reads differently by the spec (Step 3) or the class of a syntax error: | tests | what pyhocon does | |---|---| | 12 | raises pyparsing's exceptions for a syntax error; here it is a `ConfigParseException` | | 9 | flattens `x += [3, 4]` into `x`; by the spec it appends the list as one element | -| 6 | reads `-Xmx1g` unquoted; the spec does not let an unquoted string start with `-` | | 6 | `include` inside a list, `include glob(…)` and `include package(…)`, its own extensions | -| 5 | resolves a self-reference typesafe/config refuses too: `x = ${x}` with `x` only in the environment, `x = [${x}, 2]`, an object that refers into itself | +| 2 | resolves a self-reference typesafe/config refuses too: `x = [${x}, 2]` and an object that refers into itself | | 3 | accepts `.54`, and a backslash escape in an unquoted string: `abc\"test`, `abc\n` | | 2 | includes a file named without an extension; the spec tries `.conf`, `.json` and `.properties` after the name | | 1 | reads `${?nope} 4` as the number 4; by the spec it is the text `" 4"` | From 02d614299f80a139219173ff43fd6e8ea2776ec0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 18:02:24 +0100 Subject: [PATCH 10/30] fix(python): zig_hocon.pyhocon reads 0024 as pyhocon does, the number 24 libhocon reads a number pyhocon's grammar has and JSON's has not (0024, 007.5, 01e2, .54, -.5) as text, by the spec. zig_hocon.pyhocon now makes it pyhocon's int or float, in its own layer: the core and the conformance suite are as they were. A JSON number in a string was quoted in the document and stays text ("8080"); a quoted "01234" becomes 1234 as well, libhocon's JSON having no quotes left, and is in MIGRATING.md beside the quoted "30s". pyhocon's own suite: 273 of 308. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- CHANGELOG.md | 2 +- python/MIGRATING.md | 8 ++++---- python/tests/test_pyhocon_compat.py | 3 +++ python/zig_hocon/pyhocon/config_parser.py | 12 +++++++++++- 4 files changed, 19 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 420322a3..74ef8a16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ### Added -- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil), as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 272 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil) and `0024` the number 24, as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 273 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. ### Fixed diff --git a/python/MIGRATING.md b/python/MIGRATING.md index 8f165b3f..2f391c51 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -49,9 +49,9 @@ Everything of pyhocon's public API is there: `ConfigFactory.parse_file` (with `e Two tests hold it against the newest pyhocon on every change: [test_pyhocon_signatures.py](tests/test_pyhocon_signatures.py) has every public name of pyhocon's modules, with the same parameters, and [test_pyhocon_compat.py](tests/test_pyhocon_compat.py) reads documents with both and compares the values, their types and the converter's output. Each difference by design is in them with its reason. -Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. One difference: a quoted `"30s"` is a duration as well, where pyhocon keeps it text. +Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. A number pyhocon reads where JSON has none is that number, as in pyhocon: `0024` is 24, `.54` and `-.5` are 0.5 and -0.5. One difference: quoted, `"30s"` is a duration as well and `"01234"` the number 1234, where pyhocon keeps them text; `"8080"` or `"0.5"` stays text. -The measure is pyhocon's own test suite with the import swapped: 272 of its 308 tests pass. Of the 36 that do not, 35 are a config that reads differently by the spec (Step 3) or the class of a syntax error: +The measure is pyhocon's own test suite with the import swapped: 273 of its 308 tests pass. Of the 35 that do not, 34 are a config that reads differently by the spec (Step 3) or the class of a syntax error: | tests | what pyhocon does | |---|---| @@ -59,7 +59,7 @@ The measure is pyhocon's own test suite with the import swapped: 272 of its 308 | 9 | flattens `x += [3, 4]` into `x`; by the spec it appends the list as one element | | 6 | `include` inside a list, `include glob(…)` and `include package(…)`, its own extensions | | 2 | resolves a self-reference typesafe/config refuses too: `x = [${x}, 2]` and an object that refers into itself | -| 3 | accepts `.54`, and a backslash escape in an unquoted string: `abc\"test`, `abc\n` | +| 2 | accepts a backslash escape in an unquoted string: `abc\"test`, `abc\n` | | 2 | includes a file named without an extension; the spec tries `.conf`, `.json` and `.properties` after the name | | 1 | reads `${?nope} 4` as the number 4; by the spec it is the text `" 4"` | @@ -130,7 +130,7 @@ settings["db"]["host"] - **Durations and sizes are text until something asks** (not in `zig_hocon.pyhocon`, which has pyhocon's `timedelta`). pyhocon turns `30s` into a `timedelta` while it parses; here `as_dict()` has `"30s"`, and `get_duration` or the model converts it. With pydantic, a `BeforeValidator(zig_hocon.parse_duration)` on the field; [examples/durations-and-sizes](examples/durations-and-sizes) has it for pydantic and msgspec. - **A concatenation is a string.** `a = ${?nope}42` is `"42"` by the spec; pyhocon gives the number `42`. A model that wants an int converts it. - **An `include url(…)` gives up after 30 s.** pyhocon waits as long as `socket.setdefaulttimeout` says, by default for ever. Here `http_timeout=` on `parse_file` and `parse_string` sets it, and a server that does not answer in time is a `ConfigParseException` named `IncludeTimeout`. -- **A leading zero keeps the text** (the getters not in `zig_hocon.pyhocon`, whose `get_int` and `get_float` read the text as pyhocon does). A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. +- **A leading zero keeps the text** (not in `zig_hocon.pyhocon`, which reads such a number as pyhocon does). A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. - **`+=` appends one element.** `x += [3, 4]` after `x = [1, 2]` is `[1, 2, [3, 4]]`, as `x = ${?x} [[3, 4]]` is; pyhocon gives `[1, 2, 3, 4]`. To extend a list with another, `x = ${x} [3, 4]`. - **Strict syntax.** The other inputs in the table under *No silent guesses* are errors now. They are rare, and each error says where to look. diff --git a/python/tests/test_pyhocon_compat.py b/python/tests/test_pyhocon_compat.py index db40e3d7..96a864b5 100644 --- a/python/tests/test_pyhocon_compat.py +++ b/python/tests/test_pyhocon_compat.py @@ -24,6 +24,8 @@ def differs(text, why): "unquoted text": "owl = the owl hoots at midnight", # text since 0.5.3, as the spec has it; it was an error and an xfail here "-Xmx unquoted": "opts = [-Xmx1g]", + "numbers JSON has not": "a = 0024\nb = -0024\nc = 007.5\nd = 01e2\ne = .54\nf = -.5\ng = [00, 1]", + "quoted numbers": 'port = "8080"\nratio = "0.5"\nbig = "1e3"', "quoted text": 'owl = "hoot, hoot"\nempty = ""', "escapes": r'a = "tab\there"' + "\n" + r'b = "quote \" and \\ slash"', "multiline text": 'poem = """the owl\nflew home"""', @@ -67,6 +69,7 @@ def differs(text, why): # what the spec reads otherwise "quoted duration": differs('a = "30s"', 'a quoted "30s" is a duration here too: libhocon\'s JSON has no quotes left'), "+= appends one element": differs("a = [1]\na += [2, 3]", "by the spec += appends the list as one element"), + "quoted leading zero": differs('zip = "01234"', "a number pyhocon reads where JSON has none is one quoted too: libhocon's JSON has no quotes left"), "optional substitution then text": differs("a = ${?nope} 4", 'by the spec it is the text " 4", pyhocon the number 4'), "duration then text": differs("a = 5 ms foo", "pyhocon makes '0:00:00.005000foo' of it"), "bool then text": differs("a = true love", "pyhocon makes 'Truelove' of it; by the spec it is 'true love'"), diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index c8091a7b..f68dcbb0 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -11,6 +11,7 @@ import json import logging import os +import re import socket import urllib.error import urllib.request @@ -77,6 +78,12 @@ class IncludeNotFoundError(ConfigParseException, FileNotFoundError): _T = TypeVar("_T") +# A number by pyhocon's grammar, and one by JSON's: libhocon reads a text +# of the first that is not the second (0024, .54) as text, pyhocon as the +# number. One JSON has is quoted in the document, and stays text. +_PYHOCON_NUMBER = re.compile(r"-?(\d*\.\d+|\d+(\.\d+)?)([eE][+-]?\d+)?") +_JSON_NUMBER = re.compile(r"-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?") + def _call(read: Callable[[], _T]) -> _T: """``read()``, its error in pyhocon's class.""" @@ -93,8 +100,11 @@ def _call(read: Callable[[], _T]) -> _T: def _typed(value: Any) -> Any: - """A value from libhocon's JSON as pyhocon has it: ``30s`` a duration.""" + """A value from libhocon's JSON as pyhocon has it: ``30s`` a duration, + ``0024`` the number 24.""" if isinstance(value, str): + if _PYHOCON_NUMBER.fullmatch(value) and not _JSON_NUMBER.fullmatch(value): + return int(value) if value.lstrip("-").isdigit() else float(value) return duration(value) if isinstance(value, list): return [_typed(v) for v in value] From ec2bbcfb452ed9270e00078380c9811cb3b8ee21 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 18:38:04 +0100 Subject: [PATCH 11/30] feat(python): zig_hocon.pyhocon reads pyhocon's own syntax as pyhocon does The text is written the spec's way before libhocon reads it (zig_hocon/pyhocon/_source.py): x += v is x = ${?x} v, as pyhocon documents it, so a list is extended, text goes on after a space and an object merges. An include finds what pyhocon finds: a glob, sorted; package(...); a file without an extension, exactly; in a list, the included list's elements. An included file that needs any of it is inlined, its own includes made absolute. An unquoted abc\"d is abc"d. What the rewrite cannot follow it leaves as written, for libhocon. With pyparsing installed, as it is beside pyhocon, a syntax error is pyparsing's ParseException and ParseSyntaxException too; an include error is not one. pyhocon's own suite: 303 of 308, the rest the spec's answer and ConfigTree.resolve(). The core and the conformance suite are as they were. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- CHANGELOG.md | 2 +- python/MIGRATING.md | 16 +- python/README.md | 6 +- python/tests/test_pyhocon_api.py | 147 ++++++++++- python/tests/test_pyhocon_compat.py | 7 +- python/zig_hocon/pyhocon/_source.py | 292 ++++++++++++++++++++++ python/zig_hocon/pyhocon/config_parser.py | 43 +++- 7 files changed, 491 insertions(+), 22 deletions(-) create mode 100644 python/zig_hocon/pyhocon/_source.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 74ef8a16..5b061d93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,7 @@ What changed for someone using zig-hocon: the Zig package, the `hocon` command l ### Added -- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil) and `0024` the number 24, as in pyhocon. A config still reads by the spec; pyhocon's own test suite passes 273 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. +- Python: `zig_hocon.pyhocon`, pyhocon's API over zig-hocon, so moving over is a change of import. Its modules, names, arguments and output are pyhocon 0.3.63's: the tree an `OrderedDict` with `get`, `put`, `pop`, `with_fallback` and `get_list`, `ConfigFactory.parse_URL` and `from_dict`, `HOCONConverter`, and the command line as `python -m zig_hocon.pyhocon.tool`. `30s` is a `timedelta` (a `relativedelta` with python-dateutil) and `0024` the number 24, as in pyhocon. pyhocon's own syntax reads as in pyhocon: `x += [3, 4]` extends a list, `include "conf.d/*.conf"`, `include package(…)` and an include in a list work, and a syntax error is pyparsing's `ParseException` too when pyparsing is installed; the rest reads by the spec. pyhocon's own test suite passes 303 of 308 with the import swapped, and `python/MIGRATING.md` says what the rest are. Parts of it are changed from pyhocon's code and stay under its Apache License 2.0, so the package's license is `BSD-3-Clause AND Apache-2.0`, with both texts in the wheel. ### Fixed diff --git a/python/MIGRATING.md b/python/MIGRATING.md index 2f391c51..0379060e 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -5,7 +5,7 @@ This guide moves a project from pyhocon to zig_hocon. There are two ways: - **`zig_hocon.pyhocon`, a change of import.** pyhocon's API as pyhocon 0.3.63 has it: the same modules, names, arguments and output, the tree an `OrderedDict` you can `put` into and `pop` from. The section below says what it covers and what it does not. - **`zig_hocon`, the package's own API.** Typed getters on a document that stays in the library until one asks, and no tree in Python at all: Steps 1 and 2. -Either way a config reads by the spec, which in a few places is not how pyhocon reads it: Step 3. The checklist at the end finds every place to look at. +`zig_hocon` reads a config by the spec, which in a few places is not how pyhocon reads it: Step 3. `zig_hocon.pyhocon` reads pyhocon's own syntax as pyhocon does, and the rest by the spec. The checklist at the end finds every place to look at. ## Why move @@ -51,16 +51,14 @@ Two tests hold it against the newest pyhocon on every change: [test_pyhocon_sign Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. A number pyhocon reads where JSON has none is that number, as in pyhocon: `0024` is 24, `.54` and `-.5` are 0.5 and -0.5. One difference: quoted, `"30s"` is a duration as well and `"01234"` the number 1234, where pyhocon keeps them text; `"8080"` or `"0.5"` stays text. -The measure is pyhocon's own test suite with the import swapped: 273 of its 308 tests pass. Of the 35 that do not, 34 are a config that reads differently by the spec (Step 3) or the class of a syntax error: +So is its syntax, written the spec's way before the core reads the text: `x += v` is `x = ${?x} v`, as pyhocon documents it, so `x += [3, 4]` extends a list, `x += def` goes on after the text with a space and `x += { b = 2 }` merges; in a nested object, where pyhocon loses what `x` was, it is kept. An include finds what pyhocon finds: a glob (`include "conf.d/*.conf"`, in sorted order), `package("pkg:file.conf")`, a file named without an extension as it is, and in a list the included list's elements; an included file is read the same way. An unquoted `abc\"test` is `abc"test`. With pyparsing installed, as it is beside pyhocon, a syntax error is pyparsing's `ParseException` and `ParseSyntaxException` too. + +The measure is pyhocon's own test suite with the import swapped: 303 of its 308 tests pass. Of the 5 that do not, 4 are a config that reads differently by the spec: | tests | what pyhocon does | |---|---| -| 12 | raises pyparsing's exceptions for a syntax error; here it is a `ConfigParseException` | -| 9 | flattens `x += [3, 4]` into `x`; by the spec it appends the list as one element | -| 6 | `include` inside a list, `include glob(…)` and `include package(…)`, its own extensions | | 2 | resolves a self-reference typesafe/config refuses too: `x = [${x}, 2]` and an object that refers into itself | -| 2 | accepts a backslash escape in an unquoted string: `abc\"test`, `abc\n` | -| 2 | includes a file named without an extension; the spec tries `.conf`, `.json` and `.properties` after the name | +| 1 | rejects `a {} {b: 1}`; by the spec the two objects merge | | 1 | reads `${?nope} 4` as the number 4; by the spec it is the text `" 4"` | The one left is `ConfigTree.resolve()`, which is not there. Nor are a few things pyhocon has inside its parser: @@ -113,7 +111,7 @@ settings["db"]["host"] ## Step 3: configs that read differently -- **`include package(...)` is not supported.** It is pyhocon's own extension, not HOCON, and zig_hocon reports it as an error (`UnexpectedToken` where `package` starts). Include the file by its path instead, relative to the file that includes it: +- **`include package(...)` is not supported** (in `zig_hocon.pyhocon` it is). It is pyhocon's own extension, not HOCON, and zig_hocon reports it as an error (`UnexpectedToken` where `package` starts). Include the file by its path instead, relative to the file that includes it: ```hocon include "defaults.conf" @@ -131,7 +129,7 @@ settings["db"]["host"] - **A concatenation is a string.** `a = ${?nope}42` is `"42"` by the spec; pyhocon gives the number `42`. A model that wants an int converts it. - **An `include url(…)` gives up after 30 s.** pyhocon waits as long as `socket.setdefaulttimeout` says, by default for ever. Here `http_timeout=` on `parse_file` and `parse_string` sets it, and a server that does not answer in time is a `ConfigParseException` named `IncludeTimeout`. - **A leading zero keeps the text** (not in `zig_hocon.pyhocon`, which reads such a number as pyhocon does). A value that begins with a digit begins with the number JSON reads there, and the rest is text, so `zip = 01234` is `"01234"` by the spec; pyhocon gives the number 1234. An IP address, a version or a UUID is text as written, as in pyhocon. So is a value that begins with `-` and no number, `-foo` or `-Xmx2g`, and a fraction JSON has no number for: `.5` and `-.5` are the text `".5"` and `"-.5"`, where pyhocon gives 0.5 and -0.5. Text becomes a number only by the JSON rules, so `get_int("zip")` on `01234` and `get_float` on `-.5` raise `ConfigWrongTypeException`, where pyhocon gives 1234 and -0.5: write `zip = "01234"` and read it with `get_string`, or `-0.5`. -- **`+=` appends one element.** `x += [3, 4]` after `x = [1, 2]` is `[1, 2, [3, 4]]`, as `x = ${?x} [[3, 4]]` is; pyhocon gives `[1, 2, 3, 4]`. To extend a list with another, `x = ${x} [3, 4]`. +- **`+=` appends one element** (not in `zig_hocon.pyhocon`, which has pyhocon's `+=`). `x += [3, 4]` after `x = [1, 2]` is `[1, 2, [3, 4]]`, as `x = ${?x} [[3, 4]]` is; pyhocon gives `[1, 2, 3, 4]`. To extend a list with another, `x = ${x} [3, 4]`. - **Strict syntax.** The other inputs in the table under *No silent guesses* are errors now. They are rare, and each error says where to look. Everything else (variables, includes, merging) reads the same, except where pyhocon breaks the spec; the conformance report lists those cases. diff --git a/python/README.md b/python/README.md index 22098b3a..586daf2a 100644 --- a/python/README.md +++ b/python/README.md @@ -114,12 +114,12 @@ from zig_hocon.pyhocon import ConfigFactory # was: from pyhocon import ConfigF **The package's own API** has the same `ConfigFactory`, getters and exception names, without a tree in Python: `conf.as_dict()` when you need one, a typed getter or an `include` where pyhocon has `get`, `with_fallback` or `put`. -Either way, a config reads by the spec, and in a few places pyhocon reads it otherwise: +The core reads a config by the spec, and in a few places pyhocon reads it otherwise: - **Durations are text in the package's own API.** `30s` is `"30s"` until a getter or a model reads it; `zig_hocon.pyhocon` makes it a `timedelta`, as pyhocon does. -- **`+=` appends one element.** `x += [3, 4]` adds the list `[3, 4]` to `x`; `x = ${x} [3, 4]` extends it. +- **`+=` appends one element in the package's own API.** `x += [3, 4]` adds the list `[3, 4]` to `x`; `x = ${x} [3, 4]` extends it. `zig_hocon.pyhocon` has pyhocon's `+=`, which extends it. - **Strict syntax.** Input pyhocon quietly accepted, such as `a..b = 1` or `a = b:c`, is an error with its line and column. -- **No `include package(...)`.** It is pyhocon's own; include by path. +- **No `include package(...)` in the package's own API.** It is pyhocon's own; `zig_hocon.pyhocon` has it. The [migration guide](https://github.com/lejmr/zig-hocon/blob/main/python/MIGRATING.md) has every difference, with what to write instead. diff --git a/python/tests/test_pyhocon_api.py b/python/tests/test_pyhocon_api.py index e3bfc0a0..4b9debf3 100644 --- a/python/tests/test_pyhocon_api.py +++ b/python/tests/test_pyhocon_api.py @@ -401,9 +401,33 @@ def test_errors_in_pyhocon_s_classes(): assert isinstance(e.value, FileNotFoundError) and isinstance(e.value, ConfigException) assert e.value.name == "IncludeNotFound" and "IncludeNotFound" in str(e.value) # the others as zig_hocon raises them - with pytest.raises(exceptions.ConfigParseException) as e: - conf("a = {") - assert type(e.value) is exceptions.ConfigParseException + with pytest.raises(exceptions.ConfigSubstitutionException) as e: + conf("x = ${nope}") + assert type(e.value) is exceptions.ConfigSubstitutionException + + +# pyhocon ✓: a syntax error is pyparsing's ParseException and +# ParseSyntaxException too, with pyparsing installed, as pyhocon raises it +def test_syntax_errors_are_pyparsing_s(): + pp = pytest.importorskip("pyparsing") + with pytest.raises(pp.ParseException) as e: + conf("a = {\n b = [1,") + err = e.value + assert isinstance(err, pp.ParseSyntaxException) and isinstance(err, exceptions.ConfigParseException) + assert type(err).__name__ == "ConfigParseException" and err.name == "UnexpectedToken" + assert (err.line, err.column) == (err.lineno, err.col) == (2, 10) + assert str(err).startswith("2:10: UnexpectedToken") and err.msg == str(err) + + +def test_syntax_errors_without_pyparsing(monkeypatch): + monkeypatch.setitem(sys.modules, "pyparsing", None) + config_parser._syntax_error.cache_clear() + try: + with pytest.raises(exceptions.ConfigParseException) as e: + conf("a = {") + assert type(e.value) is exceptions.ConfigParseException + finally: + config_parser._syntax_error.cache_clear() def test_parse_file(tmp_path, caplog): @@ -563,3 +587,120 @@ def test_tool(tmp_path, verbosity): finally: root.handlers[:], root.level = handlers, level assert (tmp_path / "out").read_text() == "a.b = 1" + + +# pyhocon ✓: x += v is x = ${?x} v, as pyhocon documents it, so a list is +# extended, text goes on after a space and an object merges. pyhocon ✗ in +# a nested object, where it loses what x was; here it is kept +@pytest.mark.parametrize( + "text, want", + [ + ("x = [1, 2]\nx += [3, 4]", {"x": [1, 2, 3, 4]}), + ("x += [1]", {"x": [1]}), + ("x = abc\nx += def", {"x": "abc def"}), + ("x += def", {"x": " def"}), + ("x { a = 1 }\nx += { b = 2 }", {"x": {"a": 1, "b": 2}}), + ("a { b { x = [1] } }\na { b { x += [2] } }", {"a": {"b": {"x": [1, 2]}}}), + ('{ "q.k" = [1], "q.k" += [2] }', {'"q.k"': [1, 2]}), + ("a = ${?nope} { x = [1], x += [2] }", {"a": {"x": [1, 2]}}), + ('s = """x += [1]"""\nk += [2] # k += [3]', {"s": "x += [1]", "k": [2]}), + ("n = 1e+5\nx = [1] // x += [9]\nx += [2]", {"n": 1e5, "x": [1, 2]}), + # in a list, as the spec has it + ("l = [{ x = [1], x += [2] }]", {"l": [{"x": [1, [2]]}]}), + ("l = [[1], {}, 2]\nx += [1]", {"l": [[1], {}, 2], "x": [1]}), + ], +) +def test_plus_equals_is_pyhocon_s(text, want): + assert conf(text) == want + + +def test_plus_equals_unresolved_with_fallback(): + c = conf("x += [3]", resolve=False).with_fallback(conf("x = [1, 2]")) + assert c == {"x": [1, 2, 3]} + + +# pyhocon ✓: ``abc\"d`` unquoted is ``abc"d``, its escapes as in quotes +def test_escapes_in_unquoted_text(): + assert conf('a = abc\\"test\nb = x\\n y') == {"a": 'abc"test', "b": "x\n y"} + # what is no JSON escape, or forbidden, is libhocon's error as before + for text in ["a = abc\\q", "a = ab\\n!"]: + with pytest.raises(ConfigException): + conf(text) + + +# pyhocon ✓: what pyhocon finds is found: a glob, sorted; package(…); a +# file without an extension, exactly that file +def test_pyhocon_s_includes(tmp_path, monkeypatch): + (tmp_path / "d").mkdir() + (tmp_path / "d" / "a.conf").write_text("x = 1\nl = [1]") + (tmp_path / "d" / "b.conf").write_text("y = 2\nl += [2]") + (tmp_path / "plain").write_text("z = 3") + pkg = tmp_path / "owl_pkg" + pkg.mkdir() + (pkg / "__init__.py").write_text("") + (pkg / "nest.conf").write_text("hoot = true") + monkeypatch.syspath_prepend(str(tmp_path)) + c = ConfigParser.parse('include "d/*.conf"\ninclude required("plain")\ninclude package("owl_pkg:nest.conf")', str(tmp_path)) + assert c == {"x": 1, "l": [1, 2], "y": 2, "z": 3, "hoot": True} + assert conf('a { include "d/*.nothing" }\ninclude file("nowhere")', basedir=str(tmp_path)) == {"a": {}} + assert ConfigFactory.parse_file(str(tmp_path / "d" / "b.conf")) == {"y": 2, "l": [2]} + (tmp_path / "top.conf").write_text('include "plain"') + assert ConfigFactory.parse_file(str(tmp_path / "top.conf")) == {"z": 3} + # the classpath is libhocon's, and its error no syntax error + with pytest.raises(ConfigException) as e: + conf('include classpath("owl.conf")') + assert type(e.value) is exceptions.ConfigParseException and e.value.name == "UnsupportedInclude" + with pytest.raises(ImportError): + conf('include package("no_such_owl_pkg:x.conf")') + + +# pyhocon ✓: an include in a list is the included list's elements, an +# object as one; its own includes are found next to it +def test_includes_in_a_list(tmp_path): + (tmp_path / "one").write_text("[1, 2]") + (tmp_path / "sub").mkdir() + (tmp_path / "sub" / "obj.conf").write_text('{ o = 1, include "inner" }') + (tmp_path / "sub" / "inner").write_text("i = 2") + (tmp_path / "sub" / "bare.conf").write_text("b = 3") + url = (tmp_path / "one").as_uri() + text = f'''l = [ + 0, include "one" + include file("sub/obj.conf"), include "sub/*.conf" + include url("{url}"), include "missing" + ]''' + assert conf(text, basedir=str(tmp_path)) == {"l": [0, 1, 2, {"o": 1, "i": 2}, {"b": 3}, {"o": 1, "i": 2}, 1, 2]} + with pytest.raises(OSError): + conf('l = [include required("missing")]', basedir=str(tmp_path)) + # an http include in a list is left to libhocon, as text + assert conf('l = [include "http://example.invalid/x.conf"]') == {"l": ["include http://example.invalid/x.conf"]} + + +# what the rewrite cannot follow it leaves as written, for libhocon +@pytest.mark.parametrize( + "text", + ['a = """open', 'a = "open', 'a = "two\nlines"', "a = ${open", "a = [1", "a { b = 1", "a = 1 }", "] = 1", "= 1", "a ? 1", "a = b:c", "{ a = 1 } x", 'include """x"""', 'include file("x"', "a"], +) +def test_rewrite_gives_up(text): + from zig_hocon.pyhocon._source import written_spec_way + + assert written_spec_way(text + "\nx += [1]", None, str) == text + "\nx += [1]" + assert written_spec_way('x += [1]\na = "open', None, str) == 'x += [1]\na = "open' + + +def test_rewrite_of_included_files(tmp_path): + # a cycle of includes, a file not in UTF-8 and one the rewrite cannot + # follow are libhocon's to read or to report + (tmp_path / "a.conf").write_text('x += [1]\ninclude "b.conf"') + (tmp_path / "b.conf").write_text('y += [2]\ninclude "a.conf"') + with pytest.raises(ConfigException) as e: + ConfigFactory.parse_file(str(tmp_path / "a.conf")) + assert e.value.name == "IncludeCycle" + (tmp_path / "latin.conf").write_bytes(b"s = \"caf\xe9\"") + (tmp_path / "odd.conf").write_text('q = """x""""\nr = [1') + with pytest.raises(ConfigException): + conf('x += [1]\ninclude "latin.conf"', basedir=str(tmp_path)) + with pytest.raises(ConfigException): + conf('x += [1]\ninclude "odd.conf"', basedir=str(tmp_path)) + assert conf('l = [include "odd.conf"]', basedir=str(tmp_path)) == {"l": ["include odd.conf"]} + (tmp_path / "quotes.conf").write_text('q = """x""""\nr += [1]') + assert conf('include "quotes.conf"', basedir=str(tmp_path)) == {"q": 'x"', "r": [1]} diff --git a/python/tests/test_pyhocon_compat.py b/python/tests/test_pyhocon_compat.py index 96a864b5..17863b19 100644 --- a/python/tests/test_pyhocon_compat.py +++ b/python/tests/test_pyhocon_compat.py @@ -52,6 +52,11 @@ def differs(text, why): "substitution in text": 'name = owl\ngreeting = "hello "${name}" friend"', "optional substitution": "a = ${?nope}\nb = 1", "self reference append": "a = [1]\na = ${a} [2]", + # pyhocon's += is x = ${?x} v, as its docs have it; the spec's appends v + # as one element, so zig_hocon.pyhocon writes it pyhocon's way + "+= extends a list": "a = [1]\na += [2, 3]", + "+= onto text": "a = abc\na += def\nb += def", + "+= onto an object": "a { x = 1 }\na += { y = 2 }\nb += { z = 3 }", "object merge by substitution": "base { x = 1 }\nfull = ${base} { y = 2 }", "forward reference": "b = ${a}\na = 7", "text concatenation": "a = owl\nb = ${a} hoots", @@ -67,8 +72,8 @@ def differs(text, why): "missing substitution": "a = ${nope}", "substitution cycle": "a = ${b}\nb = ${a}", # what the spec reads otherwise + "+= in an object": differs("a { b { x = [1] } }\na { b { x += [2] } }", "pyhocon loses what x was in a nested object, [2]; here it is [1, 2]"), "quoted duration": differs('a = "30s"', 'a quoted "30s" is a duration here too: libhocon\'s JSON has no quotes left'), - "+= appends one element": differs("a = [1]\na += [2, 3]", "by the spec += appends the list as one element"), "quoted leading zero": differs('zip = "01234"', "a number pyhocon reads where JSON has none is one quoted too: libhocon's JSON has no quotes left"), "optional substitution then text": differs("a = ${?nope} 4", 'by the spec it is the text " 4", pyhocon the number 4'), "duration then text": differs("a = 5 ms foo", "pyhocon makes '0:00:00.005000foo' of it"), diff --git a/python/zig_hocon/pyhocon/_source.py b/python/zig_hocon/pyhocon/_source.py new file mode 100644 index 00000000..fe41a24a --- /dev/null +++ b/python/zig_hocon/pyhocon/_source.py @@ -0,0 +1,292 @@ +"""pyhocon's syntax where the spec reads it otherwise, written the spec's way +before libhocon reads the text: ``x += v`` is ``x = ${?x} v``, as pyhocon +documents it; an include pyhocon finds and libhocon would not names its +file exactly (a glob, ``package(…)``, a file without an extension); an +include in a list is the included list's elements; ``abc\\"d`` unquoted is +``abc"d``. A text it cannot follow is left as it is, for libhocon to +read or to report.""" + +import errno +import glob +import json +import os +import pathlib +import re +import urllib.parse +import urllib.request +from collections.abc import Callable + +_INCLUDE = re.compile(r'include[ \t]+(?:(required)\([ \t]*)?(?:(file|url|classpath|package)\([ \t]*)?(?=")') +# what ends an unquoted run, besides whitespace and a comment, and what +# the spec forbids in one +_RUN_END = '$"{}[]:=,#' +_FORBIDDEN = "+`^?!@*&" +_EXTENSIONS = (".conf", ".json", ".properties") + + +class _GiveUp(Exception): + pass + + +class _Scan: + def __init__(self, text: str, basedir: str | None, package_path: Callable[[str], str], chain: tuple[str, ...] = ()) -> None: + self.text = text + self.basedir = basedir + self.package_path = package_path + # the files inlined around this text, it the last; one inlined into + # another document has its includes name their files absolute, as + # they are found next to it + self.chain = chain + self.edits: list[tuple[int, int, str]] = [] + # an edit pyhocon's syntax asks for, not only an absolute path + self.changed = False + # "list", "braced" or "bare", and what is inside the brackets + self.root = "bare" + self.inner = (0, len(text)) + + def edit(self, start: int, end: int, new: str, changed: bool = True) -> None: + self.edits.append((start, end, new)) + self.changed = self.changed or changed + + def edited(self, start: int = 0, end: int | None = None) -> str: + end = len(self.text) if end is None else end + out, at = [], start + # the edits are in order, and all between start and end + for s, e, new in self.edits: + out.append(self.text[at:s]) + out.append(new) + at = e + out.append(self.text[at:end]) + return "".join(out) + + def comment(self, i: int) -> bool: + return self.text.startswith(("#", "//"), i) + + def line_end(self, i: int) -> int: + end = self.text.find("\n", i) + return len(self.text) if end < 0 else end + + def blank(self, i: int, newlines: bool = True, commas: bool = True) -> int: + """Past spaces and comments, and newlines and commas when asked.""" + text = self.text + while i < len(text): + c = text[i] + if c in " \t\r" or (newlines and c == "\n") or (commas and c == ","): + i += 1 + elif self.comment(i): + i = self.line_end(i) + else: + break + return i + + def string_end(self, i: int) -> int: + text = self.text + if text.startswith('"""', i): + end = text.find('"""', i + 3) + if end < 0: + raise _GiveUp + end += 3 + while end < len(text) and text[end] == '"': + end += 1 + return end + i += 1 + while i < len(text) and text[i] != '"': + if text[i] == "\n": + raise _GiveUp + i += 2 if text[i] == "\\" else 1 + if i >= len(text): + raise _GiveUp + return i + 1 + + def run_end(self, i: int) -> int: + """Past an unquoted run, its backslash escapes too.""" + text = self.text + while i < len(text) and not text[i].isspace() and text[i] not in _RUN_END and not self.comment(i): + i += 2 if text[i] == "\\" else 1 + return min(i, len(text)) + + def document(self) -> None: + i = self.blank(0) + if self.text.startswith(("{", "["), i): + self.root = "braced" if self.text[i] == "{" else "list" + start = i + 1 + i = self.fields(start, "", "}") if self.root == "braced" else self.items(start) + self.inner = (start, i - 1) + else: + i = self.fields(i, "", None) + if self.blank(i) < len(self.text): + raise _GiveUp + + def fields(self, i: int, prefix: str | None, close: str | None) -> int: + """An object's fields up to ``close``; ``prefix`` is its path, None + in a list.""" + text = self.text + while True: + i = self.blank(i) + if i >= len(text): + if close is not None: + raise _GiveUp + return i + if close is not None and text[i] == close: + return i + 1 + if text[i] in "}]": + raise _GiveUp + if _INCLUDE.match(text, i): + i = self.include(i, in_list=False) + continue + start = i + while i < len(text) and text[i] not in "=:{\n,}#" and not text.startswith(("+=", "//"), i): + i = self.string_end(i) if text[i] == '"' else i + 1 + key = text[start:i].strip() + if not key or i >= len(text): + raise _GiveUp + path = None if prefix is None else f"{prefix}.{key}" if prefix else key + if text.startswith("+=", i): + if path is not None: + self.edit(i, i + 2, f"= ${{?{path}}}") + i = self.value(i + 2, path) + elif text[i] in "=:": + i = self.value(i + 1, path) + elif text[i] == "{": + i = self.value(i, path) + else: + raise _GiveUp + + def items(self, i: int) -> int: + text = self.text + while True: + i = self.blank(i) + if i >= len(text): + raise _GiveUp + if text[i] == "]": + return i + 1 + if _INCLUDE.match(text, i): + i = self.include(i, in_list=True) + continue + i = self.value(i, None) + + def value(self, i: int, path: str | None) -> int: + """A field's value or a list element, up to the newline, comma or + bracket after it.""" + text = self.text + i = self.blank(i, commas=False) + while True: + i = self.blank(i, newlines=False, commas=False) + if i >= len(text) or text[i] in "\n,}]" or self.comment(i): + return i + c = text[i] + if c == "{": + i = self.fields(i + 1, path, "}") + elif c == "[": + i = self.items(i + 1) + elif c == '"': + i = self.string_end(i) + elif text.startswith("${", i): + end = text.find("}", i) + if end < 0: + raise _GiveUp + i = end + 1 + elif c in _RUN_END: + raise _GiveUp + else: + end = self.run_end(i) + run = text[i:end] + if "\\" in run and not any(f in run for f in _FORBIDDEN): + try: + json.loads(f'"{run}"') + except ValueError: + raise _GiveUp from None + self.edit(i, end, f'"{run}"') + i = end + + def include(self, i: int, in_list: bool) -> int: + text = self.text + m = _INCLUDE.match(text, i) + assert m is not None + required, kind = m.group(1), m.group(2) + end = self.string_end(m.end()) + if text.startswith('"""', m.end()): + raise _GiveUp + name: str = json.loads(text[m.end() : end]) + for paren in (kind, required): + if paren: + end = self.blank(end, newlines=False, commas=False) + if not text.startswith(")", end): + raise _GiveUp + end += 1 + if kind == "classpath" or kind == "url" and not name.startswith("file:") or kind is None and name.startswith(("http://", "https://")): + if in_list: + raise _GiveUp + return end + if kind == "url" or name.startswith("file:"): + path = urllib.request.url2pathname(urllib.parse.urlparse(name).path) + else: + if kind == "package": + name = self.package_path(name) + path = name if self.basedir is None else os.path.join(self.basedir, name) + wildcard = "*" in path or "?" in path + paths = sorted(glob.glob(path, recursive=True)) if wildcard else [path] + if in_list: + self.edit(i, end, self.elements_of(paths, required is not None and not wildcard)) + return end + parts, inlined = [], False + for p in paths: + scan = self.scan_of(p) if os.path.isfile(p) and not p.endswith((".json", ".properties")) else None + if scan is not None and scan.changed: + # pyhocon's syntax in the included file too: its text here + parts.append(scan.edited(*scan.inner)) + inlined = True + continue + if os.path.isfile(p): + target = f"url({json.dumps(pathlib.Path(os.path.abspath(p)).as_uri())})" + else: + target = f"file({json.dumps(os.path.abspath(p))})" + parts.append(f"include required({target})" if required else f"include {target}") + changed = inlined or kind == "package" or wildcard or os.path.isfile(path) and not path.endswith(_EXTENSIONS) + if changed or self.chain and kind != "url": + self.edit(i, end, ("\n" if inlined else ", ").join(parts), changed) + return end + + def scan_of(self, path: str) -> "_Scan | None": + """The file at ``path`` scanned, None when it cannot be.""" + if path in self.chain: + raise _GiveUp + try: + with open(path, encoding="utf-8") as fd: + content = fd.read() + except UnicodeDecodeError: + return None + scan = _Scan(content, os.path.dirname(path), self.package_path, (*self.chain, path)) + try: + scan.document() + except _GiveUp: + return None + return scan + + def elements_of(self, paths: list[str], required: bool) -> str: + """The elements of the lists in ``paths``, an object as one.""" + parts = [] + for p in paths: + if not os.path.isfile(p): + if required: + raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), p) + continue + scan = self.scan_of(p) + if scan is None: + raise _GiveUp + if scan.root == "list": + parts.append(scan.edited(*scan.inner)) + else: + parts.append(scan.edited() if scan.root == "braced" else "{\n" + scan.edited() + "\n}") + return "\n" + "\n".join(parts) + "\n" if parts else "" + + +def written_spec_way(text: str, basedir: str | None, package_path: Callable[[str], str]) -> str: + """``text`` with pyhocon's syntax written the spec's way; includes are + found in ``basedir``, a ``package(…)`` by ``package_path``.""" + scan = _Scan(text, basedir, package_path) + try: + scan.document() + except _GiveUp: + return text + return scan.edited() diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index f68dcbb0..32461a87 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -7,6 +7,8 @@ import codecs import contextlib +import functools +import importlib import importlib.util import json import logging @@ -21,6 +23,7 @@ from .. import config_parser as _core from .. import config_tree as _core_tree from ..exceptions import ConfigException, ConfigMissingException, ConfigParseException, ConfigSubstitutionException, ConfigWrongTypeException +from ._source import written_spec_way from .config_tree import ConfigInclude, ConfigList, ConfigQuotedString, ConfigSubstitution, ConfigTree, ConfigUnquotedString, ConfigValues, NoneValue, quoted from .period_parser import duration @@ -85,13 +88,35 @@ class IncludeNotFoundError(ConfigParseException, FileNotFoundError): _JSON_NUMBER = re.compile(r"-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][+-]?[0-9]+)?") +@functools.cache +def _syntax_error() -> type[ConfigParseException]: + """With pyparsing installed, as it is beside pyhocon, a syntax error is + pyparsing's ParseException and ParseSyntaxException too, which pyhocon + raises; ``line`` stays zig_hocon's number, ``lineno`` and ``col`` are + pyparsing's.""" + try: + pp = importlib.import_module("pyparsing") + except ImportError: + return ConfigParseException + + def init(self: Any, message: str) -> None: + ConfigParseException.__init__(self, message) + # pyparsing's own __init__ takes the message for the parsed text + self.args, self.pstr, self.loc, self.msg = (message,), "", 0, message + + namespace = {"__init__": init, "__module__": __name__, "lineno": property(lambda self: self.line), "col": property(lambda self: self.column)} + return cast(type[ConfigParseException], type("ConfigParseException", (ConfigParseException, pp.ParseException, pp.ParseSyntaxException), namespace)) + + def _call(read: Callable[[], _T]) -> _T: """``read()``, its error in pyhocon's class.""" try: return read() except ConfigException as e: cls = _AS_PYHOCON.get(e.name or "") - if cls is None: + if cls is None and type(e) is ConfigParseException and "Include" not in (e.name or ""): + cls = _syntax_error() + if cls is None or cls is ConfigParseException: raise # the message alone: OSError takes more arguments as errno and strerror err = cls(str(e)) @@ -149,6 +174,10 @@ def _read(parse: Callable[[bool], _core_tree.ConfigTree], resolve: bool, unresol return as_written(_call(lambda: parse(False).to_hocon())) +def _spec_way(text: str, basedir: Path | None) -> str: + return written_spec_way(text, None if basedir is None else os.fspath(basedir), ConfigParser.resolve_package_path) + + class ConfigFactory: """Reads a document, as pyhocon's ``ConfigFactory``. ``${NAME}`` that is not in the document is looked up in ``os.environ``.""" @@ -177,10 +206,12 @@ def parse_file( raise logger.warning("Cannot include file %s. File does not exist or cannot be read.", filename) return [] + basedir = os.path.dirname(filename) + text = _spec_way(content, basedir) # A UTF-8 file is read again by libhocon, for its name in an error. - if codecs.lookup(encoding).name == "utf-8": + if text == content and codecs.lookup(encoding).name == "utf-8": return _read(lambda r: _core.ConfigFactory.parse_file(filename, resolve=r), resolve, unresolved_value) - return cls.parse_string(content, os.path.dirname(filename), resolve, unresolved_value) + return _read(lambda r: _core.ConfigFactory.parse_string(text, basedir=basedir, resolve=r), resolve, unresolved_value) @overload @classmethod @@ -209,14 +240,16 @@ def parse_URL( if required: raise return [] - return _read(lambda r: _core.ConfigFactory.parse_string(content, resolve=r), resolve, unresolved_value) + text = _spec_way(content, None) + return _read(lambda r: _core.ConfigFactory.parse_string(text, resolve=r), resolve, unresolved_value) @classmethod def parse_string(cls, content: str, basedir: Path | None = None, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION) -> ConfigTree: """HOCON text; an include in it is found in ``basedir``, by default the working directory, as pyhocon finds it.""" where = os.getcwd() if basedir is None else basedir - return _read(lambda r: _core.ConfigFactory.parse_string(content, basedir=where, resolve=r), resolve, unresolved_value) + text = _spec_way(content, basedir) + return _read(lambda r: _core.ConfigFactory.parse_string(text, basedir=where, resolve=r), resolve, unresolved_value) @classmethod def from_dict(cls, dictionary: Mapping[str, Any], root: bool = False) -> Any: From 577217d32148605f17259db51db1aeb45b52fb9c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 18:39:00 +0100 Subject: [PATCH 12/30] perf(python): zig_hocon.pyhocon skips the rewrite where there is nothing to rewrite A document without +=, include or a backslash reads as fast as before the rewrite: 10 MB in 2.6 s, where the scan made it 4.5 s. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- python/zig_hocon/pyhocon/_source.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/python/zig_hocon/pyhocon/_source.py b/python/zig_hocon/pyhocon/_source.py index fe41a24a..d49d5d8a 100644 --- a/python/zig_hocon/pyhocon/_source.py +++ b/python/zig_hocon/pyhocon/_source.py @@ -284,6 +284,9 @@ def elements_of(self, paths: list[str], required: bool) -> str: def written_spec_way(text: str, basedir: str | None, package_path: Callable[[str], str]) -> str: """``text`` with pyhocon's syntax written the spec's way; includes are found in ``basedir``, a ``package(…)`` by ``package_path``.""" + # most documents have none of it, and are not scanned + if "+=" not in text and "include" not in text and "\\" not in text: + return text scan = _Scan(text, basedir, package_path) try: scan.document() From 116274be989bd385abfdb6ed4efca40b4186f069 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 22:02:55 +0100 Subject: [PATCH 13/30] fix(python): zig_hocon.pyhocon includes a rewritten file as the spec has it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found checking the PR against pyhocon itself: - An included file with pyhocon's syntax was inlined, so under an object its ${…} and += looked from the root; it is written into a temporary file now and included as the spec has it, and an error in it names the file and its own line, as one in a rewritten parse_file does. - A missing include in a list left its comma, a syntax error. - x += 2 onto a list was an error; a number, a boolean or null is appended, as the spec has it. - The scan gave up, and every += with it, on a key with its object on the next line and on a quoted } in a substitution; an http include in a list no longer stops it. - parse_URL looked for relative includes in the working directory; its includes are libhocon's again. - package() of a file the package does not have is pyhocon's ImportError, and a missing required include has its errno. MIGRATING.md lists where the spec answers otherwise around pyhocon's syntax, says += is pyhocon's parser's reading rather than its docs', and has the speed measured on ReleaseFast. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- python/MIGRATING.md | 17 ++- python/tests/test_pyhocon_api.py | 83 +++++++++- python/tests/test_pyhocon_compat.py | 2 +- python/zig_hocon/pyhocon/_source.py | 176 ++++++++++++++++------ python/zig_hocon/pyhocon/config_parser.py | 40 +++-- 5 files changed, 247 insertions(+), 71 deletions(-) diff --git a/python/MIGRATING.md b/python/MIGRATING.md index 0379060e..4f790dc3 100644 --- a/python/MIGRATING.md +++ b/python/MIGRATING.md @@ -49,9 +49,9 @@ Everything of pyhocon's public API is there: `ConfigFactory.parse_file` (with `e Two tests hold it against the newest pyhocon on every change: [test_pyhocon_signatures.py](tests/test_pyhocon_signatures.py) has every public name of pyhocon's modules, with the same parameters, and [test_pyhocon_compat.py](tests/test_pyhocon_compat.py) reads documents with both and compares the values, their types and the converter's output. Each difference by design is in them with its reason. -Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. A number pyhocon reads where JSON has none is that number, as in pyhocon: `0024` is 24, `.54` and `-.5` are 0.5 and -0.5. One difference: quoted, `"30s"` is a duration as well and `"01234"` the number 1234, where pyhocon keeps them text; `"8080"` or `"0.5"` stays text. +Values have pyhocon's types: `30s` is a `timedelta`, or with python-dateutil installed a `relativedelta`, as pyhocon makes it; months and years need dateutil there too. A number pyhocon reads where JSON has none is that number, as in pyhocon: `0024` is 24, `.54` and `-.5` are 0.54 and -0.5. One difference: quoted, `"30s"` is a duration as well and `"01234"` the number 1234, where pyhocon keeps them text; `"8080"` or `"0.5"` stays text. -So is its syntax, written the spec's way before the core reads the text: `x += v` is `x = ${?x} v`, as pyhocon documents it, so `x += [3, 4]` extends a list, `x += def` goes on after the text with a space and `x += { b = 2 }` merges; in a nested object, where pyhocon loses what `x` was, it is kept. An include finds what pyhocon finds: a glob (`include "conf.d/*.conf"`, in sorted order), `package("pkg:file.conf")`, a file named without an extension as it is, and in a list the included list's elements; an included file is read the same way. An unquoted `abc\"test` is `abc"test`. With pyparsing installed, as it is beside pyhocon, a syntax error is pyparsing's `ParseException` and `ParseSyntaxException` too. +So is its syntax, written the spec's way before the core reads the text: `x += v` is `x = ${?x} v`, as pyhocon's parser reads it, so `x += [3, 4]` extends a list, `x += def` goes on after the text with a space and `x += { b = 2 }` merges; in a nested object, where pyhocon loses what `x` was, it is kept. An include finds what pyhocon finds: a glob (`include "conf.d/*.conf"`), `package("pkg:file.conf")`, a file named without an extension as it is, and in a list the included list's elements. An included file with any of this syntax is read the same way, and still included as the spec has it. An unquoted `abc\"test` is `abc"test`. With pyparsing installed, as it is beside pyhocon, a syntax error is pyparsing's `ParseException` and `ParseSyntaxException` too. The measure is pyhocon's own test suite with the import swapped: 303 of its 308 tests pass. Of the 5 that do not, 4 are a config that reads differently by the spec: @@ -65,9 +65,18 @@ The one left is `ConfigTree.resolve()`, which is not there. Nor are a few things - **A value with a `${…}` is no `ConfigValues`.** A document read with `resolve=False` has its values where it resolves on its own; where it does not, it has none until `with_fallback()` or `ConfigParser.resolve_substitutions()` gives it what it waits for, and reading it before is a `ConfigSubstitutionException`. - **`unresolved_value`** takes `DEFAULT_SUBSTITUTION`, `MANDATORY_SUBSTITUTION` and `NO_SUBSTITUTION`; `STR_SUBSTITUTION` and a value of your own are a `NotImplementedError`. -- **`parse_URL`** reads a document with no `include` in it: one is an error, as it has no directory to be found in. +- **`parse_URL`** reads a document with no `include` in it: one is an error, as it has no directory to be found in. pyhocon follows a `url(…)` include and leaves out the others with a warning. -The whole document is in Python, as in pyhocon, so it is a little slower than `zig_hocon`, still far from pyhocon: a 5 KB config in 0.3 ms (pyhocon 29 ms), 10 MB in 0.38 s (pyhocon 68 s), against 0.2 ms and 0.13 s for `zig_hocon`. +Where the spec has the answer, it reads by the spec, also around pyhocon's syntax: + +- **`+=` of a number, a boolean or `null`** appends it: `x += 2` after `x = [1]` is `[1, 2]`, and after `x = 5` an error. pyhocon puts the value in place of `x`. +- **An included file under an object** looks its `${…}` and `+=` up in that object first: `n { include "f.conf" }` with `z = ${y}` in it reads `n.y` before `y`. pyhocon reads `y` at the root. +- **`include "f"`** with no file named `f` reads `f.conf`, `f.json` or `f.properties`; pyhocon includes nothing. Values from a `.properties` file are text, where pyhocon reads numbers and booleans. +- **An object file included in a list** is one element; pyhocon gives its fields as `(key, value)` tuples. +- **An include of a URL that cannot be reached** is a `ConfigParseException`; pyhocon leaves it out with a warning. A glob includes its files in sorted order, pyhocon in the file system's. +- **A text the rewrite cannot follow** is read as written, by the spec, with every `+=` in it appending one element. It is a document libhocon reports as an error, or one with HOCON the scan does not know. + +The whole document is in Python, as in pyhocon, so it is a little slower than `zig_hocon`, still far from pyhocon: a 5 KB config in 0.3 ms (pyhocon 29 ms), 10 MB in 0.4 s (pyhocon 68 s), against 0.12 ms and 0.13 s for `zig_hocon`. A document with `+=`, an include or a backslash is rewritten in Python first, which takes about four times as long. ## Step 1: imports diff --git a/python/tests/test_pyhocon_api.py b/python/tests/test_pyhocon_api.py index 4b9debf3..2d1ffb56 100644 --- a/python/tests/test_pyhocon_api.py +++ b/python/tests/test_pyhocon_api.py @@ -2,8 +2,10 @@ where it differs from pyhocon; pyhocon ✓ is the same result, checked against pyhocon 0.3.63 itself.""" +import contextlib import copy import datetime +import errno import logging import pathlib import pickle @@ -589,7 +591,7 @@ def test_tool(tmp_path, verbosity): assert (tmp_path / "out").read_text() == "a.b = 1" -# pyhocon ✓: x += v is x = ${?x} v, as pyhocon documents it, so a list is +# pyhocon ✓: x += v is x = ${?x} v, as pyhocon's parser reads it, so a list is # extended, text goes on after a space and an object merges. pyhocon ✗ in # a nested object, where it loses what x was; here it is kept @pytest.mark.parametrize( @@ -608,6 +610,14 @@ def test_tool(tmp_path, verbosity): # in a list, as the spec has it ("l = [{ x = [1], x += [2] }]", {"l": [{"x": [1, [2]]}]}), ("l = [[1], {}, 2]\nx += [1]", {"l": [[1], {}, 2], "x": [1]}), + # a number, a boolean or null is appended, as the spec has it; pyhocon + # puts it in place of the list + ("x = [1]\nx += 2\ny += true", {"x": [1, 2], "y": [True]}), + # a key and its object on the next line, a quoted } in a substitution + ("a\n{ b = 1 }\nx = [1]\nx += [2]", {"a": {"b": 1}, "x": [1, 2]}), + ('"a}b" = 1\nc = ${"a}b"}\nx = [1]\nx += [2]', {'"a}b"': 1, "c": 1, "x": [1, 2]}), + # an http include in a list is text, by the spec, and the rest is read + ('l = [include "http://example.invalid/x.conf"]\nx = [1]\nx += [2]', {"l": ["include http://example.invalid/x.conf"], "x": [1, 2]}), ], ) def test_plus_equals_is_pyhocon_s(text, want): @@ -678,13 +688,15 @@ def test_includes_in_a_list(tmp_path): # what the rewrite cannot follow it leaves as written, for libhocon @pytest.mark.parametrize( "text", - ['a = """open', 'a = "open', 'a = "two\nlines"', "a = ${open", "a = [1", "a { b = 1", "a = 1 }", "] = 1", "= 1", "a ? 1", "a = b:c", "{ a = 1 } x", 'include """x"""', 'include file("x"', "a"], + ["a, b = 1", 'a = """open', 'a = "open', 'a = "two\nlines"', "a = ${open", "a = [1", "a { b = 1", "a = 1 }", "] = 1", "= 1", "a ? 1", "a = b:c", "{ a = 1 } x", 'include """x"""', 'include file("x"', "a"], ) def test_rewrite_gives_up(text): - from zig_hocon.pyhocon._source import written_spec_way + from zig_hocon.pyhocon._source import Rewrite - assert written_spec_way(text + "\nx += [1]", None, str) == text + "\nx += [1]" - assert written_spec_way('x += [1]\na = "open', None, str) == 'x += [1]\na = "open' + with contextlib.closing(Rewrite(str)) as rewrite: + assert rewrite.text(text + "\nx += [1]", None) == text + "\nx += [1]" + assert rewrite.text('x += [1]\na = "open', None) == 'x += [1]\na = "open' + assert rewrite.text("x += [1]\na = ${open", None) == "x += [1]\na = ${open" def test_rewrite_of_included_files(tmp_path): @@ -704,3 +716,64 @@ def test_rewrite_of_included_files(tmp_path): assert conf('l = [include "odd.conf"]', basedir=str(tmp_path)) == {"l": ["include odd.conf"]} (tmp_path / "quotes.conf").write_text('q = """x""""\nr += [1]') assert conf('include "quotes.conf"', basedir=str(tmp_path)) == {"q": 'x"', "r": [1]} + + +# pyhocon ✗: an included file with pyhocon's syntax is still included as the +# spec has it, from a temporary file: under an object its += and ${…} look in +# that object first; pyhocon gives l = [1] and z = 100 +def test_included_files_are_included_by_the_spec(tmp_path): + (tmp_path / "f.conf").write_text("y = 1\nz = ${y}\nl += [1]") + (tmp_path / "g.conf").write_text("m += [2]") + c = ConfigParser.parse('y = 100\nn { l = [0], include "f.conf" }\ninclude "g.conf"', str(tmp_path)) + assert c == {"y": 100, "n": {"l": [0, 1], "y": 1, "z": 1}, "m": [2]} + + +# an error in a file that was rewritten names that file, at its own line +def test_errors_in_rewritten_files(tmp_path): + (tmp_path / "unres.conf").write_text("q = 1\nx = [1]\nx += [2]\nz = ${nope}") + (tmp_path / "top.conf").write_text('a = 1\nb = 2\ninclude "unres.conf"') + for name in ["top.conf", "unres.conf"]: + with pytest.raises(ConfigSubstitutionException) as e: + ConfigFactory.parse_file(str(tmp_path / name)) + assert e.value.file.endswith("unres.conf") and e.value.line == 4 + assert "zig_hocon-pyhocon-" not in str(e.value) + (tmp_path / "broken.conf").write_text("x = [1]\nx += [2]\na..b = 1") + with pytest.raises(exceptions.ConfigParseException) as e: + conf('include "broken.conf"', basedir=str(tmp_path)) + assert e.value.file.endswith("broken.conf") and "zig_hocon-pyhocon-" not in str(e.value) + assert getattr(e.value, "msg", str(e.value)) == str(e.value) + (tmp_path / "two.conf").write_text('include "g1.conf"\ninclude "g2.conf"') + (tmp_path / "g1.conf").write_text("a += [1]") + (tmp_path / "g2.conf").write_text("b += [2]") + assert ConfigFactory.parse_file(str(tmp_path / "two.conf")) == {"a": [1], "b": [2]} + # a file the scan cannot follow is libhocon's, as it is + (tmp_path / "odd.conf").write_text('x += [1]\na = """open') + with pytest.raises(ConfigException) as e: + ConfigFactory.parse_file(str(tmp_path / "odd.conf")) + assert e.value.file.endswith("odd.conf") + + +# pyhocon ✓: a missing required include is a FileNotFoundError with errno; +# package() of a file the package does not have is an ImportError +def test_missing_includes(tmp_path): + with pytest.raises(FileNotFoundError) as e: + conf('include required("nope.conf")', basedir=str(tmp_path)) + assert e.value.errno == errno.ENOENT and e.value.strerror + with pytest.raises(ImportError, match="Can't find nope.conf in package:json"): + conf('include package("json:nope.conf")') + + +# pyhocon ✓: a missing include in a list leaves nothing, nor its comma +def test_missing_include_in_a_list(tmp_path): + assert conf('a = [ include "nope.conf", 1 ]', basedir=str(tmp_path)) == {"a": [1]} + assert conf('a = [ 1, include "nope.conf" ]', basedir=str(tmp_path)) == {"a": [1]} + + +# a document from a URL has no directory: += is pyhocon's, an include is +# libhocon's, which refuses it +def test_parse_url_rewrites_no_include(tmp_path): + (tmp_path / "doc.conf").write_text("x = [1]\nx += [2]") + assert ConfigFactory.parse_URL((tmp_path / "doc.conf").as_uri()) == {"x": [1, 2]} + (tmp_path / "inc.conf").write_text('include "doc.conf"\nx += [2]') + with pytest.raises(ConfigException): + ConfigFactory.parse_URL((tmp_path / "inc.conf").as_uri()) diff --git a/python/tests/test_pyhocon_compat.py b/python/tests/test_pyhocon_compat.py index 17863b19..de392bf2 100644 --- a/python/tests/test_pyhocon_compat.py +++ b/python/tests/test_pyhocon_compat.py @@ -52,7 +52,7 @@ def differs(text, why): "substitution in text": 'name = owl\ngreeting = "hello "${name}" friend"', "optional substitution": "a = ${?nope}\nb = 1", "self reference append": "a = [1]\na = ${a} [2]", - # pyhocon's += is x = ${?x} v, as its docs have it; the spec's appends v + # pyhocon's += is x = ${?x} v, as its parser reads it; the spec's appends v # as one element, so zig_hocon.pyhocon writes it pyhocon's way "+= extends a list": "a = [1]\na += [2, 3]", "+= onto text": "a = abc\na += def\nb += def", diff --git a/python/zig_hocon/pyhocon/_source.py b/python/zig_hocon/pyhocon/_source.py index d49d5d8a..0ddd30fa 100644 --- a/python/zig_hocon/pyhocon/_source.py +++ b/python/zig_hocon/pyhocon/_source.py @@ -1,10 +1,11 @@ """pyhocon's syntax where the spec reads it otherwise, written the spec's way -before libhocon reads the text: ``x += v`` is ``x = ${?x} v``, as pyhocon -documents it; an include pyhocon finds and libhocon would not names its +before libhocon reads the text: ``x += v`` is ``x = ${?x} v``, as pyhocon's +parser reads it; an include pyhocon finds and libhocon would not names its file exactly (a glob, ``package(…)``, a file without an extension); an include in a list is the included list's elements; ``abc\\"d`` unquoted is -``abc"d``. A text it cannot follow is left as it is, for libhocon to -read or to report.""" +``abc"d``. An included file with any of it is written the same way into a +temporary file, which libhocon includes as the spec has it. A text it +cannot follow is left as it is, for libhocon to read or to report.""" import errno import glob @@ -12,11 +13,21 @@ import os import pathlib import re +import shutil +import tempfile import urllib.parse import urllib.request from collections.abc import Callable +from typing import TypeVar + +from ..exceptions import ConfigException + +_T = TypeVar("_T") _INCLUDE = re.compile(r'include[ \t]+(?:(required)\([ \t]*)?(?:(file|url|classpath|package)\([ \t]*)?(?=")') +# a number, a boolean or null after +=: pyhocon replaces the value with it, +# the spec appends it; the spec's way is kept +_SCALAR = re.compile(r"[ \t]*(?:-?[0-9][0-9.eE+-]*|true|false|null)[ \t]*(?:$|[\n,}\]#]|//)") # what ends an unquoted run, besides whitespace and a comment, and what # the spec forbids in one _RUN_END = '$"{}[]:=,#' @@ -28,14 +39,81 @@ class _GiveUp(Exception): pass +class Rewrite: + """One document's rewrite, and the temporary files of the included ones, + removed by ``close()``. ``includes=False`` leaves every include to + libhocon.""" + + def __init__(self, package_path: Callable[[str], str], includes: bool = True) -> None: + self.package_path = package_path + self.includes = includes + self._tmp: str | None = None + # a temporary file, as a path and as a URL, and the file it stands for + self._originals: dict[str, str] = {} + + def close(self) -> None: + if self._tmp is not None: + shutil.rmtree(self._tmp, ignore_errors=True) + + def text(self, text: str, basedir: str | None) -> str: + """``text`` written the spec's way; includes are found in ``basedir``.""" + # most documents have none of it, and are not scanned + if "+=" not in text and "include" not in text and "\\" not in text: + return text + scan = _Scan(self, text, basedir, ()) + try: + scan.document() + except _GiveUp: + return text + return scan.edited() + + def file(self, path: str, content: str) -> str: + """The file to read for the one at ``path``: itself, or its text + written the spec's way in a temporary file.""" + if "+=" not in content and "include" not in content and "\\" not in content: + return path + scan = _Scan(self, content, os.path.dirname(path), (os.path.abspath(path),)) + try: + scan.document() + except _GiveUp: + return path + return self.written(path, scan.edited()) if scan.changed else path + + def written(self, path: str, text: str) -> str: + """``text`` in a temporary file named as ``path``.""" + if self._tmp is None: + self._tmp = tempfile.mkdtemp(prefix="zig_hocon-pyhocon-") + tmp = os.path.join(self._tmp, str(len(self._originals) // 2), os.path.basename(path)) + os.makedirs(os.path.dirname(tmp)) + with open(tmp, "w", encoding="utf-8") as fd: + fd.write(text) + self._originals[tmp] = path + self._originals[pathlib.Path(tmp).as_uri()] = path + return tmp + + def read(self, parse: Callable[[], _T]) -> _T: + """``parse()``, an error in a temporary file named by the file it + stands for.""" + try: + return parse() + except ConfigException as e: + for tmp, path in self._originals.items(): + if e.file is not None and tmp in e.file: + e.file = e.file.replace(tmp, path) + e.args = (str(e.args[0]).replace(tmp, path), *e.args[1:]) + if isinstance(getattr(e, "msg", None), str): + # pyparsing's copy of the message + setattr(e, "msg", str(e.args[0])) # noqa: B010 + raise + + class _Scan: - def __init__(self, text: str, basedir: str | None, package_path: Callable[[str], str], chain: tuple[str, ...] = ()) -> None: + def __init__(self, rewrite: Rewrite, text: str, basedir: str | None, chain: tuple[str, ...]) -> None: + self.rewrite = rewrite self.text = text self.basedir = basedir - self.package_path = package_path - # the files inlined around this text, it the last; one inlined into - # another document has its includes name their files absolute, as - # they are found next to it + # the files around this text, it the last: one not read where it is + # names the files it includes absolute self.chain = chain self.edits: list[tuple[int, int, str]] = [] # an edit pyhocon's syntax asks for, not only an absolute path @@ -98,6 +176,18 @@ def string_end(self, i: int) -> int: raise _GiveUp return i + 1 + def substitution_end(self, i: int) -> int: + """Past ``${…}``, a quoted ``}`` in its path too.""" + text = self.text + i += 2 + while i < len(text) and text[i] != "}": + if text[i] == "\n": + raise _GiveUp + i = self.string_end(i) if text[i] == '"' else i + 1 + if i >= len(text): + raise _GiveUp + return i + 1 + def run_end(self, i: int) -> int: """Past an unquoted run, its backslash escapes too.""" text = self.text @@ -140,9 +230,14 @@ def fields(self, i: int, prefix: str | None, close: str | None) -> int: key = text[start:i].strip() if not key or i >= len(text): raise _GiveUp + if text[i] == "\n": + # a key, and its object on a line below + i = self.blank(i, commas=False) + if not text.startswith("{", i): + raise _GiveUp path = None if prefix is None else f"{prefix}.{key}" if prefix else key if text.startswith("+=", i): - if path is not None: + if path is not None and not _SCALAR.match(text, i + 2): self.edit(i, i + 2, f"= ${{?{path}}}") i = self.value(i + 2, path) elif text[i] in "=:": @@ -182,10 +277,7 @@ def value(self, i: int, path: str | None) -> int: elif c == '"': i = self.string_end(i) elif text.startswith("${", i): - end = text.find("}", i) - if end < 0: - raise _GiveUp - i = end + 1 + i = self.substitution_end(i) elif c in _RUN_END: raise _GiveUp else: @@ -214,49 +306,53 @@ def include(self, i: int, in_list: bool) -> int: if not text.startswith(")", end): raise _GiveUp end += 1 - if kind == "classpath" or kind == "url" and not name.startswith("file:") or kind is None and name.startswith(("http://", "https://")): - if in_list: - raise _GiveUp + remote = kind == "classpath" or kind == "url" and not name.startswith("file:") or kind is None and name.startswith(("http://", "https://")) + if remote or not self.rewrite.includes: + # libhocon's; in a list it stays the text it is by the spec return end if kind == "url" or name.startswith("file:"): path = urllib.request.url2pathname(urllib.parse.urlparse(name).path) else: if kind == "package": - name = self.package_path(name) + name = self.rewrite.package_path(name) path = name if self.basedir is None else os.path.join(self.basedir, name) wildcard = "*" in path or "?" in path paths = sorted(glob.glob(path, recursive=True)) if wildcard else [path] if in_list: - self.edit(i, end, self.elements_of(paths, required is not None and not wildcard)) + elements = self.elements_of(paths, required is not None and not wildcard) + if not elements: + # nothing is left, nor the comma after it + after = self.blank(end, newlines=False, commas=False) + end = after + 1 if text.startswith(",", after) else end + self.edit(i, end, elements) return end - parts, inlined = [], False + parts, rewritten = [], False for p in paths: - scan = self.scan_of(p) if os.path.isfile(p) and not p.endswith((".json", ".properties")) else None - if scan is not None and scan.changed: - # pyhocon's syntax in the included file too: its text here - parts.append(scan.edited(*scan.inner)) - inlined = True - continue - if os.path.isfile(p): - target = f"url({json.dumps(pathlib.Path(os.path.abspath(p)).as_uri())})" - else: + if not os.path.isfile(p): target = f"file({json.dumps(os.path.abspath(p))})" + else: + scan = None if p.endswith((".json", ".properties")) else self.scan_of(p) + if scan is not None and scan.changed: + # pyhocon's syntax in the included file too + p = self.rewrite.written(p, scan.edited()) + rewritten = True + target = f"url({json.dumps(pathlib.Path(os.path.abspath(p)).as_uri())})" parts.append(f"include required({target})" if required else f"include {target}") - changed = inlined or kind == "package" or wildcard or os.path.isfile(path) and not path.endswith(_EXTENSIONS) + changed = rewritten or kind == "package" or wildcard or os.path.isfile(path) and not path.endswith(_EXTENSIONS) if changed or self.chain and kind != "url": - self.edit(i, end, ("\n" if inlined else ", ").join(parts), changed) + self.edit(i, end, ", ".join(parts), changed) return end def scan_of(self, path: str) -> "_Scan | None": """The file at ``path`` scanned, None when it cannot be.""" - if path in self.chain: + if os.path.abspath(path) in self.chain: raise _GiveUp try: with open(path, encoding="utf-8") as fd: content = fd.read() except UnicodeDecodeError: return None - scan = _Scan(content, os.path.dirname(path), self.package_path, (*self.chain, path)) + scan = _Scan(self.rewrite, content, os.path.dirname(path), (*self.chain, os.path.abspath(path))) try: scan.document() except _GiveUp: @@ -279,17 +375,3 @@ def elements_of(self, paths: list[str], required: bool) -> str: else: parts.append(scan.edited() if scan.root == "braced" else "{\n" + scan.edited() + "\n}") return "\n" + "\n".join(parts) + "\n" if parts else "" - - -def written_spec_way(text: str, basedir: str | None, package_path: Callable[[str], str]) -> str: - """``text`` with pyhocon's syntax written the spec's way; includes are - found in ``basedir``, a ``package(…)`` by ``package_path``.""" - # most documents have none of it, and are not scanned - if "+=" not in text and "include" not in text and "\\" not in text: - return text - scan = _Scan(text, basedir, package_path) - try: - scan.document() - except _GiveUp: - return text - return scan.edited() diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index 32461a87..69a72ed3 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -7,6 +7,7 @@ import codecs import contextlib +import errno import functools import importlib import importlib.util @@ -23,7 +24,7 @@ from .. import config_parser as _core from .. import config_tree as _core_tree from ..exceptions import ConfigException, ConfigMissingException, ConfigParseException, ConfigSubstitutionException, ConfigWrongTypeException -from ._source import written_spec_way +from ._source import Rewrite from .config_tree import ConfigInclude, ConfigList, ConfigQuotedString, ConfigSubstitution, ConfigTree, ConfigUnquotedString, ConfigValues, NoneValue, quoted from .period_parser import duration @@ -121,6 +122,8 @@ def _call(read: Callable[[], _T]) -> _T: # the message alone: OSError takes more arguments as errno and strerror err = cls(str(e)) err.name, err.file, err.line, err.column, err.source_line = e.name, e.file, e.line, e.column, e.source_line + if isinstance(err, OSError): + err.errno, err.strerror = errno.ENOENT, os.strerror(errno.ENOENT) raise err from None @@ -174,8 +177,8 @@ def _read(parse: Callable[[bool], _core_tree.ConfigTree], resolve: bool, unresol return as_written(_call(lambda: parse(False).to_hocon())) -def _spec_way(text: str, basedir: Path | None) -> str: - return written_spec_way(text, None if basedir is None else os.fspath(basedir), ConfigParser.resolve_package_path) +def _rewrite(includes: bool = True) -> contextlib.closing[Rewrite]: + return contextlib.closing(Rewrite(ConfigParser.resolve_package_path, includes)) class ConfigFactory: @@ -207,11 +210,13 @@ def parse_file( logger.warning("Cannot include file %s. File does not exist or cannot be read.", filename) return [] basedir = os.path.dirname(filename) - text = _spec_way(content, basedir) - # A UTF-8 file is read again by libhocon, for its name in an error. - if text == content and codecs.lookup(encoding).name == "utf-8": - return _read(lambda r: _core.ConfigFactory.parse_file(filename, resolve=r), resolve, unresolved_value) - return _read(lambda r: _core.ConfigFactory.parse_string(text, basedir=basedir, resolve=r), resolve, unresolved_value) + with _rewrite() as rewrite: + # A UTF-8 file is read again by libhocon, for its name in an error. + if codecs.lookup(encoding).name == "utf-8": + path = rewrite.file(os.fspath(filename), content) + return rewrite.read(lambda: _read(lambda r: _core.ConfigFactory.parse_file(path, resolve=r), resolve, unresolved_value)) + text = rewrite.text(content, basedir) + return rewrite.read(lambda: _read(lambda r: _core.ConfigFactory.parse_string(text, basedir=basedir, resolve=r), resolve, unresolved_value)) @overload @classmethod @@ -240,16 +245,19 @@ def parse_URL( if required: raise return [] - text = _spec_way(content, None) - return _read(lambda r: _core.ConfigFactory.parse_string(text, resolve=r), resolve, unresolved_value) + # it has no directory: its includes are libhocon's, which refuses them + with _rewrite(includes=False) as rewrite: + text = rewrite.text(content, None) + return _read(lambda r: _core.ConfigFactory.parse_string(text, resolve=r), resolve, unresolved_value) @classmethod def parse_string(cls, content: str, basedir: Path | None = None, resolve: bool = True, unresolved_value: Any = DEFAULT_SUBSTITUTION) -> ConfigTree: """HOCON text; an include in it is found in ``basedir``, by default the working directory, as pyhocon finds it.""" where = os.getcwd() if basedir is None else basedir - text = _spec_way(content, basedir) - return _read(lambda r: _core.ConfigFactory.parse_string(text, basedir=where, resolve=r), resolve, unresolved_value) + with _rewrite() as rewrite: + text = rewrite.text(content, None if basedir is None else os.fspath(basedir)) + return rewrite.read(lambda: _read(lambda r: _core.ConfigFactory.parse_string(text, basedir=where, resolve=r), resolve, unresolved_value)) @classmethod def from_dict(cls, dictionary: Mapping[str, Any], root: bool = False) -> Any: @@ -281,14 +289,18 @@ def parse(cls, content: str, basedir: Path | None = None, resolve: bool = True, def resolve_package_path(cls, package_path: str) -> str: """``"my_package:foo/bar.conf"`` as the path of ``foo/bar.conf`` in the installed package ``my_package``. ValueError without the - ``:``, ImportError for a package that is not there.""" + ``:``, ImportError for a package that is not there or a file it does + not have, as pyhocon.""" if ":" not in package_path: raise ValueError("Expected format is 'PACKAGE:PATH'") package, path = package_path.split(":", 1) spec = importlib.util.find_spec(package) if spec is None or not spec.submodule_search_locations: raise ImportError(f"No module named {package!r}") - return os.path.join(spec.submodule_search_locations[0], path) + for directory in spec.submodule_search_locations: + if os.path.exists(os.path.join(directory, path)): + return os.path.join(directory, path) + raise ImportError(f"Can't find {path} in package:{package}") @classmethod def resolve_substitutions(cls, config: ConfigTree, accept_unresolved: bool = False) -> bool: From 7486c0b5e01cd1b2a72da0e06151a66f9453fff6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 22:16:03 +0100 Subject: [PATCH 14/30] fix(python): an error in a rewritten include names the file on Windows too libhocon writes the temporary file's path with / on Windows, so the error kept it. Rewrite maps that form as well, and scans through one helper instead of three copies; its classes and ConfigFactory say what they are for and how they work. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- python/zig_hocon/pyhocon/_source.py | 110 +++++++++++++--------- python/zig_hocon/pyhocon/config_parser.py | 17 +++- 2 files changed, 75 insertions(+), 52 deletions(-) diff --git a/python/zig_hocon/pyhocon/_source.py b/python/zig_hocon/pyhocon/_source.py index 0ddd30fa..f88be6b8 100644 --- a/python/zig_hocon/pyhocon/_source.py +++ b/python/zig_hocon/pyhocon/_source.py @@ -36,84 +36,104 @@ class _GiveUp(Exception): - pass + """Raised anywhere the scan is unsure of the text: the whole document is + then left as written, which is always safe, as libhocon reads it by the + spec or reports what is wrong with it.""" class Rewrite: - """One document's rewrite, and the temporary files of the included ones, - removed by ``close()``. ``includes=False`` leaves every include to - libhocon.""" + """What ConfigFactory uses: rewrites one document, and every file it + includes that needs it, for libhocon to read. + + libhocon reads files itself, so an included file with pyhocon's syntax + is written, rewritten, into a temporary file, and the include points + there; libhocon then includes it as the spec has it. ``read()`` names + the original file in an error from a temporary one, and ``close()`` + removes them. ``includes=False`` leaves every include as written, for a + document with no directory of its own (``parse_URL``).""" def __init__(self, package_path: Callable[[str], str], includes: bool = True) -> None: self.package_path = package_path self.includes = includes self._tmp: str | None = None - # a temporary file, as a path and as a URL, and the file it stands for - self._originals: dict[str, str] = {} + # each temporary file and the file it stands for + self._written: list[tuple[str, str]] = [] def close(self) -> None: if self._tmp is not None: shutil.rmtree(self._tmp, ignore_errors=True) def text(self, text: str, basedir: str | None) -> str: - """``text`` written the spec's way; includes are found in ``basedir``.""" - # most documents have none of it, and are not scanned - if "+=" not in text and "include" not in text and "\\" not in text: - return text - scan = _Scan(self, text, basedir, ()) - try: - scan.document() - except _GiveUp: - return text - return scan.edited() + """``text`` rewritten; its includes are found in ``basedir``.""" + scan = _scanned(self, text, basedir, ()) + return text if scan is None else scan.edited() def file(self, path: str, content: str) -> str: - """The file to read for the one at ``path``: itself, or its text - written the spec's way in a temporary file.""" - if "+=" not in content and "include" not in content and "\\" not in content: - return path - scan = _Scan(self, content, os.path.dirname(path), (os.path.abspath(path),)) - try: - scan.document() - except _GiveUp: - return path - return self.written(path, scan.edited()) if scan.changed else path + """The file to read for the one at ``path``, whose text is + ``content``: itself, or a temporary file with the text rewritten.""" + scan = _scanned(self, content, os.path.dirname(path), (os.path.abspath(path),)) + return path if scan is None or not scan.changed else self.written(path, scan.edited()) def written(self, path: str, text: str) -> str: - """``text`` in a temporary file named as ``path``.""" + """``text`` in a new temporary file with the name of ``path``.""" if self._tmp is None: self._tmp = tempfile.mkdtemp(prefix="zig_hocon-pyhocon-") - tmp = os.path.join(self._tmp, str(len(self._originals) // 2), os.path.basename(path)) + tmp = os.path.join(self._tmp, str(len(self._written)), os.path.basename(path)) os.makedirs(os.path.dirname(tmp)) with open(tmp, "w", encoding="utf-8") as fd: fd.write(text) - self._originals[tmp] = path - self._originals[pathlib.Path(tmp).as_uri()] = path + self._written.append((tmp, path)) return tmp def read(self, parse: Callable[[], _T]) -> _T: - """``parse()``, an error in a temporary file named by the file it - stands for.""" + """``parse()``; an error in a temporary file names the original.""" try: return parse() except ConfigException as e: - for tmp, path in self._originals.items(): - if e.file is not None and tmp in e.file: - e.file = e.file.replace(tmp, path) - e.args = (str(e.args[0]).replace(tmp, path), *e.args[1:]) + for tmp, path in self._written: + # libhocon writes the path with / on Windows too, or as a URL + for name in (tmp, tmp.replace("\\", "/"), pathlib.Path(tmp).as_uri()): + e.file = None if e.file is None else e.file.replace(name, path) + e.args = (str(e.args[0]).replace(name, path), *e.args[1:]) if isinstance(getattr(e, "msg", None), str): # pyparsing's copy of the message setattr(e, "msg", str(e.args[0])) # noqa: B010 raise +def _scanned(rewrite: Rewrite, text: str, basedir: str | None, chain: tuple[str, ...], always: bool = False) -> "_Scan | None": + """``text`` scanned, None when the scan gives up, or unless ``always`` + when there is nothing to rewrite in it.""" + # most documents have none of it, and are not scanned + if not always and "+=" not in text and "include" not in text and "\\" not in text: + return None + scan = _Scan(rewrite, text, basedir, chain) + try: + scan.document() + except _GiveUp: + return None + return scan + + class _Scan: + """One text, walked once from the start to find what to rewrite. + + It knows just enough HOCON for that: where a key ends (for ``+=``, whose + path it needs, hence ``prefix``), an include, an unquoted run, and how + to skip strings, substitutions and comments. Each method takes the + index it starts at and returns the index it ended at. What it finds is + recorded in ``edits`` as (start, end, new text), in order, and + ``edited()`` puts the text together with them. Anything else it does + not understand raises _GiveUp. + + ``chain`` is the files this text is included from, itself last, so an + include cycle gives up. A file not read where it lies (a temporary file, + elements put into a list) has its includes named by absolute path.""" + def __init__(self, rewrite: Rewrite, text: str, basedir: str | None, chain: tuple[str, ...]) -> None: self.rewrite = rewrite self.text = text self.basedir = basedir - # the files around this text, it the last: one not read where it is - # names the files it includes absolute self.chain = chain self.edits: list[tuple[int, int, str]] = [] # an edit pyhocon's syntax asks for, not only an absolute path @@ -343,8 +363,9 @@ def include(self, i: int, in_list: bool) -> int: self.edit(i, end, ", ".join(parts), changed) return end - def scan_of(self, path: str) -> "_Scan | None": - """The file at ``path`` scanned, None when it cannot be.""" + def scan_of(self, path: str, always: bool = False) -> "_Scan | None": + """The file at ``path`` scanned, as ``_scanned()``; None when it is + not UTF-8 too.""" if os.path.abspath(path) in self.chain: raise _GiveUp try: @@ -352,12 +373,7 @@ def scan_of(self, path: str) -> "_Scan | None": content = fd.read() except UnicodeDecodeError: return None - scan = _Scan(self.rewrite, content, os.path.dirname(path), (*self.chain, os.path.abspath(path))) - try: - scan.document() - except _GiveUp: - return None - return scan + return _scanned(self.rewrite, content, os.path.dirname(path), (*self.chain, os.path.abspath(path)), always) def elements_of(self, paths: list[str], required: bool) -> str: """The elements of the lists in ``paths``, an object as one.""" @@ -367,7 +383,7 @@ def elements_of(self, paths: list[str], required: bool) -> str: if required: raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), p) continue - scan = self.scan_of(p) + scan = self.scan_of(p, always=True) if scan is None: raise _GiveUp if scan.root == "list": diff --git a/python/zig_hocon/pyhocon/config_parser.py b/python/zig_hocon/pyhocon/config_parser.py index 69a72ed3..0e1c2328 100644 --- a/python/zig_hocon/pyhocon/config_parser.py +++ b/python/zig_hocon/pyhocon/config_parser.py @@ -91,10 +91,12 @@ class IncludeNotFoundError(ConfigParseException, FileNotFoundError): @functools.cache def _syntax_error() -> type[ConfigParseException]: - """With pyparsing installed, as it is beside pyhocon, a syntax error is - pyparsing's ParseException and ParseSyntaxException too, which pyhocon - raises; ``line`` stays zig_hocon's number, ``lineno`` and ``col`` are - pyparsing's.""" + """The class of a syntax error. With pyparsing installed, as it is beside + pyhocon, it is pyparsing's ParseException and ParseSyntaxException too, + which pyhocon raises, so an ``except`` written for pyhocon catches it. + pyparsing is no dependency of zig-hocon, so the class is made here, once, + when it is first needed. ``line`` stays zig_hocon's line number; + ``lineno`` and ``col`` give it under pyparsing's names.""" try: pp = importlib.import_module("pyparsing") except ImportError: @@ -183,7 +185,12 @@ def _rewrite(includes: bool = True) -> contextlib.closing[Rewrite]: class ConfigFactory: """Reads a document, as pyhocon's ``ConfigFactory``. ``${NAME}`` that is - not in the document is looked up in ``os.environ``.""" + not in the document is looked up in ``os.environ``. + + How: Rewrite (``_source.py``) writes pyhocon's syntax the spec's way, + libhocon parses and resolves the text, and its JSON becomes a + ConfigTree, each value with pyhocon's type (``_typed``). ``_call`` gives + an error pyhocon's class.""" @overload @classmethod From 6d4976f4d474f518a524b6874d1890a571361733 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Milo=C5=A1=20Koz=C3=A1k?= Date: Mon, 26 Oct 2026 22:16:04 +0100 Subject: [PATCH 15/30] docs(python): the migration guide's two ways, the short and the native Steps 1 to 3 are the native way's, under it, beside the short way's section; the checklist after both. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01DgbnUH2woorZ4FQv4L5W8N --- README.md | 2 +- conformance/report.html | 4 ++-- conformance/reports/zig-hocon.java.json | 4 ++-- conformance/reports/zig-hocon.spec.json | 4 ++-- python/MIGRATING.md | 16 ++++++++++------ python/README.md | 2 +- 6 files changed, 18 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index dd7aac0a..f8a43e58 100644 --- a/README.md +++ b/README.md @@ -216,7 +216,7 @@ b.step("check-config", "Check the config against Config").dependOn(&check.step); | py.hocon hocon-parser 1.14.2 (Python) | 67.9% (398/586) | 66.0% (387/586) | | pyhocon pyhocon 0.3.63 | 61.3% (359/586) | 57.8% (339/586) | | rs.hocon hocon-parser 1.14.2 (Rust) | 67.9% (398/586) | 65.5% (384/586) | -| zig-hocon zig-hocon code@64c6ebf | 100% (586/586) | 80.5% (472/586) | +| zig-hocon zig-hocon code@cb47884 | 100% (586/586) | 80.5% (472/586) | The table keeps the best parsers; the others measured, and why they are not in it: [conformance/README.md](conformance/README.md#not-in-the-table). diff --git a/conformance/report.html b/conformance/report.html index 31ed47d3..46616825 100644 --- a/conformance/report.html +++ b/conformance/report.html @@ -130,7 +130,7 @@

HOCON Conformance

-
Java
81.6%
478 of 586 rows
typesafe/config 1.4.9
hocon2
65.9%
386 of 586 rows
hocon2 0.6.3
pulse-pyhocon
54.1%
317 of 586 rows
pulse-pyhocon 0.2.1
py.hocon
67.9%
398 of 586 rows
hocon-parser 1.14.2 (Python)
pyhocon
61.3%
359 of 586 rows
pyhocon 0.3.63
rs.hocon
67.9%
398 of 586 rows
hocon-parser 1.14.2 (Rust)
zig-hocon
100%
586 of 586 rows
zig-hocon code@64c6ebf
+
Java
81.6%
478 of 586 rows
typesafe/config 1.4.9
hocon2
65.9%
386 of 586 rows
hocon2 0.6.3
pulse-pyhocon
54.1%
317 of 586 rows
pulse-pyhocon 0.2.1
py.hocon
67.9%
398 of 586 rows
hocon-parser 1.14.2 (Python)
pyhocon
61.3%
359 of 586 rows
pyhocon 0.3.63
rs.hocon
67.9%
398 of 586 rows
hocon-parser 1.14.2 (Rust)
zig-hocon
100%
586 of 586 rows
zig-hocon code@cb47884

Click a card to leave that implementation out of the tables below, and again to bring it back.

@@ -170,7 +170,7 @@

By section of the specification