Skip to content

test(conformance): a newline between a key and its { is no open question #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lejmr merged 2 commits into from
Oct 19, 2026
+168 −598
Merged

test(conformance): a newline between a key and its { is no open question #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
31a5317
test(conformance): a newline between a key and its { is no open question
lejmr Oct 19, 2026
718f084
test(conformance): a case can have more than one valid output (#23)
lejmr Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,13 +211,13 @@ b.step("check-config", "Check the config against Config").dependOn(&check.step);

| implementation | against the spec | against typesafe/config 1.4.9 |
|---|---|---|
| Java <sub>typesafe/config 1.4.9</sub> | 90% (399/445) | 99% (442/445) |
| hocon2 <sub>hocon2 0.6.3</sub> | 86% (384/445) | 80% (357/445) |
| pulse-pyhocon <sub>pulse-pyhocon 0.2.1</sub> | 71% (314/445) | 68% (303/445) |
| py.hocon <sub>hocon-parser 1.14.2 (Python)</sub> | 89% (395/445) | 88% (392/445) |
| pyhocon <sub>pyhocon 0.3.63</sub> | 70% (312/445) | 66% (295/445) |
| rs.hocon <sub>hocon-parser 1.14.2 (Rust)</sub> | 89% (395/445) | 86% (383/445) |
| zig-hocon <sub>zig-hocon code@fad796a</sub> | 99% (442/445) | 90% (399/445) |
| Java <sub>typesafe/config 1.4.9</sub> | 90% (401/445) | 100% (444/445) |
| hocon2 <sub>hocon2 0.6.3</sub> | 87% (386/445) | 81% (359/445) |
| pulse-pyhocon <sub>pulse-pyhocon 0.2.1</sub> | 71% (316/445) | 69% (305/445) |
| py.hocon <sub>hocon-parser 1.14.2 (Python)</sub> | 89% (397/445) | 89% (394/445) |
| pyhocon <sub>pyhocon 0.3.63</sub> | 71% (314/445) | 67% (297/445) |
| rs.hocon <sub>hocon-parser 1.14.2 (Rust)</sub> | 89% (397/445) | 87% (385/445) |
| zig-hocon <sub>zig-hocon code@73b5fad</sub> | 100% (444/445) | 90% (401/445) |

The table keeps the best parsers; the others measured, and why they are not in it: [conformance/README.md](conformance/README.md#not-in-the-table).

Expand Down
15 changes: 13 additions & 2 deletions conformance/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,17 @@ A runner hands your adapter the path to `main.conf`; includes resolve beside it.

A row where the reference implementation and the spec part ways carries `"java": "diverges" | "unsupported" | "lenient"` and, next to it, `"java_expect"` or `"java_error"` — what typesafe/config does with that same input. Those rows are the reason the suite exists; `maintaining/PROCESS.md` has the table of which kind means what.

Where the spec allows more than one answer, `"also_valid"` lists the others, and an output matching `expect` or any of them passes. `"rejected"` in the list lets an error pass too:

```json
{
"spec": "HOCON.md#examples-of-self-referential-substitutions",
"why": "the order a and b resolve in is undefined: both 1, both 2 or an error are all allowed, but a and b must agree",
"expect": {"a": 1, "b": 1},
"also_valid": [{"a": 2, "b": 2}, "rejected"]
}
```

A row carrying `"review"` is an open question for a human. It is neither a pass nor a failure. `--text` lists such rows apart from the failing ones, and they alone do not make `run.sh` exit 1.

Two directories assert features the specification only permits: `include-file-formats` (other formats, basename probing: "implementations may support") and `substitution-fallback-to-environment` together with `list-values-from-environment-variables` ("implementations may search external sources"). An implementation without them is still conformant; it fails those rows knowingly, and `--only` can leave them out.
Expand All @@ -102,9 +113,9 @@ Drop your own result into `reports/` and the local report picks it up.
The published table keeps the best parsers: every Python parser of its own above 70 % against the spec, and the best Rust one. These were measured too and left out:

- **hocon-rs 0.2.0** (Rust, mikai233/hocon-rs): 84 % against the spec, behind rs.hocon.
- **hocon 0.9.0** (Rust, mockersf/hocon.rs): 71 % with its strict loader, 58 % with the default one, which never fails; the repository is archived since 2023, and non-ASCII whitespace makes it panic.
- **hocon 0.9.0** (Rust, mockersf/hocon.rs): 72 % with its strict loader, 58 % with the default one, which never fails; the repository is archived since 2023, and non-ASCII whitespace makes it panic.
- **hocon_** (Rust, pepa65): a republished fork of hocon 0.9.0, not measured on its own.
- **pyhocon-fix 0.0.4** (Python, 2018) and **dd-pyhocon 0.3.61** (Python, 2023): forks of pyhocon, 71 % and 70 %, within two cases of it; dd-pyhocon hangs on one merge-stack case.
- **pyhocon-fix 0.0.4** (Python, 2018) and **dd-pyhocon 0.3.61** (Python, 2023): forks of pyhocon, both 71 %, within two cases of it; dd-pyhocon hangs on one merge-stack case.
- Not parsers of their own: `hocon` 0.3.0, `hocon-validator`, `pytyped-hocon`, `pyhocon-dantic` and `dataconf` wrap pyhocon; `atomr` (Rust/PyO3) builds its `Config` from TOML or a dict and parses no HOCON; chris-martin/hocon-python (2015) is an unfinished port of typesafe/config.

## Reusing this
Expand Down
6 changes: 3 additions & 3 deletions conformance/report.html
View file
Open in desktop

Large diffs are not rendered by default.

48 changes: 8 additions & 40 deletions conformance/reports/Java.java.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@
"mode": "java",
"java_version": "1.4.9",
"command": "conformance/adapters/oracle.sh hocon-java",
"ran_at": "2026-10-18T10:10:04Z",
"suite_digest": "2fe52df92f49aea2",
"ran_at": "2026-10-19T05:59:42Z",
"suite_digest": "cd4bdae3133b2042",
"total": 445,
"passed": 442,
"percent": 99.3,
"passed": 444,
"percent": 99.8,
"sections": {
"array-and-object-concatenation": {
"total": 19,
Expand Down Expand Up @@ -69,7 +69,7 @@
},
"key-value-separator": {
"total": 6,
"passed": 5
"passed": 6
},
"list-values-from-environment-variables": {
"total": 6,
Expand Down Expand Up @@ -105,7 +105,7 @@
},
"self-referential-examples": {
"total": 26,
"passed": 25
"passed": 26
},
"self-referential-substitutions": {
"total": 5,
Expand Down Expand Up @@ -340,7 +340,7 @@
"key-value-separator/003-omitted-separator-unquoted-key.conf": "pass",
"key-value-separator/004-value-on-the-line-after-the-separator.conf": "pass",
"key-value-separator/005-separator-cannot-be-omitted-before-array.conf": "pass",
"key-value-separator/006-newline-between-key-and-brace.conf": "open",
"key-value-separator/006-newline-between-key-and-brace.conf": "pass",
"list-values-from-environment-variables/001-list-from-indexed-variables.conf": "pass",
"list-values-from-environment-variables/002-optional-list-with-no-elements-is-undefined.conf": "pass",
"list-values-from-environment-variables/003-required-list-with-no-elements-is-error.conf": "pass",
Expand Down Expand Up @@ -436,7 +436,7 @@
"self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
"self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
"self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
"self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
"self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
"self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "pass",
"self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
"self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
Expand Down Expand Up @@ -617,15 +617,13 @@
"include-substitution/011-optional-reference-falls-back-too/main.conf": "{\"common\":{\"x\":{}},\"x\":{\"y\":0}}",
"include-substitution/014-unrelated-key-does-not-change-the-result/main.conf": "rejected",
"include-syntax/011-url-argument-must-be-a-url/main.conf": "rejected",
"key-value-separator/006-newline-between-key-and-brace.conf": "{\"foo\":{\"a\":1}}",
"omit-root-braces/006-array-root-unaffected.conf": "rejected",
"omit-root-braces/008-empty-array-root.conf": "rejected",
"omit-root-braces/009-comment-before-array-root.conf": "rejected",
"omit-root-braces/010-blank-line-before-array-root.conf": "rejected",
"paths-as-keys/008-include-cannot-begin-key.conf": "{\"include\":{\"foo\":42}}",
"plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "rejected",
"plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "rejected",
"self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
"self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "rejected",
"self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "rejected",
"string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "{\"a\":{\"b\":1}}",
Expand All @@ -651,36 +649,6 @@
"expected": "rejected",
"got": "rejected",
"open_question": "the spec says only that the string inside url() is interpreted as a URL; it does not say whether a string that is not a URL is an error or an unresolvable, hence ignorable, resource. typesafe/config rejects it"
},
{
"case": "key-value-separator/006-newline-between-key-and-brace.conf",
"spec": "HOCON.md#key-value-separator",
"rule": "the separator may be omitted before {, and the spec does not say whether a newline may stand between the key and the brace; typesafe/config and pyhocon accept it, hocon-rs rejects it",
"expected": {
"foo": {
"a": 1
}
},
"got": {
"foo": {
"a": 1
}
},
"open_question": "the spec is silent on a newline between a key and its brace; implementations disagree"
},
{
"case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
"spec": "HOCON.md#examples-of-self-referential-substitutions",
"rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
"expected": {
"a": 1,
"b": 1
},
"got": {
"a": 1,
"b": 1
},
"open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
}
]
}
48 changes: 8 additions & 40 deletions conformance/reports/Java.spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@
"version": "typesafe/config 1.4.9",
"mode": "spec",
"command": "conformance/adapters/oracle.sh hocon-java",
"ran_at": "2026-10-18T10:06:14Z",
"suite_digest": "2fe52df92f49aea2",
"ran_at": "2026-10-19T05:59:42Z",
"suite_digest": "cd4bdae3133b2042",
"total": 445,
"passed": 399,
"percent": 89.7,
"passed": 401,
"percent": 90.1,
"sections": {
"array-and-object-concatenation": {
"total": 19,
Expand Down Expand Up @@ -68,7 +68,7 @@
},
"key-value-separator": {
"total": 6,
"passed": 5
"passed": 6
},
"list-values-from-environment-variables": {
"total": 6,
Expand Down Expand Up @@ -104,7 +104,7 @@
},
"self-referential-examples": {
"total": 26,
"passed": 23
"passed": 24
},
"self-referential-substitutions": {
"total": 5,
Expand Down Expand Up @@ -339,7 +339,7 @@
"key-value-separator/003-omitted-separator-unquoted-key.conf": "pass",
"key-value-separator/004-value-on-the-line-after-the-separator.conf": "pass",
"key-value-separator/005-separator-cannot-be-omitted-before-array.conf": "pass",
"key-value-separator/006-newline-between-key-and-brace.conf": "open",
"key-value-separator/006-newline-between-key-and-brace.conf": "pass",
"list-values-from-environment-variables/001-list-from-indexed-variables.conf": "pass",
"list-values-from-environment-variables/002-optional-list-with-no-elements-is-undefined.conf": "pass",
"list-values-from-environment-variables/003-required-list-with-no-elements-is-error.conf": "pass",
Expand Down Expand Up @@ -435,7 +435,7 @@
"self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
"self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
"self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
"self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
"self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
"self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "fail",
"self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
"self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
Expand Down Expand Up @@ -616,15 +616,13 @@
"include-substitution/011-optional-reference-falls-back-too/main.conf": "{\"common\":{\"x\":{}},\"x\":{\"y\":0}}",
"include-substitution/014-unrelated-key-does-not-change-the-result/main.conf": "rejected",
"include-syntax/011-url-argument-must-be-a-url/main.conf": "rejected",
"key-value-separator/006-newline-between-key-and-brace.conf": "{\"foo\":{\"a\":1}}",
"omit-root-braces/006-array-root-unaffected.conf": "rejected",
"omit-root-braces/008-empty-array-root.conf": "rejected",
"omit-root-braces/009-comment-before-array-root.conf": "rejected",
"omit-root-braces/010-blank-line-before-array-root.conf": "rejected",
"paths-as-keys/008-include-cannot-begin-key.conf": "{\"include\":{\"foo\":42}}",
"plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "rejected",
"plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "rejected",
"self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
"self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "rejected",
"self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "rejected",
"string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "{\"a\":{\"b\":1}}",
Expand Down Expand Up @@ -1053,22 +1051,6 @@
"got": "rejected",
"open_question": "the spec says only that the string inside url() is interpreted as a URL; it does not say whether a string that is not a URL is an error or an unresolvable, hence ignorable, resource. typesafe/config rejects it"
},
{
"case": "key-value-separator/006-newline-between-key-and-brace.conf",
"spec": "HOCON.md#key-value-separator",
"rule": "the separator may be omitted before {, and the spec does not say whether a newline may stand between the key and the brace; typesafe/config and pyhocon accept it, hocon-rs rejects it",
"expected": {
"foo": {
"a": 1
}
},
"got": {
"foo": {
"a": 1
}
},
"open_question": "the spec is silent on a newline between a key and its brace; implementations disagree"
},
{
"case": "omit-root-braces/006-array-root-unaffected.conf",
"spec": "HOCON.md#omit-root-braces",
Expand Down Expand Up @@ -1154,20 +1136,6 @@
"got": "rejected (exit 1)",
"java": "unsupported"
},
{
"case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
"spec": "HOCON.md#examples-of-self-referential-substitutions",
"rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
"expected": {
"a": 1,
"b": 1
},
"got": {
"a": 1,
"b": 1
},
"open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
},
{
"case": "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf",
"spec": "HOCON.md#examples-of-self-referential-substitutions",
Expand Down
Loading
Loading