test(conformance): a case can have more than one valid output

README.md

@@ -211,13 +211,13 @@ b.step("check-config", "Check the config against Config").dependOn(&check.step);
 
 | implementation | against the spec | against typesafe/config 1.4.9 |
 |---|---|---|
-| Java <sub>typesafe/config 1.4.9</sub> | 90% (400/445) | 100% (443/445) |
-| hocon2 <sub>hocon2 0.6.3</sub> | 87% (385/445) | 80% (358/445) |
-| pulse-pyhocon <sub>pulse-pyhocon 0.2.1</sub> | 71% (315/445) | 68% (304/445) |
-| py.hocon <sub>hocon-parser 1.14.2 (Python)</sub> | 89% (396/445) | 88% (393/445) |
-| pyhocon <sub>pyhocon 0.3.63</sub> | 70% (313/445) | 67% (296/445) |
-| rs.hocon <sub>hocon-parser 1.14.2 (Rust)</sub> | 89% (396/445) | 86% (384/445) |
-| zig-hocon <sub>zig-hocon code@ca3a97c</sub> | 100% (443/445) | 90% (400/445) |
+| Java <sub>typesafe/config 1.4.9</sub> | 90% (401/445) | 100% (444/445) |
+| hocon2 <sub>hocon2 0.6.3</sub> | 87% (386/445) | 81% (359/445) |
+| pulse-pyhocon <sub>pulse-pyhocon 0.2.1</sub> | 71% (316/445) | 69% (305/445) |
+| py.hocon <sub>hocon-parser 1.14.2 (Python)</sub> | 89% (397/445) | 89% (394/445) |
+| pyhocon <sub>pyhocon 0.3.63</sub> | 71% (314/445) | 67% (297/445) |
+| rs.hocon <sub>hocon-parser 1.14.2 (Rust)</sub> | 89% (397/445) | 87% (385/445) |
+| zig-hocon <sub>zig-hocon code@107a173</sub> | 100% (444/445) | 90% (401/445) |
 
 The table keeps the best parsers; the others measured, and why they are not in it: [conformance/README.md](conformance/README.md#not-in-the-table).
 

conformance/README.md

@@ -81,6 +81,17 @@ A runner hands your adapter the path to `main.conf`; includes resolve beside it.
 
 A row where the reference implementation and the spec part ways carries `"java": "diverges" | "unsupported" | "lenient"` and, next to it, `"java_expect"` or `"java_error"` — what typesafe/config does with that same input. Those rows are the reason the suite exists; `maintaining/PROCESS.md` has the table of which kind means what.
 
+Where the spec allows more than one answer, `"also_valid"` lists the others, and an output matching `expect` or any of them passes. `"rejected"` in the list lets an error pass too:
+
+```json
+{
+  "spec": "HOCON.md#examples-of-self-referential-substitutions",
+  "why": "the order a and b resolve in is undefined: both 1, both 2 or an error are all allowed, but a and b must agree",
+  "expect": {"a": 1, "b": 1},
+  "also_valid": [{"a": 2, "b": 2}, "rejected"]
+}
+```
+
 A row carrying `"review"` is an open question for a human. It is neither a pass nor a failure. `--text` lists such rows apart from the failing ones, and they alone do not make `run.sh` exit 1.
 
 Two directories assert features the specification only permits: `include-file-formats` (other formats, basename probing: "implementations may support") and `substitution-fallback-to-environment` together with `list-values-from-environment-variables` ("implementations may search external sources"). An implementation without them is still conformant; it fails those rows knowingly, and `--only` can leave them out.

conformance/reports/Java.java.json

@@ -5,11 +5,11 @@
   "mode": "java",
   "java_version": "1.4.9",
   "command": "conformance/adapters/oracle.sh hocon-java",
-  "ran_at": "2026-10-19T05:39:43Z",
-  "suite_digest": "7647451c97c1d244",
+  "ran_at": "2026-10-19T05:59:42Z",
+  "suite_digest": "cd4bdae3133b2042",
   "total": 445,
-  "passed": 443,
-  "percent": 99.6,
+  "passed": 444,
+  "percent": 99.8,
   "sections": {
     "array-and-object-concatenation": {
       "total": 19,
@@ -105,7 +105,7 @@
     },
     "self-referential-examples": {
       "total": 26,
-      "passed": 25
+      "passed": 26
     },
     "self-referential-substitutions": {
       "total": 5,
@@ -436,7 +436,7 @@
     "self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
     "self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
     "self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
+    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "pass",
     "self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
     "self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
@@ -624,7 +624,6 @@
     "paths-as-keys/008-include-cannot-begin-key.conf": "{\"include\":{\"foo\":42}}",
     "plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "rejected",
     "plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "rejected",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "rejected",
     "self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "rejected",
     "string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "{\"a\":{\"b\":1}}",
@@ -650,20 +649,6 @@
       "expected": "rejected",
       "got": "rejected",
       "open_question": "the spec says only that the string inside url() is interpreted as a URL; it does not say whether a string that is not a URL is an error or an unresolvable, hence ignorable, resource. typesafe/config rejects it"
-    },
-    {
-      "case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
-      "spec": "HOCON.md#examples-of-self-referential-substitutions",
-      "rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
-      "expected": {
-        "a": 1,
-        "b": 1
-      },
-      "got": {
-        "a": 1,
-        "b": 1
-      },
-      "open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
     }
   ]
 }

conformance/reports/Java.spec.json

@@ -4,11 +4,11 @@
   "version": "typesafe/config 1.4.9",
   "mode": "spec",
   "command": "conformance/adapters/oracle.sh hocon-java",
-  "ran_at": "2026-10-19T05:39:43Z",
-  "suite_digest": "7647451c97c1d244",
+  "ran_at": "2026-10-19T05:59:42Z",
+  "suite_digest": "cd4bdae3133b2042",
   "total": 445,
-  "passed": 400,
-  "percent": 89.9,
+  "passed": 401,
+  "percent": 90.1,
   "sections": {
     "array-and-object-concatenation": {
       "total": 19,
@@ -104,7 +104,7 @@
     },
     "self-referential-examples": {
       "total": 26,
-      "passed": 23
+      "passed": 24
     },
     "self-referential-substitutions": {
       "total": 5,
@@ -435,7 +435,7 @@
     "self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
     "self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
     "self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
+    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "fail",
     "self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
     "self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
@@ -623,7 +623,6 @@
     "paths-as-keys/008-include-cannot-begin-key.conf": "{\"include\":{\"foo\":42}}",
     "plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "rejected",
     "plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "rejected",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "rejected",
     "self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "rejected",
     "string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "{\"a\":{\"b\":1}}",
@@ -1137,20 +1136,6 @@
       "got": "rejected (exit 1)",
       "java": "unsupported"
     },
-    {
-      "case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
-      "spec": "HOCON.md#examples-of-self-referential-substitutions",
-      "rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
-      "expected": {
-        "a": 1,
-        "b": 1
-      },
-      "got": {
-        "a": 1,
-        "b": 1
-      },
-      "open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
-    },
     {
       "case": "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf",
       "spec": "HOCON.md#examples-of-self-referential-substitutions",

conformance/reports/hocon2.java.json

@@ -5,11 +5,11 @@
   "mode": "java",
   "java_version": "1.4.9",
   "command": "pure Python, PyPI hocon2 0.6.3 (myjniak/hocon): parse the file, the tree as JSON",
-  "ran_at": "2026-10-19T05:39:14Z",
-  "suite_digest": "7647451c97c1d244",
+  "ran_at": "2026-10-19T05:59:13Z",
+  "suite_digest": "cd4bdae3133b2042",
   "total": 445,
-  "passed": 358,
-  "percent": 80.4,
+  "passed": 359,
+  "percent": 80.7,
   "sections": {
     "array-and-object-concatenation": {
       "total": 19,
@@ -105,7 +105,7 @@
     },
     "self-referential-examples": {
       "total": 26,
-      "passed": 23
+      "passed": 24
     },
     "self-referential-substitutions": {
       "total": 5,
@@ -436,7 +436,7 @@
     "self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
     "self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
     "self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
+    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "fail",
     "self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
     "self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
@@ -624,7 +624,6 @@
     "paths-as-keys/008-include-cannot-begin-key.conf": "rejected",
     "plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "{\"a\":[{\"b\":[1]}]}",
     "plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "{\"x\":{\"p\":1},\"l\":[{\"a\":{\"p\":1}}]}",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "{\"wrapper\":{},\"replacement\":{}}",
     "self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "{\"w\":{\"x\":1},\"r\":{\"x\":1},\"zzz\":1}",
     "string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "rejected",
@@ -1485,20 +1484,6 @@
       "got": "{\"x\":{\"p\":1},\"l\":[{\"a\":{\"p\":1}}]}",
       "java": "unsupported"
     },
-    {
-      "case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
-      "spec": "HOCON.md#examples-of-self-referential-substitutions",
-      "rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
-      "expected": {
-        "a": 1,
-        "b": 1
-      },
-      "got": {
-        "a": 1,
-        "b": 1
-      },
-      "open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
-    },
     {
       "case": "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf",
       "spec": "HOCON.md#examples-of-self-referential-substitutions",

conformance/reports/hocon2.spec.json

@@ -4,11 +4,11 @@
   "version": "hocon2 0.6.3",
   "mode": "spec",
   "command": "pure Python, PyPI hocon2 0.6.3 (myjniak/hocon): parse the file, the tree as JSON",
-  "ran_at": "2026-10-19T05:39:14Z",
-  "suite_digest": "7647451c97c1d244",
+  "ran_at": "2026-10-19T05:59:13Z",
+  "suite_digest": "cd4bdae3133b2042",
   "total": 445,
-  "passed": 385,
-  "percent": 86.5,
+  "passed": 386,
+  "percent": 86.7,
   "sections": {
     "array-and-object-concatenation": {
       "total": 19,
@@ -104,7 +104,7 @@
     },
     "self-referential-examples": {
       "total": 26,
-      "passed": 25
+      "passed": 26
     },
     "self-referential-substitutions": {
       "total": 5,
@@ -435,7 +435,7 @@
     "self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
     "self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
     "self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
+    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "pass",
     "self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
     "self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
@@ -623,7 +623,6 @@
     "paths-as-keys/008-include-cannot-begin-key.conf": "rejected",
     "plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "{\"a\":[{\"b\":[1]}]}",
     "plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "{\"x\":{\"p\":1},\"l\":[{\"a\":{\"p\":1}}]}",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "{\"wrapper\":{},\"replacement\":{}}",
     "self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "{\"w\":{\"x\":1},\"r\":{\"x\":1},\"zzz\":1}",
     "string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "rejected",
@@ -1122,20 +1121,6 @@
         }
       }
     },
-    {
-      "case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
-      "spec": "HOCON.md#examples-of-self-referential-substitutions",
-      "rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
-      "expected": {
-        "a": 1,
-        "b": 1
-      },
-      "got": {
-        "a": 1,
-        "b": 1
-      },
-      "open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
-    },
     {
       "case": "substitutions/031-whitespace-inside-braces-is-trimmed.conf",
       "spec": "HOCON.md#substitutions",

conformance/reports/pulse-pyhocon.java.json

@@ -5,11 +5,11 @@
   "mode": "java",
   "java_version": "1.4.9",
   "command": "pulse_pyhocon.parse_file: AstekGroup/pulse-pyhocon 0.2.1 (Rust/PyO3, falls back to pyhocon 0.3.63), HOCONConverter.to_json",
-  "ran_at": "2026-10-19T05:39:31Z",
-  "suite_digest": "7647451c97c1d244",
+  "ran_at": "2026-10-19T05:59:30Z",
+  "suite_digest": "cd4bdae3133b2042",
   "total": 445,
-  "passed": 304,
-  "percent": 68.3,
+  "passed": 305,
+  "percent": 68.5,
   "sections": {
     "array-and-object-concatenation": {
       "total": 19,
@@ -105,7 +105,7 @@
     },
     "self-referential-examples": {
       "total": 26,
-      "passed": 18
+      "passed": 19
     },
     "self-referential-substitutions": {
       "total": 5,
@@ -436,7 +436,7 @@
     "self-referential-examples/011-optional-self-reference-in-concatenation-looks-back.conf": "pass",
     "self-referential-examples/012-mutual-non-self-cycle-is-unresolvable.conf": "pass",
     "self-referential-examples/013-multi-step-cycle-is-invalid.conf": "pass",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "open",
+    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "pass",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "fail",
     "self-referential-examples/016-object-hidden-by-substituted-scalar-is-never-evaluated.conf": "pass",
     "self-referential-examples/017-object-hidden-by-substituted-concatenation-is-never-evaluated.conf": "pass",
@@ -624,7 +624,6 @@
     "paths-as-keys/008-include-cannot-begin-key.conf": "{\"include\":{\"foo\":42}}",
     "plus-equals-field-separator/006-plus-equals-inside-a-list-element.conf": "{\"a\":[{\"b\":1}]}",
     "plus-equals-field-separator/008-plus-equals-with-an-object-containing-a-merge.conf": "{\"x\":{\"p\":1},\"l\":{\"a\":{\"p\":1}}}",
-    "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf": "{\"a\":1,\"b\":1}",
     "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf": "{\"wrapper\":{},\"replacement\":{}}",
     "self-referential-examples/021-hidden-list-does-not-depend-on-key-order.conf": "{\"w\":{\"x\":1},\"r\":{\"x\":1},\"zzz\":1}",
     "string-value-concatenation/010-object-then-scalar-drops-the-scalar.conf": "rejected",
@@ -1774,20 +1773,6 @@
       "expected": "rejected",
       "got": "{\"foo\":{\"a\":1}}"
     },
-    {
-      "case": "self-referential-examples/014-order-dependent-resolution-must-agree-on-one-value.conf",
-      "spec": "HOCON.md#examples-of-self-referential-substitutions",
-      "rule": "this case has undefined behavior depending on resolution order: implementations are allowed to set both a and b to 1, both to 2, or to error, but must set both to the same value because substitutions are memoized by instance, so a and b must never diverge (e.g. a=1,b=2 is not permitted)",
-      "expected": {
-        "a": 1,
-        "b": 1
-      },
-      "got": {
-        "a": 1,
-        "b": 1
-      },
-      "open_question": "the spec allows both fields to be 1, both 2, or an error (L875-877); only 'the same value' is required, which one expected tree cannot express. typesafe/config itself gives 1 or 2 depending on key order"
-    },
     {
       "case": "self-referential-examples/015-list-hidden-by-substituted-object-is-never-evaluated.conf",
       "spec": "HOCON.md#examples-of-self-referential-substitutions",
@@ -1851,7 +1836,7 @@
         "a": "null bar"
       },
       "got": {
-        "a": "<pyhocon.config_tree.NoneValue object at 0x105c88320>bar"
+        "a": "<pyhocon.config_tree.NoneValue object at 0x103e38290>bar"
       }
     },
     {