ifctrap

An adversarial conformance corpus for IFC parsers — and a scoreboard of how every major toolkit survives it.

IFC looks solved until you feed a parser a file with a dangling #99, a coordinate at 587 123.456 m, an IFCBEAMSTANDARDCASE under an IFC4X3 header, or a wall called T\X2\01B01EDD\X0\ng. Every one of those is a real file that a real exporter has produced, and most toolkits will load all of them without a word.

This repo is a growing set of trap files plus a runner that scores toolkits against all of them and publishes the table below. The corpus is the product; the scoreboard is the pressure.

Scoreboard

Toolkit	Version	✅ correct	🚫 rejected	☠️ silently wrong	💥 crashed	➖ n/a	Score
ifcref	0.1.0	51	0	0	0	1	100%
IfcOpenShell	0.8.5	31	0	19	2	0	60%
xBIM	6.0.587	26	0	26	0	0	50%
web-ifc	0.0.77	18	0	34	0	0	35%

52 cases. ✅ gave an accepted answer, or safely refused a file that has no single valid reading · 🚫 refused a file that does · ☠️ answered confidently from a file that does not support the answer · 💥 crashed · ➖ declined to answer, excluded from the score.

• ifcref — reference reader, lives in this repo — the control column
• IfcOpenShell — load scored with ifcopenshell.validate(express_rules=True)
• web-ifc — no conformance checker; load = OpenModel returned a model
• xBIM — load scored with xBIM's own Validator at ValidationFlags.All

Per-case verdicts — every toolkit × every trap, with what it actually returned: docs/RESULTS.md.

Category	Cases	ifcref	IfcOpenShell	web-ifc	xBIM
coords	6	6/6	3/6	3/6	4/6
encoding	9	9/9	4/9	5/9	4/9
geometry	7	7/7	7/7	0/7	5/7
header	6	6/6	4/6	5/6	5/6
refs	8	8/8	4/8	2/8	2/8
schema	8	7/7	4/8	2/8	4/8
types	8	8/8	5/8	1/8	2/8

Favourite catches so far

• IfcOpenShell loads a truncated file and says ok. The file ends mid-entity, with no END-ISO-10303-21;. There is no way to know how much of the model is missing — and validate(express_rules=True), its strictest setting, reports nothing. web-ifc catches this one.
• A dangling reference and a duplicate #1 pass on all three. #1 referencing a #99 the file never defines, and #1 defined twice as two different entities, are the two most basic structural defects a STEP file can have. Nothing on the board catches either.
• IfcOpenShell deletes non-ASCII text rather than failing on it. A wall named Tường in raw UTF-8 comes back as Tng; Bê tông in ISO 8859-1 comes back as B tng; a \X2\ surrogate pair comes back as the empty string. Silently losing the diacritics of every Vietnamese room name is worse than refusing the file.
• …and for an ISO 8859-1 file it returns a string that is not text. The answer contains lone surrogates and cannot be encoded back to UTF-8 at all. The runner has to escape toolkit output before it can print it.
• web-ifc reports zero walls in a file with one wall, because the keyword is written ifcwall instead of IFCWALL. Not an error — just zero.
• xBIM loses an entity whose instance name exceeds 2³². A file with #1 and #4294967297 contains two points; xBIM reports one. Entity labels are int, and ISO 10303-21 sets no upper bound on an instance name. Every other toolkit on the board gets this right.
• All three toolkits mangle non-ASCII differently, and all three do it silently. For the same wall named Tường, IfcOpenShell drops the diacritics (Tng), xBIM double-decodes to mojibake (TÆ°á»\x9dng), and web-ifc gets it right. For an astral \X2\ surrogate pair, IfcOpenShell returns the empty string and xBIM returns two U+FFFD replacement characters — it decodes each 4-hex group separately instead of combining the pair.
• FILE_SCHEMA(('IFC4','IFC2X3')) crashes IfcOpenShell with RuntimeError: No schema loaded, and #-4 as an instance name crashes it with an OverflowError out of the SWIG binding.
• 1e6 and 1.E400 are accepted as coordinates by all three. 1e6 is not a number in ISO 10303-21 at all — a REAL there must carry a decimal point — and 1.E400 is infinity, which swallows any bounding box computed from it. NAN gets through IfcOpenShell and web-ifc too; xBIM alone refuses it.
• All three accept entities from the wrong schema version, in both directions — IFCELECTRICALELEMENT (IFC2x3 only) under an IFC4 header, and IFCINDEXEDPOLYCURVE (IFC4+) under an IFC2X3 one. A header that says IFC4X3 is, in practice, not checked against the entities beneath it.

And the good news, which matters just as much: all three read a 587 km survey coordinate and a 17-significant-digit real without losing precision; xBIM and IfcOpenShell both catch every geometry WHERE-rule trap that web-ifc misses; and web-ifc is the only one that refuses the truncated file.

How scoring works

Every case states its ground truth, and every outcome lands in one of five buckets — because how a reader fails matters far more than whether it does:

Verdict	Meaning
✅ correct	Gave an accepted answer — or safely refused a file with no single valid reading
🚫 rejected	Refused a file that does have one. Honest, but a miss
☠️ silently wrong	Answered confidently from a file that does not support the answer
💥 crashed	Raised something that was not a refusal, or died
➖ n/a	Declined to answer this kind of question. Excluded from the score

Refusing a readable file costs an afternoon. Returning plausible geometry from a file that does not contain it costs rework on site, and it is the failure you cannot see. Those are never the same bucket.

Every case asks one probe — load, schema, count:<KEYWORD> or attr:#<id>.<n> — and every toolkit answers in one canonical line of text, so a Python reader and a WASM one can be compared at all. Reals are compared with a declared tolerance rather than by spelling. Full contract: docs/PROBES.md.

Three case flavours:

• expect = "<answer>" — one valid reading. Anything else is ☠️.
• expect = "reject" — no valid reading exists. Any answer is ☠️.
• expect = "ambiguous" + accepted = [...] — several defensible readings (a BOM'd file, raw UTF-8, (0,0,0) where REALs belong). Any accepted answer ✅, refusing ✅, anything else ☠️.

Scored as generously as each toolkit allows

A benchmark that measures the wrong thing is worse than none.

• IfcOpenShell's load column is ifcopenshell.validate(express_rules=True) — its own conformance checker with every EXPRESS WHERE rule enabled. A ☠️ there means the file passed its strictest setting, not that someone forgot to switch validation on.
• xBIM's load column is its own Validator at ValidationFlags.All, the .NET equivalent of the same setting.
• web-ifc ships no validator, so load means "OpenModel returned a model". Loading a file and offering geometry from it is a claim about that file; the column is labelled so nobody misreads it as a failed check.
• Abstentions are free. A toolkit with no geometry kernel is not punished for a question it cannot be asked — and no toolkit can raise its score by declining to look, because ➖ leaves the denominator entirely.

ifcref is the control, not a competitor

ifcref/ is a strict, zero-dependency ISO 10303-21 reader that lives in this repo. It is not evidence about anything — it is the demonstration that each case's ground truth is reachable by an ordinary reader with no geometry kernel and no C++ behind it. Read its column as "is this case answerable at all".

It also abstains rather than bluffs: its schema table covers a few dozen entities, and outside them it says "not in my table" instead of pronouncing a file invalid. That table is checked against the real EXPRESS schemas on every CI run — tests/test_schema_table.py found six genuine errors in it the first time it ran, including one that had already become a corpus case.

Run it yourself

pip install -e '.[toolkits,dev]'          # ifcref + IfcOpenShell
npm --prefix bridges/web-ifc install      # web-ifc bridge
dotnet build bridges/xbim -c Release      # xBIM bridge (.NET 8 SDK)

python -m ifctrap run

python -m ifctrap case refs/dangling-reference

python -m ifcref cases/geometry/zero-length-direction.ifc

python -m ifctrap adapters shows who is on the board and who is missing; python -m ifctrap update-readme regenerates the tables; pytest runs the corpus-hygiene and reader tests with no toolkit installed at all.

Toolkits are optional — a missing one shows as "not installed" rather than quietly disappearing and making the corpus look better covered than it is.

Contributing: one failing file is a PR

The contributor unit is one .ifc and one .toml:

cases/<category>/<slug>.ifc     the artefact, byte-exact
cases/<category>/<slug>.toml    what to ask it, and the right answer

probe = "load"
expect = "reject"
why = '''
One or two sentences defending the ground truth — and, ideally, naming what
goes wrong downstream when a reader gets this one wrong.
'''

Then python -m ifctrap case <category>/<slug> to see who fails it, and open the PR. Don't touch the tables; CI regenerates them after merge.

Adding a toolkit is a bridge: a program that takes a path and a probe and writes one JSON line. Any language — see docs/PROTOCOL.md; the web-ifc one is about a hundred lines of Node.

Full guide, including the ground-truth rules and how to dispute a verdict: CONTRIBUTING.md.

Wanted

Bridges for IFC++, IfcPlusPlus, pythonOCC, BIMserver, IfcSharp and That Open Engine; cases for IFC4X3 alignment geometry, IfcIndexedPolyCurve segment indices, complex (multi-inheritance) instances, and the \S\ / \P page-switching escapes nobody implements correctly.

MIT licensed. The corpus is meant to be argued with — if you think a verdict is wrong, the PR that changes it is the point, not an inconvenience.