We tested our own tool against a linter and an AI assistant. Here is everything, limits included.

The short version

• On a small specification, an AI assistant that simply reads the file finds almost everything our rules find.
• On very large specifications, reading stops working. Computation does not. An assistant matches the reviewer only when it is told exactly what to compute.
• Our rule list is not a secret. Handed only the rule titles, an AI agent agreed with the reviewer on 73 to 88 percent of rules per run.
• Where the reviewer is different: it gives the same answer every time, finishes large files in seconds without a model, and says what it did not check.

What we ran

Test	Result
35 faults seeded into one small specification, each written to be caught by one rule	Rules: 35 of 35 caught, 27 of them pointing at the changed place. A popular linter: 8 reported for the same reason, 8 reported something else, 19 reported nothing. An AI assistant reading the file: 34 of 35. Each assistant run used a fresh copy of the file and the same prompt.
Invalid examples across 16 real public specifications, checked by an independent script	The script found 99 examples that break their own schema. The reviewer flags 95, all confirmed; the linter's example claims about 88. Four are missed by both.
Two very large specifications, an assistant that only reads	Found 6 of 78 invalid examples (6 of 48 and 0 of 30). It said which parts it had not read.
The same two, with an assistant told to validate every example with a script	Found all 78, twice each. Without that instruction, but with tools, it found 19 of 48 and 0 of 30.
Three real specifications, an AI agent given only our rule titles as a checklist	Agreed with the reviewer on 30 to 36 of 41 rules per run. Closest on structural rules, furthest on judgement calls.
About 1,800 assistant findings, mechanically checked	Pointers to the file were almost always real. Few claims could be checked by a script (under 5 percent), and those that could were not clearly wrong, except where the assistant had read instead of computed.

What these numbers do not show

• The faults are ours. We wrote them to be found by our rules, on one small specification. Treat 35 of 35 as a floor on a test we designed, not as recall on faults documentation really has.
• Precision was not measured for the assistant. On a clean copy it raised 21 to 24 findings where the reviewer raised 6. How many of those were real and worth fixing needs a second person, and we have not done that.
• One model, small runs. Mostly one run per cell, one AI model, one prompt. The open-ended part of an assistant's answer varied from run to run.
• One labeller. The person who built the scorer also did the labelling. The agreement figures are agreement, not accuracy: there is no ground truth for judgement calls.
• Possible leakage. The assistants ran inside the project that builds the reviewer and could in principle have picked up context about it. They were told to read only their file.
• The example check is the only computed check we tested at scale. Other computed checks (references, structure, consistency across schemas) were not tested the same way.

What follows from it

The finding that a prompt can reproduce most of our rules is the useful one, and it shaped what we are building. Recall on a small file is not a moat. What an assistant cannot cheaply give you is repeatability, a complete pass over a very large file, and an explicit list of what was not checked. Use both if you like. Just ask whichever you use to tell you what it did not read.

The write-up, in plain language, is in our notes.