The first call to the new API fails with a 400. The documentation never mentioned that field. The error response isn't documented either, so you can't tell whether to retry, fix the request, or give up and write to support.
Most of us have lived some version of that. It's rarely a coding problem. It's a documentation problem, and it's expensive: it's the gap between "the API exists" and "someone integrated it".
For the past months I've been building a tool that reviews API documentation for exactly these gaps. This article is about the most useful thing I've done with it so far, which was to try to prove it unnecessary.
What would make me stop
A test only counts if it can come out against you. Before the last run, I wrote down what would make me stop building this:
- An AI assistant that is given only the titles of my 41 rules agrees with the tool on 85% or more of them. Then the rule list is not the product.
- My own rules are wrong more than 15% of the time on the findings I have labelled. Then "no noise" is not true.
- On large files, a prompted assistant finds everything the tool finds. Then the tool's only edge is something other than recall.
I did not say whether "agrees on 85%" meant the average of all runs or the best single run. I'll report both, and I'll say at the end which of the three happened.
What the research says about API documentation
None of this is new. In a field study of more than 440 professional developers at Microsoft, Robillard and DeLine found that documentation and learning resources were a major obstacle to learning an API [1]. Uddin and Robillard later surveyed 323 developers and analysed 179 documentation units, and named ten recurring problems. The three they rated most severe were ambiguity, incompleteness and incorrectness, and developers described several of the problems as blockers that made them switch to a different API [2].
Others have tried to detect documentation problems automatically, for example classifiers for "documentation smells" in Java API prose [3]. The failures I care about in OpenAPI files are different: a response that says it returns JSON but never says what's in it, an example that contradicts its own schema, an endpoint that lists no errors at all.
What I built
The tool reads an OpenAPI or Swagger file (and, optionally, the developer guide) and applies 41 fixed rules. There is no language model in it. Every finding points at the exact place in your file, says what was looked for and not found, explains the impact in a sentence, and suggests a fix. This is what one looks like, trimmed:
CC-ERR-01 GET /api/fulfilment/v1/shipping-policies/{id}
documents no error responses
State: not in the specification
Evidence: not found: an error response (4xx, 5xx or default)
Impact: Developers cannot tell which failures to expect,
so error paths are discovered in production.Two design choices matter for what follows. Findings with the same cause are grouped, so 53 operations with no error responses are one problem with 53 places. And when a check can't be made, because it needs a guide you didn't supply, the report says "could not be assessed". It is never counted as a pass.
You run it like this:
irr review api.yaml --out report/ # report, printable teardown, SARIF
irr diff report-old/report.json report/report.jsonThe question I couldn't avoid
Everyone who has used an AI assistant will ask: why not paste the spec into a chat and ask it to find the gaps? It's a fair question and I didn't know the answer. So I tested it.
Test one: a small file, 35 planted faults
I took one small public specification and planted 35 faults, each the kind of thing one of my rules is meant to catch: a removed retry instruction, a response with no description, a webhook with no signature advice. I wrote the faults to match my rules, so this is the easy case for my tool. Then I gave copies to three readers, one fault per copy:
- my rules fired the right rule on all 35, and pointed at the changed place on 27 of them;
- an AI assistant, given one file and a decent prompt, found 34 of 35 (I scored that by reading its answers);
- Spectral, a widely used OpenAPI linter, with its recommended ruleset, reported something for the same reason on 8, reported something unrelated on 8 more, and said nothing on 19. It is built to check structure and to be extended with your own rules, so this is not the job it was made for.
Show the numbers
| Reader | Detail | Count |
|---|---|---|
| Rule engine | right fault, at the changed place | 27 |
| Rule engine | right rule, elsewhere in the file | 8 |
| AI assistant | found | 34 |
| AI assistant | missed | 1 |
| Spectral | flagged for the right reason | 8 |
| Spectral | flagged for another reason | 8 |
| Spectral | nothing flagged | 19 |
That is the result I had to sit with. On a small file, with a well-written prompt, an AI assistant that simply reads does almost exactly what my rules do.
Test two: when the file stops fitting in your head
Small files are the easy case. Real specifications run to megabytes. I picked two large public ones, where a script I wrote, independent of both tools, had found invalid examples: places where the example in the documentation does not match its own schema.
This is the whole check, which is the point. The field below comes from the sample report on this site, not from the two large files:
from jsonschema import Draft202012Validator
schema = {"type": "object",
"properties": {"absoluteMoneyCost": {"type": "number"}}}
example = {"absoluteMoneyCost": "1.00"} # a string, not a number
for err in Draft202012Validator(schema).iter_errors(example):
print(err.json_path, err.message)
# $.absoluteMoneyCost '1.00' is not of type 'number'An assistant that read the two files found 6 of the 78 invalid examples. It said honestly which parts it had not read. Given tools and no instruction, it did better on one file (19 of 48) and found none on the other. Then I added one sentence to the prompt:
Validate every example in the document against the schema it belongs to, with a real JSON Schema validator run by a script you write; cover the whole document, not a sample.
With that, it found all 78, twice each, in about two minutes per run. My engine finds the same 78 in seconds without a model.
Show the numbers
| Condition | Large spec A (of 48) | Large spec B (of 30) | Total (of 78) |
|---|---|---|---|
| Read only | 6 | 0 | 6 |
| Tools allowed, no instruction | 19 | 0 | 19 |
| Told to validate every example | 48 | 30 | 78 |
| Rule engine | 48 | 30 | 78 |
The engine did not start there. My first version missed some of these: a schema written in an older JSON Schema style it silently skipped, schemas above a size limit, and examples on nested properties. I fixed those, and it now reports any example it could not check instead of skipping it quietly. On one of the large files it still counts 2 examples it did not check, because they are too long to keep.
Why does reading degrade? Liu and colleagues showed that language models use information less reliably when it sits in the middle of a long input [4]. I didn't test that mechanism; I only observed that reading failed on large files and computing did not.
The lesson is not that the assistant is bad. It's that what decided the result was knowing what to check, and asking for it. That is a rubric, and a rubric is a prompt.
Test three: is the rule list the secret sauce?
I gave an AI agent only the titles of my 41 rules ("Resource-creating POST has no idempotency key") as a checklist, on three real specifications, two runs each. I ran it twice, and the first time was not clean. The agents ran inside the project folder that builds the tool. I told them to read only their file, but they could in principle have read my rule source and notes. So I reran it from an empty folder: no project files, no notes, the three specifications and the 41 titles only. I checked every transcript afterwards. No run touched anything outside that folder, and none mentioned the project. One caveat: the clean folder still ran with my usual Claude settings and plugins, though no run used them.
The agents agreed with the engine on 79.3% of the rules on average in the clean runs, and on 82.9% in the earlier runs, scored against the same current engine. Single runs ranged from 30 to 36 of 41. The leak did not inflate the result by more than the run-to-run noise.
Show the numbers
| Specification | Run | Earlier runs | Clean runs |
|---|---|---|---|
| Spec 1 | 1 | 31 | 30 |
| Spec 1 | 2 | 34 | 30 |
| Spec 2 | 1 | 35 | 33 |
| Spec 2 | 2 | 36 | 36 |
| Spec 3 | 1 | 34 | 33 |
| Spec 3 | 2 | 34 | 33 |
Agreement is closest on the structural rules and furthest on judgement calls. 19 of 41 rules agree with the engine in all six clean runs.
Show the numbers
| Rule | Runs (of 6) that agree with the engine |
|---|---|
| AUTH-01 | 6 |
| AUTH-02 | 6 |
| AUTH-03 | 4 |
| AUTH-04 | 4 |
| CC-CONFLICT-01 | 4 |
| CC-CONSIST-01 | 4 |
| CC-DEP-01 | 5 |
| CC-DESC-01 | 6 |
| CC-DESC-02 | 3 |
| CC-DESC-03 | 0 |
| CC-DESC-04 | 4 |
| CC-ENV-01 | 6 |
| CC-ERR-01 | 6 |
| CC-EX-01 | 6 |
| CC-EX-02 | 4 |
| CC-PARAM-01 | 5 |
| CC-PATH-01 | 6 |
| CC-REF-01 | 6 |
| CC-RESP-01 | 6 |
| CC-SCHEMA-01 | 6 |
| CC-SEC-01 | 6 |
| CC-TYPE-01 | 6 |
| DATA-02 | 5 |
| DATA-03 | 5 |
| DOC-CODE-01 | 4 |
| ERR-02 | 2 |
| ERR-04 | 4 |
| ERR-05 | 6 |
| IDEM-01 | 6 |
| IDEM-02 | 6 |
| LINK-01 | 2 |
| NAME-01 | 2 |
| ON-ENV-01 | 6 |
| PAG-01 | 4 |
| PAG-02 | 4 |
| RATE-01 | 4 |
| RATE-02 | 6 |
| WF-04 | 4 |
| WH-01 | 6 |
| WH-02 | 6 |
| WH-03 | 4 |
The rules where they split include weak descriptions, vague wording and naming, which are judgement, and a missing guide, which an assistant will flag and the engine can only mark "could not be assessed". Anyone who can read a list of rule titles can reproduce most of the tool.
What did differ
Not recall. What differed was everything around it:
- Repeatability. The same agent on the same file produced different numbers of findings from run to run (97 and 77 on one file, 61 and 85 on another), and not just different numbers: the two runs named the same places only about half to three-quarters of the time. That isn't surprising: Atil and colleagues found that models configured to be deterministic still gave different accuracy across repeated runs, by up to 15% on some of their tasks [5]. My rules give the same report every time, so a re-run after a fix is a fair comparison.
- Coverage statements. The assistants usually said what they hadn't read. My tool says what it couldn't check as part of every report, in the same format.
- Cost on large files. Roughly 60,000 to 70,000 tokens and a couple of minutes per run for the agent, against seconds and no model for the engine.
Show the numbers
| Specification | Findings (run 1, run 2) | Places (run 1, run 2) | Shared | Overlap |
|---|---|---|---|---|
| Large spec A | 97, 77 | 68, 55 | 53 | 0.76 |
| Large spec B | 61, 85 | 52, 69 | 39 | 0.48 |
Where my tool is wrong
I have been hard on the assistants' noise, so here is mine. I labelled the findings of the rules I could measure, as the only labeller. The result so far: 30 of the 41 rules have at least one judged finding. Across the 28 of those that are switched on by default, 4.1% of the judged findings were wrong. That is under the 15% line I set myself, but it hides a spread.
Show the numbers
| Rule | Correct | Wrong | Not sure | Out of date | Switched off by default |
|---|---|---|---|---|---|
| PAG-02 | 0 | 3 | 0 | 0 | yes |
| AUTH-03 | 0 | 1 | 1 | 0 | |
| CC-DESC-03 | 2 | 3 | 0 | 20 | yes |
| CC-CONSIST-01 | 2 | 1 | 0 | 0 | |
| CC-DESC-02 | 18 | 7 | 0 | 0 | |
| AUTH-01 | 2 | 0 | 0 | 0 | |
| AUTH-02 | 8 | 0 | 0 | 0 | |
| AUTH-04 | 1 | 0 | 0 | 0 | |
| CC-DEP-01 | 10 | 0 | 0 | 0 | |
| CC-DESC-04 | 23 | 0 | 0 | 0 | |
| CC-ERR-01 | 10 | 0 | 0 | 0 | |
| CC-EX-01 | 10 | 0 | 0 | 0 | |
| CC-EX-02 | 10 | 0 | 0 | 0 | |
| CC-PARAM-01 | 10 | 0 | 0 | 0 | |
| CC-SCHEMA-01 | 2 | 0 | 0 | 0 | |
| DATA-02 | 3 | 0 | 0 | 0 | |
| DOC-CODE-01 | 31 | 0 | 0 | 0 | |
| ERR-02 | 1 | 0 | 0 | 1 | |
| ERR-04 | 4 | 0 | 0 | 0 | |
| ERR-05 | 4 | 0 | 0 | 0 | |
| IDEM-01 | 7 | 0 | 0 | 0 | |
| LINK-01 | 8 | 0 | 0 | 18 | |
| NAME-01 | 3 | 0 | 0 | 0 | |
| ON-ENV-01 | 17 | 0 | 0 | 1 | |
| PAG-01 | 10 | 0 | 0 | 0 | |
| RATE-01 | 2 | 0 | 0 | 0 | |
| RATE-02 | 1 | 0 | 0 | 0 | |
| WF-04 | 10 | 0 | 0 | 0 | |
| WH-02 | 3 | 0 | 0 | 0 | |
| WH-03 | 3 | 0 | 0 | 0 |
- Three rules are above the line, one of them with real volume. CC-DESC-02, which flags weak descriptions, was wrong on 7 of 25 judged findings (28%). The other two, CC-CONSIST-01 and AUTH-03, have only 3 and 1 judged findings between them.
- Most measurements are thin. 19 of the 30 measured rules have fewer than 10 judged findings, and 11 rules have none.
- Two rules are off by default. One of them was right on 2 of 5 of its current judged findings, the other on 0 of 3.
- The rule-list test found a real defect. One rule counted identical error bodies behind shared response definitions as different shapes, and so overstated a problem on one large file. I found it because an assistant disagreed with the engine, checked, and the assistant was right. I fixed it the same day.
What happened to my three criteria
- Agreement of 85% or more: not by the average, once by the best run. The mean in the clean runs was 79.3%. One of the six clean runs reached 36 of 41, which is 87.8%, and so did two of the earlier runs (35 and 36 of 41). Because I had not fixed the wording, I'm reporting both. My reading is that the rule list is replicable to roughly 80%, and close to complete on the structural rules.
- Wrong more than 15%: not overall, yes for three rules. 4.1% on the rules that are on by default, but one rule at 28% and two with almost nothing judged, and the caveats above.
- A prompted assistant finds everything: yes, for example validity. Once told to compute, it found all 78, as my engine did. If recall on that check is what you need, the assistant is enough.
What I take from it
Recall on a file you can read, or on a check an assistant is told to compute, is not an advantage. I stopped treating the rule list as the product. What is left is narrower than I started with, and I think it is real:
- It computes what an assistant has to be told to compute, without being told, on large files, and without a model.
- It gives the same report every time, so a before-and-after comparison means something. On the sample on this site, one revision of the specification left 56 findings resolved, 71 still there and 1 new, and every one of those is a place you can open.
- It says what it did not check, and it says how often it is wrong. I don't get either of those from a chat answer unless I ask.
For judgement calls such as weak descriptions or naming, the assistants added things my rules don't. So I'd use both, in this order:
Show the numbers
| Step | Who | What for |
|---|---|---|
| 1 | Rule engine | Repeatable checks, an explicit list of what was not checked |
| 2 | Assistant | Judgement the rules do not make; ask what it validated by code |
| 3 | You | Decide, fix, re-run and compare |
If you do use an assistant to review your docs, ask it two things: which examples did you validate with code, and what did you not read?
What this does not show
I want to be as clear about the limits as about the results.
- The faults are mine. I wrote them to be found by my rules, on one small specification. 35 of 35 is a floor on a test I designed, not a measure of what real documentation gets wrong.
- Precision wasn't measured for the assistant. On a clean copy it raised 21 to 24 findings where my tool raised 6. How many of those were real and worth fixing needs a second person to judge, and I haven't had that.
- One model, few runs. Two runs per specification, six in each arm of Test three; one AI model (Claude) and one prompt.
- One labeller. I built the scorer and did the labelling, for the assistants and for my own rules. The agreement figures are agreement, not accuracy.
- Only one computed check was tested at scale. Example validity. Other checks weren't tested the same way.
- The clean rerun is not perfectly clean. The model has seen public API specifications and write-ups like mine in training, and the clean folder still ran with my usual settings and plugins.
If you want your API reviewed
I'm offering a free first review to a few organisations. Send an OpenAPI file or the address of your docs and I'll run it and send back the report, the teardown and what I'd fix first. I reply within one working day. It is also the next version of this test: if the report holds nothing your own assistant would not have told you, say so and I'll want to know. Ask here: apireview.dataharmoni.com/contact.
I've put the real sample outputs (a report, a printable teardown and a before-and-after diff, all on an anonymised public specification) and the full numbers with their limits on the product site.
Disclosure: I built the reviewer; this article was drafted with AI assistance and edited by me; the assistants in the study were Claude models.
References
- Robillard, M. P., and DeLine, R. (2011). A field study of API learning obstacles. Empirical Software Engineering 16, 703–732. Paper
- Uddin, G., and Robillard, M. P. (2015). How API documentation fails. IEEE Software 32(4), 68–75. DOI 10.1109/MS.2014.80. Author copy
- Khan, J. Y., Khondaker, M. T. I., Uddin, G., and Iqbal, A. (2021). Automatic detection of five API documentation smells: practitioners' perspectives. SANER 2021. arXiv:2102.08486
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the middle: how language models use long contexts. Transactions of the Association for Computational Linguistics 12, 157–173. ACL Anthology
- Atil, B., et al. Non-determinism of "deterministic" LLM settings. arXiv:2408.04667