On this page 12 sections
Use your provider’s schema-constrained output if it has one - it makes malformed JSON close to impossible. Then parse permissively, validate strictly against your own schema, and treat a validation failure as a routing decision rather than a retry. The remaining problem is not malformed output. It is a perfectly valid object containing a fabricated value, and no parser catches that.
Format reliability is largely solved. Semantic reliability is not, and teams routinely fix the first and assume they have fixed the second.
Key takeaways
- Schema-constrained decoding removes most format failures. Use it where available.
- Parse permissively, validate strictly. Tolerate cosmetic drift, reject semantic problems.
- Enums are the highest-value schema feature - they eliminate a whole class of downstream branching.
- A field that must be nullable should be nullable, or the model invents a value.
- Validity is not correctness. Validate against your systems, not just against a schema.
Who this applies to
You are getting structured data out of a model - extraction, classification, tool arguments, anything a downstream system consumes.
Use constrained decoding
Most major providers now offer some form of guaranteed-schema output, where the decoder is constrained so the result conforms by construction.
Use it. It is a configuration change, it removes almost all malformed-JSON failures, and it eliminates the retry loop that most teams build to work around them.
Two caveats worth knowing. Constrained decoding can slightly change output quality on some tasks, because the model’s generation is being restricted - measure it rather than assuming it is free. And a heavily nested schema is harder for a model to fill sensibly than a flat one, so simplify the shape before blaming the feature.
Where it is unavailable, ask for JSON in the prompt, provide an example of the exact shape, and rely on the parse-and-validate layer below.
Parse permissively, validate strictly
Two layers, and the distinction is the whole design.
Parse permissively. Strip code fences, tolerate a sentence of preamble, find the JSON object within a larger string. These are cosmetic variations, they drift between model versions, and none of them indicates anything wrong with the content. A parser that throws on a code fence creates failures out of nothing - a common source of the “it broke and nothing changed” reports after a provider update.
Validate strictly. Once parsed, check it hard against your own schema: required fields present, types correct, enums within the allowed set, numbers in plausible ranges, dates parseable. Fail loudly here.
The asymmetry matters. Permissive parsing absorbs meaningless variation; strict validation ensures a genuine problem is caught rather than passed downstream as an empty string.
Design the schema for the model
A schema written for a database is often a poor prompt-side contract.
Use enums wherever the value set is closed. "status": "shipped" | "pending" | "cancelled" rather than a free string. This is the single highest-value schema feature: it removes the downstream normalisation code that otherwise accumulates around every free-text field.
Make optional things nullable and say so. If a field may legitimately be absent, allow null and describe when. Otherwise the model, required to produce a value, invents one. A large share of “hallucinated” extraction is a schema demanding a field the document does not contain.
Keep it flat. Deeply nested structures are filled less reliably. Two shallow calls often beat one deep one. The same argument applies a level up, where the object being passed is a handoff between agents - see multi-agent systems: when the complexity pays.
Name fields the way the domain does. invoice_total is filled more accurately than amt_2. The field name is part of the instruction.
Add a confidence or a source field where it helps. Asking for the text span a value came from makes the extraction checkable, and it is often the cheapest verification available.
Handle failure as routing, not retrying
When validation fails, the instinct is to retry. Usually wrong.
If the model produced an invalid enum or omitted a required field, the same request will often produce the same problem, and you have paid twice. Retries make sense for genuinely transient issues, not for a structural mismatch.
Better: route it. A validation failure means this input was not handled correctly, so it goes to the human review queue with the raw output attached. That is the same dead-letter pattern as in n8n error handling, and the queue is also your best source of new test cases.
One retry with the validation error fed back is a reasonable middle ground - it fixes a real subset of cases. Beyond one, you are queueing failure.
The failure that matters
A valid object with wrong values.
The schema is satisfied. Types are right, enums are in range, dates parse. And the invoice total is 1,240.00 when the document says 12,400.00, because a decimal was misread. No parser catches this. No schema catches this. It flows into your ledger.
This is why format reliability is the easy half. The controls that catch semantic errors are different:
- Cross-field arithmetic. Line items sum to the subtotal, tax computes correctly. Consistency is checkable without knowing the right answer.
- Range checks against history. A value three orders of magnitude outside the normal range for this field is almost certainly wrong.
- Lookups against your systems. Does the supplier exist, does the order match, do the bank details match what you hold.
- The source span. If you asked the model for the text it extracted from, check the value actually appears there.
Covered in more depth in document extraction: what to verify. The point here is that reaching valid JSON reliably can feel like the problem is solved, and it is the point at which the real validation work starts.
Testing it
Two separate metrics, because they have different fixes.
Format validity rate. Share of responses that parse and validate. With constrained decoding this should be very close to 100%, and anything else points at a schema the model cannot fill.
Field-level accuracy. Per field, how often the value is correct. This is the number that matters and it is always lower. Report per field - averages hide the one field failing often, which is usually the one that costs money.
Include cases in your test set where a field legitimately should be null. A system that never returns null is inventing values, and you will only see it if you test for it.
What we build
Constrained decoding where the provider supports it, permissive parsing with strict validation, enums on every closed set, explicit nullability, and validation failures routed to review rather than retried.
The design rule we hold most firmly is explicit nullability, because it is where the most damaging extraction errors come from and it looks like a minor schema detail. A required field the source does not contain is a request to invent something, and models comply. Making it nullable and describing when null is correct converts a silent fabrication into an honest gap, and an honest gap can be routed to a person.
The thing we spend time on that clients sometimes see as excessive: asking for the source span alongside the value. It costs a few output tokens per field and it turns “the model said 12,400” into “the model said 12,400 and here is the text it read”. That is the difference between an extraction you can audit and one you have to trust.
Where we would push back on our own defaults: constrained decoding is not free on every task, and on a couple of extraction jobs we have seen slightly better quality from a well-prompted unconstrained call with strong validation behind it. Measure it on your own task rather than adopting either as doctrine.
When you do not need this
When the output is prose for a human. Do not force structure onto something a person reads.
When one field is enough. Asking for a single classification label does not need a schema layer - validate the enum and move on.
When the model is not in the loop. A surprising amount of “AI extraction” is regular parsing with a model added unnecessarily. If the input format is stable, parse it.
Frequently asked questions
Is constrained output slower?
Marginally in some implementations, and usually irrelevant against the retries it removes.
Should we use JSON or something else?
JSON, because it is what providers constrain against and what every downstream system reads. The format is not where your problems are.
How do we handle a field that is sometimes a number and sometimes a range?
Model it explicitly - a nullable number plus a nullable range object, or a discriminated union. Do not accept a free string and parse it downstream; that pushes the ambiguity into code that has less context.
What if the model returns valid JSON with a fabricated value?
That is the central problem here, and no schema mechanism addresses it. Use arithmetic reconciliation, range checks, lookups against your systems, and source spans.
Should the schema live with the prompt?
Yes, versioned together, because they change together. A schema change without a prompt change is a common source of quiet field-level regressions.
Next step
If your pipeline validates format but not values, the gap is where the expensive errors live. The custom AI development engagement builds extraction with schema constraints and semantic validation against your own systems.
Related: Document extraction: what to verify · Tool-calling agents: what to let them touch · n8n error handling · Custom AI development