Validate structured LLM outputs in Python: syntax, schema and evidence

A structured response needs several checks: the text must parse, the fields must satisfy the application's contract, and the values must be supported by the source. Valid JSON alone proves none of the latter requirements. Even a schema-conforming response can contain a confidently extracted wrong value.

We will validate invented outputs for the source sentence “Order: 2 notebooks.” The responses are offline fixtures, not recorded outputs from a particular language model. This keeps the boundary reproducible and lets us test failure handling before connecting any live API.

Specify the application contract

For this small example, the accepted object has exactly two fields: a nonempty item name of at most 60 characters, and an integer quantity between 1 and 100. Those are invented application requirements. They do not come from JSON itself.

We also limit input size before parsing. The validator is intentionally small enough to inspect; a larger application should use a maintained schema system with an explicit contract and equivalent tests.

import json

def reject_constant(value):
    raise ValueError("non-finite JSON constant")

def validate(raw):
    if len(raw.encode("utf-8")) > 1000:
        raise ValueError("response too large")
    record = json.loads(raw, parse_constant=reject_constant)
    if not isinstance(record, dict) or set(record) != {"item", "quantity"}:
        raise ValueError("unexpected fields")
    if not isinstance(record["item"], str) or not 1 <= len(record["item"].strip()) <= 60:
        raise ValueError("invalid item")
    if type(record["quantity"]) is not int or not 1 <= record["quantity"] <= 100:
        raise ValueError("invalid quantity")
    return record

expected = {"item": "notebooks", "quantity": 2}
candidates = [
    '{"item":"notebooks","quantity":2}',
    '{"item":"notebooks","quantity":"two"}',
    '{"item":"notebooks","quantity":true}',
    '{"item":"notebooks","quantity":20}',
    '{not json}',
]
for index, raw in enumerate(candidates):
    try:
        record = validate(raw)
    except ValueError:
        print(index, "parse or contract failed")
        continue
    print(index, "accepted" if record == expected else "source check failed")

Expected output:

0 accepted
1 parse or contract failed
2 parse or contract failed
3 source check failed
4 parse or contract failed

Case 3 is the important boundary: 20 is a valid integer in the permitted range, but it contradicts the source sentence. Stronger structural guarantees would not correct that factual error by themselves.

Keep parsing, contract and evidence checks distinct

Parsing establishes a data structure from text. Contract validation checks the shape and allowed values. Evidence checking asks whether the extraction matches the source. In this example a human-written expected record supplies the answer key. We have not automated understanding of arbitrary source text.

The exact equality check is appropriate for these fixed fixtures. A real extraction task may require normalization, evidence spans, field-level labels and adjudication of ambiguous inputs. Design those checks around the application rather than treating this tiny comparator as a universal factuality evaluator.

Be deliberate about types and parser behavior

The quantity check uses type(value) is int. In Python, bool is a subclass of int, so an ordinary isinstance(value, int) check would also accept True. JSON booleans and numeric quantities have different meanings in this contract.

Python's JSON decoder can also accept non-finite constants by default; our callback rejects them. Duplicate object keys require a separate policy: the default decoder keeps the final value. This example does not detect duplicates, so a stricter production contract should use an appropriate pairs hook or parser configuration if duplicates must be rejected. The Python JSON reference documents these behaviors.

Avoid repairing malformed outputs with broad string substitutions that silently change meaning. Code-fence stripping, coercing strings into numbers and filling missing fields all change the accepted language. If such normalization is intentional, define and test it as its own step.

Handle failures without hiding them

Record which stage failed and retain suitable diagnostic information without logging sensitive source material unnecessarily. A bounded retry can ask for a corrected structure, but repeated retries consume resources and do not independently verify correctness. Keep an explicit failure or review outcome when the application cannot establish a valid result.

Parsing and validation also do not authorize external actions. A schema-valid destination, identifier or tool argument still needs the application's ordinary permission and business-rule checks. Treat model-produced data as a proposal entering those existing boundaries.

Extend the fixture set

Add missing fields, extra fields, a zero quantity, a floating-point quantity, an empty item, an oversized response and duplicate keys. Decide the intended outcome for every case before modifying the validator. Include ambiguous or unanswerable source sentences in an actual extraction evaluation, rather than assuming every request has a clean answer.

Use RAG retrieval evaluation to check whether the source evidence reaches a system in the first place. The Generative AI resources connect these separate checks into a reliable application workflow.