Output formatting and JSON mode: function calling, Pydantic, Instructor, Outlines

What is structured output formatting?

Structured output formatting forces a language model to produce its answer in a machine-readable schema – typically JSON, sometimes XML or a specific data class. Instead of free text "The client is Müller, has 1500 CHF due on 15.06.2026" the model returns `{"client": "Müller", "amount": 1500, "due_date": "2026-06-15"}`.

The difference sounds small but is fundamental. Free text must be parsed by fragile string parsers or a second model call. JSON output can feed directly into a database, an API endpoint or a bookkeeping system. As of May 2026 structured output is a must-have pattern for every productive LLM integration into business processes.

Multiple techniques compete. First: JSON mode – the provider guarantees the answer is syntactically valid JSON (OpenAI, Anthropic, Mistral have offered it since 2024). Second: function calling or tool use – the model is instructed to "call" a function with defined parameters, effectively enforcing the schema. Third: Pydantic output parsing – a Python library validates output against a Pydantic model and forces a repair on errors. Fourth: Instructor – a wrapper library embedding Pydantic models directly into OpenAI/Anthropic API calls. Fifth: Outlines – a library for local models (Llama, Mistral) enforcing constrained decoding at the token level.

As of May 2026 the major providers support structured output natively: GPT-4.1 has `response_format` with JSON-Schema enforcement, the current top Claude model has `tool_use` with strict schema, Mistral has `function_calling` since 2024. Local models can be forced into structured output via Outlines, LMQL or llama.cpp grammars.

Why it matters

Without structured outputs, LLM integration is a tinkering project. An SME fiduciary building AI receipt capture needs reliably `{"date": "...", "amount": 0.0, "vat_rate": 7.7, "category": "..."}` – not "The receipt from xx.xx.xxxx totals about ...". Without schema guarantee every second receipt fails on a parse error.

Production pipeline stability: before JSON mode (pre-mid-2024) parse error rates ran 15-25% depending on output complexity. With OpenAI structured outputs as of May 2026: < 0.1%. That is the difference between "we can deploy this for clients" and "we redo 30% manually".

Data integrity on mandatory fields: when a receipt requires a VAT number and the model cannot show one, the system should return a clear error message, not an invented VAT number. Structured output with required fields plus Pydantic validation enforces that.

Cost and latency: models with strictly enforced schemas often produce shorter outputs (less verbosity) and finish faster. In our fiduciary pipelines output token usage drops 30-50% compared to free-text answers.

Error handling: when a mandatory field cannot be filled (e.g. uncertain date), the schema can include a `confidence` score and an `unsure_fields` array. The pipeline then knows exactly where human review is needed.

Compliance: for automated individual decisions reproducibility matters. Structured outputs are easy to write into audit logs and trace who decided what when. Free-text answers fit this poorly.

How it works – methods and tools May 2026

OpenAI Structured Outputs. Since November 2024 OpenAI guarantees 100% schema conformity with `response_format: {"type": "json_schema", "json_schema": ...}`. The model is constrained at the token level – tokens that would violate the schema are not even sampled. As of May 2026 with GPT-4.1: schema validation 100% reliable, performance hit under 5%.

Claude Tool Use. Anthropic does not have a direct "JSON mode", but `tool_use` with `input_schema` serves the same purpose. You define a tool with JSON schema, the model must call it with validly formatted parameters. As of May 2026 with Claude Opus: high schema conformity (98-99%), rare cases where the model refuses the tool and returns text instead.

Mistral Function Calling. Mistral Large 2.1 and Mistral Small 3 support function calling in OpenAI-compatible format. Schema conformity 95-97%, slightly weaker than OpenAI/Anthropic, but EU-hosted (France).

Pydantic + output parser. Python pattern: you define a Pydantic model (`class Receipt(BaseModel): date: date; amount: float; vat_rate: float`) and use it as schema source. Without strict JSON mode, Pydantic parses model output, validates, and raises `ValidationError` on failure. Your pipeline triggers repair logic or human review.

Instructor (jxnl/instructor). Open-source library embedding Pydantic models directly into OpenAI and Anthropic API calls. Very popular as of May 2026 (over 8000 GitHub stars). Three lines of code: `client.chat.completions.create(model="gpt-current", response_model=Receipt, messages=[...])`. We use Instructor in nearly every Fairlane project for receipt parsing, dunning classification, client routing.

Outlines (dottxt-ai/outlines). Library for local models. Constrained decoding at the token level enforces regex, JSON Schema or CFG (context-free grammars). Works with Llama, Mistral, Qwen locally via Hugging Face. As of May 2026 the default solution for EU-sovereignty setups with a local model.

LMQL (Language Model Query Language). ETH Zurich. Domain-specific language for prompts with built-in constraints. More research-oriented but powerful – e.g. "answer must have exactly 3 bullet points, each between 10 and 30 words".

llama.cpp grammars. When running llama.cpp locally, you can enforce output via GBNF (Grammar in Backus-Naur Form). Efficient, low-level, very fast. Not for fast prototyping but good for production setups.

Practical May 2026 recommendation. For OpenAI: native structured outputs. For Anthropic: tool_use with strict input_schema. For Mistral: function_calling. Wrapper layer: Instructor in Python, or your own Pydantic adapter in TypeScript (e.g. zod-to-json-schema). For local models: Outlines.

Introduce structured output in 6 steps

1. 01Define the schema with Pydantic (Python) or Zod (TypeScript): required fields, optional fields, types, validations.
2. 02Choose the API method: OpenAI structured_outputs, Claude tool_use, Mistral function_calling, or Instructor wrapper.
3. 03Add confidence and unsure fields: `confidence_score: float`, `unsure_fields: list[str]` as human-review triggers.
4. 04Cover edge cases in the test set: empty fields, ambiguous inputs, wrong formats – the schema must handle them.
5. 05Fallback logic for validation errors: repair call (model sees the error and corrects) or human-review escalation.
6. 06CI test: Pydantic validation against 100+ test outputs, schema conformity >= 99% as pass threshold.

When structured outputs make sense

In every use case where the output flows into a data system, structured outputs are mandatory. Examples:

Receipt capture (date, amount, VAT, category → accounting DB), client mail classification (priority, category, case worker → workflow system), contract clause extraction (clause type, risk score, recommendation → contract reviewer), dunning tier determination (tier 1/2/3, tone, due date → dunning system), lead scoring (score 0-100, reasoning, next action → CRM).

For function-calling setups (agent architectures): always structured outputs because the tool schema demands it.

For audit-trail requirements (Art. 957a CO, EU AI Act): structured outputs are a precondition because audit logs must be machine-readable.

For multilingual pipelines: the schema can be language-independent (fields in English), the values language-specific (content in DE/FR/IT). Eases cross-language analysis.

For A/B-testing different prompts: comparable output structures allow automatic diff testing.

For JSON mode specifically: everywhere the answer lands in a database, API or structured bookkeeping system. As of May 2026 that is the majority of all productive SME LLM pipelines.

When free text is better

Pure generation tasks without machine-to-machine processing benefit from free text. Example: an AI-generated draft letter to a client. Here natural language should flow, not be squeezed into fields.

Creative tasks (slogan generation, image concepts, narrative summaries) need free text. Forcing a slogan into JSON yields worse slogans.

Beware of overly strict schemas: an extremely detailed schema with 50 fields, 80% of them optional, can overwhelm the model. It forgets fields, hallucinates values or refuses the answer. Rule of thumb: a schema with over 15 fields needs hierarchical structure (nested objects) or must be split across multiple LLM calls.

Also problematic: fields that are "almost always empty". When a field is null in 95% of cases, the model often hallucinates something into it. Fix: mark the field as optional or work with an explicit `is_missing` Boolean flag.

Cost reasons: for very simple tasks (e.g. "is this text German or French?"), a fully structured schema is overhead. A simple Boolean return or one-word output suffices.

Trade-offs

FAQ

Related topics

Sources

1. OpenAI – Structured Outputs (response_format json_schema) · 2026-04
2. Anthropic – Tool Use and Strict Input Schemas (the current top Claude model) · 2026-04
3. Mistral – Function Calling Guide · 2026-03
4. Instructor – Pydantic-first LLM library (docs) · 2026-05
5. Outlines – Structured generation for local LLMs · 2026-05
6. ETH Zurich – LMQL: Language Model Query Language · 2026-02