Structured Output: JSON Schema, Tool Use and Production Prompt Design

Free-form prompting is a prototype, structured output is a product

A weekend hackathon prototype can use free-form responses. You say "write me 5 product recommendations" and copy the returned markdown. But a feature you take to production does not work this way. The reason: the model sometimes returns 5 products, sometimes 4, sometimes 6; sometimes JSON, sometimes a list, sometimes a table; sometimes a complete answer, sometimes cut off halfway.

In production, a model output becomes the input to the next step: it is written to a database, another API is called, it is rendered in the user interface. If there is a 3-10% chance of a parse error in every output, your system ends up in a faulty state. Writing error-recovery code becomes more expensive than the prompt itself.

Structured output solves this problem at its root. You define the schema of the model output, and the model produces in that schema with 100% conformance. In this article I explain the three main approaches, when you should choose which one, and the retry / cost / validation patterns.

Approach 1: OpenAI Structured Outputs (2024 Aug)

In Aug 2024, OpenAI added json_schema support to the response_format parameter. This feature forces the model to conform to the JSON Schema you provide during token sampling. It cannot be bypassed.

from openai import OpenAI
from pydantic import BaseModel
from typing import Literal

client = OpenAI()

class ProductExtraction(BaseModel):
    name: str
    category: Literal["electronics", "clothing", "home", "books"]
    price_usd: float
    in_stock: bool
    tags: list[str]

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "Extract product info from text."},
        {"role": "user", "content": "iPhone 15 Pro, 999 USD, electronics, available, [smartphone, apple, 5g]"}
    ],
    response_format=ProductExtraction,
)

product = response.choices[0].message.parsed
print(product.price_usd)  # 999.0 guaranteed

Advantages:

• 100% schema conformance (OpenAI's guarantee)
• Type safety with Pydantic
• IDE autocomplete
• Automatic validation

Limitations:

• Only on OpenAI models (gpt-4o-2024-08-06+)
• Some complex schema patterns are not supported (recursive references, anyOf without discriminator)
• There is a schema compile cost on the first call (the first request is ~2s slower)

Approach 2: Anthropic Tool Use (Forced)

In Anthropic Claude, the official method for structured output is tool use. A single tool is defined, and the model is required to use this tool.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4",
    max_tokens=1024,
    tools=[{
        "name": "extract_invoice",
        "description": "Extract structured invoice data",
        "input_schema": {
            "type": "object",
            "properties": {
                "invoice_number": {"type": "string"},
                "vendor_name": {"type": "string"},
                "total_amount": {"type": "number"},
                "currency": {"type": "string", "enum": ["TRY", "USD", "EUR"]},
                "line_items": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "description": {"type": "string"},
                            "quantity": {"type": "number"},
                            "unit_price": {"type": "number"}
                        },
                        "required": ["description", "quantity", "unit_price"]
                    }
                },
                "kdv_rate": {"type": "number"}
            },
            "required": ["invoice_number", "vendor_name", "total_amount", "currency"]
        }
    }],
    tool_choice={"type": "tool", "name": "extract_invoice"},
    messages=[{"role": "user", "content": "[invoice PDF text]"}]
)

result = response.content[0].input  # dict, conforms to schema

Advantages:

• Very strong extraction with Claude 3.5 Sonnet + Opus 4.7
• More flexible JSON Schema (recursive, anyOf, conditional)
• Tool use requires no extra prompt engineering

Limitations:

• No hard schema enforcement (the model can sometimes produce the wrong type, very rarely)
• Manual Pydantic conversion is required (for type safety)
• A practical validation layer needs to be added

Approach 3: Model-Agnostic (Instructor, Outlines, Marvin)

If you are working with multiple models (cost optimization, fallback, A/B testing), provider-agnostic libraries:

Instructor (most common)

import instructor
from openai import OpenAI
from pydantic import BaseModel

class Meeting(BaseModel):
    title: str
    date: str
    attendees: list[str]
    duration_minutes: int
    action_items: list[str]

client = instructor.from_openai(OpenAI())

meeting = client.chat.completions.create(
    model="gpt-4o-mini",
    response_model=Meeting,
    max_retries=3,
    messages=[
        {"role": "user", "content": "Meeting transcript: ..."}
    ]
)

Core feature: automatic retry on validation error with max_retries=3. If the model returns the wrong format, it feeds the error back to the model, which corrects it.

Outlines (lower level)

Regex/grammar constraint at the token sampling level. The most powerful solution for local models (Llama, Mistral, Qwen):

import outlines

model = outlines.models.transformers("meta-llama/Llama-3.3-70B")
generator = outlines.generate.json(model, Meeting)

result = generator("Transcript: ...")

100% conformance with open-source models like Llama / Mistral.

Marvin (Pydantic-AI approach)

import marvin

@marvin.fn
def extract_meeting(transcript: str) -> Meeting:
    """Extract meeting info from transcript"""

meeting = extract_meeting("Meeting transcript: ...")

Less code, magic decorator. Practical for small projects.

Design Patterns

Pattern 1: Use Enums

The LLM writes freely into "string" fields. If the downstream system expects one of 3-5 different values, always use an enum:

class TicketCategory(str, Enum):
    BUG = "bug"
    FEATURE_REQUEST = "feature_request"
    QUESTION = "question"
    BILLING = "billing"
    OTHER = "other"

class Ticket(BaseModel):
    title: str
    category: TicketCategory  # NOT a free string
    priority: Literal["low", "medium", "high", "critical"]

Pattern 2: Optional + Default

The model sometimes cannot find the information. Instead of keeping it required, make it optional:

class CustomerInfo(BaseModel):
    name: str
    email: str
    phone: Optional[str] = None
    company: Optional[str] = None
    estimated_arr_usd: Optional[float] = None

Pattern 3: Discriminated Union

Handle outputs with multiple different structures in a single schema:

class TextResponse(BaseModel):
    type: Literal["text"]
    content: str

class TableResponse(BaseModel):
    type: Literal["table"]
    columns: list[str]
    rows: list[list[str]]

class ChartResponse(BaseModel):
    type: Literal["chart"]
    chart_type: Literal["bar", "line", "pie"]
    labels: list[str]
    values: list[float]

class APIResponse(BaseModel):
    response: Union[TextResponse, TableResponse, ChartResponse] = Field(discriminator="type")

The model decides which schema to fill based on the "type" field.

Pattern 4: Chain of Thought Field

An extra "reasoning" field to force the model to think:

class Diagnosis(BaseModel):
    differential_diagnoses: list[str]
    reasoning: str  # ask it to fill this in first
    primary_diagnosis: str
    confidence: Literal["low", "medium", "high"]
    recommended_tests: list[str]

Pydantic field order = JSON Schema field order = the order the model will fill. If the reasoning field comes BEFORE primary_diagnosis, the model reasons first, then decides. If you reverse the order, the model decides first, then fabricates reasoning (post-hoc rationalization). This difference produces a 5-10% impact on accuracy.

Pattern 5: Citation / Source Tracking

In RAG systems, a source for each claim:

class Claim(BaseModel):
    statement: str
    source_chunk_ids: list[int]  # which retrieved chunk it came from
    confidence: float

class Answer(BaseModel):
    claims: list[Claim]
    summary: str

The result turns into a citation display in the UI where you can click on each claim.

Validation and Retry Strategies

Output that does not conform to the schema can still come through (rarely, but it happens in production). Two strategies:

Strategy A: Soft retry

from instructor import OpenAISchema

@instructor.patch(OpenAI())
def extract(text: str):
    return client.chat.completions.create(
        model="gpt-4o",
        response_model=Product,
        max_retries=2,  # on error, feed back to the model to correct it
        messages=[{"role": "user", "content": text}]
    )

Instructor feeds the error back to the model like this:

Previous attempt failed validation:
ValidationError: price_usd must be a positive number, got -50

Please try again, fixing the error.

Typical success: 92% on the 1st attempt, 98% on the 2nd, 99.5% on the 3rd.

Strategy B: Hard fail + fallback model

Safer for production:

try:
    result = extract_with_gpt4o_mini(text)
except ValidationError:
    # fall back to a stronger model
    result = extract_with_gpt4o(text)
except ValidationError:
    # if it still fails, queue for human review
    queue_for_human_review(text)
    return None

For cost optimization: 95% of traffic on the cheap model, 5% fallback. Total cost at least 3x lower.

Cost Optimization

Structured output is cost-effective on its own. Because:

1. A shorter system prompt is sufficient (the schema guides the model)
2. Fewer retries (the output is already clean)
3. Less downstream parse code

Practical techniques:

• Field ordering: Put the simplest fields first (the model's attention is fresh). Complex reasoning fields at the end.

• Description hints: Explain what you want to the model with Pydantic Field(description="..."). A shorter system prompt is enough.

  class Invoice(BaseModel):
      total: float = Field(description="Total amount BEFORE tax, in original currency")
      tax: float = Field(description="Tax amount, calculated as total * tax_rate")
      grand_total: float = Field(description="Total + Tax")
  

• Keep optional fields limited: Every optional field is a branch where the model decides "should I leave it empty or fill it in". Too many optional fields lower accuracy.

• Nested depth: Accuracy drops in schemas nested 3+ levels. 2 levels is optimal.

Production Monitoring

After structured output goes live:

Conclusion

Production-grade LLM integration is not sustainable without structured output. There are three main approaches: OpenAI Structured Outputs (hard enforcement, OpenAI-only), Anthropic tool use (flexible schema, Claude), Instructor / Outlines (model-agnostic, retry built in).

The right patterns (enum, optional + default, discriminated union, reasoning field, citation tracking) increase accuracy by 10-30% and eliminate downstream parse logic. With retry + fallback, the production error rate drops below 1%. Cost optimization (field ordering, smaller model with fallback) reduces total OPEX by 3x.

The most critical difference between prototype and production is not the technical quality of the prompt, but schema discipline. The schema is your contract; the LLM is a part of an integrated system.

For one-on-one support in prompt engineering, you can take a look at the individual mentoring option.