Get valid JSON from LLMs every time with structured outputs

If your LLM integration breaks because the model returned a stray sentence, a missing field or a trailing comma, stop fixing it with regular expressions and retries. Both the Claude API and the OpenAI API now offer structured outputs: you send a JSON Schema and the model's output is constrained to match it. Turn it on, keep your schema inside the supported subset, and handle the two cases where it can still fail: refusals and truncated output.

This guide explains the usual causes of broken JSON, shows the current request shapes for both providers (checked against their documentation in October 2026), and lists the schema rules that trip people up.

Why "please reply in JSON" keeps failing

Asking for JSON in the prompt works most of the time, which is exactly why it is dangerous: the failures show up in production, not in testing. The common symptoms are:

• Prose around the JSON. "Here is the result:" followed by a code fence, which JSON.parse rejects.
• Valid JSON, wrong shape. A field renamed (order_id instead of orderId), a value outside your enum, or an extra key your code does not expect.
• Truncated output. The response hit the token limit halfway through an array.
• A refusal. The model declined the request and wrote an explanation instead of data.

Structured outputs remove the first two classes of failure. The last two still need code, and they are covered below.

How structured outputs work

With structured outputs, the provider compiles your JSON Schema into a grammar and uses it to constrain generation, so the model can only produce tokens that keep the output valid against the schema. This is different from older JSON mode, which guarantees syntactically valid JSON but not your schema. OpenAI's guide says it directly: both produce valid JSON, but only Structured Outputs adheres to your schema, and it recommends Structured Outputs whenever possible.

Two practical consequences follow from the grammar step. The first request with a new schema is slower while it compiles, and later requests reuse it. Anthropic documents that compiled grammars are cached for 24 hours from last use, and that changing the schema structure invalidates the cache while changing only a name or description does not.

Claude API: output_config.format

On the Claude API, structured outputs are generally available and need no beta header. You pass a schema in output_config.format with "type": "json_schema", and the JSON arrives in the text block of the response. The older output_format parameter is deprecated and only works with the structured-outputs-2025-11-13 beta header, so new code should use output_config.

{
  "model": "claude-haiku-5-5",
  "max_tokens": 1024,
  "messages": [{ "role": "user", "content": "Order 4471 arrived broken, please refund." }],
  "output_config": {
    "format": {
      "type": "json_schema",
      "schema": {
        "type": "object",
        "properties": {
          "category": { "type": "string", "enum": ["billing", "bug", "feature_request", "other"] },
          "orderId": { "type": ["string", "null"] },
          "summary": { "type": "string" }
        },
        "required": ["category", "orderId", "summary"],
        "additionalProperties": false
      }
    }
  }
}

In TypeScript the SDK can build that schema from Zod and validate the response for you. messages.parse() returns a typed parsed_output:

import Anthropic from "@anthropic-ai/sdk";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";
import { z } from "zod";

const Ticket = z.object({
  category: z.enum(["billing", "bug", "feature_request", "other"]),
  urgency: z.enum(["low", "normal", "high"]),
  orderId: z.string().nullable(),
  summary: z.string(),
});

const client = new Anthropic();

export async function triage(email: string) {
  const response = await client.messages.parse({
    model: "claude-haiku-5-5",
    max_tokens: 1024,
    system: "Classify the support email. Use null when no order ID is present.",
    messages: [{ role: "user", content: email }],
    output_config: { format: zodOutputFormat(Ticket) },
  });

  if (response.stop_reason === "refusal") throw new Error("Model refused");
  if (response.stop_reason === "max_tokens") throw new Error("Output truncated; raise max_tokens");
  return response.parsed_output; // typed as z.infer<typeof Ticket>
}

In Python the equivalent is client.messages.parse(..., output_format=YourPydanticModel), which returns response.parsed_output. Structured outputs on Claude also work with streaming and the Message Batches API, and they can be combined with strict tool use ("strict": true on a tool definition) in the same request.

OpenAI Responses API: text.format

On OpenAI's Responses API the schema goes in text.format with type: "json_schema", a name and strict: true. The JavaScript SDK has a Zod helper, and responses.parse() exposes the result as output_parsed:

import OpenAI from "openai";
import { zodTextFormat } from "openai/helpers/zod";
import { z } from "zod";

const Ticket = z.object({
  category: z.enum(["billing", "bug", "feature_request", "other"]),
  urgency: z.enum(["low", "normal", "high"]),
  orderId: z.string().nullable(),
  summary: z.string(),
});

const openai = new OpenAI();

export async function triage(email: string) {
  const response = await openai.responses.parse({
    model: "gpt-6-astra",
    input: [
      { role: "system", content: "Classify the support email." },
      { role: "user", content: email },
    ],
    text: { format: zodTextFormat(Ticket, "ticket") },
  });
  if (response.status !== "completed") {
    throw new Error(`Incomplete: ${response.incomplete_details?.reason}`);
  }
  return response.output_parsed;
}

In Python, pass text_format=YourPydanticModel to client.responses.parse(). A refusal appears as a separate output item of type refusal rather than as data, so check for it before you use the parsed value.

Schema rules that cause most 400 errors

Neither provider accepts the whole JSON Schema specification. The rules below come from each provider's documentation; an unsupported schema is rejected with an error rather than silently ignored.

Two habits avoid most surprises. First, model "unknown" as null rather than as an omitted key: it works on both providers and forces the model to make an explicit choice. Second, keep business rules such as "summary under 200 characters" out of the grammar and in your own validation. The Anthropic TypeScript SDK's schema transform does something similar for you: constraints the API cannot enforce are moved into the field's description as hints, and parse() then validates the response against your original Zod schema.

The two failures structured outputs cannot prevent

Refusals

If the model declines a request for safety reasons, Claude returns stop_reason: "refusal" with HTTP 200, you are billed for the tokens, and the output may not match your schema. Anthropic also notes that a schema property asking for step-by-step reasoning can itself trigger a refusal, so ask for a short explanation field instead. On OpenAI, look for the refusal output item. In both cases, log the event and route the request to a fallback path rather than parsing it.

Truncation

If generation stops at the token limit, the JSON may be incomplete. Claude reports stop_reason: "max_tokens"; OpenAI sets status to "incomplete" with the reason max_output_tokens. OpenAI's guidance is to parse only when the status is "completed". The fix is usually to raise the limit or reduce the output: return IDs rather than full records, or paginate large extractions.

Keep a validation layer anyway

Structured outputs make the shape reliable, but your code should still validate at the boundary. You may run the same parser against a provider or local model that only offers JSON mode, a fallback path might bypass the SDK helper, and the schema sent to the model may be looser than your real business rules. A small wrapper costs nothing:

import { z } from "zod";

const Ticket = z.object({
  category: z.enum(["billing", "bug", "feature_request", "other"]),
  urgency: z.enum(["low", "normal", "high"]),
  orderId: z.string().nullable(),
  summary: z.string().max(200),
});

export function parseTicket(text) {
  let data;
  try {
    data = JSON.parse(text);
  } catch {
    return { ok: false, error: "not JSON" };
  }
  const result = Ticket.safeParse(data);
  return result.success
    ? { ok: true, value: result.data }
    : { ok: false, error: z.prettifyError(result.error) };
}

Given {"category":"refund",...}, this returns ok: false with the message "Invalid option: expected one of ... at category", which is precise enough to log, alert on, or feed back to the model in a single retry.

Limitations and gotchas

• Feature conflicts on Claude. JSON outputs cannot be combined with citations (the API returns a 400) or with assistant message prefilling.
• Changing the format breaks prompt caching. Anthropic notes that changing output_config.format invalidates the prompt cache for that conversation thread, so keep one schema per conversation where you can.
• No sensitive data in schemas. Anthropic's documentation warns that compiled schemas are cached separately from message content, so do not put personal or health data in property names, enum values or patterns.
• Model support varies. Check each provider's supported-model list before switching models; older OpenAI models fall back to JSON mode only.
• Correct shape is not correct content. A perfectly typed category can still be the wrong category. Measure accuracy with an evaluation set, not just parse rates.

What to do next

1. Find every place your code calls JSON.parse on model output and list the schema it expects.
2. Move each one to structured outputs, starting with the call that fails most often in your logs.
3. Add explicit handling for refusals and truncation, with metrics for both.
4. Keep the Zod or Pydantic validation at the boundary and delete the regex clean-up code.