LLM Error Handling in Production: Rate Limits, Timeouts, and Fallbacks in Next.js

Production AI apps fail in ways that are predictable but rarely handled well: the provider returns a 429 rate limit, the model takes longer than your timeout allows, or the response is truncated because you hit the token limit. None of these need to crash your app or hang a user's browser. With a short set of patterns in your Next.js route handler, every transient failure either retries silently or falls back to a clean degraded state.

The three failure classes to plan for

LLM errors cluster into three categories, each with a different recovery strategy:

• Rate limits (429). The provider is throttling you. These are always transient — wait and retry with exponential backoff. The Retry-After header tells you exactly how long to wait.
• Timeouts (408 / 504). The model took too long. This can be a transient spike or a sign that your context window is too large for the latency budget. Retry once; if it fails again, return a fallback response.
• Content errors. The model refused the request, hit a token limit, or returned malformed JSON. These are not transient — retrying the same payload will produce the same result. Log and fall back immediately without retrying.

A typed error classifier

The Vercel AI SDK throws structured errors. A small classifier converts them into your three categories so the rest of the handler stays readable:

// src/lib/llm-error.ts
import { APICallError } from "ai";

export type LlmErrorKind = "rate_limit" | "timeout" | "content";

export interface ClassifiedError {
  kind: LlmErrorKind;
  retryAfterMs: number | null;
  original: unknown;
}

export function classifyLlmError(err: unknown): ClassifiedError {
  if (APICallError.isInstance(err)) {
    if (err.statusCode === 429) {
      const header = err.responseHeaders?.["retry-after"];
      const retryAfterMs = header ? parseInt(header, 10) * 1_000 : 5_000;
      return { kind: "rate_limit", retryAfterMs, original: err };
    }
    if (err.statusCode === 408 || err.statusCode === 504) {
      return { kind: "timeout", retryAfterMs: null, original: err };
    }
  }
  return { kind: "content", retryAfterMs: null, original: err };
}

Retry with backoff in the route handler

With the classifier in place, a small retry loop handles 429s and timeouts transparently. The caller never knows a retry happened:

// src/app/api/chat/route.ts
import { streamText } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import { classifyLlmError } from "@/lib/llm-error";

const MAX_ATTEMPTS = 3;

async function withRetry<T>(attempt: () => Promise<T>): Promise<T> {
  for (let i = 0; i < MAX_ATTEMPTS; i++) {
    try {
      return await attempt();
    } catch (err) {
      const { kind, retryAfterMs } = classifyLlmError(err);
      if (kind === "content" || i === MAX_ATTEMPTS - 1) throw err;
      const delay = retryAfterMs ?? 2 ** i * 1_000;
      await new Promise((res) => setTimeout(res, delay));
    }
  }
  throw new Error("unreachable");
}

export async function POST(req: Request) {
  const { messages } = await req.json();

  try {
    const result = await withRetry(() =>
      streamText({
        model: anthropic("claude-haiku-4-5-20251001"),
        messages,
        maxTokens: 1_024,
      })
    );
    return result.toDataStreamResponse();
  } catch (err) {
    const { kind } = classifyLlmError(err);
    console.error("llm_error", { kind, err });
    return Response.json(
      {
        degraded: true,
        reason: kind,
        message: "The AI service is temporarily unavailable. Please try again.",
      },
      { status: kind === "rate_limit" ? 429 : 503 }
    );
  }
}

Three details matter here. Rate-limit retries respect the Retry-After header so you do not compound your own backpressure. Timeout retries use exponential backoff (1 s, 2 s, 4 s) so a slow model has time to recover. Content errors skip the loop entirely because the same payload will produce the same refusal — retrying wastes money and delays the fallback.

Surfacing degraded state in the client

The fallback response includes a degraded: true flag. The client checks for it and shows a specific message instead of a generic spinner hang:

// src/components/Chat.tsx ("use client")
const { messages, append } = useChat({
  onError(err) {
    let uiMessage = "Something went wrong. Please try again.";
    try {
      const body = JSON.parse(err.message);
      if (body?.degraded) uiMessage = body.message;
    } catch {
      // fall through to generic message
    }
    setUiError(uiMessage);
  },
});

What to log — and when

Log every classified error with its kind and status code before retrying, not just on final failure. A spike in rate_limit logs that resolves after one retry tells you your concurrency headroom is shrinking and you should investigate before it starts causing final failures. A spike in content errors tells you a recent prompt change is generating responses the model is refusing.

Neither pattern shows up if you only log on final failure. The classifier keeps this cheap: the kind field is a stable enum you can filter and alert on in any log aggregator without parsing freeform error messages.

One thing to add on top: a circuit breaker

The pattern above handles individual failures. If a provider has an outage that lasts minutes, you will exhaust all retry attempts on every request, adding latency before each failure. A circuit breaker tracks consecutive failures and opens fast — skipping the retry loop and returning the fallback immediately — until the provider recovers. The AI SDK Middleware pattern (see AI SDK Middleware: Logging, Retries, and Rate-Limiting) is the right place to put one: it runs for every call without touching individual route handlers.