Model Routing in Next.js: Directing Requests to the Right LLM

Not every LLM request needs your most capable model. A request to reword a sentence, classify a support ticket, or check grammar costs the same compute as a request to architect a multi-step workflow — if you route everything to Claude Sonnet or GPT-4o. In practice, a fast cheap model handles the simple work well, and the expensive model is reserved for the problems that actually need it. That split is model routing, and it compounds into significant savings at any real traffic volume.

The three signals that tell you which model to use

You rarely have direct insight into "complexity" — but you have proxies. Three are reliable enough to act on:

• Message length.Short user messages (<100 tokens) are usually simple tasks. Long messages with detailed context tend to require more reasoning.
• Explicit hints from the UI.A "quick reply" button signals that the user wants speed over depth. A "detailed analysis" button signals the opposite. You control these.
• Conversation history depth. Multi-turn conversations with tool calls and long histories usually benefit from a capable model. A single-turn zero-context request usually does not.

Combine all three into a scoring function that returns a model identifier:

// src/lib/select-model.ts
import type { CoreMessage } from "ai";

type ModelTier = "fast" | "capable";

interface RoutingHints {
  explicitMode?: ModelTier;
}

function estimateTokens(messages: CoreMessage[]): number {
  return messages.reduce((sum, m) => {
    const text = typeof m.content === "string" ? m.content : JSON.stringify(m.content);
    return sum + Math.ceil(text.length / 4);
  }, 0);
}

export function selectModel(
  messages: CoreMessage[],
  hints: RoutingHints = {}
): string {
  if (hints.explicitMode === "capable") return "claude-sonnet-4-6";
  if (hints.explicitMode === "fast") return "claude-haiku-4-5-20251001";

  const totalTokens = estimateTokens(messages);
  const turnCount = messages.filter((m) => m.role === "user").length;

  // Multi-turn or long context → capable model
  if (turnCount > 4 || totalTokens > 3_000) return "claude-sonnet-4-6";

  return "claude-haiku-4-5-20251001";
}

Wiring routing into the route handler

The route handler reads an optional mode field from the request body and passes it as a hint. The model selection happens before the streamText call — nothing else in the handler changes:

// src/app/api/chat/route.ts
import { streamText } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import type { CoreMessage } from "ai";
import { selectModel } from "@/lib/select-model";

export const runtime = "nodejs";
export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages, mode }: { messages: CoreMessage[]; mode?: "fast" | "capable" } =
    await req.json();

  const modelId = selectModel(messages, { explicitMode: mode });

  const result = streamText({
    model: anthropic(modelId),
    messages,
    onFinish({ usage }) {
      // Log model + token usage together so you can track cost by tier
      console.log(`[${modelId}] prompt: ${usage.promptTokens}, completion: ${usage.completionTokens}`);
    },
  });

  return result.toDataStreamResponse();
}

Falling back when the fast model misses

Cheap models occasionally produce thin or off-target output. A retry against the capable model — gated on a lightweight quality check — recaptures those cases without routing everything up-tier by default. Keep the check fast: a token count, a regex for known failure patterns, or a second shortgenerateTextcall asking "is this answer complete?" are all reasonable:

import { generateText } from "ai";
import { anthropic } from "@ai-sdk/anthropic";
import type { CoreMessage } from "ai";

async function withFallback(messages: CoreMessage[]): Promise<string> {
  const { text: fastReply } = await generateText({
    model: anthropic("claude-haiku-4-5-20251001"),
    messages,
  });

  // Too short or ends mid-sentence → retry with the capable model
  if (fastReply.length < 80 || !fastReply.match(/[.!?]$/)) {
    const { text: capableReply } = await generateText({
      model: anthropic("claude-sonnet-4-6"),
      messages,
    });
    return capableReply;
  }

  return fastReply;
}

This pattern is most useful for non-streaming endpoints where you can inspect the full reply before returning it. For streaming routes, prefer conservative up-front routing — it is simpler and adds no latency.

Measuring the split

Routing without measurement is guesswork. Add a custom header to every response that names the model used, then correlate it with user satisfaction signals (thumbs up/down, follow-up questions, session length) in your analytics tool. A high rate of follow-up questions on fast-model responses is a signal that your routing threshold is too aggressive.

Most teams discover the same thing: 60–75% of requests can be served by the lighter model without any noticeable quality drop. The 25–40% that genuinely need the capable model are easy to identify once you start measuring — they cluster around specific features, not random users.

What not to route cheaply

A few categories are worth hard-coding to the capable model regardless of apparent complexity: anything that writes to a database or calls an external API via tool use, anything in a legal or medical context, and anything where the user has explicitly asked for a careful or thorough answer. The cost of a wrong action or a missed nuance in those cases is not worth the savings.