Skip to main content
Jodei
Insights

Context Window Management: Keeping Long Conversations Under the Token Limit

Every token in your conversation history costs money and slows down the model. Here's how to track usage, implement a sliding-window strategy, and compress history into a rolling summary — all in a Next.js 16 route handler.

6 min read

A multi-turn AI chat feels cheap until you look at the token count after a dozen exchanges. Every message in the conversation history is re-sent to the model on every request — the context window is not cumulative, it's reconstructed from scratch each time. A 20-message conversation that averages 200 tokens per message sends 4,000 tokens of history before the user's new question even arrives. At scale, that's real money, and past a certain length the model starts to degrade — it loses track of early instructions, and latency climbs.

The Vercel AI SDK gives you the tools to manage this. Here's a practical approach — usage tracking, sliding-window truncation, and summary compression — implemented in a Next.js 16 route handler.

Reading token usage from the response

The AI SDK's streamText and generateText both return a usage object that reports promptTokens, completionTokens, and totalTokens. Log it on every request so you know where you stand:

// src/app/api/chat/route.ts
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import type { CoreMessage } from "ai";

export const runtime = "nodejs";
export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages }: { messages: CoreMessage[] } = await req.json();

  const result = streamText({
    model: openai("gpt-4o-mini"),
    messages,
    onFinish({ usage }) {
      console.log(
        `tokens — prompt: ${usage.promptTokens}, completion: ${usage.completionTokens}`
      );
    },
  });

  return result.toDataStreamResponse();
}

Once you can see the numbers, set a threshold. A sensible limit for GPT-4o-mini is 12,000 prompt tokens — well inside the 128k context window but cheap enough to sustain high-volume use.

Strategy 1: Sliding window

The simplest approach is to drop the oldest messages when the history grows too long. Keep the system prompt, discard early turns, and retain the most recent n exchanges. A utility function keeps the route handler clean:

// src/lib/truncate-messages.ts
import type { CoreMessage } from "ai";

const APPROX_CHARS_PER_TOKEN = 4;

function estimateTokens(messages: CoreMessage[]): number {
  return messages.reduce((sum, m) => {
    const text = typeof m.content === "string" ? m.content : JSON.stringify(m.content);
    return sum + Math.ceil(text.length / APPROX_CHARS_PER_TOKEN);
  }, 0);
}

export function slidingWindow(
  messages: CoreMessage[],
  maxTokens = 10_000
): CoreMessage[] {
  // Always keep the system message if present
  const system = messages[0]?.role === "system" ? [messages[0]] : [];
  const turns = messages[0]?.role === "system" ? messages.slice(1) : messages;

  let kept = [...turns];
  while (kept.length > 2 && estimateTokens([...system, ...kept]) > maxTokens) {
    kept = kept.slice(2); // drop one user+assistant pair
  }

  return [...system, ...kept];
}

The character-to-token estimate (~4 chars/token) is a rough heuristic — good enough to avoid catastrophic overruns. For exact counts, call the model's tokeniser or use the experimental_countTokens helper the AI SDK exposes on some providers.

Strategy 2: Summary compression

Truncation loses context. A better approach for long-running conversations is to replace dropped messages with a single summary generated by the model itself. When the history exceeds the threshold, compress the oldest half into a "memory" message before discarding it:

// src/lib/compress-history.ts
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import type { CoreMessage } from "ai";

export async function compressOldTurns(
  messages: CoreMessage[],
  keepRecent = 6
): Promise<CoreMessage[]> {
  if (messages.length <= keepRecent) return messages;

  const toCompress = messages.slice(0, messages.length - keepRecent);
  const recent = messages.slice(messages.length - keepRecent);

  const { text: summary } = await generateText({
    model: openai("gpt-4o-mini"),
    prompt:
      "Summarise the following conversation in 3–5 bullet points. " +
      "Preserve every factual detail the user stated about themselves or their goals.\n\n" +
      toCompress.map((m) => `${m.role}: ${m.content}`).join("\n"),
  });

  const memoryMessage: CoreMessage = {
    role: "user",
    content: `[Earlier conversation summary]\n${summary}`,
  };

  return [memoryMessage, ...recent];
}

The summary call adds one small LLM round-trip, but it runs only when the conversation is long — and it saves many more tokens than it spends on every subsequent request.

Wiring it into the route handler

Combine both strategies: compress first, then apply the sliding window as a safety net if the compressed history is still large:

// src/app/api/chat/route.ts
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import type { CoreMessage } from "ai";
import { compressOldTurns } from "@/lib/compress-history";
import { slidingWindow } from "@/lib/truncate-messages";

export const runtime = "nodejs";
export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages }: { messages: CoreMessage[] } = await req.json();

  const compressed = await compressOldTurns(messages, 6);
  const trimmed = slidingWindow(compressed, 10_000);

  const result = streamText({
    model: openai("gpt-4o-mini"),
    messages: trimmed,
  });

  return result.toDataStreamResponse();
}

What to keep, what to drop

A few rules that have saved us from subtle context-quality regressions:

  • Never drop the system prompt.It contains the model's persona, constraints, and tool definitions. Losing it mid-conversation produces subtly wrong answers that are hard to debug.
  • Drop in user+assistant pairs. Removing a single side of an exchange leaves the model with a dangling reference — it can see a response but not the question that prompted it.
  • Preserve tool results. If the model called a function and received a result, keep that pair together. A tool call without its result confuses structured-output models significantly.

The takeaway

Context window management is one of those problems that only becomes visible in production, after a user has had a long session and starts getting incoherent replies — or after you open the billing dashboard. Addressing it early, with a sliding window as a safety net and summary compression for quality, keeps both costs and response quality predictable as conversations grow.