LLM Observability with OpenTelemetry: Tracing AI Calls in Next.js

An LLM call is a black box by default. You know it succeeded because text came back, and you know it failed because an exception was thrown — but you don't know how long the model spent on the prompt, how many tokens you consumed per route, or which user triggered the expensive 4,000-token response. In a production AI-native app, that gap becomes painful fast.

The Vercel AI SDK has built-in OpenTelemetry support. Enable it with one option and every streamText / generateText / generateObjectcall emits a span carrying the model name, token counts, and prompt content. Here's how to wire it into Next.js 16.

Enable telemetry on a call

Pass experimental_telemetry to any AI SDK generation function. The minimum config is just isEnabled: true:

// src/app/api/chat/route.ts
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";

export const runtime = "nodejs";
export const maxDuration = 30;

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = streamText({
    model: openai("gpt-4o-mini"),
    messages,
    experimental_telemetry: {
      isEnabled: true,
      functionId: "chat",           // shows up as the span name
      metadata: { route: "/api/chat" },
    },
  });

  return result.toDataStreamResponse();
}

With telemetry enabled the SDK wraps each call in an OpenTelemetry span named after functionId. The span carries standard attributes: gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens, gen_ai.request.model, and — if you opt in — gen_ai.prompt and gen_ai.completion.

Register the OpenTelemetry SDK in Next.js 16

Next.js 16 supports the OpenTelemetry SDK through the instrumentation.ts convention. Create the file at the project root (next to next.config.ts):

// instrumentation.ts
import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node";

export function register() {
  if (process.env.NEXT_RUNTIME !== "nodejs") return;

  const sdk = new NodeSDK({
    traceExporter: new OTLPTraceExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
    }),
    spanProcessors: [
      new SimpleSpanProcessor(
        new OTLPTraceExporter({
          url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
        })
      ),
    ],
  });

  sdk.start();
}

Enable the instrumentation hook in next.config.ts:

// next.config.ts
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  experimental: {
    instrumentationHook: true,
  },
};

export default nextConfig;

The NEXT_RUNTIME !== "nodejs" guard is important — the instrumentation file runs in both the Edge and Node runtimes, but @opentelemetry/sdk-node only works in Node. Skipping the guard causes a build error on Edge routes.

What spans look like

A single streamText call produces a parent span for the AI SDK operation and a child span for the underlying HTTP request to the model provider. The parent carries the high-level attributes you care about:

{
  "name": "chat",
  "attributes": {
    "gen_ai.system": "openai",
    "gen_ai.request.model": "gpt-4o-mini",
    "gen_ai.usage.prompt_tokens": 312,
    "gen_ai.usage.completion_tokens": 87,
    "gen_ai.response.finish_reasons": ["stop"],
    "route": "/api/chat"
  },
  "duration_ms": 1843
}

The metadata object you pass is merged into span attributes, so adding userId or sessionId is straightforward — and lets you slice usage by user in your backend.

Controlling prompt recording

By default the SDK does not record prompt or completion text — only token counts and metadata. To capture the full content for debugging, set recordInputs and recordOutputs:

experimental_telemetry: {
  isEnabled: true,
  functionId: "chat",
  recordInputs: process.env.NODE_ENV !== "production",
  recordOutputs: process.env.NODE_ENV !== "production",
}

Gate both on NODE_ENV !== "production". Prompt content can include PII — user queries, names, internal documents — so shipping it to a third-party collector in production creates a data-handling risk that rarely passes a security review.

Sampling in production

LLM calls are low-frequency compared with web requests, so a 100% sample rate is usually fine. If your app handles thousands of AI calls per minute, add a TraceIdRatioBased sampler to the SDK config and keep the trace volume manageable. Token counts are aggregated in most APM tools regardless of sampling, so you keep cost visibility without the span storage overhead.

The takeaway

OpenTelemetry + the AI SDK's built-in telemetry option gives you per-call latency, token usage, and model attribution with about 30 lines of setup. In practice it's the fastest path from "the AI feature feels slow" to "the p95 latency on this route is 4 s and 80% of it is time-to-first-token" — the kind of signal that turns a vague complaint into a concrete fix.