Multimodal Inputs with the Vercel AI SDK: Sending Images to Vision Models in Next.js

Text-only LLMs are giving way to vision models that can read screenshots, interpret product photos, and extract data from scanned documents. The Vercel AI SDK supports multimodal inputs, but the API is different from a plain text prompt — images travel as typed content parts inside each message, not as top-level parameters. Getting that structure right is the whole trick.

How content parts work

Every message in the AI SDK can contain an array of parts rather than a plain string. A text part carries prose; an image part carries either a URL or a base64-encoded buffer with an explicit MIME type. Vision models receive the entire array and can reason across all parts simultaneously:

import { generateText } from "ai";
import { anthropic } from "@ai-sdk/anthropic";

const { text } = await generateText({
  model: anthropic("claude-opus-4-5"),
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "What errors appear in this screenshot?" },
        {
          type: "image",
          image: new URL("https://example.com/screenshot.png"),
          // or: image: Buffer.from(base64String, "base64"), mimeType: "image/png"
        },
      ],
    },
  ],
});

The image field accepts a URL object, a Node.js Buffer, a base64 string, or an ArrayBuffer. For server-side routes, passing a Buffer from an uploaded file is the most reliable path — it avoids ephemeral URLs and works in any runtime that has access to the filesystem or an in-memory upload.

Accepting image uploads in a Next.js route

Multipart form data is the natural transport for file uploads. Next.js 16 parses it via the standard Request.formData() API — no extra library needed:

// app/api/analyze/route.ts
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";

export async function POST(req: Request) {
  const form = await req.formData();
  const file = form.get("image") as File | null;
  const prompt = (form.get("prompt") as string) ?? "Describe what you see.";

  if (!file) return new Response("No image", { status: 400 });

  const bytes = await file.arrayBuffer();
  const buffer = Buffer.from(bytes);

  const result = streamText({
    model: openai("gpt-4o"),
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: prompt },
          { type: "image", image: buffer, mimeType: file.type as "image/jpeg" | "image/png" | "image/webp" },
        ],
      },
    ],
  });

  return result.toDataStreamResponse();
}

The mimeType field is required when you pass a Buffer — the SDK cannot infer it from raw bytes. The type union above is intentionally explicit: most vision models accept JPEG, PNG, and WebP; GIF support varies by provider.

Streaming the response to the client

On the client, useCompletion works with multipart POSTs through a small adapter. Because useCompletion sends JSON by default, you need to override the fetch option to build the FormData yourself:

"use client";
import { useCompletion } from "@ai-sdk/react";
import { useRef } from "react";

export function ImageAnalyzer() {
  const fileRef = useRef<HTMLInputElement>(null);
  const { completion, isLoading, complete } = useCompletion({
    api: "/api/analyze",
    fetch: async (url, init) => {
      const body = JSON.parse(init!.body as string) as { prompt: string };
      const form = new FormData();
      form.append("prompt", body.prompt);
      if (fileRef.current?.files?.[0]) {
        form.append("image", fileRef.current.files[0]);
      }
      return fetch(url, { method: "POST", body: form });
    },
  });

  return (
    <div>
      <input ref={fileRef} type="file" accept="image/jpeg,image/png,image/webp" />
      <button
        onClick={() => complete("What errors appear in this image?")}
        disabled={isLoading}
      >
        {isLoading ? "Analyzing…" : "Analyze"}
      </button>
      <p style={{ whiteSpace: "pre-wrap" }}>{completion}</p>
    </div>
  );
}

Keeping images off the edge

Edge functions cap request body size (Vercel enforces 4 MB by default). For anything larger than a small screenshot, run the route in the Node.js runtime:

// at the top of app/api/analyze/route.ts
export const runtime = "nodejs";

For production workloads where users upload high-resolution images, upload directly to object storage (S3, R2, Vercel Blob) from the client, then pass the resulting URL to the vision model. That keeps your API route thin and avoids proxying megabytes through your server on every request.

Multiple images in one turn

Nothing limits you to a single image part per message. Passing two images lets the model compare them, spot regressions between screenshots, or extract data from a multi-page scan:

content: [
  { type: "text", text: "What changed between the before and after screenshots?" },
  { type: "image", image: beforeBuffer, mimeType: "image/png" },
  { type: "image", image: afterBuffer, mimeType: "image/png" },
]

Token costs scale with image size — most providers tile large images into fixed-size patches and charge per patch. Resize images server-side to the smallest dimensions that still preserve the relevant detail, especially if you're running many analyses in parallel.

The practical takeaway

Multimodal inputs are one of the fastest ways to add real intelligence to an existing Next.js app — screenshot analysis, document parsing, and product photo tagging all become tractable with a few dozen lines of code. The key is getting the content-parts structure right and making sure large payloads stay in the Node.js runtime or route through object storage rather than your API handler.