RAG with the Vercel AI SDK: Grounding LLM Responses in Your Own Data

Fine-tuning teaches a model new facts by retraining its weights — expensive, slow, and stale the moment your data changes. Retrieval-Augmented Generation (RAG) takes a different approach: at query time, you find the documents most relevant to the user's question and paste them into the prompt. The model's weights stay untouched; only the context window changes. It's cheaper, fresher, and easier to debug.

The Vercel AI SDK ships embedding helpers that make the retrieval half straightforward. Here's how to build a working RAG pipeline in Next.js 16 — embedding documents, searching them, and grounding streamText responses in the results.

The three-step loop

Every RAG system runs the same loop at query time:

1. Embed the query.Turn the user's question into a vector using the same model that embedded your documents.
2. Retrieve. Find the top-k document chunks whose vectors are closest to the query vector.
3. Generate. Pass the retrieved chunks as context into the system prompt, then call the LLM.

The embedding model maps text to a high-dimensional vector space where semantically similar text lands close together. Cosine similarity is the standard distance metric — two vectors pointing in the same direction have a score of 1; orthogonal vectors score 0.

Embedding your documents at build time

Use the AI SDK's embedMany function to turn a list of text chunks into vectors. Run this once — when documents are ingested — and persist the result wherever you store data.

// scripts/embed-docs.ts  (run with: npx tsx scripts/embed-docs.ts)
import { embedMany } from "ai";
import { openai } from "@ai-sdk/openai";
import { writeFileSync } from "fs";

const docs = [
  { id: "intro",    text: "Jodei builds AI-native web products for Bay Area teams…" },
  { id: "services", text: "We offer product strategy, design, and full-stack engineering…" },
  { id: "pricing",  text: "Engagements start at $8k/month for a dedicated pod…" },
];

const { embeddings } = await embedMany({
  model: openai.embedding("text-embedding-3-small"),
  values: docs.map((d) => d.text),
});

const store = docs.map((doc, i) => ({ ...doc, embedding: embeddings[i] }));
writeFileSync("src/lib/doc-embeddings.json", JSON.stringify(store, null, 2));
console.log(`Embedded ${store.length} documents.`);

For a real app, replace the JSON file with a vector database like pgvector, Pinecone, or Turso. The shape is the same: each record holds the original text and its embedding array.

Cosine similarity search

A dot product between two unit-normalised vectors equals their cosine similarity. The AI SDK's cosineSimilarity utility handles the math:

// src/lib/vector-search.ts
import { cosineSimilarity } from "ai";
import store from "./doc-embeddings.json";

interface DocRecord {
  id: string;
  text: string;
  embedding: number[];
}

export function topK(queryEmbedding: number[], k = 3): DocRecord[] {
  return (store as DocRecord[])
    .map((doc) => ({ doc, score: cosineSimilarity(queryEmbedding, doc.embedding) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k)
    .map(({ doc }) => doc);
}

The RAG route handler

Embed the incoming question, retrieve the closest chunks, and inject them into the system prompt before calling streamText. Everything fits in one Next.js 16 route handler:

// src/app/api/ask/route.ts
import { embed, streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { topK } from "@/lib/vector-search";

export const runtime = "nodejs";
export const maxDuration = 30;

export async function POST(req: Request) {
  const { question } = await req.json();

  // 1. Embed the query with the same model used at ingest
  const { embedding } = await embed({
    model: openai.embedding("text-embedding-3-small"),
    value: question,
  });

  // 2. Retrieve the top-3 document chunks
  const chunks = topK(embedding, 3);
  const context = chunks.map((c) => `[Source: ${c.id}]\n${c.text}`).join("\n\n");

  // 3. Generate with grounded context
  const result = streamText({
    model: openai("gpt-4o-mini"),
    system: `You are a helpful assistant for Jodei, an AI web consultancy.
Answer the user's question using ONLY the context below.
If the answer is not in the context, say you don't know.

CONTEXT:
${context}`,
    prompt: question,
  });

  return result.toDataStreamResponse();
}

The "use only the context" instruction is the most important line. Without it, the model blends retrieved facts with parametric knowledge — confidently producing answers that contradict your documents. Strict grounding makes hallucinations auditable: if the model says something wrong, the retrieved chunks show you exactly what it was working from.

Surfacing results in a Server Component

The client side is the same useCompletionhook you'd use for any streamed response. Nothing RAG-specific leaks into the UI layer:

"use client";
import { useCompletion } from "@ai-sdk/react";
import { useState } from "react";

export function AskJodei() {
  const [question, setQuestion] = useState("");
  const { completion, complete, isLoading } = useCompletion({ api: "/api/ask" });

  return (
    <div>
      <textarea
        value={question}
        onChange={(e) => setQuestion(e.target.value)}
        placeholder="Ask anything about our services…"
        rows={3}
      />
      <button
        onClick={() => complete(question, { body: { question } })}
        disabled={isLoading || !question.trim()}
      >
        {isLoading ? "Thinking…" : "Ask"}
      </button>
      {completion && <p>{completion}</p>}
    </div>
  );
}

Chunking strategy matters

The retrieval quality depends heavily on how you split documents before embedding. A few rules of thumb:

• Chunk size: 300–500 tokens. Smaller chunks produce more precise retrieval but lose surrounding context. Larger chunks risk exceeding the context window when you concatenate several.
• Overlap: 50–100 tokens.Overlapping adjacent chunks means a sentence that lands at the boundary of one chunk still appears fully in the next — retrieval won't miss it.
• Preserve structure. Split on paragraph or section boundaries before splitting by token count. A chunk that contains half a table or a dangling code block embeds poorly.

Moving to production

The JSON file approach works for a few dozen documents. At scale, replace it with pgvector (add a vector column to your Postgres table, query with <->distance) or a managed vector store like Pinecone or Supabase Vector. The route handler doesn't change — only the topK implementation swaps from in-memory to a database query.

Two other production concerns worth handling early: re-embed on document updates (stale embeddings silently degrade retrieval), and log which chunks were retrieved alongside the model's response so you can audit where wrong answers came from.

The takeaway

RAG is not a magic bullet — garbage-in retrieval produces garbage-out answers — but it solves a real problem elegantly. Your data stays in your database, updates propagate without retraining, and every answer is traceable to a source document. For AI-native apps that need to reason over proprietary content, it's the right starting point before you even consider fine-tuning.