How to build a RAG app in Next.js with a local AI model

RAG, or retrieval-augmented generation, lets a language model answer from a selected collection of documents instead of relying only on knowledge learned during training. The application retrieves passages related to a question and then provides those passages to the model as context.

This tutorial builds a small local RAG pipeline in Next.js. The documents, embeddings and answer generation can remain on your own computer. That does not automatically make the system enterprise-secure, but it gives you direct control over where the data travels and which components process it.

What we are building

The minimal application performs five steps:

1. loads prepared document chunks,
2. creates an embedding for the question,
3. compares it with document embeddings,
4. selects the most relevant passages,
5. sends the question and retrieved context to a local model.

You can later add file uploads, a vector database, authentication, source citations and systematic answer evaluation.

What you need

• a Next.js project using the App Router,
• Node.js 20 or newer,
• Ollama installed,
• a model for answer generation,
• an embedding model such as embeddinggemma.

Ollama exposes a local HTTP API, so a Next.js server can communicate with the models like any other service. Its official embeddings documentation recommends models including embeddinggemma, qwen3-embedding and all-minilm.

Download two example models:

ollama pull gemma3:4b
ollama pull embeddinggemma

A smaller generation model is easier to run on a laptop. A larger model may improve answer quality, but it will require more system memory or VRAM.

How the RAG architecture works

A RAG system contains two separate workflows.

Indexing runs when documents are added or changed. The application cleans the text, splits it into chunks, converts each chunk into an embedding and stores the vector together with its text and metadata.

Retrieval and generation run after a user asks a question. The application embeds the question, compares it with the index and places the best passages in the model prompt.

Keeping these workflows separate matters. Re-embedding every document for every question would add needless latency and computation.

Step 1: prepare the documents

For the prototype, use a small array of passages. In a real system they might come from manuals, knowledge-base articles, database records or PDF files.

type KnowledgeChunk = {
  id: string;
  source: string;
  content: string;
  embedding?: number[];
};
 
const documents: KnowledgeChunk[] = [
  {
    id: "returns",
    source: "returns-policy.md",
    content: "Customers can return a product within 30 days of delivery.",
  },
  {
    id: "shipping",
    source: "shipping.md",
    content: "Standard domestic delivery takes between two and four business days.",
  },
];

Every chunk should have a stable identifier and a source. This allows the interface to show readers where an answer came from.

Step 2: choose a sensible chunking strategy

There is no universally correct chunk size. Tiny chunks lose context, while oversized chunks reduce retrieval precision and consume more of the model context window.

For text documentation, a useful starting point is:

• 300–600 tokens per chunk,
• 10–20% overlap between neighboring chunks,
• splitting by headings and paragraphs before applying a hard limit,
• preserving the document title in metadata.

Avoid cutting text mechanically in the middle of a sentence. A document's structure often carries more useful meaning than a fixed character count.

Step 3: create embeddings

An embedding is a numeric representation of meaning. Semantically similar questions and passages should occupy nearby positions in vector space.

async function createEmbedding(input: string): Promise<number[]> {
  const response = await fetch("http://127.0.0.1:11434/api/embed", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "embeddinggemma",
      input,
    }),
  });
 
  if (!response.ok) {
    throw new Error(`Embedding request failed: ${response.status}`);
  }
 
  const data = (await response.json()) as { embeddings: number[][] };
  return data.embeddings[0] ?? [];
}

Call this function for every chunk during indexing and store the result. The Ollama embeddings documentation describes the API endpoint and models intended for semantic search.

Step 4: retrieve the closest passages

Cosine similarity is sufficient for a small prototype. At thousands or millions of chunks, use a vector database with an approximate nearest-neighbor index.

function cosineSimilarity(a: number[], b: number[]): number {
  if (a.length !== b.length || a.length === 0) return 0;
 
  let dot = 0;
  let normA = 0;
  let normB = 0;
 
  for (let i = 0; i < a.length; i += 1) {
    dot += a[i]! * b[i]!;
    normA += a[i]! ** 2;
    normB += b[i]! ** 2;
  }
 
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}
 
function retrieve(
  queryEmbedding: number[],
  chunks: Required<KnowledgeChunk>[],
  limit = 3,
) {
  return chunks
    .map((chunk) => ({
      ...chunk,
      score: cosineSimilarity(queryEmbedding, chunk.embedding),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, limit);
}

A similarity score is not proof that a passage is relevant. In production, define a minimum threshold, test different values of limit and measure whether the required passage appears among the returned results.

Step 5: create a Next.js Route Handler

With the App Router, the request can be handled in app/api/ask/route.ts. Next.js defines Route Handlers as custom endpoints built with the standard Web Request and Response APIs.

export async function POST(request: Request) {
  const { question } = (await request.json()) as { question?: string };
 
  if (!question || question.length > 1_000) {
    return Response.json({ error: "Invalid question" }, { status: 400 });
  }
 
  const queryEmbedding = await createEmbedding(question);
  const matches = retrieve(queryEmbedding, indexedDocuments, 3);
 
  const context = matches
    .map((item, index) => `[${index + 1}] ${item.content}`)
    .join("\n\n");
 
  const response = await fetch("http://127.0.0.1:11434/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "gemma3:4b",
      stream: false,
      messages: [
        {
          role: "system",
          content:
            "Answer only from the supplied context. " +
            "If the context does not contain the answer, say that you do not know.",
        },
        {
          role: "user",
          content: `Context:\n${context}\n\nQuestion: ${question}`,
        },
      ],
    }),
  });
 
  if (!response.ok) {
    return Response.json({ error: "Model unavailable" }, { status: 503 });
  }
 
  const result = await response.json();
 
  return Response.json({
    answer: result.message?.content ?? "",
    sources: matches.map(({ source, score }) => ({ source, score })),
  });
}

This is an educational example, not a complete production endpoint. You still need a request body limit, rate limiting, access control, timeouts and robust error handling.

How to reduce hallucinations

RAG does not remove hallucinations automatically. A model can still ignore a document or combine a correct passage with an unsupported conclusion.

The most useful safeguards are:

• explicitly allowing the model to say it does not know,
• rejecting retrieval results below a tested relevance threshold,
• returning source names with the answer,
• testing questions that have no answer in the knowledge base,
• separating document content from system instructions,
• treating prompt injection stored inside documents as untrusted input.

Evaluate retrieval and generation separately. If the correct passage was never retrieved, swapping the generation model is unlikely to solve the real problem.

Context, memory and performance

In a local application, context length directly affects memory use. The Ollama context documentation explains that larger context windows increase memory requirements, while agent and coding workloads may need substantially more context than a simple chat.

Do not send every loosely related document to the model just in case. A few precise passages often produce better answers than dozens of marginal matches.

When to add a vector database

An in-memory array is enough for learning and a small demo. A vector database becomes useful when:

• the corpus is too large to keep conveniently in process memory,
• the index must survive application restarts,
• retrieval needs filters for user, date or document type,
• documents must be updated without rebuilding the entire index,
• the application runs across multiple server instances.

The specific database matters less than embedding consistency. Documents and questions must use the same embedding model and compatible configuration.

Pre-deployment checklist

• Every document has a source, identifier and update date.
• Chunking respects headings and paragraph boundaries.
• Document embeddings are not recalculated for every question.
• The endpoint validates input type and length.
• The interface displays sources used in the answer.
• The system can decline when the retrieved context is weak.
• Data belonging to different users remains isolated.
• The evaluation set includes answerable, ambiguous and unanswerable questions.

Is local RAG always the best choice?

No. A local model gives you more control over data and per-query costs, but it also makes you responsible for hardware, updates, security and observability.

Start with a small, measurable prototype. Build a test set of several dozen questions, verify retrieval quality and only then choose a larger model or a more sophisticated database. In RAG systems, document quality and retrieval design usually matter more than an impressive chat interface.