Retrieval-Augmented Generation (RAG) over enterprise documents

01The problem

Large language models are excellent at language but they don't know your private, current, or domain-specific content — and when asked, they will confidently make something up. In regulated domains like insurance, a wrong answer about a policy clause or a claims rule is not an inconvenience, it is a liability.

RAG closes that gap: instead of relying on what the model memorised, we retrieve the most relevant passages from a trusted knowledge base at query time and ask the model to answer only from that retrieved context, with citations back to the source.

02Key terms in plain English

The handful of words that everything else in this page is built from.

03Minimal worked example — the whole thing, broken down

RAG stripped to its essence: no framework, no vector DB — just the five moves that make it work.

• 1. Knowledge base. A few trusted text snippets (in real life: your chunked documents).
• 2. Embed. Turn every snippet and the question into vectors with the same model.
• 3. Retrieve. Score each snippet against the question by cosine similarity, keep the top one.
• 4. Ground. Put the retrieved snippet into the prompt and instruct the model to answer only from it.
• 5. Answer. The model responds using the evidence, not its memory.

# RAG in ~20 lines — the whole loop, no libraries beyond an embed + chat call
import numpy as np

docs = [                                            # 1. knowledge base (toy chunks)
    "Policy A-1024 renews annually on 1 March.",
    "Claims under 500 EUR are auto-approved.",
    "Flood damage is excluded from basic cover.",
]
question = "When does policy A-1024 renew?"

doc_vecs = [embed(d) for d in docs]                  # 2. embed the corpus
q_vec = embed(question)                              #    embed the query (same model!)

def cosine(a, b):                                   # 3. similarity = closeness of meaning
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

best = max(range(len(docs)), key=lambda i: cosine(q_vec, doc_vecs[i]))
context = docs[best]                                 #    retrieve top-1 passage

prompt = (                                           # 4. ground the model in that passage
    "Answer ONLY from the context. If unknown, say so.\n"
    f"Context: {context}\nQuestion: {question}"
)
print(chat(prompt))                                  # 5. -> "It renews annually on 1 March."

04Architecture at a glance

Two pipelines: an offline indexing pipeline that prepares the knowledge base, and an online query pipeline that answers in real time.

05Components & the decisions behind them

06Indexing — annotated snippet

Conceptual, framework-style pseudocode (LangChain-flavoured) to show the shape of the solution.

# --- Offline indexing: build the knowledge base once (refresh on change) ---
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import AzureOpenAIEmbeddings
from langchain_community.vectorstores import FAISS

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800, chunk_overlap=120,          # overlap preserves context across boundaries
)
chunks = []
for doc in load_documents("./knowledge_base"):   # parse + clean upstream
    for piece in splitter.split_text(doc.text):
        chunks.append({
            "text": piece,
            "metadata": {"doc_id": doc.id, "section": doc.section, "page": doc.page},
        })

embeddings = AzureOpenAIEmbeddings(model="text-embedding-3-large")
store = FAISS.from_texts(
    texts=[c["text"] for c in chunks],
    embedding=embeddings,
    metadatas=[c["metadata"] for c in chunks],   # keep metadata for citations + filtering
)
store.save_local("./index")

07Query — annotated snippet

# --- Online query: retrieve, ground, generate with citations ---
from langchain_openai import AzureChatOpenAI

retriever = store.as_retriever(search_kwargs={"k": 8})       # wide net, then re-rank
top_docs = rerank(query, retriever.invoke(query))[:4]          # cross-encoder keeps the best 4

context = "\n\n".join(
    f"[{i+1}] ({d.metadata['doc_id']} p.{d.metadata['page']})\n{d.page_content}"
    for i, d in enumerate(top_docs)
)
system = (
    "Answer ONLY from the context. Cite sources as [n]. "
    "If the context is insufficient, say you don't know."      # anti-hallucination rule
)
llm = AzureChatOpenAI(temperature=0)                            # low temp = deterministic, factual
answer = llm.invoke([
    {"role": "system", "content": system},
    {"role": "user", "content": f"Question: {query}\n\nContext:\n{context}"},
])

08Key design trade-offs

09Evaluation & LLMOps

A RAG system is only trustworthy if it is measured. I separate retrieval quality from generation quality:

10Limitations & where this goes next

• Honest scope: this is an architecture / PoC-level walkthrough to show how I reason about RAG, not a production deployment.
• Quality ceiling = data quality: parsing and chunking dominate outcomes; tables and scanned forms need extra care.
• Next step — agentic RAG: let the system decide when to retrieve, reformulate weak queries, and verify its own answer — covered in my LangGraph orchestration and agentic systems case studies.
• Guardrails: add PII handling, access control on the index, and prompt-injection defenses before real use.