Project 22: Chat with Your PDF

Introduction

"Chat with your PDF" is the app that made RAG (Retrieval-Augmented Generation) famous — and it is far simpler than the buzzwords suggest. The recipe: split the document into chunks, embed the chunks, find the few chunks relevant to the user's question, and hand those to an LLM as context. This app does the full pipeline with Mistral's free API tier, building on the chatbot patterns from the Mistral chatbot tutorial.

The key insight: the LLM never reads the whole PDF. It reads a handful of retrieved paragraphs — which is why this works on a 300-page manual without a 300-page context window.

Features

PDF upload — any text-based document, parsed with pdfplumber.

Smart chunking — overlapping paragraphs that keep context intact.

Embedding search — cosine similarity over chunk vectors.

Grounded answers — the model answers only from retrieved chunks.

Citations — every answer shows the chunks it was built from.

Prerequisites

Python 3.9+ — from python.org.

A free Mistral API key — from console.mistral.ai.

Dependencies:

code

pip install streamlit pdfplumber requests numpy

Step 1: Create the Script

Save as pdf_chat.py:

code

import streamlit as st
import requests
import numpy as np
import pdfplumber
import re

st.set_page_config(page_title="Chat with PDF", page_icon="💬")
st.title("💬 Chat with Your PDF")

api_key = st.secrets.get("MISTRAL_API_KEY", "") or st.text_input("Mistral API key", type="password")


def split_chunks(text, size=900, overlap=150):
    text = re.sub(r"\s+", " ", text)
    chunks = []
    for i in range(0, len(text), size - overlap):
        chunks.append(text[i:i + size])
    return chunks


@st.cache_data(show_spinner=False)
def embed_texts(texts, key):
    out = []
    for t in texts:
        r = requests.post(
            "https://api.mistral.ai/v1/embeddings",
            headers={"Authorization": f"Bearer {key}"},
            json={"model": "mistral-embed", "input": t}, timeout=30)
        out.append(r.json()["data"][0]["embedding"])
    return np.array(out)


def ask_mistral(question, context, key):
    r = requests.post(
        "https://api.mistral.ai/v1/chat/completions",
        headers={"Authorization": f"Bearer {key}"},
        json={
            "model": "mistral-small-latest",
            "messages": [{"role": "user", "content":
                f"Answer using ONLY this context. If it's not in the context, say you don't know.\n\nContext:\n{context}\n\nQuestion: {question}"}],
        }, timeout=60)
    return r.json()["choices"][0]["message"]["content"]


uploaded = st.file_uploader("Upload a PDF", type=["pdf"])

if uploaded and api_key:
    if "chunks" not in st.session_state:
        with pdfplumber.open(uploaded) as pdf:
            text = "\n".join(p.extract_text() or "" for p in pdf.pages)
        st.session_state.chunks = split_chunks(text)
        st.session_state.vectors = embed_texts(st.session_state.chunks, api_key)
        st.success(f"Indexed {len(st.session_state.chunks)} chunks")

    question = st.text_input("Ask a question", placeholder="What is the main takeaway?")
    if st.button("Ask", type="primary") and question.strip():
        q_vec = embed_texts([question], api_key)[0]
        sims = st.session_state.vectors @ q_vec / (
            np.linalg.norm(st.session_state.vectors, axis=1) * np.linalg.norm(q_vec))
        top = np.argsort(sims)[-3:][::-1]

        context = "\n---\n".join(st.session_state.chunks[i] for i in top)
        with st.spinner("Reading the relevant pages..."):
            answer = ask_mistral(question, context, api_key)

        st.markdown(answer)
        with st.expander("📎 Sources used"):
            for i in top:
                st.caption(f"Chunk {i}: {st.session_state.chunks[i][:200]}...")

Step 2: Run the App

code

streamlit run pdf_chat.py

Upload a report or manual, ask "what are the key findings?", then open the sources expander — you'll see exactly which paragraphs the answer came from.

How It Works

Chunking with overlap is the quiet hero: a 900-character window stepping 750 characters means every sentence appears in a chunk with some surrounding context, and no fact gets sliced in half at a boundary. Too-small chunks lose meaning; too-big chunks dilute the embedding.

Embeddings turn each chunk into a ~1024-dimension vector where similar meanings point in similar directions. The question gets embedded the same way, and cosine similarity ranks chunks by relevance — vectors @ q_vec with normalization is the entire search engine, one matrix product. The top-3 idiom is the same numpy ranking used in the digit recognizer.

The prompt guardrail does the RAG heavy lifting: "answer ONLY from this context; say you don't know otherwise." That instruction converts a creative writer into a document reader — the difference between grounded answers and confident hallucinations. Citations make the grounding auditable.

Common Errors & Fixes

401 from Mistral — the key isn't reaching the header; prefer st.secrets over the text input for anything beyond local testing.

Empty text from the PDF — it's scanned; OCR first with the OCR app pipeline, then feed that text here.

Answers ignore the context — your chunks are too small to carry meaning; raise size to 1200 and re-index.

Embedding loop is slow for big PDFs — the free tier rate-limits; batch the embedding request (input: [list of texts]) or cache aggressively (already done via @st.cache_data).

Key Concepts

RAG pipeline — chunk → embed → retrieve → generate.

Overlapping chunks — context insurance at boundary lines.

Cosine similarity — meaning-space search as one dot product.

Grounded prompting — restrict the model to retrieved evidence.

What to Try Next

Add chat history — follow-up questions with conversation context in the messages list.

Show page numbers in citations by tracking chunk → page mapping during extraction.

Swap embeddings for a local model (sentence-transformers) to drop the API dependency for retrieval.

Combine with the resume parser — chat across a folder of resumes.

FAQ

Why retrieve instead of pasting the whole PDF into the prompt?

Context windows are finite and attention degrades over long inputs. Retrieval feeds the model exactly the relevant 2,700 characters — cheaper, faster, and more accurate.

Does this cost money?

Mistral's free tier covers the API used here for reasonable personal use. The only heavy cost is embedding a large PDF once, which the cache absorbs.

Can it handle tables and images in the PDF?

Tables extract as jumbled text (partially usable); images don't extract at all. For image-heavy documents, look at vision-capable multimodal models.