A completely offline, private AI assistant that answers questions using the actual text of your own PDF documents — built from scratch using Python, without relying on any paid API (no OpenAI, no internet needed after setup).
Imagine you have a friend who has read every document you give them cover to cover, and whenever you ask a question, they:
- Flip to the exact page that talks about your topic
- Read it
- Explain it back to you in their own words
That's exactly what this project does — except the "friend" is a small AI model running on my own laptop, and the "flipping to the right page" happens automatically in milliseconds.
This is called RAG — Retrieval-Augmented Generation. Instead of an AI just guessing answers from what it vaguely remembers (which can be wrong), it first looks up the real content, then answers based on what it found.
You can point this at any set of PDFs — textbooks, research papers, manuals, notes, reports — and it turns them into a searchable, question-answerable knowledge base.
Great question. Here's the difference:
| ChatGPT / Cloud AI | This Project | |
|---|---|---|
| Needs internet | Yes | Only during setup |
| Costs money | Sometimes (API/subscription) | 100% free |
| Your data | Sent to a company's server | Stays on your laptop |
| Knows your exact documents | No — answers from general training, can be wrong or make things up | Yes — reads and quotes your actual uploaded PDFs |
| Can it lie confidently? | Yes (this is called "hallucination") | Much less — if it can't find the answer in your documents, it says "I don't know" instead of guessing |
I actually ran into this problem while building this — I asked a plain (non-RAG) local AI model about a specific document's content, and it completely made up a fake answer that didn't exist in the real source. That's exactly the problem RAG solves.
Think of the whole system as a 4-step assembly line:
Your PDF documents
↓
[1] CHOP into small text pieces (chunks)
↓
[2] CONVERT each piece into a list of numbers (embeddings)
↓
[3] STORE those numbers in a searchable database (FAISS)
↓
When you ask a question:
→ your question also becomes numbers
→ the database finds the 3 most similar chunks
→ those chunks + your question are sent to the AI model
→ the AI reads them and writes you a real answer
Computers can't "understand" meaning the way we do, but they're great at math. An embedding model converts text into a list of numbers (a vector) that represents its meaning. Sentences with similar meaning end up with similar numbers — even if they use totally different words. This is how the system finds relevant content even if you phrase your question completely differently from the source text.
| Tool | Role | Analogy |
|---|---|---|
| Ollama | Runs the AI language model on my laptop | The "brain" that reads and writes answers |
| Llama 3.2 | The actual AI model (open-source, made by Meta) | The specific person doing the thinking |
| LangChain | Python framework that connects all the pieces together | The wiring/glue between all the parts |
| PyMuPDF | Extracts raw text out of PDF files | The person reading each document page by page |
| sentence-transformers (all-MiniLM-L6-v2) | Converts text into embeddings (numbers) | Translator that turns sentences into "meaning-fingerprints" |
| FAISS (by Meta) | Stores embeddings and searches them super fast | The librarian who instantly finds the right page |
| Python | The programming language tying it all together | — |
Everything here is free and open-source. No API keys, no subscriptions, no credit card.
DocuMind/
├── data/
│ ├── folder1/ ← put related PDFs here
│ └── folder2/ ← put another category of PDFs here
├── faiss_index/ ← auto-generated searchable database (created by ingest.py)
├── rag_env/ ← Python virtual environment (keeps dependencies isolated)
├── ingest.py ← Step 1 script: reads PDFs, builds the database
├── query.py ← Step 2 script: lets you ask questions
└── README.md ← this file
The data/ folder can be organized however makes sense for your use case — by subject, by project, by year, etc. The script reads every PDF inside it, including subfolders.
- Python installed
- Ollama installed, with a model pulled:
ollama pull llama3.2
python -m venv rag_env
rag_env\Scripts\activate # Windows
pip install langchain langchain-community langchain-ollama langchain-huggingface langchain-text-splitters langchain-classic faiss-cpu sentence-transformers pymupdfDrop your PDF files into folders inside data/.
python ingest.pyThis reads every PDF, breaks it into chunks, converts them to embeddings, and saves a faiss_index/ folder. You only need to re-run this if you add new PDFs.
python query.pyType any question about your documents and get an answer grounded in the actual content. Type quit to exit.
Ask a question (or 'quit'): what is bowman's capsule
Answer: According to the source text, Bowman's capsule is a cup-shaped
end of a coiled tube that collects the filtrate from the cluster of
very thin-walled blood capillaries, also known as nephrons, in the kidney.
Ask a question (or 'quit'): what is abscisic acid
Answer: I don't know what the answer to the question "What is abscisic
acid?" is based on the provided context. The context doesn't mention
anything about abscisic acid.
Notice the second answer — it honestly says it doesn't know rather than making something up, because that topic isn't covered in the documents I loaded. This is intentional and is actually a good sign — it means the system is staying grounded in real source material instead of guessing.
# 1. Load every PDF from the data folder
documents = []
for root, dirs, files in os.walk("data"):
for file in files:
if file.endswith(".pdf"):
loader = PyMuPDFLoader(os.path.join(root, file))
documents.extend(loader.load())
# 2. Split the extracted text into small overlapping chunks
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(documents)
# 3. Convert chunks into embeddings and store them in FAISS
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = FAISS.from_documents(chunks, embeddings)
vectorstore.save_local("faiss_index")Why chunk size 800 with 100 overlap? Chunks need to be small enough to be specific (so retrieval finds exactly the relevant bit), but not so small that they lose context. The overlap ensures a sentence isn't awkwardly cut in half between two chunks.
# Load the saved database
vectorstore = FAISS.load_local("faiss_index", embeddings, allow_dangerous_deserialization=True)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3}) # fetch top 3 matches
# Connect to the local AI model
llm = OllamaLLM(model="llama3.2")
# Chain retrieval + generation together
qa_chain = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)
# Ask in a loop
while True:
query = input("Ask a question: ")
result = qa_chain.invoke({"query": query})
print(result["result"])k=3 means: for every question, grab the 3 most relevant chunks from the documents and hand them to the AI as reference material before it answers.
Building this wasn't just copy-pasting code — here's what actually broke and how I debugged it:
ModuleNotFoundError: langchain.text_splitter— LangChain restructured its packages in newer versions; the text splitter moved to a standalonelangchain_text_splitterspackage.pypdf.errors.LimitReachedError— the default PDF loader (pypdf) has a safety limit that some image-heavy PDFs (with diagrams) exceeded. Fixed by switching toPyMuPDF, a more robust PDF text extractor.RetrievalQAimport errors — in LangChain 1.x, several chains moved into a separatelangchain-classicpackage.
These are the kinds of real-world dependency/versioning issues you run into with any actively-developed library — not bugs in the core logic.
- Show which PDF and page number each answer came from (source citation)
- Build a simple web interface using Streamlit instead of the terminal
- Support more file formats (Word docs, plain text, web pages)
- Experiment with different chunk sizes to improve answer accuracy
- Add a fallback mode that clearly labels answers as "from documents" vs "general knowledge"
- LLM (Large Language Model): An AI trained on huge amounts of text that can understand and generate human-like language. Llama 3.2 is the LLM used here.
- Embedding: A way of converting text into numbers that represent its meaning, so similar ideas end up with similar numbers.
- Vector Database: A database built specifically to store and search embeddings quickly. FAISS is the one used here.
- Chunking: Breaking long documents into smaller pieces so they're easier to search and fit into the AI's context.
- Hallucination: When an AI confidently generates false information instead of admitting it doesn't know. RAG helps reduce this.
- Retrieval: The process of searching for and pulling out the most relevant pieces of information before generating an answer.
I wanted to understand how modern AI tools like ChatGPT-with-your-documents actually work under the hood, instead of just using no-code tools. Building this from raw Python taught me about embeddings, vector search, LLM orchestration, and real dependency-debugging — skills that a drag-and-drop tool wouldn't have taught me.
