architecture · Personal project

DocuMind — A Production RAG Platform

A multi-tenant RAG platform with streamed, cited answers — hybrid BM25 + embedding retrieval with reranking, a Redis semantic cache, and RAGAS scoring on every change.

Role: Sole designer and builder — ingestion, retrieval, caching, evaluation, and the app
Team: Solo
Scope: Multi-tenant question answering over PDFs and wiki pages, with streamed answers that cite their sources
Level: Personal project

Retrieval

vector-onlyhybrid + rerank

Method: BM25 and embedding retrieval merged and reranked with a cross-encoder, compared with a vector-only baseline on the evaluation set.
Window: by design

Quality tracking

spot checksRAGAS every change

Method: Faithfulness and context recall are scored with RAGAS on an evaluation set for every change.
Window: every change

Repeated questions

full pipelinesemantic cache

Method: Redis semantic cache returns answers for semantically repeated questions, cutting latency and token cost.
Window: by design

Context

"Chat with your documents" is easy to demo and hard to make good. DocuMind is a personal project to build it the way a production system needs to be built: multi-tenant, answers streamed and cited, retrieval quality measured, and costs kept under control. It answers questions over PDFs and wiki pages.

Code: repository link coming soon.

Retrieval is the product

Vector search is good at meaning and bad at exact terms — error codes, product names, section titles. Keyword search is the opposite. DocuMind uses hybrid retrieval: BM25 and embeddings run together, and a cross-encoder reranker scores each candidate against the actual question before anything reaches the model. That improves relevance over vector-only search.

One store for vectors and tenants

Embeddings live in PostgreSQL with pgvector, next to tenant and document metadata. Every query must only ever see the asking tenant's documents, and keeping both in one store means tenant scoping is part of the same query that does the search.

Cheaper repeats

People ask the same question in different words. A Redis semantic cache catches those rephrasings, so repeats come back faster and cost no generation tokens.

Measured, not guessed

Every change — chunking, retrieval, reranking, prompts — is scored with RAGAS on an evaluation set: faithfulness (does the answer stick to the sources?) and context recall (did retrieval find what was needed?). Separating the two tells you whether to fix retrieval or generation.

What broke

The first cache matched only on question similarity, which meant a similar question from one tenant could in principle get another tenant's cached answer. I caught it in testing, and cache keys are now scoped by tenant and document-set version, with a test that tries to read across tenants. Separately, fixed-size chunking split tables and headings from their content, and context recall made that visible — so chunking now follows document structure.

Outcomes

Streamed, cited answers over PDFs and wiki pages; hybrid retrieval with reranking; a tenant-safe semantic cache cutting latency and token cost; and answer quality tracked with RAGAS on every change.

Decision Records

DR-001 · architecture

Combining BM25 keyword search with embeddings and reranking the candidates with a cross-encoder improves relevance over vector-only search

Context

Document Q&A questions mix fuzzy intent with exact terms — product names, error codes, section titles. Vector search handles meaning well but can miss exact-term matches; keyword search does the reverse.

Options considered

✗
Vector-only similarity search
Simple and good at paraphrase, but weak on exact identifiers and rare terms — the questions where users most expect a precise answer.
✓
BM25 + embeddings, merged and reranked by a cross-encoder
Keyword and semantic retrieval cover each other's blind spots; the cross-encoder scores each candidate against the actual question, pushing the best passages to the top.

Tradeoffs

Two retrieval paths to maintain and an extra reranking step that adds latency per query. Relevance is the product, so it's worth it.

Outcome

Retrieval relevance improved over the vector-only baseline, measured with the evaluation suite (DR-004).

DR-002 · architecture

Storing embeddings in PostgreSQL alongside tenant and document metadata keeps multi-tenant filtering and data management in one place

Context

The platform is multi-tenant: every query must only ever see the asking tenant's documents. Retrieval needs to filter by tenant and document metadata as part of the search itself.

Options considered

✗
A separate dedicated vector database
Another system to run and keep in sync with the relational metadata, and tenant filtering split across two stores.
✓
PostgreSQL with pgvector, next to the relational data
Embeddings, tenants, and document metadata live together, so tenant scoping is part of the same query and there's one store to back up and reason about.

Tradeoffs

A dedicated vector database may scale further for very large corpora. For this workload, one well-understood store is the better trade.

Outcome

Retrieval is tenant-scoped in the same query that does the search.

DR-003 · architecture

Serving semantically repeated questions from a Redis cache cuts latency and token cost without hurting answer quality

Context

Users in the same tenant often ask the same question in slightly different words. Running full retrieval and generation for each one wastes time and tokens.

Options considered

✗
No caching — run the full pipeline every time
Every repeat pays full latency and full token cost.
✗
Exact-match cache on the question text
Misses rephrasings, which are most of the repeats.
✓
Semantic cache in Redis keyed on question similarity
Rephrased repeats hit the cache, so they return quickly and cost no generation tokens.

Tradeoffs

Cached answers can go stale when documents change, and similarity thresholds need tuning so different questions don't collide. Cache keys must be scoped correctly (see failure).

Outcome

Repeated questions return faster and at lower token cost.

DR-004 · bet

Tracking faithfulness and context recall with RAGAS on every change is what turns 'it seems better' into a measured improvement

Context

Chunking, retrieval, reranking, and prompt changes all affect answer quality, often in opposite directions. Without measurement, it's impossible to know which changes actually help.

Options considered

✗
Judge quality by reading a few answers
Doesn't separate retrieval failures from generation failures, and misses regressions outside the examples you happen to read.
✓
RAGAS metrics on an evaluation set, tracked on every change
Faithfulness measures whether answers stick to the retrieved sources; context recall measures whether retrieval found what was needed. Both are tracked change over change.

Tradeoffs

Building and maintaining an evaluation set takes effort, and automated metrics need occasional sanity checks by hand. Worth it to iterate with confidence.

Outcome

Every change to chunking, retrieval, or prompts is scored on faithfulness and context recall before it's kept.

Where I Got It Wrong

Failure 1

The first version of the semantic cache matched only on question similarity, so in principle a similar question from one tenant could be served another tenant's cached answer — caught in testing before it shipped

Cost: A redesign of the cache keys, and a reminder that in a multi-tenant system every shortcut is a potential data leak

Lesson: In a multi-tenant system, scope every cache key by tenant and by the version of the document set — and write a test that tries to read across tenants

Failure 2

Fixed-size chunking split tables and section headings away from their content, which showed up as low context recall on questions about structured parts of documents

Cost: Answers that missed information sitting just across a chunk boundary

Lesson: Chunk along document structure — headings, sections, tables — and let the context-recall metric tell you when chunking is the problem rather than the prompt

Long-Term Impact

A reference implementation for production RAG: hybrid retrieval, tenant-safe caching, cited answers, and quality measured on every change.