Why We Moved Our AI Context to MongoDB (And How It Reduced Our Token Usage by 73%)
Rebuilding AI context from conversation history on every session is expensive and fragile. Here's the architecture we built — storing context in a database and fetching only what's relevant — and what it actually cost us.
Every AI session that starts with “let me catch you up on what we’ve been working on” is paying twice: once for the conversation that established the context originally, and again for the tokens spent re-establishing it.
We ran this pattern for six months before we got frustrated enough to build something better. Here’s what we learned.
The Problem With Conversation-Based Context
Conversational AI is stateless by default. Each session starts fresh. The model doesn’t remember your name, your preferences, your prior decisions, or what you were working on yesterday.
The traditional workaround: start every session with a long context-setting message. Or work in one continuous conversation until it gets so long the model starts forgetting things at the beginning (context window limits) or the session becomes unwieldy.
Neither approach scales. The long context-setting message costs 1,000-3,000 tokens per session. The never-ending conversation eventually breaks or becomes too expensive to continue.
What a Context Database Actually Is
A context database stores structured records of things the AI should know — persistently, queryably, across sessions and across machines.
In our implementation, each record has:
- A type (user preference, project decision, feedback, reference)
- A description used to decide relevance
- A body with the actual content
- Tags for filtering
When a session starts, instead of pasting in a giant context block, the system queries: “what context records are relevant to this task?” and retrieves only those — typically 200-500 tokens instead of 2,000-3,000.
The Architecture
We use MongoDB (FerretDB in our homelab setup) to store context records. A session hook automatically saves checkpoints as work progresses — decisions made, patterns observed, key discoveries. A query function retrieves relevant context at session start using keyword matching against descriptions and tags.
The result: context that was being re-established conversationally at 2,000-3,000 tokens per session now costs 200-400 tokens to retrieve from the database. That’s a 73-85% reduction in context-overhead tokens.
What This Enables Beyond Cost Savings
Persistent context across machines. The context database is shared — work started on one computer is immediately available on another, without copying conversation history.
Structured context is more reliable than conversational context. A database record that says “never use heredocs over SSH — they fail in /bin/sh contexts” is clearer and more reliably applied than a conversation memory that said roughly the same thing six weeks ago.
Historical context is searchable. Instead of trying to remember when you made a decision or why you adopted a particular pattern, you can search the database.
Implementation Notes
The query approach matters as much as the storage. Dumping everything into the context is almost as bad as re-establishing conversationally. The key is fetching only what’s relevant — which requires good descriptions and tags on stored records, and a retrieval mechanism that matches task-specific queries against those descriptions.
We built a context_search tool that does semantic keyword matching against record descriptions. Typical retrieval: 3-8 relevant records out of several hundred stored — enough to reestablish context without overwhelming the token budget.
The source code for our implementation is on the Enthropic Data GitHub.