LLM wiki vs RAG vs vector memory: when to compile

Updated · By the Cozy team

Use RAG or vector memory when you have a large, changing pile of documents and need fast lookups of specific passages. Use an LLM wiki when knowledge is reused often, needs to stay consistent, and should be readable and correctable by you. An LLM wiki compiles sources into linked pages once; RAG re-derives answers from chunks every time.

Three ways to give an agent knowledge

RAG (retrieval-augmented generation) splits documents into chunks, indexes them, and fetches the best matches at question time. The model reads those chunks and answers. Nothing is synthesized in advance.

Vector memory is a store of extracted facts or snippets that an agent writes to and reads from, usually ranked by embedding similarity. Products such as Mem0, Zep and Letta sit in this area. The memories are typically records inside a service, not files you open.

An LLM wiki is a folder of Markdown pages that an agent maintains. Sources go in, the agent writes and updates linked pages, and later questions are answered from those pages. The pattern was popularized by Andrej Karpathy's LLM Wiki gist, which describes three layers (raw sources, an LLM-owned wiki, and a schema file) and three operations: ingest, query and lint.

Where compiling wins

Compiling does the synthesis work once and keeps it. That matters in a few situations.

Where retrieval wins

Be honest about the costs of compiling. Writing and maintaining pages takes agent effort, and the gist's own comment thread asks whether ingest cost pays off against direct retrieval.

Side by side

FactorRAG over chunksVector memoryLLM wiki
Unit of storageDocument chunksExtracted memoriesLinked Markdown pages
SynthesisAt query timeAt write time, per factAt ingest and maintenance time
Human-readableRarelyVaries by productYes
Handles huge corporaYesModerateNeeds search alongside
ContradictionsLeft to the modelOften overwrittenCan be recorded
ProvenanceChunk sourceVariesCitations to sources

They combine well

Most real setups use more than one. A common split is to keep raw material as sources, compile the parts you reuse into pages, and search across both. The wiki becomes the curated layer and retrieval covers the long tail.

How Cozy fits

We make Cozy, a private wiki and cloud file space for each person and their AI agents. It sits on the compile side of this comparison, with a few design choices worth stating plainly.

Mem0 and similar services are built to be embedded in apps and return memories through an API. Their pricing page lists graph memory and consolidation features on a custom tier. Cozy stores readable files you can export and edit, but it does not offer the vector search those products are built around.

A practical rule

  1. Keep everything you might need as a source.
  2. Compile a page when you notice the same facts being re-explained or re-fetched.
  3. Let retrieval handle the rest until a topic earns a page.
  4. Run a health check regularly so compiled pages do not drift from their sources.

For the broader pattern, read about the LLM wiki and how it differs from built-in AI memory.

Frequently asked questions

Is an LLM wiki a replacement for RAG?

Not usually. An LLM wiki suits knowledge you reuse and want to keep consistent, while RAG suits large or fast-changing collections where you need passage lookup. Many setups keep both.

Does Cozy use embeddings or a vector database?

No. Cozy search is lexical and paginated, and the link graph is built from Markdown conventions. Cozy also runs no model itself, so you bring your own agent.

How big can an LLM wiki get before it needs search?

Karpathy has said index-based navigation works at roughly 100 articles or 400,000 words. Past that, add search over the wiki and its sources. Cozy provides lexical search through the MCP server.

How do I keep compiled pages from going stale?

Cite sources on each page and run lint regularly. Cozy's health check flags pages whose cited source changed after the page was last edited.

Give your agents a memory worth keeping

Cozy gives every account a private wiki that Claude, Codex, ChatGPT and other MCP clients can read and maintain.

Start your wiki