LLM wiki lint: what to check and how Cozy scores it

Updated · By the Cozy team

LLM wiki lint is a set of automated checks that find problems in a wiki an AI agent maintains: broken links, stale pages, conflicts, orphans and unsourced claims. In Cozy, the cozy_health tool reports these checks, rolls them into a score from 0 to 100, and lists fixes in priority order so an agent can repair the wiki.

Why an LLM wiki needs linting

In the LLM wiki pattern, an agent ingests sources, writes Markdown pages, and answers questions from them. Karpathy's LLM Wiki gist names three operations: ingest, query and lint. Ingest and query get most of the attention. Lint is the step that keeps the wiki trustworthy after hundreds of agent edits.

Without it, small errors pile up. A renamed page leaves dead links. A source changes and the summary written from it goes quietly out of date. Two pages record different values for the same fact. Nobody notices until an agent repeats the wrong one.

Cozy calls its lint pass Health. It is exposed to agents as the cozy_health tool on the hosted MCP server.

What the checks catch

Each check maps to a failure that is common when agents write most of the pages.

CheckWhat it flags
Broken linksA wikilink that resolves to no page
Duplicate titles or aliasesTwo pages that answer to the same name
Stale pagesA cited source changed after the page was last edited
ConflictsPages carrying a CONFLICT: line
OrphansPages that nothing links to
Unreachable pagesPages that wiki/index.md cannot reach through links
Unsourced pagesPages that cite no source
Missing type or descriptionFrontmatter that is incomplete
Oversized pagesPages too large to read comfortably
Uncited sourcesEvidence in sources/ that no page uses

The stale-page check depends on source citations. A page that links into sources/ is recorded as citing that source. When the source is re-imported with new content, the page is older than its evidence and gets flagged.

Cozy rebuilds the graph from Markdown on every save, so link checks reflect the current files. A name link resolves by path, path without extension, file stem, title or alias, ignoring case and accents. A link that matches none of these counts as broken. Links inside code are ignored, so examples in a page do not create false alarms.

Frontmatter keys whose values are wikilinks become typed relations, and relation:: [[target]] lines work too. The typed links page covers the full set of conventions.

How the health score works

Each page starts at 1.0 and loses weight for every issue found on it. The score is the average across pages, multiplied by 100. A wiki with no issues scores 100. The tool also returns a next list that orders fixes by priority, so an agent does not have to decide where to start.

Cozy does not publish per-issue weights on this page, and the score is only meaningful relative to your own wiki over time. Use it as a trend line and read the issue list for specifics.

Running a health pass with an agent

Cozy runs no model of its own, so your agent does the fixing. A typical loop looks like this:

  1. Call cozy_health and read the score and the next list.
  2. Use cozy_read or cozy_context on the first flagged page to see its neighbors, evidence and warnings.
  3. Fix the issue with cozy_edit, passing the page's expectedRevision.
  4. Call cozy_health again and confirm the score moved.

Edits require expectedRevision, and concurrent edits return a conflict instead of overwriting. If an agent makes a bad fix, cozy_history and cozy_restore bring back any of the last 50 revisions.

For broken links after a rename, cozy_move keeps history and returns the links that need repair. For duplicates, cozy_resolve maps names to pages and surfaces likely duplicates.

Lint and dreaming

Health tells you what is wrong. Dreaming is the offline consolidation pass your own agent runs to fix it. A dream run orients with cozy_health, scopes work to cozy_changes since the last run, verifies claims against sources, keeps superseded values with dates, records CONFLICT: lines, merges duplicates, writes typed relations, and keeps wiki/index.md short. The last-run time lives in agents/dream/state.md. See memory consolidation for the longer explanation.

What lint cannot do

Lint is structural. It can tell you a page cites no source, but not whether the page is true. It can detect that a source changed, but not whether the page's claim still holds. A human or an agent still has to read the source and decide.

It also does not replace review of what agents write. Scoped credentials limit what an agent can touch, and the revision history lets you undo it, but neither judges the content.

If you keep a wiki in plain Markdown elsewhere, you can approximate some of these checks with scripts. Link checkers and orphan finders are easy to write. Source-versus-page staleness and conflict counting are harder to bolt on, which is where a built-in pass helps. Obsidian and similar tools have strong plugin ecosystems for local vaults, so compare options on the compare page before deciding.

Getting started

Sign in at cozy.cloud, connect an agent to https://cozy.cloud/mcp over MCP. Then ask it to run cozy_health and work through the next list. The FAQ covers setup questions.

Frequently asked questions

What is LLM wiki lint?

It is an automated pass over a wiki that an AI agent maintains, looking for broken links, stale pages, conflicts, orphans and unsourced claims. The term comes from the lint operation in Karpathy's LLM Wiki pattern. In Cozy it is the cozy_health tool.

How is the Cozy health score calculated?

Each page starts at 1.0 and loses weight for every issue on it. The score is the average across pages times 100. The tool also returns a next list of fixes in priority order.

What makes a page stale in Cozy?

A page is stale when a source it cites changed after the page's last edit. The page links into sources/, so Cozy can compare the two timestamps. See source citations.

Does Cozy fix the problems it finds?

No. Cozy runs no model itself, so your own agent reads the report and makes the edits. The offline memory consolidation pass called dreaming is one way to do that on a schedule you choose.

Can lint tell me whether a page is correct?

No. It checks structure, links, citations and timestamps. A person or an agent still has to compare claims with the sources.

Give your agents a memory worth keeping

Cozy gives every account a private wiki that Claude, Codex, ChatGPT and other MCP clients can read and maintain.

Start your wiki