How the chat works: RAG, grounded in my own writing

Why ground a chatbot at all

A raw language model will happily answer “what does Peter do?” from whatever it half-remembers from training — which, for a specific private person, is mostly nothing, so it fills the gap with plausible fiction. That’s the worst possible failure mode for a personal site: a confident, wrong biography.

Grounding flips the contract. The model is told, in effect: here are the only passages you’re allowed to use; answer from these or admit you don’t know.Every claim traces back to something I actually wrote, and the answer shows you the sources. When I want the chat to know something new, I edit the writing — not a prompt, not a database row.

What RAG actually is

RAG — Retrieval-Augmented Generation — is two steps glued together. Retrieval: find the passages most relevant to the question. Generation:give those passages to the model and have it write the answer. The “augmented” part is that the model’s own knowledge is augmented, at question time, with text it didn’t see during training.

The trick that makes retrieval work is embeddings: turning text into a list of numbers (a vector) such that passages about similar things land near each other in space. Find the question’s neighbors, and you’ve found the relevant passages. So yes — to answer the obvious question directly: this chat is RAG.

From writing to vectors

Everything the chat can cite lives in MDX files under content/ — the About page, the six case studies, the essays. A build step chunks and embeds them:

• Chunk by heading. Each file is split on its ## sections, so one chunk is one coherent topic. The frontmatter (title + summary) becomes its own lead-in chunk. Anything over 4,000 characters is split further on paragraph breaks.
• Embed each chunk. Every chunk goes through openai/text-embedding-3-small (via the Vercel AI Gateway) and comes back as a 1,536-number vector.
• Only re-embed what changed.Each chunk’s text is hashed (SHA-256). On the next build, unchanged chunks keep their existing vector; only edited ones get re-embedded. Editing one section of one file re-embeds a handful of chunks, not the whole corpus.

  about: +2 -2 =6
  work/bonvivant: +0 -0 =5
  writing/the-software-house-of-one: +1 -1 =4
  Done. Added 3, deleted 3, unchanged 64.

That’s a real run from editing two sections: 3 chunks changed, 64 untouched. No wasted embedding calls, no wasted tokens.

Where the vectors live

Chunks and their vectors sit in a single Postgres table (Supabase + the pgvector extension). The vector column is indexed with HNSW using cosine distance— an approximate-nearest-neighbor index that finds the closest vectors fast without scanning every row. A scope column lets two knowledge bases share the table: peterwd for this chat, and pico-demo for the embedded restaurant demo. Every search filters on scope, so the two never bleed into each other.

Answering a question, step by step

When you send a message, the chat route runs this pipeline before a single word is generated:

1. 1.Embed the question with the same model used at index time, so it lands in the same vector space as the chunks.
2. 2.Vector search for the top 5 chunks whose cosine similarity beats 0.3. This is the core retrieval call.
3. 3.Check confidence. If the best match is below 0.45, a second search (threshold 0.15, titles only) collects nearby section names, so the model can point you toward adjacent topics instead of just saying “nothing matches.”
4. 4.Build the prompt.The retrieved chunks, a voice profile (style only, no biography), and grounding rules (“use only this context; don’t invent”) go into the system prompt.
5. 5.Generate & cite. anthropic/claude-sonnet-4-6 streams the answer. The sources behind it attach as citation chips — suppressed when the top match is below 0.25, so a thin retrieval never wears a citation it didn’t earn.

Four questions, four retrievals

These are real results from the live vector search — the actual chunks and similarity scores that come back for four questions. Watch the threshold do its job, including the last one, where the right answer is to retrieve nothing.

What grounding buys you

• Honest gaps.When the corpus doesn’t cover something, the chat says so — it doesn’t fabricate a confident answer.
• Real citations.Every grounded answer shows the sources it drew from, and they’re suppressed when retrieval was too thin to count.
• Edits flow from the writing.To change what the chat knows, I edit an MDX file and re-run the embeddings. The rendered page and the chat’s knowledge come from the same source.
• No hallucinated biography. The voice profile controls how it talks; the indexed content controls what it can claim. Identity facts stay grounded, never improvised.