“RAG” — retrieval-augmented generation — has accumulated a lot of vendor mystique. It’s more than worth a solid definition: find the documents relevant to a question, paste them into the prompt, ask the model to answer from them.
It’s not really a trick; just how you get an LLM to answer questions about your content. This means the relevant data sources are your own docs, wikis, and knowledge bases. Responses have direct citations rather than hallucinations reconstructed from training data.
rag-over-markdown is a reference implementation of that loop over a folder of markdown files, built on the stack from the last several articles: ext-infer for embeddings and generation, ext-turbovec for retrieval, ai-toolkit for the plumbing. Every stage local.
Point it at your docs and ask questions.
Stage 1: Chunk
Whole documents embed poorly — one vector averaging ten topics matches none of them well. The corpus gets split with ai-toolkit’s recursive chunker, which tries paragraph boundaries first and only degrades toward sentence and character splits when a section runs too long.
Markdown’s explicit structure (headings, paragraphs) makes those boundaries unusually clean, which is exactly why it’s the right corpus for a reference implementation. Each chunk keeps its source path and heading trail as metadata to help construct citations.
Stage 2: Embed and index
<?php
$embedder = Model::load('models/Qwen3-Embedding-0.6B-Q8_0.gguf');
$index = new IdMapIndex(dim: 1024, bitWidth: 4);
foreach ($chunks as $id => $chunk) {
$vector = $embedder->embed($chunk->text);
$index->addWithIds(Vectors::pack($vector->toArray()), [$id]);
}
$index->write('corpus.tvim'); // index once, query forever
Indexing is a one-time (or on-change) batch job; the persisted index loads instantly afterward. Chunk ids map back to a side table of text and metadata.
Stage 3: Retrieve
A question comes in. Embed it with the same model, search the index, look up the winners:
<?php
$query = $embedder->embed($question);
$result = $index->search(Vectors::pack($query->toArray()), k: 5);
$context = array_map(
fn (int $id) => $chunkStore->get($id),
$result->ids(),
);
Single-digit milliseconds against the quantized index, no network anywhere.
Stage 4: Generate, grounded
The retrieved chunks are added to the prompt as context, with instructions that aid in the anti-hallucination work:
<?php
$prompt = Prompt::system(
'Answer using ONLY the provided context. ' .
'Cite sources by file path. ' .
'If the context does not contain the answer, say so.'
)->withUser($contextBlock . "\n\nQuestion: " . $question);
$response = $generator->chat(
$prompt,
maxTokens: 512,
temperature: 0.0
);
echo $response->answer();
Three specific configuration flags help ground answers in reality:
- temperature 0 (grounded Q&A wants determinism, not creativity)
- the escape hatch (“say so” beats a confident fabrication when retrieval misses)
- citations by construction — the model cites the metadata we attached in stage 1, so every answer is verifiable against the source files.
What a small model can and can’t do
To be honest and up front, a sub-1B generator is not a frontier chatbot.
But RAG changes the job description — the model doesn’t need to actually know anything; it needs to read five paragraphs and synthesize an answer from them. Reading comprehension over supplied context is exactly where small models punch above their weight, and retrieval quality matters more than generator size.
When you do need stronger synthesis, the ai-contracts work from last week lets you swap the Generator to a larger local model — or even a cloud one — while retrieval stays local and private. Your corpus never leaves the machine even if the final generation step does.
This pipeline handles anything already in text. Next we’ll feed it something harder: hours of audio.
