What RAG does and what it does not
RAG stands for retrieval-augmented generation. The model answers from passages the system retrieves from a document collection at runtime, not from what it saw in training. The sources can change without anyone retraining a model, and every statement points to a file you can check it against.
RAG does not make the model smarter. If the answer is in no document, or retrieval does not find the passage, the best model cannot help. The stages before the model call therefore decide most of the quality, and this text is about them.
Preparing: read, split, embed
Before anyone asks, TippelPi reads the documents (text, Markdown, PDF, Word, Excel, e-mail) and splits them into chunks. It joins paragraphs until they reach 700 characters and never cuts mid-sentence. With every chunk it stores the heading the chunk sits under. For the embedding model it puts the document title and the heading in front of the chunk, because a paragraph under “Belgium” in a text on dismissal protection contains neither word. It stores and cites the chunk without this prefix.
We kept the 700 characters short on purpose. On the Pi the model writes a few tokens per second and first has to read every chunk it is given. Four short chunks therefore allow an answer in acceptable time, and four long ones do not. Context is lost in exchange, because a table or a long list breaks into several chunks, of which retrieval may find only one. Chunking Strategies for RAG describes the trade-off.
Each chunk then goes into two indexes. The embedding model multilingual-e5-small turns it into a vector that a local vector database stores, and a full-text index on SQLite FTS5 takes the same text. The system also stores the source file, the running number of the chunk and the groups allowed to read it.
We swapped the embedding model after a comparison on 74 German questions (31 asked directly, 25 paraphrased, 18 off-topic). For paraphrased questions the English model used before put the right chunk among the first three in only 72 % of cases, the multilingual model in 92 %. The English model also could not separate off-topic questions from answerable ones at any threshold.
Two retrieval channels, fused by rank
Vector search finds chunks that mean the same as the question even when no word matches. It easily misses the chunk in which a part number, a section reference or a proper name appears literally, because such strings carry little meaning in vector space. Full-text search behaves the other way round. TippelPi therefore queries both channels and merges the result lists with reciprocal rank fusion:
def _rrf_fuse(dense, lex, top_k, k=60): scores, by_key = {}, {} for rank, h in enumerate(dense): scores[h["id"]] = scores.get(h["id"], 0.0) + 1.0 / (k + rank + 1) by_key[h["id"]] = h for rank, h in enumerate(lex): scores[h["id"]] = scores.get(h["id"], 0.0) + 1.0 / (k + rank + 1) by_key.setdefault(h["id"], h) fused = sorted(by_key.values(), key=lambda h: -scores[h["id"]]) return fused[:top_k]
The excerpt leaves out the original’s fallback key for chunks without an id. The function computes with rank positions only, because cosine distance and BM25 score sit on different scales and cannot be added in any meaningful way, whereas their ranks can. A chunk that both channels find receives two terms and moves up. We want that, because two independent searches that find the same chunk give the strongest signal this pipeline has. The constant 60 dampens the gap between the top positions, so that a first place in one channel does not outvote every double hit.
The threshold: no chunk rather than a weak one
A vector search always returns hits. When the collection says nothing about the question, it returns the least unsuitable chunks. A model that receives those chunks phrases an answer from them and attaches a source reference that looks real.
TippelPi therefore discards, in both channels, every hit whose cosine distance to the question exceeds 0.18; a full-text hit has to clear the same threshold. In an earlier version one word shared with the question was enough for a full-text hit, so the system could never refuse. We set the value for this embedding model on part of the test questions and confirmed it on the held-out rest. There the right chunk was always among the first three for directly asked questions and in 92 % of cases for paraphrased ones; the system refused 93 % of the off-topic questions and no answerable one. An allowance of just 0.04 for full-text hits cut the refusal rate in the measurement on the whole question set from 89 to 44 %. If the best hit lies above 90 % of the threshold, the answer notes that the documents cover the question only in part. The prompt takes at most four chunks.
Refusing without asking the model
If neither channel then returns a chunk, the system searches a second time with a differently built query. If that pass comes back empty as well, the server returns a fixed refusal sentence and does not call the model. The code makes this decision, not the prompt, because a model that is never asked cannot invent an answer.
The prompt repeats the rule for the case where chunks exist but the answer is not in them. It also requires that every statement carries a source, that a partly answerable question names the missing part explicitly, and that two chunks contradicting each other both appear with their source. We rely on this second layer less than on the first, because a model can follow an instruction and a server has to. LLM Hallucinations has more on this.
Follow-ups: the raw question searches first
A follow-up such as “and the short version?” makes no sense on its own, which is why the conversation is usually condensed into a search query. In testing the condensed query steered retrieval back to the old topic. Three questions about firewalls were followed by one about Docker, and the condensed query matched firewall chunks only, while the raw question alone would have hit.
Since then the raw question searches first. If it fills all four slots, the system does not issue the condensed query. If slots stay open, which for a pure follow-up means all of them, the condensed query fills only those.
Permissions on the chunk, in both channels
Every chunk carries the groups allowed to read it. The vector index filters on them inside the query itself. Full-text search does not know this filter and needs the same check a second time, otherwise a leak opens that no test of the vector search uncovers. We tested this on 10 October 2026 with 1,200 queries in which a locked chunk itself was the question: 200 chunks from each of three groups, asked by the other two. On the test corpus no locked chunk came through, and without the filter the search found the source document in all 600 control cases. The figures are also on the project page.
How good the answers are
We measured the answers as well, on 127 questions: 77 answerable ones with a reviewed reference, 28 off-topic ones and 22 that ask for company knowledge the documents do not contain. With Qwen3-8B on a GPU server, 82 % of the answers were fully correct and 90 % fully supported by the chunks. The system refused all off-topic questions, 89 % of them without calling the model, and 95 % of the company-specific ones. More chunks in the prompt raised correctness a little and doubled the invented answers to company-specific questions, which is why it stays at four.
Limits
A reranker is provided for but switched off, because the multilingual model for it takes up a good gigabyte and costs seconds per question on the Pi. We measured answer quality on a GPU server and not on the Pi, and a second, larger model judged it, not a person; differences of one to three questions are within noise. The threshold of 0.18 holds for this embedding model, and whoever changes the model has to determine it again. Paraphrased questions remain the weakest point, because in 8 % of cases the right chunk is not among the first three. The first embedding pass manages about 6.4 chunks per second on the Pi, so 10,000 documents take roughly four hours once.