Build a local RAG pipeline with checked source citations

A RAG pipeline retrieves evidence, supplies it to a generator and returns an answer connected to that evidence. Retrieval success, answer correctness and citation validity are separate checks. A citation-looking string alone proves none of them.

This complete example uses three original fictional club notes, TF-IDF retrieval and a real local SmolLM2-135M-Instruct model. It downloads public model weights on first use and runs generation on CPU. No API key or private document is needed. The questions test one answerable case and one case absent from the notes.

Choose an auditable answer contract

To keep validation strict, the output must be a literal quotation found inside the retrieved note. The model proposes text; the application checks that proposal and attaches the source ID itself. A fluent paraphrase is rejected under this deliberately narrow contract, even if a person might consider it correct.

The check establishes that the published quotation appears in the selected note. It does not establish that the note is true or that any quoted sentence answers the question. The answerable fixture separately checks the expected room. A production system needs a broader evaluation set, source quality checks and a rubric appropriate to its answer format.

Our retrieval gate rejects a query with zero vocabulary overlap. This is useful for the supplied absent-topic fixture, but it is not a calibrated answerability detector: a misleading query can share words with irrelevant evidence.

Run retrieval, generation and publication separately

Use an isolated environment with NumPy, scikit-learn, PyTorch CPU and Transformers. The verified versions are in the batch manifest. Model revision, greedy decoding, evaluation mode and a single CPU thread are explicit. Model weights stay in the normal local Hugging Face cache; they are not website assets.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

torch.set_num_threads(1)
name = 'HuggingFaceTB/SmolLM2-135M-Instruct'
revision = '12fd25f77366fa6b3b4b768ec3050bf629380bac'
tokenizer = AutoTokenizer.from_pretrained(name, revision=revision, token=False)
model = AutoModelForCausalLM.from_pretrained(name, revision=revision, token=False).eval()
notes = {
 'club-1': 'The Saturday workshop meets in the Cedar Room.',
 'club-2': 'The loan desk lends blue toolkits on Tuesdays.',
 'club-3': 'The library closes at six on Fridays.',
}
ids = list(notes)
vectorizer = TfidfVectorizer()
index = vectorizer.fit_transform(notes.values())

def answer(question):
    scores = cosine_similarity(vectorizer.transform([question]), index).ravel()
    best = int(scores.argmax())
    if scores[best] == 0:
        return 'I do not know.', None
    source_id, source = ids[best], notes[ids[best]]
    messages = [
      {'role': 'system', 'content': 'Answer using only the source. Include the source id in square brackets. If not answered, say I do not know.'},
      {'role': 'user', 'content': f'[{source_id}] {source}\nQuestion: {question}'}]
    inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                           return_tensors='pt', return_dict=True)
    with torch.inference_mode():
        generated = model.generate(**inputs, max_new_tokens=40, do_sample=False,
                                    pad_token_id=tokenizer.eos_token_id)
    proposal = tokenizer.decode(generated[0, inputs['input_ids'].shape[1]:],
                                 skip_special_tokens=True).strip()
    if not proposal or proposal not in source:
        return 'I do not know.', source_id
    return f'{proposal} [{source_id}]', source_id

for question in ['Where does the Saturday workshop meet?', 'Volcanic mineral composition?']:
    response, retrieved = answer(question)
    print('question:', question)
    print('retrieved:', retrieved, 'answer:', response)
    if 'Saturday' in question:
        assert 'Cedar Room' in response and '[club-1]' in response
    else:
        assert response == 'I do not know.'

Checked output:

question: Where does the Saturday workshop meet?
retrieved: club-1 answer: The Saturday workshop meets in the Cedar Room. [club-1]
question: Volcanic mineral composition?
retrieved: None answer: I do not know.

Inspect what the model actually did

On the verified run, the small model produced the source sentence without a citation marker despite the instruction. The application still returned a valid source ID because it controls citation construction. This is precisely why output validation belongs outside the prompt.

The model's successful quotation on one question does not establish reliability across paraphrases, long documents or conflicting passages. Small changes in model revision, inference libraries or hardware can alter generated output. The strict publication check should remain even when the example is moved to a larger generator.

The unsupported question is rejected before generation because its vocabulary has no overlap. Add a question such as “What time is the Saturday workshop?”: it shares useful words but lacks a supported answer. That harder case belongs in the evaluation set and may expose a quotation that is real but not responsive.

Expand the pipeline one boundary at a time

For longer documents, preserve stable document and passage IDs, record the selected evidence and respect the formatted token budget. Chunking can separate an answer from a qualification, so evaluate passage boundaries rather than merely increasing top-k.

Use RAG retrieval evaluation for evidence ranking and an LLM evaluation dataset for answer, citation and abstention checks. The Generative AI syllabus connects these components; lessons are being published. Prompt-injection boundaries become essential when retrieved notes contain instructions as well as facts.

References

Original RAG paper — Lewis and colleagues describe retrieval-augmented generation; this miniature pipeline is an original teaching implementation.

SmolLM2 model card — Model distribution, usage and limitations.

Transformers chat templates — Formatting the actual model input.