Build an LLM evaluation dataset that separates retrieval from answers

An LLM evaluation dataset needs a task definition, representative inputs and a way to judge outputs. For a retrieval-based assistant, record which evidence should be found separately from what a correct answer should say. Otherwise a single pass rate hides whether the failure belongs to retrieval, generation, citations or abstention.

Our task answers factual questions about three original fictional club notes. The dataset includes development and test splits, answerable questions and questions unsupported by those notes. It is a small teaching fixture, not a representative benchmark of any deployed model.

Define the fields before writing a scorer

Each record contains an ID, split, question, expected short answer and expected source ID. Unsupported questions have no answer or source. The source collection is: club-1: “The Saturday workshop meets in the Cedar Room.”; club-2: “The loan desk lends blue toolkits on Tuesdays.”; club-3: “The library closes at six on Fridays.” Keep those IDs stable when running the fixture.

The source collection and downloadable JSONL records are public original teaching material; no private conversations are used.

The rubric has separate checks. Retrieval succeeds when the expected source ID appears among retrieved IDs. Exact answer match uses a declared lowercase-and-whitespace normalization. Citation validity checks that a cited ID was actually retrieved. For this closed task, source correctness also requires the expected source ID. Unsupported cases pass only when the response abstains and provides no citation.

Exact match is intentionally narrow. A legitimate paraphrase may fail it, and a wider task may need a human rubric or a separately validated judge. Keep that limitation beside the score rather than calling a string match a universal factuality test.

Score deliberate failures

The predictions below are manually authored fixtures. They demonstrate the scorer; they are not claimed outputs from a live LLM. The failures include correct retrieval with a wrong answer, a right answer with an invented citation and an unsupported answer instead of abstention.

import json
records = [
 {'id':'d1','split':'development','question':'Where is the Saturday workshop?','answer':'Cedar Room','source':'club-1'},
 {'id':'d2','split':'development','question':'Which day are toolkits lent?','answer':'Tuesdays','source':'club-2'},
 {'id':'d3','split':'development','question':'What is the annual membership fee?','answer':None,'source':None},
 {'id':'t1','split':'test','question':'When does the library close on Friday?','answer':'six','source':'club-3'},
 {'id':'t2','split':'test','question':'What color are the loan toolkits?','answer':'blue','source':'club-2'},
 {'id':'t3','split':'test','question':'Who sponsors the club?','answer':None,'source':None},
]
# (retrieved IDs, answer, cited ID): deterministic scorer test cases.
predictions = [(['club-1'],'Cedar Room','club-1'),
               (['club-2'],'Mondays','club-2'),
               ([],None,None),
               (['club-3'],'six','invented-source'),
               (['club-1'],'blue','club-1'),
               (['club-1'],'A local company','club-1')]
def normalize(value):
    return ' '.join(value.lower().split()) if value is not None else None
for row, (retrieved, answer, cited) in zip(records, predictions):
    unsupported = row['source'] is None
    retrieval_ok = None if unsupported else row['source'] in retrieved
    answer_ok = normalize(answer) == normalize(row['answer'])
    citation_valid = cited in retrieved if cited is not None else unsupported
    source_ok = cited == row['source']
    passed = answer_ok and citation_valid and source_ok
    print(row['id'], f'retrieval={retrieval_ok} answer={answer_ok}',
          f'citation_valid={citation_valid} source_correct={source_ok} pass={passed}')
encoded = '\n'.join(json.dumps(row, sort_keys=True) for row in records)
assert [json.loads(line) for line in encoded.splitlines()] == records
assert len({r['id'] for r in records}) == len(records)

Checked output:

d1 retrieval=True answer=True citation_valid=True source_correct=True pass=True
d2 retrieval=True answer=False citation_valid=True source_correct=True pass=False
d3 retrieval=None answer=True citation_valid=True source_correct=True pass=True
t1 retrieval=True answer=True citation_valid=False source_correct=False pass=False
t2 retrieval=False answer=True citation_valid=True source_correct=False pass=False
t3 retrieval=None answer=False citation_valid=True source_correct=False pass=False

Read failures by component

Record d2 finds the right note and cites it, but says Mondays instead of Tuesdays. More retrieval tuning would not directly repair that answer error. Record t1 gives the expected short answer but cites an ID that was never retrieved. Record t2 happens to give the expected answer while retrieving and citing the wrong note; answer match alone would reward unsupported behavior.

Record t3 invents an answer to an unsupported question. A system that always produces something fluent can look helpful while failing the actual task. Include absent-answer questions and score abstention explicitly.

These six fixture outcomes are checks on the rubric, not a final model score. Before reporting a model result, run actual model predictions over the frozen test split, retain the evaluation version and describe the population the cases represent.

Expand coverage without leaking the test set

Use development cases to improve prompts and retrieval. Keep test questions and expected outputs out of prompt examples and tuning decisions. Split near-duplicate questions, source versions and related conversations as groups so paraphrases do not leak across partitions.

Add categories for ambiguity, conflicting evidence, outdated sources, malformed output and instructions embedded in documents. Weight or report slices according to the intended application rather than letting abundant easy questions dominate an average.

The Generative AI syllabus covers evaluation; lessons are being published. Pair this rubric with retrieval metrics and run the local RAG pipeline against additional frozen questions. Validate structured outputs separately when a schema is part of the contract.

References

HELM evaluation paper — Liang and colleagues motivate explicit scenarios, metrics and multidimensional evaluation.

Original RAG paper — Retrieval and generation are distinct components in the architecture.