Module 9 of 11 · Advanced · 11 min read

Testing prompts: golden sets, rubrics and judges

Replace vibes with evidence: build a golden set, write rubrics, and use model judges without being fooled by them.

Key takeaways

1A golden set of real inputs with expected results is the foundation of every reliable prompt.
2Prefer cheap programmatic checks; use rubrics for what code can't judge.
3Model judges have known biases (position, length, self-preference). Mitigate and calibrate them.
4Re-run your evals whenever you change the prompt or the model.

01Why vibes fail

Reading a few outputs and deciding they look good is how most prompts are tested, and it is why many fail in production. People notice fluent prose and miss subtle errors; they test the cases they thought of, not the ones users send; and they remember the last output better than the first.

An evaluation (eval) is just a repeatable way to measure whether a prompt meets its criteria on a fixed set of inputs. It does not need to be elaborate to be valuable.

02Build a golden set

A spreadsheet with columns for input, expected result and pass or fail per version is enough to start.

Start with 20 to 50 real inputs, and more as the prompt matters more.
Cover common cases, known hard cases, and adversarial ones such as empty input, the wrong language or injected instructions.
For each, record either the expected output or the criteria a good output must meet.
Add every real failure you discover in production to the set.
Keep some cases aside that you never tune against, to catch overfitting.

03Choose the right kind of check

Most good eval suites combine several types.

Exact or programmatic: valid JSON, correct label, contains a citation, under N words, code compiles, tests pass. Cheap and objective, so use them wherever possible.
Reference comparison: does the answer contain the key facts from a reference answer?
Rubric grading: a person or model scores against explicit criteria.
Pairwise preference: which of two versions is better for this input? Good for comparing prompt versions.
Human review: the final word for subjective quality and high-stakes outputs.

04Using a model as judge

Model judges let you grade hundreds of outputs quickly, and Zheng et al. (2023) found that strong model judges can agree with human preferences reasonably well. The same work documented biases you must design around: position bias (favouring an answer because of where it appears), verbosity bias (favouring longer answers) and self-enhancement bias (favouring outputs from the same model).

Use specific, preferably binary criteria rather than a vague 1 to 10 score.
Ask for the evidence or reasoning before the verdict.
In pairwise comparisons, run both orders and only count consistent wins.
Calibrate: grade a sample yourself and check the judge agrees with you.
Where possible, use a different model family as the judge.

05Example judge prompt

Binary criteria with quoted evidence are much easier to audit than a single score.

Example

You are grading an answer to a customer question. Be strict. <question>{{question}}</question> <sources>{{help articles}}</sources> <answer>{{answer}}</answer> For each criterion, quote the relevant part of the answer, then write PASS or FAIL: 1. Answers the actual question in the first two sentences. 2. Every factual claim is supported by the sources. 3. Does not promise refunds, discounts or timelines not in the sources. 4. Under 150 words. Finish with OVERALL: PASS only if all four pass.

06Make it a habit

Run the eval on every prompt change, every model change and periodically on live samples. Track the pass rate per version. When a change improves one criterion and hurts another, you see it immediately instead of hearing about it from users.

Tip

For non-engineers: a spreadsheet with inputs in column A, outputs from versions 1 and 2 in B and C, and a pass or fail column per criterion is a real eval. Start there.