Module 8 of 11 · Advanced · 9 min read

Meta-prompting and prompt templates

Have models write and improve your prompts, and turn the winners into reusable templates with variables.

Key takeaways

1Models are good at drafting and critiquing prompts. Use them, then test the result.
2Feed real failures back in: here is the prompt, the bad output and what I wanted.
3Templates with named {{variables}} make prompts reusable and testable.
4Automated prompt optimisation exists, but it needs a good evaluation set to optimise against.

01Let the model write the first draft

Describing a task to a strong model and asking it to write the prompt is often faster than writing it yourself, and the result tends to include things you would forget: edge cases, output formats and clarifying rules. Treat the output as a draft to test, not a finished product.

Example

I need a prompt for this task: {{task}}. It will be used by {{who}}, with inputs like the samples in <samples>. Good output looks like: {{description or example}}. Common failure I want to avoid: {{failure}}. Write the prompt. Use tags for the variable inputs, explain the reasons behind important rules, and keep it as short as it can be while covering the edge cases in the samples. Then list three test inputs most likely to break it.

02Improve prompts from real failures

The most effective meta-prompt shows the model what went wrong. Give it the current prompt, the input, the output you got and what you wanted, and ask it to diagnose which part of the prompt caused the gap before proposing a fix. Diagnosis first stops it from rewriting everything.

Example

<prompt>{{current prompt}}</prompt> <input>{{input}}</input> <output>{{bad output}}</output> <wanted>{{what good looks like}}</wanted> Explain which instruction, missing context or example most likely caused the gap. Then propose the smallest change to the prompt that would fix it without breaking other cases. Show the change as a diff.

03Templates and variables

Once a prompt works, turn it into a template: fixed instructions plus named placeholders such as {{audience}} or {{document}}. Templates make prompts shareable, let you swap inputs during testing, and keep the tested wording from drifting as people copy and edit it.

Name variables for what they contain, not where they go: {{customer_email}}, not {{input2}}.
Wrap user-supplied variables in tags so their content is treated as data.
Give defaults for optional variables, or remove the section when empty.
Version templates and note which test set they passed.

04Automated prompt optimisation

Researchers have shown models can search for better prompts automatically. Automatic Prompt Engineer (Zhou et al., 2022) generates and scores candidate instructions; OPRO (Yang et al., 2023) uses a model as an optimiser over prompts and their scores; and frameworks such as DSPy (Khattab et al., 2023) compile prompt pipelines against a metric.

All of these need the same thing: a representative evaluation set and a metric that captures what you care about. Without that, an optimiser happily overfits to whatever is easiest to score.

05Pitfalls

Meta-prompting speeds up the process; it does not replace testing.

Bloat: model-written prompts tend to be long and generic. Ask for the shortest version that passes your tests.
Overfitting: tuning a prompt until it aces five examples can make it worse on everything else. Hold back some test cases.
Lost reasons: rewrites often drop the 'because' behind rules. Put them back.