Evaluation

Testing a prompt on a fixed set of inputs and scoring the outputs against acceptance criteria, so improvements can be measured instead of guessed.

Last updated: November 9, 2026 · Markdown version

← Back to glossary

How to apply it in prompts

Use this concept to reduce ambiguity and improve consistency. If you need strict formatting, combine it with a JSON guide and a per-model template.

Related terms:Hallucinations Constraints

Strict JSON Models Packs

What it means

Definition: prompt evaluation means running the same prompt over a small, fixed set of test inputs — including awkward edge cases — and grading the outputs with a rubric.
Why it matters: one good answer proves little. Without a fixed test set, every edit is a coin flip and a change that fixes one case can quietly break another.
How to apply it: write 5–20 inputs that cover typical and difficult cases, define what a correct output looks like (format, facts, tone), score each criterion (for example 0–2) and rerun after every prompt change.
Common mistake: tuning the prompt against the same three examples you keep staring at. Keep a few test inputs aside that you never tune on.

Copy-paste example

Create an evaluation set for this task: [task description]
Return:
1) A rubric with 5 criteria, each scored 0–2, with what each score means.
2) 8 test inputs: 3 typical, 3 edge cases, 2 adversarial.
3) For each input, what a perfect output must contain.

Opens the analyzer with the prompt prefilled.

Open in Promptea

FAQ

How many test cases do I need?

Five to twenty is enough to start. Quality matters more than count: include the cases that have failed before.

Is a prompt score the same as an evaluation?

No. Promptea's 0–100 score rates how well specified the prompt itself is, using fixed rules. An evaluation judges the outputs a prompt produces on your own test inputs. They complement each other.