How to test one prompt across ChatGPT, Claude, Gemini, Grok and DeepSeek

A practical method to compare AI models on your own task: fix the prompt and the inputs, write pass/fail criteria before you look at answers, run a small sample, and decide on evidence instead of impressions. Includes templates for test cases, a scoring rubric and a side-by-side review.

← Back to guides

Why public rankings aren't enough

Benchmarks measure specific tasks under specific conditions. Your task — your documents, your format, your language — may behave differently.
Models change: new versions ship and older ones are retired, so a comparison you ran months ago may no longer apply.
Cost and speed matter as much as quality for repeated work: a slightly weaker answer that is much faster may be the right choice.
Use rankings to pick 2–3 candidates, then test those candidates on your own examples.

Set up a fair test

Write the criteria before running anything: what a correct answer must contain, what it must not contain, and the required format.
Collect 5–10 real inputs, including at least two hard ones (ambiguous request, missing data, very long input, the edge case that broke things before).
Keep the task and the inputs identical across models. If you adapt the wording per model, adapt only phrasing and structure — never the requirements.
Start each run in a fresh chat so earlier messages or memory don't leak into the result.
Remove confidential data from test inputs, or use a tool your organization has approved.

Score the answers, not your impression

Grade each answer against the criteria with pass/fail per item, and note the reason for every fail.
Hide which model wrote which answer while you grade, when you can; it reduces brand bias.
Run each input more than once if consistency matters: the same prompt can produce different answers.
Track failures by type (wrong fact, missing field, broken format, ignored rule). A pattern tells you whether to change the model or the prompt.
If every model fails the same item, the prompt is usually the problem: clarify it and re-run before blaming the models.

Adapting the prompt per model without changing the test

Claude responds well to clearly delimited material (for example XML-style tags) and explicit priorities.
GPT and Gemini models follow numbered steps and an explicit output format reliably; state the format once, near the end.
Grok and DeepSeek benefit from the same basics: task first, constraints listed, an example of the expected output.
Promptea's analyzer applies these per-model adjustments for you: pick the target AI and compare the optimized versions while keeping your requirements fixed.

Templates

Turn a task into test cases

I want to compare AI models on this task:

<task>
[describe the task and paste the prompt you use today]
</task>

Help me build a small test set:
1. List 6 test inputs: 3 typical, 3 hard (ambiguous, missing information, unusually long or an edge case). Describe each in one line; I will supply the real content.
2. For each input, write 3–5 pass/fail criteria that a correct answer must meet.
3. List 2 things that should count as an automatic fail (for example an invented figure or a broken format).

Don't solve the task itself. Ask me if something about the task is unclear.

Opens the analyzer with the prompt prefilled.

Open in Promptea

Grade one answer against a rubric

Grade the answer below against the criteria. You are a strict reviewer.

<criteria>
[paste the pass/fail criteria]
</criteria>

<answer>
[paste one model's answer, without saying which model wrote it]
</answer>

Return JSON only:
{"results": [{"criterion": "...", "pass": true, "reason": "one sentence quoting the answer"}], "auto_fail": false, "summary": "one sentence"}

Rules:
- Judge only what is in the answer; don't give credit for what it might have meant.
- If a criterion can't be checked from the answer, mark pass as false and say why.

Opens the analyzer with the prompt prefilled.

Open in Promptea

Side-by-side review of two answers

Compare two answers to the same request. They are labeled A and B; don't guess which model wrote them.

<request>
[paste the original prompt and input]
</request>

<answer_a>
[paste]
</answer_a>

<answer_b>
[paste]
</answer_b>

1. List factual differences between A and B and say which one is supported by the input. If neither is, say so.
2. List any requirement from the request that either answer missed.
3. Say which answer you would ship as is, which one needs fewer edits, and why, in at most 4 sentences.

Opens the analyzer with the prompt prefilled.

Open in Promptea

FAQ

How many test cases do I need to compare models?

For a personal or team decision, 5–10 real inputs with written criteria usually reveal clear differences, as long as a few of them are hard cases. If the decision is expensive or the task runs at scale, grow the set over time with every failure you find in real use.

Can I use an AI model to grade the answers?

Yes, as a first pass, if the criteria are concrete (contains X, follows format Y, doesn't invent figures). Hide which model wrote each answer, ask the grader to quote the answer for every verdict, and spot-check its grades yourself, since a model grader can also make mistakes.

Should I use the exact same prompt for every model?

Keep the task, the inputs and the requirements identical. Small per-model adjustments to structure (delimiters, where the format goes) are fine and often fair, because each provider documents its own prompting recommendations. What you must not do is give one model extra information or looser rules.

Where do Promptea's Benchmarks fit in?

The Benchmarks page shows how models rank on public benchmarks, one ranking per benchmark. Use it to shortlist candidates, then run the test in this guide on your own task to make the final choice.