What is an AI evaluation (eval)?

An evaluation, or eval, is a repeatable test that measures how well an AI model, prompt or agent does a specific task.

Key takeaways

  • An eval is a set of test cases with a way to score each result.
  • It tells you if a change to a prompt or a model made things better or worse.
  • A small eval of 10 to 20 real cases beats a general feeling that the output looks fine.

How it works

You collect real examples of the task, each with a known good answer or a clear standard. You run the AI on all of them and score the results, by hand, with a rule or with a second model that grades. You run the same set again each time you change something and compare the scores.

Example

Before a consultant switches the model behind a proposal prompt, they run both models on 12 past client briefs and grade each draft against a five-point checklist. The new model wins on nine briefs, so they switch.

Why it matters

Models and prompts change often. An eval lets you decide with evidence, and it catches a quality drop before your clients do.

Common mistake

A public benchmark score tells you little about your task. Test with your own cases and your own standard.

Guides that use this term

Related terms

See all terms in the AI Encyclopedia, the numbers on the AI statistics pages, or build a prompt with the prompt builder.