Evaluation
How do you know it works?
A demo shows that something can work. Evaluation shows how often it works, and whether a change made it better or worse.
The checks from earlier lessons, such as schema validation and source checks, become the building blocks of an evaluation.
1. Collect real examples
Start with 20 to 50 inputs that look like real use, including hard and unusual ones. Add every failure you find in production to the set.
2. Decide what good looks like
Write down what a correct output must contain. Some checks are exact: valid JSON, the right category, a cited source. Others need judgement: is the answer helpful and correct?
3. Choose a grader for each check
Exact checks: format, category, required fields.
Fast, cheap, and consistent.A second model scores the output against a written rubric.
Scales well. Compare it with human judgement first.People review a sample of the outputs.
Slow, but the reference for quality.4. Run it on every change
Run the same set before and after each change to a prompt, model, or tool. Compare the scores, then read the examples that got worse.
A change that fixes one case can break another. The set catches it before users do.
5. Keep watching in production
Offline tests cannot cover everything. Log real inputs and outputs, review a sample regularly, and turn new kinds of failure into test cases.
When is it useful?
As soon as other people depend on the system, or you plan to change prompts and models over time. A small set you run often is more useful than a large one you never run.