Evals that don't lie.

The first eval suite I ever wrote for an agent had 24 cases. They all passed. I shipped the agent. It broke in production within a week.

The next morning, I wrote a 25th eval case — the exact failure that had happened in prod. It also passed.

That's when I realized: my evals weren't broken. They were performing. The agent was very good at memorizing the small set of inputs I'd given it, and very bad at handling the long tail of inputs it would actually see.

This piece is a list of the patterns I now use to avoid that trap.

Pattern 1: never write the eval and the prompt the same week

If you write both at the same time, you'll unconsciously align them. The prompt will mention the things the evals check; the evals will check the things the prompt mentions. The model will learn to thread that needle and tell you it's general.

Wait a week. Come back with fresh eyes and write evals based on what you'd actually want the agent to do — without re-reading the prompt. The mismatches you find are the real bugs.

Pattern 2: tolerances, not exact matches

LLM outputs are stochastic. An eval that requires output === "Yes." will fail half the time for reasons unrelated to whether the agent did the right thing.

Use tolerances:

• "Does the output contain a positive answer?" (regex)
• "Is the structured output parseable as the expected schema?"
• "Is the tool call's input within these bounds?"
• "Does the LLM-judge rate this response as 'good enough'?" (judge-driven evals, used sparingly)

Pattern 3: include adversarial inputs

Half of your eval set should be inputs you don't expect the agent to handle well. Inputs in the wrong language. Inputs with the wrong format. Inputs with prompt injection. Inputs where the right answer is "I can't do this."

Without these, your eval pass rate is meaningless. With them, you can answer the question that matters: how does this agent fail?

Pattern 4: separate "regression" evals from "shipping" evals

Regression evals are bugs you've fixed. They live forever. Every model bump, every prompt change runs them.

Shipping evals are new things you're trying to teach the agent. They might be flaky. They get pruned or promoted.

Mix the two and you'll either ship slow (because you're waiting on flaky tests) or ship breaking changes (because the suite is too forgiving).

Pattern 5: version everything

Prompt versions. Model versions. Skill versions. Eval suite versions. The point isn't that you'll roll back — though you will. The point is that when something breaks in prod, you can answer "what changed?" in under five minutes.

I keep a single versions.json per project that gets bumped on every release. The agent logs its versions on every run. The dashboard groups failures by version triple. This has paid for itself many, many times.

Evals are the most under-appreciated artifact in the entire agent stack. They are also the only thing standing between "this looks great in the demo" and "this still works on a random Tuesday in November."

Build them first. Run them constantly. Trust the ones that lie to you the least.