Practical AI Evals for Product Managers

What are AI evals?

AI features in digital products stopped being a wow factor a while ago. They're the new norm, and as the hype wave subsides, only the AI use cases that deliver real value remain.

If you recently added an "AI assistant" to your product just because it's trendy, try removing it in an A/B experiment. There's a good chance you'll see the same or even improved business metrics (yes, but be careful showing that to your CEO). Most of the time, good old web UI is the best solution, and adding "assistants" here and there does harm. A PM friend at an internet bank told me about many attempts to introduce chat helpers across the bank's verticals. In the end, they scaled it all back because it didn't prove useful for users.

In some cases, though, new AI applications do create value. Agents can be more effective and consistent than humans in customer support. And sometimes it's simply easier for users to say what they're looking for in natural language than to browse the options (Amazon Rufus for shopping, for example).

If you own an agent that plays an important role in solving the customer's problem, you probably already know about evals. Evals are basically test cases for your ML solution, with a few important caveats. Yes, yes, the good old software tests that have existed for decades, in a new form (say that to an eval hyper and watch them get angry).

Deterministic evals

Take an input, run it through the agent, and compare the output directly with a fixed expected answer. This part is simple: the answer is either correct or not.

Let's say the AI support chat in a food delivery app like Wolt or DoorDash first classifies every customer message, so it can be routed to the right flow. Output is the label the AI chose, and ground truth is the label you, the product manager, defined. Keep scrolling and the eval runs row by row.

The result is a simple accuracy score: 3 out of 5 in this table, so accuracy is 60%. You can track this metric over time and check every new release against the set of evals.

And by the way, that was your first eval, so congrats. Not so magical and complex so far, right?

LLM as a judge

Unfortunately, this approach only works for agents with fixed outputs: making a decision, routing a request, calling a structured API. The limitation is obvious, since most of what an agent says to customers doesn't have a single correct answer. Some checks on free text can be deterministic too (valid JSON, banned words, length), but "is this a good answer?" can't.

Imagine our support chat can now also reply to customers on its own. You still want (I hope!) to measure how well it does that. Say the agent writes "Sorry about that, we'll refund the fries" or "That's on us, the fries amount is refunded to your card". Are these answers fine? They look fine to me, but how do you check that at scale, with 10k customer chats per day?

For open-ended answers we use another, usually more powerful LLM as a judge. Instead of an exact expected output, the ground truth becomes a general description of the expected outcome, written by... a product manager! Try being the judge yourself.

A few things about judge LLMs I learned the hard way:

• Create binary true/false criteria. "Does the answer avoid making claims about allergens that are not in the menu data? Yes/No." This gives you something you can act on when it fails.
• One criterion per judge. A judge that checks tone, accuracy and policy compliance at once will be vague about all three. Split them into three judges. The table above bundles several checks into one description to keep the example short. In a real setup, split them.
• Ask the judge to explain the verdict. The reasoning is often more useful than the verdict itself when you're debugging a failing trace. Plus, you can use it in self-improving agentic loops.
• Evaluate the judge. Don't laugh (or panic), but the judge is also an LLM application, so if it becomes critical and complex, you need evals for it too. Yes, sometimes we need evals for our evals. For this one, you can manually label 50-100 examples and run a deterministic eval.

Managing eval sets

• Synthetic use cases. If you're starting a greenfield use case, you'll most likely have to write the first set of evals from scratch. I don't like this period, because the fantasies rarely match what customers actually ask for.

  Written at your desk

  Two margaritas from Luigi's.

  Written by a real user

  same as last time but no onions and faster pls

  Use synthetic evals only to get to real production requests as quickly as possible.

• Every serious bug becomes an eval. When the agent drops an allergy note in production, add this exact case to the set first. Then fix the prompt. This way the same bug never slips through unnoticed, at least not in the same form. (Again, your devs have been doing the same with software tests for decades.)
• Create themed sets. If your case brings value and you start scaling it (congratulations, by the way!), your set of evals will grow fast. Structure it by subject: insufficient data, unsafe topics, prompt injection.

• Version evals together with prompts. An eval score only makes sense next to the prompt, model and eval set version it was measured on. Whaaat? Let me explain.

  Your support classifier scores 80% on 50 test cases. A week later you rerun it and get 70%, and the team starts asking what broke in the prompt. But nothing broke.

  Eval runs

  40 of 5080%

  prompt v4 × model X × eval set v1 → 80%

  Add 20 cases from last week's logs

  Someone added 20 new cases from last week's logs, many of them tricky ones like "My fries were missing" that the bot already struggled with. The prompt and model are identical, only the test got harder.

  If you store each result in that format, the difference is obvious immediately. When you do change the prompt, a fair comparison means running the old and new versions on the same eval set. Read that one twice, it isn't obvious.

Monitoring production with evals

Offline evals help you understand whether the new pipeline version is better than the previous one. Most importantly, they help you catch it when you accidentally break something else while fixing a certain step.

In production, users will keep coming up with cases you never thought were even possible. Then you have two options:

In any case, business signals matter much more than evals.

That's still a bad use case. LLM judges can sometimes help you understand why customers aren't impressed, but they're a tool and never the end goal.

Let's sum up

• Evals are test cases for your AI agent.
• Deterministic evals compare the output with a fixed expected answer. They're simple and cheap, but only work for structured outputs: classification, routing and so on.
• LLM as a judge can critically evaluate open-ended answers against a high-level description of the expected outcome.
• Start with synthetic evals, but move to real production cases as soon as possible. Group evals by theme and version them together with prompts.
• In production there's no ground truth, but you can still run evals that don't need it on a sample of conversations.
• Business metrics are always more important than eval scores.