AI evaluations look like the least designerly corner of product development. They are the tests that decide whether an AI feature is ready to ship, and they run on scripts and dashboards full of percentages. No wonder companies hand them to engineers and product managers.
The more I read about evals, the more familiar they look. Behind each one, there is a decision about what “good” means for the person using the product. They all seem to answer questions about user experience. What should the AI do when it isn’t sure? Which mistakes can people live with?
Designers make calls like these every day. However in AI evals, we tend to have a say after someone else has written the criteria, if we are considered at all. As catastrophic as it may sound, this can lead to shipping a product that is technically functional but fails its users.
Here are five reasons why designers should always be in the loop, right next to engineers and product managers.
1. TASTE AND JUDGMENT CATCH WHAT THE NUMBERS MISS
Providing users with the right answers might be the hardest quality score to quantify in evals. Even user ratings can mislead: a Stanford study found that people trusted flattering AI answers more than ones that pushed back.
To decide what the right answer looks like, evaluation teams need a professional with trained taste and judgment. As designers, we develop such attributes by working with users every day. We look at a product through a human lens, so we judge an answer by whether it helps the person who asked.
Teams can train AI to grade other AI outputs, but someone with taste has to teach it what to do, like jargon a new user won’t understand or a cheerful reply to a complaint. Details like these can slip past an engineer, for instance, but a professional in UX/UI design knows to look for them.
2. DECIDE WHAT HAPPENS WHEN THE AI DOESN’T KNOW THE ANSWER
AI often sounds too confident and seems to have all the right answers for the person prompting it, even when it’s wrong. In some cases, chatbots mix good and bad medical advice, and people struggle to tell the difference.
Hallucinations and false information aren’t a good user experience. They make the product feel untrustworthy, pushing users to drop sessions or switch to a competitor.
Whenever the AI doesn’t know the answer, I’d have it say so and point the user in the right direction. That way, the user can rest assured they will still get help, even if the AI can’t give it at that particular moment.
Designers have worked on error messages and dead ends for years. An AI not knowing the answer is just another dead end. We should set this criterion in evals: The AI must clarify what’s missing in its answer and, if possible, where to find it.
3. REAL USERS ARE MESSIER THAN ANY TEST
One skill people working in design have is knowing how to recruit the right set of people to run tests. I’m talking especially about UX researchers here.
The products we built, way before AI was even a thing, required heavy research. We understand how people talk, how they mumble and jump from one topic to another, how unique each answer can be, and how much can depend on mood, time of day, and other external factors.
Some teams running AI evals build test sets with synthetic users. I don’t see that working, and I’m not alone: 88% of researchers in one recent survey raised concerns about the quality and accuracy of AI-generated personas.
4. QUESTION THE ASSUMPTIONS BEHIND THE EVAL
Evals are built on assumptions about what users want, and they’re rarely written down. Most of the time, they come from business goals or from whatever is easiest to measure.
For example, a team building a customer service chatbot to cut support costs may assume people would rather not talk to a human. In that case, any handoff to a live agent counts as a failed test, and the bot that never hands anyone over gets the best score, even when people are getting nowhere.
If you’ve ever been stuck in a loop with a support bot, you know sometimes you just want to talk to a real person.
As designers, we write these assumptions down and check with real users before designing a feature. I’d recommend doing the same with evals.
5. SOME EVAL PROBLEMS ARE DESIGN PROBLEMS
When an eval fails, the first reaction is usually to tweak the prompt or switch models. Sometimes that’s the right call, but other times the problem is the product itself.
Hamel Husain, a machine learning engineer with more than 20 years of experience who now teaches AI evals, wrote this summer: If an AI’s answers are hard for a team to check, they’re often hard for users too.
Imagine a travel assistant that books the wrong date because a user typed “next Friday,” which can mean two different days. A better prompt won’t fix that, but asking the user to confirm the exact date will. Now the answer is easier to check. That’s a design decision.
We catch problems like this in usability tests. So before a team spends another week refining prompts, I’d ask a designer to review the failed cases.
FINAL THOUGHTS
Engineers and product managers will keep running AI evals, as they should. But most of the decisions behind them are about the people using the product. Aren’t designers among the professionals who would know users the best?
My advice to design leaders is to help write the eval criteria before they’re final. After all, deciding which mistakes users can live with is our job, too.
Arin Bhowmick is chief design officer at SAP.