Evals: How AI Architects Measure If AI Works

6 July 2026 · Bedrock AI

An AI architect measures whether AI is working by running evals: structured tests that feed an AI system real or representative inputs and grade the outputs against a defined standard of success. A demo that looks impressive once tells you nothing about consistency. Evals tell you whether a system succeeds reliably, on the cases that matter, under conditions that resemble production, not the cherry-picked ones from a sales call.

Most businesses adopt AI without ever answering this question. They ship a chatbot, an agent, or a summarisation tool, watch it work in a handful of test conversations, and declare victory. Then it fails silently on the twentieth real customer query, and nobody notices until a complaint arrives. The architect’s job is to close that gap before it opens: to build the measurement system alongside the AI system, not after something breaks.

What is an eval, exactly?

An eval is a test that gives an AI system an input, applies grading logic to its output, and records whether the result met the bar. That is the entire mechanism: input, output, grade. The complexity lives in getting each of those three parts right for a specific business problem, not in the mechanism itself.

A good eval is built from real failure cases, not hypothetical ones. Architects start by collecting 20 to 50 examples pulled from actual transcripts, support tickets, or early pilot runs, prioritising the inputs that broke the system or nearly did. A test suite built entirely from easy cases will pass with a 98% score and tell you nothing useful; a test suite anchored in real edge cases tells you where the system actually degrades.

Why do demos pass but production fails?

Demos pass because they are run once, on inputs the builder chose, under no adversarial pressure. Production fails because it runs thousands of times, on inputs a stranger chose, often with typos, ambiguity, or intent the system was never shown during development.

This gap is well documented in agentic systems specifically, where multi-step reasoning compounds small errors across each step. A model that is 95% reliable per step drops to roughly 60% reliability over eight sequential steps, purely from error compounding, even though every individual step “worked” in isolation. This is why architects distinguish between two related but different questions when scoring agent runs: pass@k, the probability that at least one of k attempts succeeds, and pass^k, the probability that all k attempts succeed. Pass@k is the right measure when a single working output is enough. Pass^k is the measure that matters for anything running unattended in production, because it captures whether the system can be trusted to succeed every time, not just sometimes.

What should an eval actually measure?

An eval should measure the outcome the business cares about, not a proxy that is easy to compute but only loosely correlated with it. Architects typically build evals across three layers, each answering a different question.

Skipping straight to business metrics without quality metrics underneath makes failures hard to diagnose: you know the number moved, but not why. Skipping business metrics and stopping at quality scores risks optimising a system that looks technically excellent but never changes a real outcome. Architects build all three layers because each one catches a different kind of silent failure.

How do you actually grade the output?

Grading is where most eval programmes fail, because grading itself needs to be reliable before it can be trusted to judge anything else. There are three grader types, and each has a distinct failure mode.

1. Code-based graders check outputs against a rule: does the JSON parse, does the answer contain the required field, does the number fall within range. They are fast, cheap, and perfectly reproducible, but brittle: a correct answer phrased differently than the grader expects will fail even though the system worked.
2. Model-based graders use a second AI model to judge the first model’s output against a rubric. They handle nuance code cannot, judging tone, helpfulness, or reasoning quality, but they are non-deterministic and need calibration against human judgment before they can be trusted at scale.
3. Human graders remain the gold standard for subjective or high-stakes judgments, but they are slow and expensive, which makes them unsuitable as the only method for continuous measurement.

The architect’s job is not to pick one. It is to combine all three deliberately: code-based graders for objective checks that scale infinitely, model-based graders calibrated against a sample of human judgments for everything subjective, and human review reserved for the highest-stakes or most ambiguous cases. This layered approach is sometimes described using the Swiss Cheese Model from safety engineering: no single layer catches every failure, but stacked together, the holes rarely line up.

Does an eval suite ever stop changing?

No. An eval suite that stays frozen after launch measures yesterday’s problems while today’s real failures go uncaught. Production data drifts as user behaviour changes, as the business adds new use cases, and as the underlying model provider ships updates that shift performance in ways a static test suite will not detect until someone notices something feels off.

Architects treat evals as a living system with four ongoing inputs: automated evals run pre-deployment and on every change to catch regressions before release; production monitoring flags distribution drift, the gap between the inputs a system was tested on and the inputs it is actually receiving; A/B testing validates that a change which scores better on paper also improves the real user outcome; and regular manual transcript review keeps the humans building the system honest about what “good” looks like, because graders calibrated once and never revisited quietly drift out of sync with reality. The discipline is reading transcripts on a schedule, not only when something has already gone wrong.

Why does this matter more than which model you choose?

Because model choice is reversible and measurement is not. A business that swaps GPT for Claude or upgrades to a newer model version can do so in an afternoon if the underlying eval suite is solid: run the new model through the same tests, compare the scores, decide with evidence. A business with no eval suite has no way to know if a model swap helped, hurt, or did nothing, and every future AI decision inherits that same blindness.

This is precisely why architecture has to come before tool selection. An organisation that picks a model first and figures out measurement later is optimising the reversible decision and ignoring the durable one. Diagnose first: define what “working” means for this specific business process, in terms that connect to a real outcome, before evaluating a single vendor. Build second: only after that measurement system exists does model or tool selection become a question with a defensible answer.

FAQ

What is the difference between an eval and testing in traditional software? Traditional software tests check that code behaves deterministically against a known specification: same input, same output, every time. Evals grade probabilistic outputs against a quality bar, because the same input to an AI system can produce different valid outputs, so the test has to judge quality rather than exact match.

How many test cases does a business need to start? Twenty to fifty is enough to start, provided they come from real failures rather than invented scenarios. A small suite built from genuine edge cases catches more real problems than a large suite built from easy, hypothetical ones.

Can evals be fully automated? Most of the volume can be, using code-based and model-based graders. But model-based graders need periodic calibration against human judgment, and the highest-stakes cases still warrant human review, so “fully automated and unattended” is not a safe end state.

Who should own the eval suite inside a business: engineering or the business team? Both, structured through the three-layer model. Engineering typically owns system and quality metrics; the business owns defining which business metric the AI is meant to move. An AI architect’s role is connecting the two, so a quality score always ties back to a business outcome someone actually asked for.

What is the single biggest mistake businesses make with evals? Building the eval suite after the system is already in production, using it as an afterthought rather than a design input. The teams that get this right define what success looks like before they build, so the eval shapes the system rather than just grading it after the fact.

Bedrock AI maps your systems, team and workflows to show where AI actually pays, before you spend a pound building. Book a strategy call.