Before You Test AI: The Engine, the Car and the Pit Wall
Everyone is being asked to test AI, but almost nobody explains what they'd be testing. This article explains the five ideas behind AI engineering (agent runs, the harness, memory, loop engineering, and LLM Ops and evals) through one car analogy, real incidents and what each means for quality engineers.
Everyone is talking about testing AI. Conference tracks, LinkedIn threads, vendor webinars, internal town halls. Every quality engineer (QE) I speak to is interested, and most have already been told some version of "we need you to test the AI".
What almost nobody tells you is what to do before you test it.
I've been testing AI agents this year, and the hardest part wasn't writing evals. It was the meeting before that. Someone says harness. Someone says loop engineering. Someone says we need LLM Ops and evals before go-live. Everyone nods. Everyone means something different. And the testers in the room are expected to build a test strategy for a system nobody has explained.
We'd never accept that anywhere else. We don't test a web app without understanding how it's built. We don't run production without understanding DevOps and SRE. AI deserves the same order: understand the engineering, then the operations, then test both.
This article is that missing step. Five ideas, one picture to hold them together, one map of how an AI system is built, proved and run, and what each idea means for you as a quality engineer.
The Five Words, in One Line Each
If you only read this section, you'll follow most AI engineering conversations.
- Agent run: one user prompt in, one reply out, however many steps happen in between.
- Harness: everything your team builds around the model, so it does your job, safely.
- Memory: facts and history the model wasn't trained on, stored outside it and fetched when needed.
- Loop engineering: designing how an agent repeats think, act, check, and when it must stop.
- LLM Ops and evals: tracing, scoring and improving the whole system once real users arrive.
The rest of this article places each one on the AI engineering flow, then unpacks it with a real incident and what it means for testing.
One Picture: The Engine and the Car
Think of a large language model (LLM) as a world-class engine. Enormous power, built by someone else (OpenAI, Anthropic, Google), and on its own it goes nowhere useful. It doesn't know your roads, your passengers or your speed limits.
What your team builds is the car around it. Car makers even have a name for the bundle that connects the engine to the sensors, controls and dashboard: the wiring harness. An AI harness does the same job. It wires the model to instructions, memory, tools and controls so that raw power turns into one useful, safe journey.

This is also how cars are tested. Crash-test programmes like ANCAP don't award safety stars by running an engine on a bench. They test the whole car, with the engine inside it. Testing an AI agent works the same way.
You don't crash-test the engine. You crash-test the car.
The AI Engineering Flow: Build It, Prove It, Run It
The car tells you what the parts are. The AI engineering flow tells you the order they are built, proved and run in, and where testing sits in that order. It is the map the rest of this article follows.

Read the flow top to bottom in three stages. Build it: a model, wrapped with tools, knowledge and memory, fed the right context on every turn, and set running as an agent. Some agents hand work to other agents, which multiplies the loops to get right. Prove it: evals check every layer above them before anything ships. They are the gate between building an agent and running it. Run it: production and ops run the agent for real users, trace every run and watch the scores. A failure found in production comes back as a new eval case, so the gate gets stronger every time something slips through.
Underneath everything sits system design: how the parts fit together, how they scale, and how they stay safe when one of them misbehaves.
Most testers picture evals as a last phase before go-live. On this map they sit in the middle, as the gate every change passes through in both directions: on the way out to production, and on the way back when production finds something the evals missed. When someone in a meeting says "harness", "loop" or "LLM Ops", point at the box on this map. Most of the confusion goes away once everyone is looking at the same layer.
The Agent Run: One Journey, One Test Case
Start with the smallest unit. An agent run begins when a user sends a prompt and ends when the agent replies. Inside, the agent might call five tools and search memory twice. It's still one run, like one journey from A to B however many turns it takes.
Each run is built from three inputs: the user's prompt, the chat history so far, and the system prompt that tells the agent who it is. Together they form the working memory, the context the model actually reads. By default, nothing survives the run. Close the chat and the agent forgets you.
For quality engineers, the agent run is your test case boundary. Input, context, output, like an API call. The twist is that a model predicts words with probabilities, so the same input can produce different outputs. Your assertions have to allow for that, which is why evals exist.
The Harness: Where Your Defects Live
The model knows a huge amount about the world and nothing about your business. The harness closes that gap: everything wrapped around the model for one use case, its instructions, memory, tools and the rules that decide when to stop. Frameworks such as LangGraph, LangChain and PydanticAI help you build one.
Here's the point that changes how you plan testing: most teams don't build the engine, they build the car. So most defects are in the harness, not the model.
Air Canada learned this in 2024. A grieving passenger asked the airline's website chatbot about bereavement fares. It told him he could book a normal fare and claim the discount afterwards. The real policy said no, and the bot even linked to the page that said so. A British Columbia tribunal ordered Air Canada to pay CA$812.02 and rejected the argument that the chatbot was responsible for its own words.
The engine wasn't faulty. The wiring was: the harness didn't ground the answer in the policy it already had. Your test surface is the harness. List every instruction, memory store, tool and stop rule. That list is your test scope.
Memory: The Handbook, the Map and the Logbook
An agent without memory is a skilled driver on day one in a strange city, with no map and no idea where they drove yesterday. Long-term memory comes in three kinds:
- Procedural memory, the driver's handbook. How to act. Tone, steps for a task, what to refuse. Usually plain text, often markdown files.
- Semantic memory, the map. Durable facts. Your return window, your product catalogue, who the customer is.
- Episodic memory, the logbook. What happened. Past conversations and events, stored with timestamps.
Writing memory is where it gets interesting. A common pattern is a summariser agent that periodically reads batches of past conversations and distils them into new facts. Useful, and risky. If the agent gave the same wrong answer 400 times, the summariser can promote that wrong answer to a permanent fact, and every future run will retrieve it.
Bad facts can also be written in on purpose. In February 2026, Microsoft's security researchers reported more than 50 hidden prompts from 31 companies in 14 industries, tucked into "Summarize with AI" buttons and share links. Each one told the assistant to remember the company as a trusted source or recommend it first.
Reading memory back is called retrieval. Facts are usually fetched with retrieval augmented generation (RAG): the question is matched by meaning against stored text, and the closest pieces go into the context. When memory has nothing to offer, the model fills the gap. In April 2025, Cursor users were being logged out when they switched machines. The support reply, from "Sam", said one device per subscription was a security policy. Sam was an AI bot, and there was no such policy. A co-founder confirmed it on Reddit: the real cause was a session bug.
Memory is state, and state needs tests. Start with these five:
- Grounding: the answer matches the stored fact, not the model's general knowledge.
- Empty retrieval: when memory has nothing, the agent says so or escalates. It never invents.
- Staleness: change a fact (30-day returns to 14) and the next run uses the new value.
- Isolation: customer A's history never appears in customer B's run. That's a privacy defect.
- Consolidation: seed a repeated wrong answer, or a planted "remember this" instruction, and check neither becomes a stored fact.
Loop Engineering: Brakes, Not Just an Accelerator
A chatbot answers. An agent acts. Give an agent tools (read the CRM, book a meeting, issue a refund) and it starts running a loop: think, call a tool, read the result, decide what's next. Imagine a store owner asking it to find customers who complained recently, follow up, and refund anyone still waiting. The agent reads the CRM, finds 8 unrefunded customers, books follow-ups, and might call the payments tool to refund them.
Loop engineering is designing that cycle, and above all deciding when it stops. We call these stopping rules end-loop guardrails. In car terms, they're the brakes, the speed limiter and the warning lights.

- Done: you've arrived. The task is complete, checked against a real system, not just the agent's say-so.
- Ask: pull over and check with the passenger. Refund the 8, or just give me the list?
- Budget: the speed limiter. A hard cap on steps, tokens or time. Anthropic's guide Building effective agents notes that a maximum number of iterations is a common way to keep control.
- Stuck: the warning light. The same failure keeps coming back, so stop and report.
What happens without real brakes? In May 2026, PocketOS, a booking platform used by car rental firms, lost its data in nine seconds. A coding agent hit a credential mismatch during a routine task in staging, found an API token it had not been given, and deleted a storage volume. The latest backups sat on the same volume and went with it. PocketOS had written safety rules for the agent, but nothing in the harness enforced them: the API had no confirmation step and no environment scoping.
It's not a one-off. When UC Berkeley researchers annotated 1,642 multi-agent traces for the MAST study, repeating steps and not knowing when to stop together accounted for over a quarter of failures. In July 2026, researchers scanned 6,549 agent repositories and confirmed 68 infinite-loop defects in 47 projects.
A guardrail is a rule about when the system stops and what state it's in. We already have a format for that:

Assert on the sequence of tool calls, not only the final reply. A perfect summary written after a refund that should never have happened is still a failed test.
LLM Ops and Evals: Telemetry and the Pit Wall
You can build a great car and still not know how it performs on lap 40,000, in the rain, with a driver nobody trained. Motorsport solved this long ago: the car streams telemetry, engineers on the pit wall watch it, and no setup change goes out without being checked. LLM Ops (large language model operations) is the same discipline for AI: evaluate the system, diagnose problems, ship fixes, repeat. If you know DevOps and SRE, you already know the shape of it. Logs become a trace per agent run, CI tests become offline evals, synthetic monitoring becomes online evals, and a release gate becomes an eval gate on prompt and config changes.

Trace. Record every run as a tree of events: what was asked, what memory was fetched, which tools ran with which arguments, how long each step took and how many tokens it used. OpenTelemetry now has GenAI conventions that standardise agent and tool spans across vendors.
Evaluate. Ask two questions. Was it a good run? Some of that is plain code: if the job was to book a meeting, does the meeting exist? For fuzzier qualities like tone, teams use LLM-as-a-judge, a second model scoring the reply against a rubric. Was it a healthy run? Response time, tokens, errors. Because the same input can give different outputs, run each case several times. Anthropic's guide to agent evals separates pass@k, the agent succeeds at least once in k tries, from pass^k, it succeeds every time. An agent that succeeds 3 runs in 4 will succeed three times in a row only 42% of the time. pass^k is what a customer feels.
Treat a judge like any measuring instrument. The paper that popularised the approach found strong judges agreed with human preferences over 80% of the time, but also documented biases toward longer answers and toward whichever answer is shown first. A June 2026 study of 21 judges found agreement dropped by 33 to 41 points once corrected for chance. Calibrate your judge against human-labelled examples from your own domain before you trust its scores.
Diagnose. Scores say that something is wrong; traces say where. The meeting never got booked: did the agent skip the tool, pass bad arguments, or ignore an error?
Gate. If the fix is a prompt, a model setting or a retrieval parameter, change it, run the evals, ship it if they pass. If something is deeply broken, fix the code and go round the whole loop again. Either way, nothing ships without passing, and every live failure goes into the eval set so it can't ship twice.
Three stories show what happens without that gate. In January 2024, a system update to DPD's chat assistant let a frustrated customer get it to swear and write poems mocking the company. Klarna's assistant handled 2.3 million chats in its first month before its CEO admitted cost had been "a too predominant evaluation factor" and Klarna began hiring human agents again. And in September 2026, Citizens Advice found that more than half of people using chatbots at UK energy, banking and broadband providers lost time, felt stress or waited longer for a fix.
You get the system your evals reward. If the dashboard only shows speed, volume or cost, that's what the system will optimise for. Put quality on the scorecard from day one, and gate prompt changes as seriously as code.
Where to Start on Monday
You don't need a platform or a big budget to start testing agents properly. You need one agent and a week.
- Draw the car. List every instruction, memory store, tool and stop rule in your agent. That's your test surface.
- Turn on tracing for that one agent, so every run leaves a record.
- Write five deterministic checks. Was the right tool called? With the right arguments? Did the run stop within budget?
- Label 50 real traces by hand. Good or bad, and why. You'll learn more from those 50 than from any framework comparison.
- Calibrate one LLM judge against your labels before trusting it on anything else.
- Gate prompt and config changes the way you gate code. No eval pass, no release.
The Vocabulary Is New. The Discipline Isn't.
Put it together and the picture is simple. A powerful engine you didn't build. A car your team built around it: instructions, memory, tools and brakes. And a pit wall watching every lap, scoring it, finding faults and deciding what goes back on track.
Almost every idea here has a testing equivalent we already know: test boundaries, state, acceptance criteria, observability, oracles, release gates. Quality engineers aren't late to AI. We just need the map before we start driving.
Frequently Asked Questions
What Is an AI Harness?
An AI harness is everything a team builds around a language model so it does one job safely: its instructions, memory, tools and stop rules. Because most teams build the harness rather than the model, most AI defects sit in the harness.
What Is the Difference Between pass@k and pass^k?
pass@k measures whether an agent succeeds at least once in k attempts. pass^k measures whether it succeeds every time. An agent that succeeds 3 runs in 4 passes three times in a row only 42% of the time, which is closer to what customers experience.
Where Should a Quality Engineer Start Testing an AI Agent?
Start with one agent. List its instructions, memory, tools and stop rules, turn on tracing, write a few deterministic checks, and hand-label 50 real traces before calibrating an LLM judge against them.

Stay Informed
Your data is used to send you our monthly newsletter.