I shipped a silent failure. It wasn't a crash; it was a subtle, incorrect output that would have cost a client thousands in wasted time. My production AI agent, designed to extract specific data from unstructured text, started hallucinating dates. The unit tests passed. The integration tests passed. The agent thought it was doing its job, but it was wrong. This failure, caught only by a human in a pre-production environment, solidified my decision: I needed an AI agent evaluation harness before writing another feature.
My current harness runs 131 tests across four distinct layers for every agent change. Each full run costs $0.03. Without it, I would have shipped that date hallucination and countless other subtle regressions. This isn't about testing code; it's about testing behavior in a non-deterministic system.
The Fundamental Flaw of Unit Tests for AI Agents
Unit tests verify functions. Integration tests verify component interaction. Neither fundamentally addresses the core challenge of an AI agent: its ability to correctly interpret intent, execute a multi-step plan, and produce a useful, accurate output in a non-deterministic environment.
My initial agent for a client involved extracting specific entities from long-form documents. I had unit tests for my parsing functions, integration tests for API calls to Groq and Claude, and even end-to-end tests that checked if any output was generated. But when I swapped a prompt template or fine-tuned a retrieval step, the agent's reasoning could subtly shift.
The date hallucination was a perfect example. A minor change in the RAG prompt, intended to improve entity linking, inadvertently made the agent overconfident in inferring dates from ambiguous phrases like "next quarter." The output format was correct, the API calls succeeded, but the content was wrong. A unit test couldn't catch this because the underlying parsing logic was sound. An integration test couldn't catch it because the LLM returned a date, just the wrong one.
My 4-Layer Evaluation Harness Architecture
My evaluation harness is built on Python, leveraging pytest for structure and custom runners for execution. It's designed to be lightweight and run on Oracle Cloud Infrastructure (OCI) serverless functions, triggered by Git pushes.
1. Input Validation Layer (28 tests):
* Purpose: Ensure the agent correctly identifies invalid or out-of-scope user inputs.
* Examples:
* "Can you book me a flight to Mars?" (Expected: "I can only assist with [domain].")
* "What's the weather in 1999?" (Expected: "I can only provide current weather.")
* Inputs exceeding token limits (Expected: "Your request is too long.")
* Mechanism: Predefined prompts with expected negative responses or specific error messages. This layer uses regex matching on the agent's final output.
2. Tool Invocation Layer (45 tests):
* Purpose: Verify the agent correctly selects and uses its available tools (e.g., database lookup, external API call, code interpreter).
* Examples:
* "What's the current stock price of AAPL?" (Expected: tool_call(name='stock_api', args={'symbol': 'AAPL'}))
* "Summarize this PDF." (Expected: tool_call(name='pdf_summarizer', args={'url': '...'})
* "Convert 10 USD to EUR." (Expected: tool_call(name='currency_converter', args={'amount': 10, 'from': 'USD', 'to': 'EUR'}))
Mechanism: I intercept the tool calls before* execution. The test asserts against the name and args of the proposed tool call. This is critical for preventing expensive or destructive tool calls during evaluation. I use mock tools that return predefined responses, ensuring the test focuses solely on the agent's decision-making.
3. Output Accuracy Layer (50 tests):
* Purpose: The core of the harness. Verify the agent's final output is correct, relevant, and formatted as expected for a given valid input.
* Examples:
* "What's the capital of France?" (Expected: "Paris")
* "Extract all dates from 'Meeting on Jan 15, 2024, rescheduled to Feb 2, 2024.'" (Expected: ['Jan 15, 2024', 'Feb 2, 2024'])
* "Summarize this article: [URL]" (Expected: summary containing key entities from a known ground truth summary).
* Mechanism: This layer uses a combination of exact string matching, regex, and semantic similarity (using an embedding model like text-embedding-ada-002 for longer, more nuanced responses). For structured extractions, I parse the JSON output and assert against specific key-value pairs. This is where the date hallucination was finally caught. The test case for date extraction had a ground truth of ['Jan 15, 2024', 'Feb 2, 2024'], and the agent's output was ['Jan 15, 2024', 'Feb 2, 2024', 'next quarter'], failing the exact match.
4. Performance & Cost Layer (8 tests):
* Purpose: Monitor token usage, latency, and API call count to ensure the agent remains within operational budgets and performance SLAs.
* Examples:
* "Summarize [long text]." (Expected: token usage < 5000, latency < 5s)
* "Complex query requiring 3 tool calls." (Expected: API calls = 3)
* Mechanism: I wrap the LLM and tool invocation calls with custom metrics collectors. These metrics are then asserted against predefined thresholds. This layer helps me catch regressions where a prompt change might inadvertently increase token usage or lead to an infinite loop of tool calls. My routing logic between Groq (for speed) and Claude (for complexity) is heavily influenced by these metrics.
The $0.03/Run Cost Breakdown
Running 131 tests across these layers involves multiple LLM calls. My strategy to keep costs low ($0.03 per full run) involves:
- Strategic LLM Selection: For simple input validation and tool invocation tests, I often use smaller, faster models like Groq's Llama 3 8B. For accuracy tests requiring complex reasoning, I route to Claude 3 Sonnet or Opus. My routing logic is part of the agent's design, and the eval harness tests this routing.
- Caching: For tests that involve identical prompts and expected deterministic outputs (e.g., simple fact retrieval), I cache LLM responses. This is a delicate balance, as over-caching defeats the purpose of testing non-determinism, but it's effective for baseline checks.
- Parallel Execution: The harness runs tests in parallel using
pytest-xdist. My OCI serverless function (Oracle Functions) can handle concurrent invocations, speeding up the overall run time. - Mocking Tools: As mentioned, tool invocation tests mock the actual tool execution. This saves API call costs to external services.
A typical run involves around 15,000 tokens processed (input + output) across various models. At an average cost of $0.50/M tokens (blended rate across Groq, Claude, and some OpenAI for embeddings), this comes out to $0.0075 for LLM inference. The remaining cost is for OCI Functions execution time and embedding model calls for semantic similarity checks. The total is consistently around $0.03.
What AI Agent Tests Catch That Unit Tests Cannot
The core difference lies in the nature of the system. Unit tests assume deterministic functions. AI agents, especially those leveraging large language models, are inherently non-deterministic. They reason, plan, and generate.
1. Reasoning Drift: A prompt change intended to improve one aspect might subtly degrade another. My date hallucination was a perfect example. A unit test on the date parsing function would still pass, but the agent's decision to include "next quarter" as a date was a reasoning error.
2. Tool Selection Logic: An agent might correctly implement a tool, but incorrectly decide to use it, or use the wrong arguments. My tool invocation layer catches this. A unit test on stock_api.get_price(symbol='AAPL') passes, but if the agent calls stock_api.get_price(symbol='APPL') due to a typo in its reasoning, that's a failure.
3. Output Nuance: Beyond simple correctness, AI agent outputs have nuance. Is the tone appropriate? Is the summary comprehensive? Is the explanation clear? While harder to automate, semantic similarity checks and keyword presence can catch gross deviations.
4. Robustness to Edge Cases: How does the agent handle ambiguous inputs, malformed requests, or out-of-domain queries? My input validation layer specifically targets these. A unit test might check a specific error handler, but the agent's ability to classify an input as an error is a higher-level function.
Building this evaluation harness was a significant upfront investment. It took me two weeks of focused development. But it has paid for itself many times over by preventing silent failures, maintaining output quality, and giving me the confidence to iterate rapidly on my AI agents. It's not just about catching bugs; it's about understanding and controlling the emergent behavior of complex systems.
Frequently Asked Questions
Q: How do you handle non-deterministic LLM outputs in your accuracy layer?
A: For highly variable outputs like summaries, I use semantic similarity (cosine similarity of embeddings) against a ground truth embedding, with a threshold of 0.85. For structured extractions, I enforce strict JSON schema validation and exact key-value matching.
Q: What if the LLM itself changes (e.g., a new model version from Groq or Claude)?
A: This is a major risk. My harness acts as a regression suite. When an LLM provider updates their model, I run the full harness. If tests fail, it indicates a behavioral shift, and I either adjust my prompts/agent logic or revert to an older model version if available.
Q: How do you manage the ground truth data for 131 tests?
A: Ground truth is stored in YAML files alongside the test cases. For simple cases, it's direct string or JSON. For complex summaries, I generate a human-verified "golden" summary once and store its embedding. This dataset is version-controlled with the agent code.
Q: Isn't $0.03 per run too expensive for CI/CD?
A: For a solo developer, $0.03 per push is negligible. If I were running this 100 times a day, it would be $3.00, which is still acceptable for ensuring production quality. The cost of shipping a silent failure is orders of magnitude higher.
Q: How do you prevent prompt injection from your test inputs?
A: My test inputs are controlled strings, not user-provided. The harness itself is not exposed to external users. The goal is to test the agent's robustness to user prompt injection, which is a specific set of test cases within the input validation layer.