Stop CI Flakiness: Test the Side Effects of LLM Validators

Your CI pipeline turns red at 2 AM. The test suite for your LLM validator passed 47 times yesterday, but now it fails because the model returned a slightly different JSON structure. The validator logic is fine. The model just decided to wrap the output in a markdown code block this time. You rerun the pipeline. It passes. You rerun it again. It fails differently.

This is the core problem with testing LLM validators against live models. The model is non-deterministic, the network is unreliable, and your validator's behavior depends on both. You're not testing your code. You're testing OpenAI's or Anthropic's or Qwen's willingness to return the same shape twice.

Why Real Model Calls Break Your CI

When you test a validator against a live model, you're coupling two systems with completely different reliability profiles. Your validator is deterministic code. The model is a probabilistic function approximator that can change behavior between calls, even at temperature 0.

We've seen this pattern across our on-device AI work and voice AI pipelines. A validator that checks whether a model's response contains required fields passes 95% of the time in CI. That 5% failure rate is enough to erode trust in the entire pipeline. Engineers start ignoring red builds. Real regressions slip through.

The problem compounds when your validator has side effects. Maybe it writes to a database when validation passes. Maybe it triggers a webhook. Maybe it logs a rejection reason to an observability backend. If you test the side effects against a live model, you're testing whether the model happened to return something that triggered the side effect, not whether the side effect logic itself is correct.

What a Fake Model Client Actually Does

A fake model client is a test double that implements the same interface as your real LLM client but returns canned responses. Instead of calling an API, it returns a pre-configured string, JSON object, or error. You control exactly what the model "says," so you can test how your validator handles every possible response shape.

The approach is straightforward. You define a client interface, implement it for your real provider, and implement a fake version for tests. Your validator takes the client as a dependency. In production, you inject the real client. In tests, you inject the fake. Source: dev.to

Here's how the pipeline shifts when you introduce the fake client:

flowchart TD
    A[CI Test Run] --> B[Fake Model Client]
    B --> C[Programmed Response]
    C --> D[Validator Logic]
    D --> E{Side Effects}
    E -->|Pass| F[DB Write / Webhook]
    E -->|Fail| G[Log Rejection]
    F --> H[Assert Expected State]
    G --> H

The fake client makes the entire chain deterministic. You know what the model returns. You know what the validator should do. You can assert on the side effects with confidence.

Building the Fake Client

Start with a simple interface. Your model client needs a method that takes a prompt (or messages) and returns a response. In Python, that might look like a class with a complete method. The fake version stores a queue or mapping of responses to return.

The simplest fake returns a fixed string every time. That's enough for basic happy-path tests. But the real value comes from programming different responses for different test cases. You want a fake that can return malformed JSON on one call, a valid response on the next, and an API error on the third.

We use a pattern where the fake client takes a list of responses and pops them off one at a time. This lets us simulate a sequence of calls, which matters when your validator retries on failure or makes multiple calls to validate different aspects of the output. For custom model deployments where the model might behave differently across versions, this sequence testing catches regressions that a single-response fake would miss.

A more sophisticated fake can inspect the prompt it receives and return a response based on the input. This is useful when your validator constructs different prompts for different validation stages. The fake can pattern-match on the prompt content and return the appropriate canned response.

Keep the fake simple. If it starts parsing prompts, generating responses, or simulating reasoning, you've built a second model. The complexity belongs in the validator, not the test double.

Failure Modes the Fake Client Exposes

This is where the fake model client earns its keep. These are the failure modes that are nearly impossible to reproduce reliably against a live model, but they happen in production all the time.

Malformed JSON. Models love to wrap JSON in markdown code blocks. They add trailing commas. They include comments. Your validator needs to handle all of these. With a fake client, you program the exact malformed response and verify your validator either parses it correctly or rejects it with the right error.

Unexpected keys and structure. A model might return a valid JSON object with the right fields but also include extra keys you didn't ask for. Or it might nest the data one level deeper than expected. Your validator should either ignore unknown keys or reject them explicitly. Test both behaviors.

Empty or null responses. Sometimes the model returns an empty string. Sometimes it returns null or None. Sometimes it returns a valid JSON object where every value is null. These are distinct failure modes and your validator should handle each differently.

Prompt injection leakage. If your validator passes user input into the prompt, the model might follow instructions embedded in that input. With a fake client, you can program a response that looks like a successful injection attempt and verify your validator catches it. This is especially relevant for AI agent systems where untrusted user input flows through multiple model calls.

Side effect errors. What happens when the validator passes validation but the database write fails? What happens when the webhook times out? These are side effect failures, not model failures, and they need their own tests. The fake client lets you separate the two concerns completely.

Structuring the Validator Test Suite

A good validator test suite has three tiers. Each tier tests a different aspect of the system, and each tier has different determinism requirements.

The first tier is unit tests with the fake client. These are fast, deterministic, and run on every commit. They cover the validator logic, the side effect behavior, and the error handling. This is your core CI signal. If these tests fail, you have a real bug.

The second tier is integration tests with the fake client and real downstream services. The fake client provides the model response, but the database, webhook, and observability backend are real (or at least realistic test doubles). These tests verify that side effects actually fire correctly. They're slower but still deterministic because the model response is controlled.

The third tier is smoke tests against the real model. These run on a schedule or on merge to main, not on every commit. They verify that the real model still returns responses your validator can handle. These tests are allowed to be flaky. When they fail, you investigate, but they don't block deployment.

The separation matters. If you mix all three tiers into one CI pipeline, the non-determinism of the real model pollutes the determinism of the fake client tests. Engineers can't tell which failures are real bugs and which are just model variance. Source: dev.to

Integrating Into Your CI Pipeline

The integration is simpler than you might think. Your test framework already supports dependency injection or mocking. The fake model client is just another test double.

In a typical setup, your CI pipeline runs the unit and integration tiers on every pull request. The fake client is instantiated in the test setup, programmed with the appropriate responses, and injected into the validator. The validator runs, the test asserts on the return value and the side effects, and the pipeline reports pass or fail.

For the smoke test tier, use a separate job that triggers on merge to main or on a cron schedule. This job uses the real model client and runs a small set of representative prompts. If it fails, it sends a notification but doesn't block the merge. This is the same pattern we recommend for AI consulting engagements: separate the deterministic validation from the probabilistic model behavior.

One thing to watch for: if your validator makes multiple model calls in a single validation pass, your fake client needs to handle the sequence correctly. A single fixed response will work for the first call but might not make sense for the second. Use a response queue or a prompt-matching strategy to keep the responses coherent across calls.

What to Assert On

The fake client gives you control over the model response. Now you need to assert on what your validator does with that response. There are two categories of assertions: return values and side effects.

Return value assertions are straightforward. Did the validator return True or False? Did it return the parsed JSON or an error object? Did it return the expected error code? These are the same assertions you'd write for any function.

Side effect assertions are where most validator test suites fall short. When validation passes, does the validator write to the database? Does it call the webhook? Does it increment a metric? When validation fails, does it log the rejection reason? Does it send an alert? These are the side effects you need to test.

Use a spy or mock for each side effect target. Assert that the spy was called with the expected arguments. Assert that it was called the right number of times. Assert that it was not called when it shouldn't have been. The fake model client makes all of this deterministic because you control the input that triggers the side effect.

For RunHotel, our on-device voice AI for hotels, the validators check whether a model response is safe to speak aloud in a guest's room. The side effects include logging rejected responses for audit and triggering a fallback to a canned response. Testing these side effects with a fake model client means we can verify the audit log and fallback behavior without depending on the model's willingness to produce unsafe content on demand.

Common Mistakes

The most common mistake we see is testing only the happy path. Teams program the fake client to return a perfect response, assert that the validator passes, and call it done. The happy path is the least interesting case. The bugs live in the edge cases, the malformed responses, and the side effect failures.

Another mistake is making the fake client too smart. If your fake client parses prompts, generates responses, and tries to simulate real model behavior, you've built a second model. Keep the fake simple. It returns what you tell it to return. The complexity belongs in the validator, not the test double.

A third mistake is skipping the side effect tests entirely. Teams test that the validator returns the right value and assume the side effects will follow. They won't. A validator can return True and still fail to write to the database because of a connection error or a bug in the side effect code. Test the side effects separately and explicitly.

When to Use a Fake vs a Real Model

The fake model client is not a replacement for real model testing. It's a complement. Use the fake client for deterministic CI tests that verify your validator logic and side effect behavior. Use the real model for smoke tests that verify the model still produces responses your validator can handle.

If you're building a new validator, start with the fake client. Get the logic right. Cover the edge cases. Then add a small set of real model smoke tests. If you're maintaining an existing validator that's flaky in CI, refactor to introduce the fake client and move the real model calls to a separate job. Use the AI cost estimator to understand what the real model smoke tests will cost you per run, then decide how frequently to run them.

The decision is not fake or real. It's deterministic core with probabilistic peripheral tests. Your CI signal should be green when your code is correct, regardless of what the model decides to do on a given Tuesday. If you're struggling with this separation, reach out and we can help you structure the pipeline.

The validators we build for production systems handle dozens of failure modes, and new ones appear as models update. A fake model client lets you add a test for each new failure mode the moment you discover it, without waiting for the real model to reproduce the exact response that triggered it. That alone justifies the setup cost.

Sources

• Test the side effects of your LLM validator with a fake model client