How to Write E2E Tests for AI Chatbot Interfaces That Actually Work
Hitesh Sondhi · November 5, 2026 · 8 min read
Your chatbot passes every unit test. The API returns 200. The streaming parser handles SSE chunks correctly. Then a user types "what can you help me with?" and the bot responds with a preamble about being an AI assistant, followed by a bulleted list nobody asked for. Your E2E test didn't catch this because it was asserting expect(response).toContain("Hello").
E2E testing for AI chatbots fails when you treat the LLM output as a string to match rather than a behavior to verify. The interface has moving parts that traditional E2E patterns don't account for: streaming tokens, multi-turn state, tool call rendering, and outputs that change when the model gets updated or the temperature shifts.
Here's how we approach it at Cropsly, and where the standard Playwright and Cypress patterns break down.
Why Standard E2E Assertions Fail on Chatbot Output
The first mistake teams make is hitting the real LLM API in their E2E suite. You get flaky tests because the model rephrased a greeting, latency spiked past a timeout, or the provider rate-limited your CI pipeline. We've seen suites with flake rates so high that developers stopped trusting the results entirely.
The second mistake is mocking the LLM so aggressively that you're testing your mock, not your interface. If your mock always returns {"content": "Hello!"} in one chunk, you'll never catch the bug where your streaming renderer drops the first token or mishandles a \n\n separator.
The fix is a middle layer: deterministic mock responses that mimic real LLM output shape, including streaming chunk patterns, tool calls, and error states.
Here's the flow we use:
flowchart TD
A[E2E Test Runner] --> B[Mock LLM Server]
B --> C{Response Type}
C -->|Text| D[Stream chunks with delays]
C -->|Tool Call| E[Return function schema]
C -->|Error| F[Return 429 or 500]
D --> G[Chatbot UI]
E --> G
F --> G
G --> H[Assertions on UI state]
What to Actually Assert Against
Stop asserting on the raw text content of LLM responses. Assert on the structural and behavioral properties of the interface instead.
Check that a message bubble rendered with the correct role, user versus assistant. Verify the typing indicator appeared during streaming and disappeared when it completed. Confirm markdown was parsed into the right DOM structure: a code block became a <pre>, not a <p> with backticks. Make sure the scroll container pinned to the bottom during streaming and stayed there when a new message arrived.
For content assertions, use semantic matching instead of exact strings. If the bot should recommend hotels, assert that the response contains a hotel name from a known set, not that it equals a specific sentence. We use a custom matcher that checks for category presence: "does this response contain at least one entity of type 'hotel'?" rather than "does this response contain 'The Grand Hotel'?" This approach aligns with the semantic query philosophy in Testing Library, which encourages asserting on what users actually perceive rather than implementation details.
This is especially important for our work on RunHotel, where the chatbot recommends specific properties. The model might format the hotel name differently across runs, with or without a suffix, different capitalization, and your test needs to tolerate that.

Streaming: Where Most Bugs Hide
Streaming is the hardest part to test and where we see the most production bugs. The issues are specific and mechanical.
Your SSE parser might buffer incorrectly, concatenating two chunks into one token. Your UI might render partial markdown that looks broken for the fraction of a second before the chunk completes. Your abort button might not actually cancel the fetch request. Your reconnection logic might replay the entire conversation when a WebSocket drops.
Test these scenarios explicitly. We configure our mock server to stream a known response in chunks with configurable delays. A typical test streams a short response across several chunks, then asserts the final rendered text matches the concatenated chunks. Another test sends chunks out of order to verify the buffer handles it. A third kills the connection mid-stream and checks that the UI shows a retry button.
Playwright's page.waitForSelector with { state: 'visible' } handles most of the timing, but for streaming you need page.waitForFunction to poll for content completion. We use a data attribute data-streaming="complete" that the UI sets when the stream finishes, which is more reliable than waiting for the typing indicator to disappear. The Playwright documentation covers the polling mechanics in detail.
Multi-Turn Conversations Break State Management
Single-message tests are easy. Multi-turn conversations are where state management bugs surface. The user sends a follow-up that references "that hotel you mentioned" and the UI needs to pass conversation history to the API. Your test needs to verify the request payload includes prior messages, not just the current one.
We test this by intercepting the outgoing API request and asserting on the message array length and roles. After three turns, the payload should contain five messages: system prompt plus two exchanges plus the new user message. If it contains one, you've got a state bug.
This matters more when you're building AI agents that maintain context across tool calls. A dropped message in history means the agent forgets what it just did and calls the same tool again.
Testing Tool Calls and Function Rendering
When the LLM returns a tool call, your UI needs to render it in a way that makes sense to the user. This might be a "Looking up availability..." indicator, a card with structured data, or a collapsed JSON view for debugging.
Your E2E test should verify the full cycle: the tool call appears, the UI shows a loading state, the tool result comes back, and the final response incorporates the result. This requires your mock server to handle the two-step flow where the first response is a tool call and the second is the final answer. The OpenAI function calling guide documents this two-step pattern.
If you're working with custom models that have non-standard tool schemas, the mock needs to match that schema exactly. A mismatch between your mock and the real model's tool call format is a bug factory.
Latency and Timeout Testing
Real LLM APIs have latency that varies wildly. First-token latency for production models can range from a few hundred milliseconds to several seconds depending on load, as documented in OpenAI's latency optimization guide. Your UI needs to handle both gracefully.
We add a test that configures the mock to delay the first chunk significantly and verifies the UI shows a loading state, doesn't freeze, and allows the user to cancel. Another test sends the response instantly and checks that no loading flicker occurs.
For on-device AI deployments where the model runs locally, latency is more predictable but still varies with input length. A prompt of moderate length on Qwen3-8B might take just over a second on a Jetson Orin but several seconds on a lower-end device. Your E2E suite for on-device should test against the slowest target, not the fastest. The same applies to voice AI interfaces where latency budgets are tighter because users expect conversational pacing.
The Mock Server Pattern That Works
Don't use a JSON fixture file. Don't stub window.fetch. Run a real HTTP server on a random port that mimics the LLM API contract.
We use a lightweight Express server that reads test scenarios from a config object. Each scenario defines the response type (text, tool call, error), the chunk pattern, delays, and status codes. The test runner points the chatbot's API client at this server via an environment variable.
This approach catches issues that fetch stubbing misses: CORS handling, header parsing, connection lifecycle, and retry logic. It's more setup than a mock function, but it's the difference between testing your interface and testing your test code.
If you're setting this up for the first time, our AI consulting team can help you identify which response patterns to mock for your specific chatbot. Use our AI cost estimator to model what running real API calls in CI would cost before you commit to the mock approach. If you want to talk through your architecture, reach out.
The mock server config becomes your test contract. When a new model behavior appears in production, you add a scenario for it. When a scenario starts failing after a UI change, you know exactly which response shape broke and can reproduce it in isolation.





