In 2024 every SaaS adds AI. "Write with AI." Smart suggestions. A chatbot. Autocomplete. And almost everyone ships these features without automated tests because "AI cannot be tested automatically."
That is a myth. More precisely, it is a half-truth that became an excuse.
Model answer quality is hard to test automatically. But most AI bugs users see are not "the model gave a bad answer". They are "the app crashed when the model was slow", "the user saw an empty screen instead of an error", or "the interface froze on empty input".
Those are ordinary bugs. They are tested with ordinary methods.
What to test in AI features
An AI feature is an integration: your frontend, your backend, and an external model API. Each layer has its own responsibility.
answer quality, relevance, safety, accuracy. This belongs to evaluation pipelines, human review, and red-teaming. Not Playwright.
everything around the model: UI, error handling, edge cases, loading states, and input validation. This is ordinary functional testing.
Five scenarios every AI feature needs
The user enters a valid prompt. The answer arrives and renders correctly.
The user clicks without entering text. The UI should show validation or disable the button. No API request should be sent.
The model takes too long. The spinner must not run forever. Show a clear error and allow retry.
The provider returns 500, 429, or a network error. The app should handle it gracefully and preserve user input.
The user pastes ten thousand characters. The product should validate, warn, or limit clearly. Silent loss is a bug.
Five tests. Not two hundred. These cover most AI bugs users actually see.
How to mock an AI API in tests
Testing against the real provider is expensive, slow, and non-deterministic. Mock the model API instead.
await page.route('**/api/openai/chat**', route => {
route.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify({
choices: [{ message: { content: 'Mocked AI response' } }]
})
})
})Now the test does not depend on OpenAI, runs instantly, and is deterministic. You are not testing the model. You are testing how your app handles the model response.
For error states, mock error responses:
await page.route('**/api/openai/chat**', route => {
route.abort('timedout')
})Streaming responses
Many AI features stream responses token by token. That adds UI scenarios:
All of this is UI behavior. Standard tools can test it.
Why special AI testing tools are often unnecessary
Many tools promise "AI testing with AI". Most evaluate generation quality. That may matter if answer quality is your main product: AI assistants, content generators, RAG systems.
For most SaaS products with one AI feature, the first need is simpler: make sure the app does not crash, freeze, lose user data, or hide errors.
Five mocked functional scenarios will catch most production AI bugs.
FAQ
How do you test AI features automatically?
Separate model quality from product behavior. Use evals for quality, and functional tests for UI, errors, loading states, and input validation.
Can Playwright test AI features?
Yes. Playwright is good for testing AI feature UI behavior and can mock network requests to AI providers.
How do you mock OpenAI in automated tests?
Use route interception, return a fixed response for success, and return 500, 429, or timeout for error paths.
Do AI apps need special AI testing tools?
Only when model quality is the core product. Most SaaS AI features first need ordinary functional tests around the integration.
What breaks most often in AI features?
Loading states, empty input, timeouts, API errors, long input, and interrupted streaming. These are ordinary automation scenarios.