Every team that abandoned automation tells the same story. They wrote tests. The tests worked. Then they started failing: one by one, then in batches. Evenings went into investigation. The team got tired and deleted them. The conclusion: "automation does not work for us."
Unstable tests are not a feature of automation. They are a symptom of four known architectural problems. Each has a concrete fix.
Reason 1: The test is in a hurry
The most common cause of flaky tests is a test that clicks before the page is ready.
The test opens a page, immediately looks for a button, does not find it because the page is still loading, and fails. On the next run the server responds faster and the test passes.
The wrong fix is sleep(2000). It works only until the server or network becomes slower.
wait for state, not time.
// Wrong
await page.waitForTimeout(2000)
await page.click('.checkout-btn')
// Better
await page.waitForSelector('.checkout-btn', { state: 'visible' })
await page.click('.checkout-btn')
// Best in Playwright
await page.click('.checkout-btn') // auto-wait is built inReason 2: Tests depend on each other
Test A creates a user. Test B reads that user and checks their data. Run B before A and B fails. Run in parallel and it fails randomly depending on order.
That is shared state. It is a slow death for a test suite.
Signs:
each test creates everything it needs and cleans up after itself.
Use setup and teardown. Each test should be an isolated unit that leaves the world as it found it.
Reason 3: Fragile locators
Some locators survive refactoring. Some do not survive the next sprint.
div:nth-child(3) > span.text-gray-500
//div[@class='form-group'][2]/input
.container > .row > .col-md-6 > buttonThey describe where an element is in the DOM, not what it means.
[data-testid="submit-order-button"]
[aria-label="Submit order"]
button:has-text("Submit order")They describe what the element does. Add data-testid to key elements when you can. If you cannot, use text and ARIA roles before CSS structure.
Reason 4: External dependencies are not mocked
Your test checks that a user receives an email after signup. The logic is correct, but it uses the real email service. The service slows down and the test fails. The problem was not your code.
mock external dependencies.
await page.route('**/api/payments/**', route => {
route.fulfill({ status: 200, body: JSON.stringify({ status: 'success' }) })
})The test becomes stable. It checks your code, not a third-party system.
The system metric: flakiness rate
If you do not measure test stability, you do not manage it.
Flakiness rate is the share of tests that sometimes pass and sometimes fail without code changes. Acceptable: below two percent. Above five percent, the suite is unreliable and will soon be ignored.
Good tools show this automatically. If yours does not, calculate it from run history once per sprint.
FAQ
Why do automated tests fail without code changes?
Usually because of fixed timeouts, shared state between tests, fragile locators, or unmocked external services.
What are flaky tests?
Flaky tests sometimes pass and sometimes fail on the same code. Use auto-wait, isolate tests, and switch to reliable locators.
How should we choose locators?
Use data-testid first. Then ARIA attributes and roles. Avoid DOM-position XPath and CSS chains.
Why do tests pass locally but fail in CI?
CI is slower, browser versions differ, parallel runs expose shared state, and external services behave differently.
How do we isolate tests?
Each test should create its own data before it runs and remove or reset it after it finishes.