What Is a Flaky Test? A Plain Guide for AI-Built Software

A flaky test is an automated check that sometimes passes and sometimes fails on the very same code. Nothing about the product changed between the two runs, yet the answer did. If your software is built with automated checks, whether by people or by AI engineers, you will meet flaky tests. How the team handles them says a lot about how far you can trust a green result.

This guide explains what flaky tests are, why they matter to people who do not write code, and what a sensible team does about them.

What is a flaky test, in plain English?

An automated test is a small script that uses your product the way a person would and checks the outcome. For example: "Log in with a valid password, and the dashboard should appear." If the dashboard appears, the test passes. If not, it fails.

A flaky test gives different answers without any change to the product. Martin Fowler, a well-known software author, describes such tests as ones that pass sometimes and fail sometimes, "without any noticeable change in the code, tests, or environment". He calls this behaviour non-deterministic, which just means "not predictable".

Think of a bathroom scale that reads 70 kg, then 74 kg, then 70 kg again while you stand still. You would not change your diet based on it. You would doubt the scale.

Why do tests become flaky?

The idea behind the test is usually fine. Something around it is unreliable. Fowler names five common causes: lack of isolation, asynchronous behaviour, remote services, time, and resource leaks. Each has an everyday version:

• Tests that disturb each other (lack of isolation). One test leaves data behind, and the next test trips over it. It is like two people cooking in one kitchen without cleaning up.
• Waiting badly (asynchronous behaviour). A test clicks a button and checks the result before the page has finished loading. Sometimes the page is quick enough, sometimes not.
• Outside services (remote services). The test depends on something it does not control, such as a payment provider's test system, which may be slow or down.
• Time. A test that depends on today's date or the current minute can behave differently at midnight or at the end of a month.
• Leaks. A test borrows something, such as a connection to a database, and does not hand it back. A later test then finds none left.

A busy computer can add to the problem. If the machine is working hard, a page may load more slowly than the test expected, and the test fails even though the product is fine.

Why should a founder or manager care?

Three reasons.

It wastes time and money. A failing test is a signal to go and fix something. If the failure is not real, a person or an AI agent can spend hours looking for a bug that does not exist, or rework code that was correct. With AI engineers, that effort counts against your plan or budget.

It slows releases. Finished work waits while someone reruns the checks and wonders whether the red result is real.

It destroys trust. This is the serious one. Fowler warns that flaky tests are "a virulent infection that can completely ruin your entire test suite". The reason is human: people learn to shrug at red results. Then, on the day a real failure appears, it gets shrugged at too.

Can I just rerun the test until it passes?

One rerun is a reasonable first step. It is not a fix.

A careful rerun works like this. When a small number of tests fail and nothing else looks wrong, run them once more on the same code before blaming anyone. If they now pass, the first failure was probably noise. If they fail again, treat the failure as real and send it back to whoever built the change.

Be wary of teams that rerun three or four times until something turns green. A real bug that only shows up now and then can slip through by luck. And a rerun never ends the story, because the flaky test is still there and will fail again.

What does a healthy way to handle flaky tests look like?

Fowler's advice centres on quarantine. In plain terms, it works in four steps:

1. Confirm it is flaky. Same code, different results.
2. Set it aside, visibly. Move it out of the main set of checks so the healthy tests keep giving clear answers. This is called quarantine. It protects everyone from noise, but it is only temporary.
3. Cap the quarantine. Fowler suggests two limits: allow only a fixed number of tests in quarantine (he gives 8 as an example), and put a time limit on how long a test may stay there (he gives a week as an example). Without limits, set-aside tests get forgotten.
4. Fix the cause. Make the test wait properly, give it clean data, or control the clock.

One rule matters most for non-coders: a quarantined test must never be reported as a clean pass. "All green, except four tests we set aside" is honest. "All green" is not.

How does Atlas handle flaky tests?

Atlas is a software team of AI engineers built on Claude Code. You talk to PA, the project assistant, in plain words, and PA shares the work out to specialists who design, build, test and security-check it.

Atlas does not yet have special handling for flaky tests, so here is what it does today. Its "landing train" merges up to six finished branches and runs the project's own checks once on the combined result. If that batch goes red, Atlas splits it until it finds the branch that broke it, and returns the work to its builder. Atlas also refuses to call work "done" until it is on the main line (or queued to land), a screen has its test cases and screenshots, and the product's rules pass. The Tests screen holds a library of test cases, runs, screenshots and a step-by-step replay, so you can see what a run actually did.

Dealing with flaky tests specifically, such as telling noise from a real failure and keeping a visible list of set-aside tests, is coming to Atlas, built in, so it needs no setup on your side.

Until then, you can do the checking yourself. Ask PA in plain words to look at a failed run and tell you whether the same test has passed on the same code before. Use the replay and screenshots on the Tests screen to see what happened. Keep your own short list of tests you suspect, with an owner and a date for each. See the Atlas home page for what the team does.

What should I ask my team?

You do not need to read a test. Ask these:

• How many flaky tests do we have right now? Zero is fine. "We don't know" is a warning sign.
• Who owns fixing them? Each one needs a named owner, human or AI engineer.
• How long can one stay set aside? A deadline keeps quarantine from becoming a graveyard.
• Did a rerun ever hide a real failure? A good team can tell you how it knows the difference.
• Are we chasing failures that turned out to be noise? If someone has spent hours on one, that is a cost worth naming.

Conclusion

Flaky tests are common and are not a sign that your team is bad. They become a problem when nobody names them, when reruns hide real faults, or when someone, human or AI, burns hours chasing a ghost. Ask for a visible list, an owner and a deadline for each one, and you keep the one thing a test suite is for: an answer you can believe.