What I listen for when I ask about a flaky test

I interview SDET candidates. My favourite opening question isn't about Playwright and it isn't a coding puzzle:

Tell me about the last flaky test you fixed. Walk me through how you found the root cause.

I like it because there's no way to prepare for it. You either have the story or you don't. It can't be answered from a tutorial, and it reveals more in three minutes than a whiteboard exercise does in thirty.

The weak answer

I've heard this many times, close to verbatim: it was failing sometimes, so I added a wait. Occasionally the wait is named out loud — sleep(5000). The test went green, everyone moved on.

What makes this a poor answer isn't the wait itself. It's that the story has no investigation in it. Something was wrong, an action was taken, the symptom disappeared. There's no point at which the candidate asked why.

And it usually can't survive a follow-up. If I ask what was actually racing, or whether it failed alone or only in parallel, the answer runs out. Not because the candidate is unintelligent — because the question was never asked at the time.

The strong answer

It sounds like an investigation. Roughly:

Reproduced it locally first, running it repeatedly rather than once.
Checked whether it failed on its own or only when the suite ran in parallel.
Narrowed it down — two tests quietly sharing the same test user, or leftover data from a previous run, or a genuine race in the application.
Fixed the cause rather than the symptom. Isolated the data instead of slowing the test.
Then ran it fifty times before believing it was gone.

Notice what that answer contains that the weak one doesn't: a hypothesis, a way of testing the hypothesis, and a way of knowing when to stop. It's the shape of debugging, and it transfers to every problem the person will meet on the job — including the ones nobody has written a tutorial about.

What the question is actually measuring

Not Playwright knowledge. I can teach someone a framework in a few weeks.

What I can't teach is the willingness to keep pulling on a thread until the real cause comes out. Some people find an unexplained failure intolerable and will chase it. Others find a green pipeline sufficient. Both types pass a coding screen. Only one of them is useful when a suite starts failing for reasons nobody understands.

There's a second signal in there too. A candidate who says "it turned out to be my own bad assumption about the data" tells me they can be wrong out loud. That matters more on a team than almost any technical skill, and it's very hard to fake.

How to have a real answer

If you're preparing for SDET interviews and you don't have this story, the fix isn't to invent one. Interviewers follow up, and invented stories collapse under the second question.

The fix is to go and have the experience. Build a suite, run it in CI, and wait — flakiness will arrive on its own. When it does, resist the wait, find the cause, and write down what happened while it's fresh: what you saw, what you suspected, what it turned out to be, how you verified the fix.

That's one afternoon of work and it's worth more in an interview than a month of watching course videos. It's also why "fix one flaky test properly, and write the story down" is a milestone in the 90-day roadmap rather than an afterthought.

Same tools on the resume. Very different engineers in the room.

If you want the diagnostic routine itself rather than the interview angle, that's in flaky tests don't get fixed with waits.