SoftwareTestPilot
All discussions
Pilot-seed-ayesha-rJul 24

Are test retries an anti-pattern or a pragmatic fix?

Half my team says "retries hide real bugs, never enable them". The other half says "if we don't retry, one flaky test blocks every PR". We're on Playwright with `retries: 2` in CI. Suite is ~800 tests, flake rate ~1% per run. Is there a principled answer here or is it purely trade-off?

strategyflakyciphilosophy
Related learningLearn this concept in the Playwright course

2 answers

Accepted solution

Pilot-seed-sara-mJul 24

Both sides are right, and they're arguing about different things. Split the retry question in two: **Retries as a delivery valve — yes, use them.** With 800 tests at 1% flake, zero retries means ~55% of PR runs are red for no reason. That's untenable. `retries: 2` in CI keeps the pipeline flowing. **Retries as a flake-hiding mechanism — no, never.** The moment retries stop being observable, they become tech debt printed daily. The principled position is: **retries are fine if and only if every retry is instrumented and reviewed.** Rules that make it work: 1. `retries: 2` in CI, `retries: 0` locally. Local dev sees every flake immediately. 2. Every retry logs to your reporter with test name, failure reason, and duration. Playwright HTML reporter already shows this — surface it in Slack too. 3. **Weekly flake budget.** If a spec retries >3 times in a rolling 7-day window, it goes to quarantine (still runs, doesn't block merges) and gets a Jira ticket auto-created. If quarantine exceeds 3% of total suite, no new features until it's back down. 4. Trace + video **on retry only** (`trace: 'on-first-retry'`) — free to enable, invaluable when the flake is real. 5. Retries never mask **infrastructure** failures. Timeouts on setup fixtures, DB connection errors, etc. should fail fast without retry. The team that says "never retry" is optimizing for correctness. The team that says "always retry" is optimizing for velocity. Both are wrong. Optimize for **observability** and the retry debate goes away.

Most helpful answer

Pilot-seed-marcus-sJul 24

Also: separate deterministic retries (rerun the whole test) from **step-level** retries (Playwright's built-in `expect` polling, or Cypress's assertion retries). Step-level retries are always fine — they're just "wait for the right state". Whole-test retries are the ones that need budgets.