SoftwareTestPilot
Automation TestingPublished: 13 min read

Best Flaky Test Detection Tools to Stabilise Your CI

Compare ACCELQ, Trunk, BuildPulse and Launchable for detecting flaky tests, investigating failures and quarantining them so CI stays green and trustworthy.

SoftwareTestPilot
QA Editorial Team
Share:XLinkedInWhatsApp
Best Flaky Test Detection Tools to Stabilise Your CI: Automation Testing article cover, SoftwareTestPilot.com
Best Flaky Test Detection Tools to Stabilise Your CI: Automation Testing article cover, SoftwareTestPilot.com

Quick answer: the best flaky test detection tool is the one that reads the results your CI already produces, tells you which tests are unstable, and keeps them from blocking the team while someone fixes them. A practical shortlist:

  • ACCELQ if your UI automation lives in ACCELQ and brittle element locators are a main cause of your flakiness.
  • Trunk Flaky Tests if you want detection plus automatic quarantine wired into CI.
  • BuildPulse if you want a dedicated flaky-test dashboard that ranks tests by impact.
  • Launchable if you also care about test selection and want flakiness scores as part of wider test insights.

This guide is for automation testers, SDETs and QA leads whose pipelines go red for no clear reason. It explains how detection works, compares four tools on the same criteria, and walks through a worked example of finding and containing one flaky test.

How this was written: features come from each vendor's documentation, linked in the text. We did not benchmark the tools ourselves, so there are no accuracy rankings, and prices are left out on purpose. Check vendor sites for current plans.

What makes a test flaky, and why detection needs history

A flaky test passes and fails on the same code with no change in between. Common causes are timing and waits, shared test data, test order, network calls and slow environments. We cover fixes for each in 15 causes of flaky tests and how to fix them.

The hard part is noticing. One red run looks exactly like a real bug. You only see flakiness when you compare many runs of the same test on the same commit or branch. Humans are bad at remembering that; tools are good at it.

Detection tools usually use one or more of these signals:

  • Pass on retry: a test fails, is retried in the same job, and passes.
  • Mixed results on the same commit: the test passed in one CI run and failed in another without code changes.
  • Failure rate over time: a test fails a small, steady share of runs on the main branch.

How to judge a flaky test detection tool

1. Detection quality

Which signals does it use, and can you tune them? A tool that only counts retries misses tests that fail without being retried.

2. Investigation help

When a test is flagged, can you see its run history, error messages and which commits or jobs it failed on? Grouping by error saves the most time.

3. Containment

Can it quarantine a flaky test so its failure no longer fails the build, while still running and reporting it? Is there a clear way to release it from quarantine?

4. Fit with your pipeline

Does it accept your result format (often JUnit XML), your CI provider and your languages? How much code change does setup need?

5. Team workflow

Can you assign owners, link tickets and see whether the flaky count is going down over weeks?

The four tools at a glance

Documented capabilities only. "Strong fit" describes the scenario each vendor's documentation is built around, not a measured score.

ToolMain approachInvestigationContainmentStrong fit
ACCELQPrevent locator-driven flakiness with self-healing element identificationRun reports and scenario reliability insights in the platformFewer brittle failures at the sourceTeams automating UI in ACCELQ
Trunk Flaky TestsDetect from uploaded CI results, including pass-on-retryPer-test history and failure detailsAutomatic quarantineTeams that want merges unblocked fast
BuildPulseDetect flaky tests from CI resultsDashboard ranking flaky tests by impactQuarantine options described in its docsTeams wanting a focused flaky-test view
LaunchableFlakiness scores within wider test insightsUnhealthy test insightsPair with your own CI rulesTeams also interested in test selection

ACCELQ: cutting flakiness at the source

ACCELQ is a codeless automation platform, so its angle is prevention rather than after-the-fact detection. Its documentation describes self-healing element identification: when a page changes, the platform adapts how it finds elements instead of failing on a stale locator. Its Scenario Insights add reliability analytics per scenario.

Where it fits: UI suites built in ACCELQ, where broken or brittle locators cause much of the noise.

Limitations to check: it does not analyse tests you run in other frameworks. Self-healing addresses locator problems, not shared-data or environment causes.

Trunk Flaky Tests: detect and quarantine

Trunk's Flaky Tests product ingests test results uploaded from CI and identifies unstable tests. Its docs describe detection monitors such as a pass-on-retry monitor, plus pages for managing detected flaky tests.

The key feature is quarantining: a known flaky test keeps running, but its failure no longer fails the CI job. That unblocks merges without deleting the test.

Where it fits: teams where flaky failures regularly block pull requests and slow everyone down.

Limitations to check: quarantine is only safe with discipline. Give every quarantined test an owner and review the list weekly, or real bugs can hide there.

BuildPulse: a focused flaky-test dashboard

BuildPulse is built around flaky tests. Its documentation explains how it reads CI results to find flaky tests, and its Flaky Tests Dashboard shows which tests are flaky and how much they disrupt builds, so you can fix the worst first.

Where it fits: teams that want a single, clear list of flaky tests ordered by impact, without adopting a broader platform.

Limitations to check: confirm support for your CI provider and result format, and how it treats tests that are renamed or moved.

Launchable: flakiness inside wider test insights

Launchable is known for predictive test selection: running the tests most likely to fail for a change first. It also provides Unhealthy Test Insights, which surface flaky tests along with other problem tests, and its changelog describes flakiness scores that are recalculated regularly.

Where it fits: teams with long test suites who want faster feedback and a flakiness view from the same data.

Limitations to check: if you only need flaky-test detection and quarantine, a narrower tool may be simpler to adopt.

Worked example: from red build to contained flaky test

Illustrative example, not a real customer story.

A team's checkout suite fails about once in every eight nightly runs. The failure is always order-summary.spec.ts, with a TimeoutError waiting for the total price.

  1. Measure before guessing. Run the test many times on the same commit. In Playwright, the documented --repeat-each option does this: npx playwright test order-summary.spec.ts --repeat-each=30. It fails 4 of 30 times. That is flakiness, not a regression.
  2. Turn on retries, but record them. Playwright marks a test that fails and then passes on retry as flaky in its report, so retries give you a signal instead of hiding the problem.
  3. Send results to a detection tool. Upload the JUnit XML from every CI run. Within a week the tool flags order-summary.spec.ts as flaky across several branches.
  4. Contain. Quarantine the test with an owner and a target date. The build stays green; the test still runs and reports.
  5. Investigate. History shows failures only on runs where the pricing API mock responded late. The fix: wait for the network response that loads the total instead of a fixed sleep.
  6. Release. After the fix passes 30 repeated runs and a week of CI, take it out of quarantine.

The tool did steps 3 and 4. The person did the measuring, the diagnosis and the fix. For an AI-assisted way to cluster and triage many flaky failures at once, see our AI flaky test triage playbook.

A CI-friendly starting configuration

Most detection tools read standard result files. This Playwright config retries only in CI and writes JUnit XML that tools can ingest. It uses documented options but was not executed as part of writing this article.

// playwright.config.ts
import { defineConfig } from "@playwright/test";

export default defineConfig({
  retries: process.env.CI ? 2 : 0,             // retry in CI only, so flaky tests get flagged
  reporter: [
    ["list"],
    ["junit", { outputFile: "results/junit.xml" }], // upload this file to your detection tool
  ],
  use: { trace: "on-first-retry" },             // capture a trace when a retry happens
});

Then follow your chosen tool's guide for the upload step in CI. For pipeline setup, see the GitHub Actions pipeline guide for QA.

Decision table

Your main painStart by trialling
UI tests in ACCELQ break when pages changeACCELQ self-healing
Flaky failures block pull requests every dayTrunk Flaky Tests
You need one ranked list of what to fix firstBuildPulse
Suite is slow and also flakyLaunchable
Small suite, a handful of flaky testsFramework retries plus a good report; a tool may be premature

If you also need dashboards and failure categories across all failures, not just flaky ones, compare test automation reporting tools too.

Mistakes and edge cases

  • Infinite quarantine. Set a review date. A test quarantined for months is effectively deleted.
  • Unstable test names. Parameterised tests with timestamps or random IDs in their names split history and defeat detection.
  • Flaky environment, not flaky test. If dozens of unrelated tests flake at the same time, look at the environment first.
  • Order dependence. Tests that pass alone but fail in a full run point to shared state; run them in random order to confirm.
  • Counting found, not fixed. Track how many flaky tests were fixed each month and how long quarantine lasted.

Why this matters in interviews

"How do you handle flaky tests?" is a standard SDET interview question. A strong answer covers measuring with repeated runs, recording retries as flaky, quarantining with an owner, and fixing the root cause. Build those foundations with the free QA courses, and go deeper on Playwright framework design in the SoftwareTestPilot Academy.

Your next step

Pick the test that annoys your team most and run it 30 times on the same commit. If it fails even once, enable recorded retries, start uploading JUnit results, and trial one tool from the decision table for two weeks.

Got a flaky test you cannot pin down? Post the error and what you have tried in the QA Community.

Key takeaways

  1. 1A flaky test passes and fails on the same code; you need many runs, not one, to spot it.
  2. 2Detection tools read results from CI over time and score each test's instability.
  3. 3Quarantine keeps flaky failures from blocking merges while the test is fixed, but must have an owner and an exit date.
  4. 4Pick a tool that reads your existing result files (usually JUnit XML) and fits your CI.
  5. 5Detection does not fix anything: track time-to-fix, not just the number of flaky tests found.

Common mistakes to avoid

  • Adding blanket retries, which hides flakiness instead of measuring it.
  • Quarantining tests forever so real regressions slip through unnoticed.
  • Judging flakiness from a single failed run.
  • Uploading results without stable test names, so history splits into duplicates.
  • Blaming the tool when the real issue is shared test data or a slow environment.

References

  1. 1
  2. 2
    Trunk Docs: Quarantining— docs.trunk.io
  3. 3
  4. 4
  5. 5
    ACCELQ: Self Healing— support.accelq.com
  6. 6
    Playwright: Retries— playwright.dev