Flaky tests make a CI/CD pipeline difficult to trust. This repeatable triage workflow helps engineering teams separate product defects from test defects, infrastructure failures, and nondeterministic behavior, then assign evidence-based fixes instead of relying on repeated reruns.
Overview
A flaky test does not produce the same result when the code, test inputs, and intended environment are apparently unchanged. One run passes and another fails, or the test fails only under particular timing, load, ordering, or parallel-execution conditions. Flakiness is not merely an inconvenience: it consumes review time, slows releases, and teaches people to ignore pipeline failures.
The practical goal is not to label every intermittent failure as “flaky.” First classify the failure. A real product defect can appear intermittent when a test reaches it through a race or an unstable dependency. An infrastructure failure can look like an application regression when a runner loses network access. A test defect may be exposed only when tests run in a different order. Good triage preserves this distinction.
| Failure category | Typical signals | First response |
|---|---|---|
| Product defect | The failure is reproducible with controlled inputs, or the application violates an expected behavior. | Reproduce outside the pipeline, create or update a defect, and keep the test active. |
| Infrastructure failure | Runner outages, unavailable services, resource exhaustion, DNS errors, or dependency timeouts appear in logs. | Compare runner and environment telemetry, then repair the platform or dependency path. |
| Test defect | Weak waits, shared state, order dependence, unstable selectors, or incorrect cleanup are present. | Fix the test design and add a focused reproduction. |
| Nondeterministic behavior | The same test varies with timing, concurrency, scheduling, random data, or external state without a clear single cause. | Capture repeated-run evidence and narrow the variable before changing production code. |
Use a single owner for each investigation, even when several teams contribute evidence. Ownership means coordinating the investigation and documenting the decision; it does not imply that one person must implement every fix.
Checklist by scenario
1. The test failed once
- Record the commit, pipeline identifier, job name, runner or executor type, test command, dependency versions, and environment name.
- Save the complete test output, screenshots, video, traces, browser logs, container logs, and relevant service logs before the retention window removes them.
- Check whether the failure is a test assertion, setup error, timeout, process termination, network error, or infrastructure status failure.
- Re-run the smallest useful scope. A targeted test rerun is more informative than repeatedly rerunning the entire pipeline.
- Compare the failed run with a nearby successful run, including test order, parallel shard, generated data, and environment configuration.
GitHub Actions and GitLab CI jobs can expose different metadata, but the principle is the same: preserve job context and artifacts as part of the failure record. A rerun that passes is evidence of intermittence, not proof that the test is safe to ignore.
2. The same test fails intermittently
- Run the test repeatedly in the same environment and record pass and fail counts, duration, shard, worker count, and failure message.
- Repeat with parallel execution disabled. If the failure disappears, investigate shared state, port collisions, database records, files, locks, and service capacity.
- Repeat with a fixed random seed when the test framework supports it, then vary the seed deliberately to determine whether generated data affects the outcome.
- Compare cold-start and warm-start behavior. A dependency that is ready eventually may still violate the test’s startup assumptions.
- Inspect time-sensitive code: clock boundaries, time zones, token expiry, polling intervals, retries, asynchronous jobs, and eventual consistency.
Do not increase every timeout as a first fix. A longer timeout can hide a readiness or performance problem and make the pipeline slower. Prefer an explicit readiness check, deterministic fixture, bounded polling strategy, or controlled test clock where appropriate.
3. Failures appear only in parallel jobs
- Identify whether parallel workers share a database, queue, filesystem path, browser profile, service account, port, or test data record.
- Give each worker isolated namespaces, schemas, temporary directories, or resource identifiers where practical.
- Check that cleanup runs after both passing and failing tests. Cleanup that happens only in the success path leaves contamination for later workers.
- Review tests that depend on execution order. A test suite should establish its own preconditions rather than inheriting state from a previous test.
- Compare resource limits and scheduling. CPU throttling, memory pressure, and container eviction can change timing without changing application code.
For Kubernetes-based preproduction suites, inspect pod events, service endpoints, readiness conditions, ingress behavior, and configuration before blaming the test. The Kubernetes preproduction troubleshooting guide provides a useful companion checklist.
4. Failures depend on the preproduction environment
- Compare environment variables, feature flags, secrets references, service versions, schemas, network policies, and external integrations.
- Verify that the test data exists in the expected state and is safe to reset or recreate.
- Confirm that preproduction is sufficiently representative for the test’s purpose, while recognizing that it may not match production in scale or configuration.
- Check for configuration drift between the pipeline definition and the deployed environment.
- Use logs, metrics, and traces to correlate the test timestamp with application and dependency behavior.
Review the preproduction environment checklist, secrets and configuration workflow, and test data management guide when the environment itself is part of the uncertainty.
5. The failure blocks a release
- Apply the team’s documented failure policy rather than deciding ad hoc under delivery pressure.
- Separate “test failed” from “test is known flaky.” A temporary quarantine should require a ticket, owner, reason, linked evidence, and expiry or review date.
- Use a rerun only when it is an approved diagnostic or recovery action. Record the original failure and rerun result together.
- Escalate if the test covers a high-risk path, security control, data migration, or release gate that should not be bypassed casually.
What to double-check
Before closing an investigation, ask whether the evidence rules out the most common hidden variables:
- Time: Did the test depend on wall-clock time, delayed jobs, eventual consistency, or an unbounded wait?
- Order: Does it pass alone but fail after another test, or does it depend on a particular shard sequence?
- State: Are database rows, queues, files, caches, browser sessions, or feature flags isolated and cleaned up?
- Concurrency: Can two workers update the same resource, bind the same port, or consume the same message?
- Dependencies: Did a service, DNS record, certificate, container image, third-party API, or test double behave differently?
- Capacity: Were CPU, memory, disk, connections, or rate limits exhausted?
- Observability: Can the team connect the test failure to application logs, metrics, and traces using a timestamp, request ID, or job ID?
- Reproduction: Can another engineer run the smallest failing case using the recorded inputs and environment details?
Use the preproduction observability guide to improve the evidence available during the next failure. If the pipeline itself failed before tests ran, follow the CI/CD pipeline failure decision tree instead of classifying the event as test flakiness.
Flaky-test review template
Copy this record into the team’s issue tracker or quality dashboard:
- Test and owner: test name, repository, owning team, and responsible engineer.
- Scope: unit, integration, end-to-end, contract, browser, deployment, or smoke test.
- Evidence: failing and passing run links, commit, runner, shard, seed, duration, artifacts, and relevant logs.
- Classification: product defect, infrastructure failure, test defect, or nondeterministic behavior.
- Hypothesis: the specific timing, state, dependency, or concurrency condition being investigated.
- Containment: rerun policy, temporary quarantine, release impact, and review date.
- Permanent fix: code or configuration change, added diagnostic coverage, and validation runs.
- Exit criteria: for example, the cause is documented, the fix is merged, the test passes under normal and parallel execution, and no new unexplained failures appear during the agreed observation period.
Common mistakes
- Rerunning until green: This hides the failure rate and removes useful evidence. Capture the original run first and define how many diagnostic reruns are appropriate.
- Quarantining without ownership: A disabled test becomes permanent debt when no person or team is accountable for its return.
- Blaming the application too early: Validate runner health, environment configuration, test data, and dependency availability before opening a product regression.
- Blaming the environment too early: A race condition or shared fixture can surface only on a busy runner, but the test may still be defective.
- Using sleeps as synchronization: Fixed delays are often fragile across machines and load levels. Wait for a meaningful state or event with a bounded timeout.
- Changing several variables at once: If the team changes the test, runner size, retry count, and environment simultaneously, it becomes difficult to identify the cause.
- Discarding artifacts too quickly: Retain enough logs and diagnostic output to compare failures across runs while controlling sensitive data and storage costs.
For broader release decisions, connect test-health findings with the team’s environment strategy and promotion rules in staging versus preproduction versus production. A flaky test is a workflow signal, not only a test-code problem.
When to revisit
Review this workflow before seasonal planning cycles, when test suites are expanded, and whenever the CI/CD platform, runner image, test framework, browser, container base image, deployment model, or preproduction architecture changes. Revisit it after repeated failed builds, a new parallelization strategy, a migration to ephemeral environments, or an incident in which a pipeline failure was incorrectly ignored.
Make test health part of a short weekly review. Examine newly quarantined tests, repeat offenders, mean time to triage, failure categories, and tests that require frequent reruns. Do not optimize for a green dashboard alone; optimize for failures that provide trustworthy information.
To act now, choose the most frequently rerun test, create the review record, preserve its next failure’s artifacts, and run it under both normal and parallel conditions. Assign an owner and an exit date for any quarantine. After the permanent fix is merged, update the pipeline documentation and review whether the same isolation or observability pattern exists elsewhere.