# Test Flakiness Reproduction and Stabilization Workflow

Public URL: https://amo.ng/prompts/test-flakiness-reproduction-stabilization-workflow

Summary: Reproduce intermittent test failures, isolate nondeterministic causes, implement a cause-specific fix, and measure reliability without hiding product defects.

Use this for: Reproducing flaky tests across seeds, order, timing, concurrency, environments, and dependencies before implementing and measuring a cause-specific stabilization.

Category: Codex & Coding
Tool: Codex
Difficulty: Expert
Prompt type: testing

## Best Use Cases

1. Intermittent CI Failure Reproduction
2. Test Order Dependency Isolation
3. Concurrency and Timing Diagnosis
4. Flaky Test Quarantine Review
5. Stabilization Reliability Measurement

## Prompt Body

You are a senior test-infrastructure and software-reliability engineer experienced in nondeterminism, concurrency, asynchronous systems, test isolation, CI environments, experiment design, and regression-safe code changes.

Your task is to reproduce an intermittent test failure, identify the responsible nondeterministic mechanism, implement the smallest cause-specific stabilization within the allowed scope, and measure whether reliability improved without weakening the product invariant.

Produce a flakiness evidence report, controlled reproduction protocol, hypothesis experiment ledger, root-cause determination, focused stabilization, and before-and-after reliability assessment.

Do not call a test stable merely because it passed after a retry or completed several successful runs.

## Context to Provide

Replace every bracketed placeholder. If a blocking input is missing, ask one consolidated set of questions before editing or reaching a root-cause conclusion. Continue with clearly labelled assumptions only when the missing information is non-blocking.

- [Investigation objective and affected test identifiers]
- [Repository context and project instructions]
- [Relevant test, fixture, helper, and product files]
- [Failure output, logs, traces, and artifacts]
- [Test contract, current behavior, and expected invariant]
- [Test runner, commands, configuration, and versions]
- [CI and local environment details without secrets]
- [Failure history, retries, and quarantine status]
- [Known seed, order, worker, timing, and resource clues]
- [Allowed files, commands, and mutation boundaries]
- [Repetition budget and reliability acceptance criteria]
- [Definition of done]

## Evidence and Repository Rules

- Inspect repository instructions, version-control status, relevant code, runner configuration, and available artifacts before proposing edits.
- Preserve unrelated, pre-existing, staged, untracked, and user-authored work.
- Stay within the supplied file, command, environment, and mutation boundaries.
- Do not reset branches, discard work, delete artifacts, rewrite history, push, publish, deploy, change protected CI settings, or mutate external services unless explicitly authorized.
- Separate confirmed evidence, assumptions, hypotheses, unknowns, risks, recommendations, changes, and observed results.
- Do not invent files, configurations, versions, seeds, commands, logs, traces, run counts, pass rates, failure rates, owners, approvals, or product behaviour.
- Use `Not provided`, `Not inspected`, `Not run`, or `To be agreed` when evidence is unavailable.
- Preserve conflicting evidence. Record the source, commit, environment, time, scope, and discriminating check needed to resolve each conflict.
- Report exact commands, parameters, working directory, relevant environment controls, exit status, run count, and failures.
- Redact secrets, credentials, tokens, customer data, private endpoints, and confidential values.
- Prefer focused diagnostic instrumentation and reversible experiments before changing product or test behaviour.
- Do not describe an experiment as controlled when the commit, configuration, dependencies, data, external state, or environment changed without being recorded.
- Tie every change to a supported root-cause hypothesis, preserved invariant, verification method, and acceptance condition.

## Flakiness Definition and Classification

Treat a test as flaky only when materially comparable executions of the relevant code and test can produce different outcomes without an intentional behavioural change.

Distinguish:

1. **Test defect** — the test, fixture, helper, assertion, setup, teardown, or isolation boundary is nondeterministic or incorrect.
2. **Product defect** — the test correctly exposes an intermittent race, data-integrity problem, state transition error, or other real behavioural defect.
3. **Environment defect** — runner resources, operating system behaviour, process isolation, filesystem, ports, containers, clocks, or CI infrastructure cause inconsistent results.
4. **Dependency defect** — a browser, database, network, external service, emulator, library, cache, or other dependency behaves inconsistently.
5. **Mixed cause** — more than one layer is required to produce the failure.
6. **Not reproduced** — the supplied or completed experiments did not reproduce the symptom sufficiently to establish flakiness or cause.

Do not classify a deterministic failure that occurs only under a specific recorded configuration as flaky merely because another configuration passes.

## Preserve the Test Contract

Before changing code, state:

- the behaviour or invariant under test;
- why the invariant matters;
- whether the assertion represents a user-visible, data-integrity, security, concurrency, or internal contract;
- what evidence proves the expected behaviour;
- which behaviours may change;
- which behaviours must remain unchanged.

Do not weaken, remove, skip, mock away, or broaden tolerances around the invariant simply to obtain a passing test.

If the expected behaviour is disputed, stop before stabilization and identify the product or domain owner needed to resolve the contract.

## Baseline Evidence

Create a baseline before applying a fix.

Record:

- commit and working-tree state;
- test identifier and source location;
- runner, runtime, browser, dependency, operating-system, container, and architecture versions;
- runner configuration, retries, timeouts, sharding, parallelism, and worker count;
- command and environment controls;
- isolation level;
- number of first attempts;
- automatic retry attempts recorded separately;
- passes, failures, skips, timeouts, crashes, and inconclusive runs;
- seed, order, worker, process, shard, and attempt number;
- start time, duration, time zone, locale, and clock controls;
- resource and dependency state;
- failure signature and artifact references.

Treat the first-attempt result as distinct from retry outcomes. A test that fails initially and passes on retry remains evidence of unresolved flakiness.

If retries cannot be disabled safely for the measurement run, preserve each attempt separately and explain the limitation.

## Reproduction Harness

Design a bounded harness that can repeat the affected test or scenario while collecting enough information to compare passing and failing executions.

The harness should capture, where relevant:

- run and attempt ID;
- commit and configuration fingerprint;
- seed and randomized values;
- test collection and execution order;
- worker, shard, thread, and process;
- timestamps and durations for meaningful state transitions;
- temporary paths, ports, resource identifiers, and database records;
- clock, time zone, locale, and date boundary;
- relevant logs, traces, screenshots, videos, network evidence, and process output;
- setup and teardown result;
- external dependency status;
- failure signature;
- cleanup completion.

Do not add logging that changes timing or behaviour without recording the risk of observer effects.

Keep artifacts from representative passing and failing runs when comparison may reveal the cause.

## Experiment Design

Start with the cheapest safe experiment that can distinguish the leading hypotheses.

Use materially comparable trials and vary one dimension at a time initially. Test interactions only when individual experiments do not explain the evidence or when the suspected mechanism requires multiple conditions.

Consider these experiment dimensions where relevant:

1. Affected test alone versus its containing file, suite, or full dependency chain.
2. Same process repeated versus a fresh process for each run.
3. Original order versus reverse, randomized, targeted predecessor, and targeted successor order.
4. Fixed seed versus multiple recorded seeds.
5. Serial execution versus normal and increased parallelism.
6. One worker versus multiple workers or shards.
7. Local environment versus a reproducible CI-like environment.
8. Warm versus cold caches and reused versus fresh state.
9. Controlled clock, time zone, locale, and boundary times.
10. Unique versus shared database records, files, ports, accounts, queues, or namespaces.
11. Normal versus constrained CPU, memory, storage, or connection capacity.
12. Real dependency versus a contract-preserving controlled dependency.
13. Automatic retry disabled versus retry results recorded by attempt.
14. Suspected predecessor tests included versus excluded.

Do not indiscriminately run every combination. Prioritize experiments by explanatory value, cost, safety, and likelihood of producing a discriminating result.

## Hypotheses to Test

Treat each possible cause as a hypothesis, not a conclusion.

Evaluate whether:

- setup, teardown, fixtures, helpers, or tests leak mutable state;
- the test depends on execution order or an unrecorded predecessor;
- an assertion observes asynchronous or eventually consistent state before the required condition is true;
- a fixed sleep or timeout substitutes for a meaningful readiness condition;
- the system clock, date boundary, time zone, locale, or timer resolution affects behaviour;
- random values, seeds, identifiers, paths, ports, or generated data are uncontrolled;
- multiple workers contend for shared data, files, services, queues, processes, or configuration;
- a transaction, cache, singleton, module, environment variable, browser context, or global configuration survives between tests;
- cleanup runs late, fails silently, or overlaps the next execution;
- a database, browser, network, emulator, container, or external dependency is unstable;
- CI resources, caching, architecture, or load differ materially from local execution;
- automatic retries, quarantine, broad exception handling, or permissive assertions conceal the first failure;
- the test correctly exposes a real intermittent product race or data defect.

For each hypothesis, state:

- predicted signal;
- evidence for;
- evidence against;
- controlled manipulation;
- comparison group;
- run budget;
- result;
- limitation;
- status: supported, unresolved, unlikely, rejected, or confirmed.

## Root-Cause Standard

Do not declare a root cause merely because a change makes the symptom disappear.

A confirmed root cause should satisfy the following where feasible:

- it explains the observed failure signature;
- it explains relevant seed, order, timing, worker, process, environment, or dependency patterns;
- manipulating the suspected cause changes the failure behaviour as predicted;
- removing or controlling the cause eliminates or materially reduces failures in comparable trials;
- the explanation identifies the responsible test, product, dependency, or environment boundary;
- competing explanations have been tested or remain explicitly unresolved;
- the proposed fix operates at the responsible layer.

If reproduction remains unsuccessful, provide the strongest supported hypotheses and improve future artifact capture instead of inventing a conclusion.

## Stabilization Hierarchy

Choose the narrowest fix that addresses the confirmed or best-supported mechanism.

Prefer, where applicable:

- deterministic setup and teardown;
- unique and isolated test resources;
- explicit state reset;
- controlled clock, seed, locale, and identifiers;
- synchronization on an observable condition;
- runner-supported retrying assertions for genuine asynchronous state;
- correct transaction and cleanup boundaries;
- removal of hidden order dependence;
- thread-safe or process-safe product behaviour;
- bounded polling with a meaningful condition and timeout;
- contract-preserving dependency control;
- cause-specific CI resource correction;
- improved diagnostic capture for unresolved recurrence.

Do not use these as a final stabilization without cause-specific justification:

- unconditional retries;
- arbitrary or longer sleeps;
- broadly increased timeouts;
- weakened assertions;
- ignored exceptions;
- indiscriminate mocking;
- global serialization of an otherwise parallel-safe suite;
- excessive resource increases;
- permanent quarantine;
- skipping the test.

Record why rejected masking approaches would hide rather than repair the problem.

## Quarantine Controls

Quarantine is temporary containment, not stabilization.

If quarantine is necessary, define:

- affected test and protected invariant;
- evidence supporting containment;
- user, product, merge, or release risk;
- owner;
- tracking issue;
- start date;
- expiry or review date;
- CI visibility and reporting behaviour;
- whether the test still runs outside the blocking path;
- artifact-retention requirement;
- removal criteria;
- escalation when the expiry is reached.

Do not silently convert a required failing test into a passing, skipped, allowed-to-fail, or non-blocking check.

## Reliability Measurement

After implementing the stabilization, repeat comparable baseline and stress conditions.

Report separately:

- first-attempt runs;
- retry attempts;
- serial runs;
- parallel runs;
- randomized-order runs;
- fixed-seed and varied-seed runs;
- affected test runs;
- related test and product regression runs;
- environments tested;
- passes, failures, timeouts, skips, and inconclusive results.

Calculate observed failure rates only from clearly defined comparable trials.

When reporting statistical confidence or an upper failure-rate bound:

- state the method and confidence level;
- state any independence and stationarity assumptions;
- explain whether repeated runs share a process, worker, cache, fixture, or external dependency;
- do not imply that zero observed failures proves a zero true failure probability;
- do not combine materially different environments or experiment conditions into one rate;
- do not compare before and after results when instrumentation, population, or configuration differences make them non-comparable.

If the repetition budget is too small to support the requested reliability claim, report the achieved evidence and the additional run budget needed.

## Workflow

1. Inspect repository instructions, version-control status, test code, product path, runner configuration, CI configuration, history, retries, and artifacts.
2. Define the test contract, failure signature, affected scope, product risk, and reliability acceptance criteria.
3. Freeze and record the baseline commit, environment, configuration, and measurement method.
4. Build or adapt a bounded reproduction harness without weakening the invariant.
5. Establish the baseline first-attempt failure rate and preserve representative passing and failing artifacts.
6. Rank hypotheses and run controlled experiments against the highest-value nondeterministic dimensions.
7. Determine whether the defect belongs to the test, product, dependency, environment, or multiple layers.
8. Implement the smallest cause-specific change within the allowed files.
9. Run focused verification before broader regression and stress checks.
10. Measure before-and-after reliability using comparable conditions.
11. Review any quarantine, retry, timeout, mock, serialization, or resource change for masking risk.
12. Report changed files, preserved behaviour, evidence, limitations, remaining risk, rollback steps, and the smallest safe next action.

## Output Contract

Use concise prose for conclusions and tables for experimental comparisons, commands, evidence, and status.

### 1. Input Sufficiency and Investigation Boundary

State the test, failure signature, repository scope, allowed files, available artifacts, missing inputs, assumptions, prohibited actions, repetition budget, and acceptance criteria.

### 2. Test Contract and Flakiness Profile

Provide:

| Test identifier | Expected invariant | Observed failure | First known occurrence | Affected environments | Known frequency | Product risk | Current retry or quarantine |
|---|---|---|---|---|---|---|---|

### 3. Baseline Run Ledger

Provide:

| Run cohort | Command and configuration | Process model | Order/seed/workers | Runs | First-attempt passes | First-attempt failures | Retry passes | Other outcomes | Artifact reference |
|---|---|---|---|---:|---:|---:|---:|---|---|

Do not merge automatic retries into first-attempt passes.

### 4. Hypothesis Experiment Ledger

Provide:

| Priority | Hypothesis | Predicted signal | Controlled change | Comparison | Runs | Observed result | Limitation | Status |
|---:|---|---|---|---|---:|---|---|---|

### 5. Root-Cause Determination

State:

- classification;
- nondeterministic mechanism;
- responsible boundary;
- evidence supporting the mechanism;
- evidence against competing causes;
- affected product or test behaviour;
- confidence and limitations;
- whether the original symptom was reproduced.

### 6. Stabilization Change

Provide:

- changed files;
- exact behavioural change;
- why the change addresses the cause;
- invariant preserved;
- diagnostic evidence added;
- risks and compatibility considerations;
- rejected masking approaches;
- rollback method.

### 7. Verification Results

Provide:

| Check | Command | Environment | Runs | Expected result | Observed result | Exit status | Evidence | Status |
|---|---|---|---:|---|---|---|---|---|

Mark checks that were not executed as `Not run`.

### 8. Before-and-After Reliability Assessment

Provide:

| Comparable cohort | Before runs/failures | Before observed rate | After runs/failures | After observed rate | Statistical method or caveat | Interpretation |
|---|---|---:|---|---:|---|---|

Do not claim zero residual flakiness from zero observed failures.

### 9. Quarantine or Temporary Containment Record

Include this section only when containment remains necessary. Record the owner, issue, reason, CI behaviour, protected risk, expiry, monitoring, and removal criteria.

### 10. Changed Files and Remaining Risk

Summarize modified files, preserved behaviour, unrun checks, unresolved hypotheses, environmental limitations, and rollback steps.

### 11. Smallest Safe Next Action

End with the single next action that most reduces remaining uncertainty or recurrence risk.

## Verification Checklist

Before finalizing, confirm that:

- the expected product invariant is explicit and preserved;
- the original symptom is reproduced or accurately marked `Not reproduced`;
- baseline and post-change trials use documented, comparable conditions;
- first attempts remain separate from retry outcomes;
- seeds, order, workers, processes, timing, environments, and artifacts are recorded where relevant;
- experiments distinguish correlation from a cause-specific mechanism;
- test, product, dependency, and environment defects were considered separately;
- the stabilization operates at the responsible boundary;
- retries, sleeps, timeouts, mocks, serialization, and quarantine do not conceal unresolved failures;
- related product behaviour and regression tests were checked where relevant;
- reliability claims disclose run counts, failures, method, assumptions, and sample limitations;
- zero observed failures is not presented as proof of zero future failures;
- changed files, rollback steps, unrun checks, and remaining risks are reported;
- every conclusion is supported by supplied evidence or explicitly labelled;
- no file, command, result, rate, approval, or product behaviour was invented.

Begin by reviewing the supplied context for blocking gaps. If none remain, inspect the repository and build the Test Contract and Baseline Run Ledger before proposing changes.

## Variables to Replace

1. Investigation objective and affected test identifiers
2. Repository context and project instructions
3. Relevant test, fixture, helper, and product files
4. Failure output, logs, traces, and artifacts
5. Test contract, current behavior, and expected invariant
6. Test runner, commands, configuration, and versions
7. CI and local environment details without secrets
8. Failure history, retries, and quarantine status
9. Known seed, order, worker, timing, and resource clues
10. Allowed files, commands, and mutation boundaries
11. Repetition budget and reliability acceptance criteria
12. Definition of done

## How to Use

Replace every placeholder and paste the completed prompt into Codex from the repository root. Include the exact test identifiers, failure output, CI run links or sanitized logs, retained traces, test and product code, runner configuration, relevant versions, known seeds or worker IDs, allowed files, safe commands, repetition budget, and reliability criteria.

Allow Codex to inspect the repository and existing artifacts before editing. Review the proposed reproduction matrix before approving expensive repeated runs. Require manual review of any retry, timeout, mock, serialization, resource increase, assertion change, or quarantine. Do not accept “passed after retry” or a small number of successful runs as proof of stabilization.

## Example Use Case

A team provides a CI test that fails on approximately three first attempts per 100 comparable runs, along with failing seeds, worker IDs, traces, test and fixture code, product logs, runner configuration, recent changes, and a target of zero observed failures across 1,000 defined post-fix trials.

## Tags

1. test-flakiness
2. automated-testing
3. ci
4. nondeterminism
5. concurrency
6. test-isolation
7. codex
8. debugging
9. stabilization
10. regression-testing

## Dates

Published: 2026-08-03
Updated: 2026-08-03