Reusable AI capability
Isolate and Stabilize Flaky Automated Test Failures
Diagnose intermittent CI or local test failures, identify the supported nondeterministic cause, and define a cause-specific stabilization plan with reliability evidence instead of hiding product defects.
This Skill is a reusable set of instructions based on the linked Amo.ng Prompt or Workflow. It tells you what to provide, how to apply the method and how to know when the task is complete. It does not run automatically.
# Isolate and Stabilize Flaky Automated Test Failures Skill ID: AMO-S-000049 Skill URL: https://amo.ng/skills/isolate-and-stabilize-flaky-automated-test-failures Purpose: Apply a repeatable flakiness investigation method that reproduces intermittent failures, separates product defects from test defects, traces timing, ordering, concurrency, isolation, fixture, environment, and data-dependency causes, and produces a minimal stabilization and verification plan. Required inputs: - Failing test names, file paths, and relevant assertions or stack traces. - CI run history or local run evidence showing pass/fail frequency, timing, environment, commit range, and any correlated changes. - Relevant test code, fixtures, setup/teardown, mocks, clocks, seeds, network or database dependencies, and shared-state behavior. - Repository test commands and constraints, including whether repeated runs, sharding, randomization, or stress runs are authorized. - Any prior attempted fixes, quarantines, skips, or known product defects related to the test. How to use: When to use: - A test fails intermittently in CI or locally and the team needs evidence before quarantining, rewriting, or trusting it. - A recent change appears unrelated but correlates with sporadic test failures. - A test suite has order dependencies, shared state, timing assumptions, race conditions, external service variance, or environment-specific failures. - A team needs a reliability measurement plan after a proposed flaky-test repair. When not to use: - The failure is deterministic and already has a clear root cause; use ordinary debugging instead. - The goal is to suppress, delete, or quarantine failing tests without investigating whether they reveal a real product defect. - No failure logs, run history, test code, environment details, or reproduction attempts are available. - The issue is a production incident rather than a test-suite reliability problem. Instructions: 1. Open AMO-P-000236, supply its required inputs, and run the complete source Prompt in Codex. Review its evidence register and limitations before using the result. Use the authorized repository environment; a text-provider relay cannot inspect or change the repository for you. Inspect repository instructions and existing changes first, preserve unrelated work, and keep edits, commands and external actions within explicit authorization. 2. Separate supplied facts, observed run evidence, assumptions, inferences, missing information, and uncertainty before recommending a fix. 3. Build a hypothesis table covering likely nondeterministic causes such as timing, concurrency, async waits, test ordering, shared mutable state, stale fixtures, external dependencies, random data, clocks, retries, and environment differences. 4. Distinguish product behavior regressions from test harness defects; do not label a test flaky merely because it fails intermittently. 5. Define the smallest cause-specific repair or test-design change, plus any product-code investigation needed if the test is exposing a real defect. 6. Specify verification runs proportionate to risk, such as repeated local runs, CI reruns, randomized order, stress loops, affected-suite execution, and regression checks for nearby behavior. 7. For merge, quarantine, deletion, or CI policy changes, present evidence and options for human review rather than treating the AI recommendation as authorization. Expected output: A flakiness investigation brief containing reproduction evidence, ranked root-cause hypotheses, product-defect versus test-defect assessment, proposed minimal stabilization, test/CI verification plan, residual risks, and a human-reviewed merge or quarantine recommendation where applicable. Constraints and boundaries: - Do not claim tests were run, failures reproduced, or code changed unless execution evidence is supplied. - Do not recommend weakening assertions, adding arbitrary sleeps, or broad retries unless tied to a supported cause and verified as safe. - Do not hide suspected product defects by changing only the test. - Require human authorization for merge decisions, test quarantine, CI policy changes, or production-impacting code changes. - Preserve privacy and secrets in logs; redact tokens, credentials, customer data, and sensitive environment values. Based on Prompt: Test Flakiness Reproduction and Stabilization Workflow Source ID: AMO-P-000236 https://amo.ng/prompts/test-flakiness-reproduction-stabilization-workflow Completion criteria: Complete when: - The output cites concrete failure evidence, run IDs, logs, files, commands, or supplied excerpts for each material finding. - At least one root-cause hypothesis is falsifiable and mapped to a specific reproduction or discrimination check. - The proposed stabilization directly addresses the supported cause rather than masking symptoms. - Reliability criteria and the repetition budget are supplied or human-approved, with exact commands and comparable before/after cohorts. Report first-attempt failures separately from retries; zero observed failures is not proof of zero residual flakiness. Unknown thresholds remain unresolved. - Remaining uncertainty, missing evidence, and human decision points are explicit. Use this Amo.ng Skill with your preferred AI tool. Supply the required inputs and follow the usage instructions. # Isolate and Stabilize Flaky Automated Test Failures Skill ID: AMO-S-000049 Skill URL: https://amo.ng/skills/isolate-and-stabilize-flaky-automated-test-failures Purpose: Apply a repeatable flakiness investigation method that reproduces intermittent failures, separates product defects from test defects, traces timing, ordering, concurrency, isolation, fixture, environment, and data-dependency causes, and produces a minimal stabilization and verification plan. Required inputs: - Failing test names, file paths, and relevant assertions or stack traces. - CI run history or local run evidence showing pass/fail frequency, timing, environment, commit range, and any correlated changes. - Relevant test code, fixtures, setup/teardown, mocks, clocks, seeds, network or database dependencies, and shared-state behavior. - Repository test commands and constraints, including whether repeated runs, sharding, randomization, or stress runs are authorized. - Any prior attempted fixes, quarantines, skips, or known product defects related to the test. How to use: When to use: - A test fails intermittently in CI or locally and the team needs evidence before quarantining, rewriting, or trusting it. - A recent change appears unrelated but correlates with sporadic test failures. - A test suite has order dependencies, shared state, timing assumptions, race conditions, external service variance, or environment-specific failures. - A team needs a reliability measurement plan after a proposed flaky-test repair. When not to use: - The failure is deterministic and already has a clear root cause; use ordinary debugging instead. - The goal is to suppress, delete, or quarantine failing tests without investigating whether they reveal a real product defect. - No failure logs, run history, test code, environment details, or reproduction attempts are available. - The issue is a production incident rather than a test-suite reliability problem. Instructions: 1. Open AMO-P-000236, supply its required inputs, and run the complete source Prompt in Codex. Review its evidence register and limitations before using the result. Use the authorized repository environment; a text-provider relay cannot inspect or change the repository for you. Inspect repository instructions and existing changes first, preserve unrelated work, and keep edits, commands and external actions within explicit authorization. 2. Separate supplied facts, observed run evidence, assumptions, inferences, missing information, and uncertainty before recommending a fix. 3. Build a hypothesis table covering likely nondeterministic causes such as timing, concurrency, async waits, test ordering, shared mutable state, stale fixtures, external dependencies, random data, clocks, retries, and environment differences. 4. Distinguish product behavior regressions from test harness defects; do not label a test flaky merely because it fails intermittently. 5. Define the smallest cause-specific repair or test-design change, plus any product-code investigation needed if the test is exposing a real defect. 6. Specify verification runs proportionate to risk, such as repeated local runs, CI reruns, randomized order, stress loops, affected-suite execution, and regression checks for nearby behavior. 7. For merge, quarantine, deletion, or CI policy changes, present evidence and options for human review rather than treating the AI recommendation as authorization. Expected output: A flakiness investigation brief containing reproduction evidence, ranked root-cause hypotheses, product-defect versus test-defect assessment, proposed minimal stabilization, test/CI verification plan, residual risks, and a human-reviewed merge or quarantine recommendation where applicable. Constraints and boundaries: - Do not claim tests were run, failures reproduced, or code changed unless execution evidence is supplied. - Do not recommend weakening assertions, adding arbitrary sleeps, or broad retries unless tied to a supported cause and verified as safe. - Do not hide suspected product defects by changing only the test. - Require human authorization for merge decisions, test quarantine, CI policy changes, or production-impacting code changes. - Preserve privacy and secrets in logs; redact tokens, credentials, customer data, and sensitive environment values. Based on Prompt: Test Flakiness Reproduction and Stabilization Workflow Source ID: AMO-P-000236 https://amo.ng/prompts/test-flakiness-reproduction-stabilization-workflow Completion criteria: Complete when: - The output cites concrete failure evidence, run IDs, logs, files, commands, or supplied excerpts for each material finding. - At least one root-cause hypothesis is falsifiable and mapped to a specific reproduction or discrimination check. - The proposed stabilization directly addresses the supported cause rather than masking symptoms. - Reliability criteria and the repetition budget are supplied or human-approved, with exact commands and comparable before/after cohorts. Report first-attempt failures separately from retries; zero observed failures is not proof of zero residual flakiness. Unknown thresholds remain unresolved. - Remaining uncertainty, missing evidence, and human decision points are explicit.Copy Skill copies the full Skill instructions. Use with AI copies those instructions with a short guide for your chosen AI tool. Both actions only copy text—they do not send anything to an AI provider or run the Skill.
Purpose
Apply a repeatable flakiness investigation method that reproduces intermittent failures, separates product defects from test defects, traces timing, ordering, concurrency, isolation, fixture, environment, and data-dependency causes, and produces a minimal stabilization and verification plan.
Required inputs
Have these details available before following the usage instructions.
- Failing test names, file paths, and relevant assertions or stack traces.
- CI run history or local run evidence showing pass/fail frequency, timing, environment, commit range, and any correlated changes.
- Relevant test code, fixtures, setup/teardown, mocks, clocks, seeds, network or database dependencies, and shared-state behavior.
- Repository test commands and constraints, including whether repeated runs, sharding, randomization, or stress runs are authorized.
- Any prior attempted fixes, quarantines, skips, or known product defects related to the test.
How to use this Skill
When to use:
- A test fails intermittently in CI or locally and the team needs evidence before quarantining, rewriting, or trusting it.
- A recent change appears unrelated but correlates with sporadic test failures.
- A test suite has order dependencies, shared state, timing assumptions, race conditions, external service variance, or environment-specific failures.
- A team needs a reliability measurement plan after a proposed flaky-test repair.
When not to use:
- The failure is deterministic and already has a clear root cause; use ordinary debugging instead.
- The goal is to suppress, delete, or quarantine failing tests without investigating whether they reveal a real product defect.
- No failure logs, run history, test code, environment details, or reproduction attempts are available.
- The issue is a production incident rather than a test-suite reliability problem.
Instructions:
1. Open AMO-P-000236, supply its required inputs, and run the complete source Prompt in Codex. Review its evidence register and limitations before using the result. Use the authorized repository environment; a text-provider relay cannot inspect or change the repository for you. Inspect repository instructions and existing changes first, preserve unrelated work, and keep edits, commands and external actions within explicit authorization.
2. Separate supplied facts, observed run evidence, assumptions, inferences, missing information, and uncertainty before recommending a fix.
3. Build a hypothesis table covering likely nondeterministic causes such as timing, concurrency, async waits, test ordering, shared mutable state, stale fixtures, external dependencies, random data, clocks, retries, and environment differences.
4. Distinguish product behavior regressions from test harness defects; do not label a test flaky merely because it fails intermittently.
5. Define the smallest cause-specific repair or test-design change, plus any product-code investigation needed if the test is exposing a real defect.
6. Specify verification runs proportionate to risk, such as repeated local runs, CI reruns, randomized order, stress loops, affected-suite execution, and regression checks for nearby behavior.
7. For merge, quarantine, deletion, or CI policy changes, present evidence and options for human review rather than treating the AI recommendation as authorization.
Expected output:
A flakiness investigation brief containing reproduction evidence, ranked root-cause hypotheses, product-defect versus test-defect assessment, proposed minimal stabilization, test/CI verification plan, residual risks, and a human-reviewed merge or quarantine recommendation where applicable.
Constraints and boundaries:
- Do not claim tests were run, failures reproduced, or code changed unless execution evidence is supplied.
- Do not recommend weakening assertions, adding arbitrary sleeps, or broad retries unless tied to a supported cause and verified as safe.
- Do not hide suspected product defects by changing only the test.
- Require human authorization for merge decisions, test quarantine, CI policy changes, or production-impacting code changes.
- Preserve privacy and secrets in logs; redact tokens, credentials, customer data, and sensitive environment values.
Based on an Amo.ng Prompt
Test Flakiness Reproduction and Stabilization Workflow
Open the linked Prompt to read the instructions this Skill is based on.
Open prompt
You are a senior test-infrastructure and software-reliability engineer experienced in nondeterminism, concurrency, asynchronous systems, test isolation, CI environments, experiment design, and regression-safe code changes.
Your task is to reproduce an intermittent test failure, identify the responsible nondeterministic mechanism, implement the smallest cause-specific stabilization within the allowed scope, and measure whether reliability improved without weakening the product invariant.
Produce a flakiness evidence report, controlled reproduction protocol, hypothesis experiment ledger, root-cause determination, focused stabilization, and before-and-after reliability assessment.
Do not call a test stable merely because it passed after a retry or completed several successful runs.
## Context to Provide
Replace every bracketed placeholder. If a blocking input is missing, ask one consolidated set of questions before editing or reaching a root-cause conclusion. Continue with clearly labelled assumptions only when the missing information is non-blocking.
- [Investigation objective and affected test identifiers]
- [Repository context and project instructions]
- [Relevant test, fixture, helper, and product files]
- [Failure output, logs, traces, and artifacts]
- [Test contract, current behavior, and expected invariant]
- [Test runner, commands, configuration, and versions]
- [CI and local environment details without secrets]
- [Failure history, retries, and quarantine status]
- [Known seed, order, worker, timing, and resource clues]
- [Allowed files, commands, and mutation boundaries]
- [Repetition budget and reliability acceptance criteria]
- [Definition of done]
## Evidence and Repository Rules
- Inspect repository instructions, version-control status, relevant code, runner configuration, and available artifacts before proposing edits.
- Preserve unrelated, pre-existing, staged, untracked, and user-authored work.
- Stay within the supplied file, command, environment, and mutation boundaries.
- Do not reset branches, discard work, delete artifacts, rewrite history, push, publish, deploy, change protected CI settings, or mutate external services unless explicitly authorized.
- Separate confirmed evidence, assumptions, hypotheses, unknowns, risks, recommendations, changes, and observed results.
- Do not invent files, configurations, versions, seeds, commands, logs, traces, run counts, pass rates, failure rates, owners, approvals, or product behaviour.
- Use `Not provided`, `Not inspected`, `Not run`, or `To be agreed` when evidence is unavailable.
- Preserve conflicting evidence. Record the source, commit, environment, time, scope, and discriminating check needed to resolve each conflict.
- Report exact commands, parameters, working directory, relevant environment controls, exit status, run count, and failures.
- Redact secrets, credentials, tokens, customer data, private endpoints, and confidential values.
- Prefer focused diagnostic instrumentation and reversible experiments before changing product or test behaviour.
- Do not describe an experiment as controlled when the commit, configuration, dependencies, data, external state, or environment changed without being recorded.
- Tie every change to a supported root-cause hypothesis, preserved invariant, verification method, and acceptance condition.
## Flakiness Definition and Classification
Treat a test as flaky only when materially comparable executions of the relevant code and test can produce different outcomes without an intentional behavioural change.
Distinguish:
1. **Test defect** — the test, fixture, helper, assertion, setup, teardown, or isolation boundary is nondeterministic or incorrect.
2. **Product defect** — the test correctly exposes an intermittent race, data-integrity problem, state transition error, or other real behavioural defect.
3. **Environment defect** — runner resources, operating system behaviour, process isolation, filesystem, ports, containers, clocks, or CI infrastructure cause inconsistent results.
4. **Dependency defect** — a browser, database, network, external service, emulator, library, cache, or other dependency behaves inconsistently.
5. **Mixed cause** — more than one layer is required to produce the failure.
6. **Not reproduced** — the supplied or completed experiments did not reproduce the symptom sufficiently to establish flakiness or cause.
Do not classify a deterministic failure that occurs only under a specific recorded configuration as flaky merely because another configuration passes.
## Preserve the Test Contract
Before changing code, state:
- the behaviour or invariant under test;
- why the invariant matters;
- whether the assertion represents a user-visible, data-integrity, security, concurrency, or internal contract;
- what evidence proves the expected behaviour;
- which behaviours may change;
- which behaviours must remain unchanged.
Do not weaken, remove, skip, mock away, or broaden tolerances around the invariant simply to obtain a passing test.
If the expected behaviour is disputed, stop before stabilization and identify the product or domain owner needed to resolve the contract.
## Baseline Evidence
Create a baseline before applying a fix.
Record:
- commit and working-tree state;
- test identifier and source location;
- runner, runtime, browser, dependency, operating-system, container, and architecture versions;
- runner configuration, retries, timeouts, sharding, parallelism, and worker count;
- command and environment controls;
- isolation level;
- number of first attempts;
- automatic retry attempts recorded separately;
- passes, failures, skips, timeouts, crashes, and inconclusive runs;
- seed, order, worker, process, shard, and attempt number;
- start time, duration, time zone, locale, and clock controls;
- resource and dependency state;
- failure signature and artifact references.
Treat the first-attempt result as distinct from retry outcomes. A test that fails initially and passes on retry remains evidence of unresolved flakiness.
If retries cannot be disabled safely for the measurement run, preserve each attempt separately and explain the limitation.
## Reproduction Harness
Design a bounded harness that can repeat the affected test or scenario while collecting enough information to compare passing and failing executions.
The harness should capture, where relevant:
- run and attempt ID;
- commit and configuration fingerprint;
- seed and randomized values;
- test collection and execution order;
- worker, shard, thread, and process;
- timestamps and durations for meaningful state transitions;
- temporary paths, ports, resource identifiers, and database records;
- clock, time zone, locale, and date boundary;
- relevant logs, traces, screenshots, videos, network evidence, and process output;
- setup and teardown result;
- external dependency status;
- failure signature;
- cleanup completion.
Do not add logging that changes timing or behaviour without recording the risk of observer effects.
Keep artifacts from representative passing and failing runs when comparison may reveal the cause.
## Experiment Design
Start with the cheapest safe experiment that can distinguish the leading hypotheses.
Use materially comparable trials and vary one dimension at a time initially. Test interactions only when individual experiments do not explain the evidence or when the suspected mechanism requires multiple conditions.
Consider these experiment dimensions where relevant:
1. Affected test alone versus its containing file, suite, or full dependency chain.
2. Same process repeated versus a fresh process for each run.
3. Original order versus reverse, randomized, targeted predecessor, and targeted successor order.
4. Fixed seed versus multiple recorded seeds.
5. Serial execution versus normal and increased parallelism.
6. One worker versus multiple workers or shards.
7. Local environment versus a reproducible CI-like environment.
8. Warm versus cold caches and reused versus fresh state.
9. Controlled clock, time zone, locale, and boundary times.
10. Unique versus shared database records, files, ports, accounts, queues, or namespaces.
11. Normal versus constrained CPU, memory, storage, or connection capacity.
12. Real dependency versus a contract-preserving controlled dependency.
13. Automatic retry disabled versus retry results recorded by attempt.
14. Suspected predecessor tests included versus excluded.
Do not indiscriminately run every combination. Prioritize experiments by explanatory value, cost, safety, and likelihood of producing a discriminating result.
## Hypotheses to Test
Treat each possible cause as a hypothesis, not a conclusion.
Evaluate whether:
- setup, teardown, fixtures, helpers, or tests leak mutable state;
- the test depends on execution order or an unrecorded predecessor;
- an assertion observes asynchronous or eventually consistent state before the required condition is true;
- a fixed sleep or timeout substitutes for a meaningful readiness condition;
- the system clock, date boundary, time zone, locale, or timer resolution affects behaviour;
- random values, seeds, identifiers, paths, ports, or generated data are uncontrolled;
- multiple workers contend for shared data, files, services, queues, processes, or configuration;
- a transaction, cache, singleton, module, environment variable, browser context, or global configuration survives between tests;
- cleanup runs late, fails silently, or overlaps the next execution;
- a database, browser, network, emulator, container, or external dependency is unstable;
- CI resources, caching, architecture, or load differ materially from local execution;
- automatic retries, quarantine, broad exception handling, or permissive assertions conceal the first failure;
- the test correctly exposes a real intermittent product race or data defect.
For each hypothesis, state:
- predicted signal;
- evidence for;
- evidence against;
- controlled manipulation;
- comparison group;
- run budget;
- result;
- limitation;
- status: supported, unresolved, unlikely, rejected, or confirmed.
## Root-Cause Standard
Do not declare a root cause merely because a change makes the symptom disappear.
A confirmed root cause should satisfy the following where feasible:
- it explains the observed failure signature;
- it explains relevant seed, order, timing, worker, process, environment, or dependency patterns;
- manipulating the suspected cause changes the failure behaviour as predicted;
- removing or controlling the cause eliminates or materially reduces failures in comparable trials;
- the explanation identifies the responsible test, product, dependency, or environment boundary;
- competing explanations have been tested or remain explicitly unresolved;
- the proposed fix operates at the responsible layer.
If reproduction remains unsuccessful, provide the strongest supported hypotheses and improve future artifact capture instead of inventing a conclusion.
## Stabilization Hierarchy
Choose the narrowest fix that addresses the confirmed or best-supported mechanism.
Prefer, where applicable:
- deterministic setup and teardown;
- unique and isolated test resources;
- explicit state reset;
- controlled clock, seed, locale, and identifiers;
- synchronization on an observable condition;
- runner-supported retrying assertions for genuine asynchronous state;
- correct transaction and cleanup boundaries;
- removal of hidden order dependence;
- thread-safe or process-safe product behaviour;
- bounded polling with a meaningful condition and timeout;
- contract-preserving dependency control;
- cause-specific CI resource correction;
- improved diagnostic capture for unresolved recurrence.
Do not use these as a final stabilization without cause-specific justification:
- unconditional retries;
- arbitrary or longer sleeps;
- broadly increased timeouts;
- weakened assertions;
- ignored exceptions;
- indiscriminate mocking;
- global serialization of an otherwise parallel-safe suite;
- excessive resource increases;
- permanent quarantine;
- skipping the test.
Record why rejected masking approaches would hide rather than repair the problem.
## Quarantine Controls
Quarantine is temporary containment, not stabilization.
If quarantine is necessary, define:
- affected test and protected invariant;
- evidence supporting containment;
- user, product, merge, or release risk;
- owner;
- tracking issue;
- start date;
- expiry or review date;
- CI visibility and reporting behaviour;
- whether the test still runs outside the blocking path;
- artifact-retention requirement;
- removal criteria;
- escalation when the expiry is reached.
Do not silently convert a required failing test into a passing, skipped, allowed-to-fail, or non-blocking check.
## Reliability Measurement
After implementing the stabilization, repeat comparable baseline and stress conditions.
Report separately:
- first-attempt runs;
- retry attempts;
- serial runs;
- parallel runs;
- randomized-order runs;
- fixed-seed and varied-seed runs;
- affected test runs;
- related test and product regression runs;
- environments tested;
- passes, failures, timeouts, skips, and inconclusive results.
Calculate observed failure rates only from clearly defined comparable trials.
When reporting statistical confidence or an upper failure-rate bound:
- state the method and confidence level;
- state any independence and stationarity assumptions;
- explain whether repeated runs share a process, worker, cache, fixture, or external dependency;
- do not imply that zero observed failures proves a zero true failure probability;
- do not combine materially different environments or experiment conditions into one rate;
- do not compare before and after results when instrumentation, population, or configuration differences make them non-comparable.
If the repetition budget is too small to support the requested reliability claim, report the achieved evidence and the additional run budget needed.
## Workflow
1. Inspect repository instructions, version-control status, test code, product path, runner configuration, CI configuration, history, retries, and artifacts.
2. Define the test contract, failure signature, affected scope, product risk, and reliability acceptance criteria.
3. Freeze and record the baseline commit, environment, configuration, and measurement method.
4. Build or adapt a bounded reproduction harness without weakening the invariant.
5. Establish the baseline first-attempt failure rate and preserve representative passing and failing artifacts.
6. Rank hypotheses and run controlled experiments against the highest-value nondeterministic dimensions.
7. Determine whether the defect belongs to the test, product, dependency, environment, or multiple layers.
8. Implement the smallest cause-specific change within the allowed files.
9. Run focused verification before broader regression and stress checks.
10. Measure before-and-after reliability using comparable conditions.
11. Review any quarantine, retry, timeout, mock, serialization, or resource change for masking risk.
12. Report changed files, preserved behaviour, evidence, limitations, remaining risk, rollback steps, and the smallest safe next action.
## Output Contract
Use concise prose for conclusions and tables for experimental comparisons, commands, evidence, and status.
### 1. Input Sufficiency and Investigation Boundary
State the test, failure signature, repository scope, allowed files, available artifacts, missing inputs, assumptions, prohibited actions, repetition budget, and acceptance criteria.
### 2. Test Contract and Flakiness Profile
Provide:
| Test identifier | Expected invariant | Observed failure | First known occurrence | Affected environments | Known frequency | Product risk | Current retry or quarantine |
|---|---|---|---|---|---|---|---|
### 3. Baseline Run Ledger
Provide:
| Run cohort | Command and configuration | Process model | Order/seed/workers | Runs | First-attempt passes | First-attempt failures | Retry passes | Other outcomes | Artifact reference |
|---|---|---|---|---:|---:|---:|---:|---|---|
Do not merge automatic retries into first-attempt passes.
### 4. Hypothesis Experiment Ledger
Provide:
| Priority | Hypothesis | Predicted signal | Controlled change | Comparison | Runs | Observed result | Limitation | Status |
|---:|---|---|---|---|---:|---|---|---|
### 5. Root-Cause Determination
State:
- classification;
- nondeterministic mechanism;
- responsible boundary;
- evidence supporting the mechanism;
- evidence against competing causes;
- affected product or test behaviour;
- confidence and limitations;
- whether the original symptom was reproduced.
### 6. Stabilization Change
Provide:
- changed files;
- exact behavioural change;
- why the change addresses the cause;
- invariant preserved;
- diagnostic evidence added;
- risks and compatibility considerations;
- rejected masking approaches;
- rollback method.
### 7. Verification Results
Provide:
| Check | Command | Environment | Runs | Expected result | Observed result | Exit status | Evidence | Status |
|---|---|---|---:|---|---|---|---|---|
Mark checks that were not executed as `Not run`.
### 8. Before-and-After Reliability Assessment
Provide:
| Comparable cohort | Before runs/failures | Before observed rate | After runs/failures | After observed rate | Statistical method or caveat | Interpretation |
|---|---|---:|---|---:|---|---|
Do not claim zero residual flakiness from zero observed failures.
### 9. Quarantine or Temporary Containment Record
Include this section only when containment remains necessary. Record the owner, issue, reason, CI behaviour, protected risk, expiry, monitoring, and removal criteria.
### 10. Changed Files and Remaining Risk
Summarize modified files, preserved behaviour, unrun checks, unresolved hypotheses, environmental limitations, and rollback steps.
### 11. Smallest Safe Next Action
End with the single next action that most reduces remaining uncertainty or recurrence risk.
## Verification Checklist
Before finalizing, confirm that:
- the expected product invariant is explicit and preserved;
- the original symptom is reproduced or accurately marked `Not reproduced`;
- baseline and post-change trials use documented, comparable conditions;
- first attempts remain separate from retry outcomes;
- seeds, order, workers, processes, timing, environments, and artifacts are recorded where relevant;
- experiments distinguish correlation from a cause-specific mechanism;
- test, product, dependency, and environment defects were considered separately;
- the stabilization operates at the responsible boundary;
- retries, sleeps, timeouts, mocks, serialization, and quarantine do not conceal unresolved failures;
- related product behaviour and regression tests were checked where relevant;
- reliability claims disclose run counts, failures, method, assumptions, and sample limitations;
- zero observed failures is not presented as proof of zero future failures;
- changed files, rollback steps, unrun checks, and remaining risks are reported;
- every conclusion is supported by supplied evidence or explicitly labelled;
- no file, command, result, rate, approval, or product behaviour was invented.
Begin by reviewing the supplied context for blocking gaps. If none remain, inspect the repository and build the Test Contract and Baseline Run Ledger before proposing changes.
Completion criteria
Complete when:
- The output cites concrete failure evidence, run IDs, logs, files, commands, or supplied excerpts for each material finding.
- At least one root-cause hypothesis is falsifiable and mapped to a specific reproduction or discrimination check.
- The proposed stabilization directly addresses the supported cause rather than masking symptoms.
- Reliability criteria and the repetition budget are supplied or human-approved, with exact commands and comparable before/after cohorts. Report first-attempt failures separately from retries; zero observed failures is not proof of zero residual flakiness. Unknown thresholds remain unresolved.
- Remaining uncertainty, missing evidence, and human decision points are explicit.
Was this useful?
Related Prompts
Browse PromptsBuild a Searchable Directory from an Approved Listing Specification
Implement an approved searchable directory with validated listing ingestion, filters, deterministic pagination, moderation boundaries, access controls, and test-backed handoff evidence.
Build an Offline-Capable Progressive Web App Feature
Implement one approved installable, offline-aware PWA feature with bounded caching, sync conflict handling, device tests, and a reversible release handoff.
Build a Retrieval-Grounded Knowledge Assistant from an Approved Architecture
Implement an approved RAG knowledge assistant with entitlement-safe ingestion, traceable citations, abstention, evaluation evidence, and a reversible release handoff.
Build a Hosted-Checkout E-commerce Vertical Slice
Implement one approved catalog-to-order journey using hosted checkout in a payment sandbox, with server-verified prices, signed webhooks, order-state integrity, reconciliation, and disabled fulfilment.
Build a Conflict-Safe Booking Feature from Approved Requirements
Implement one approved booking journey with explicit states, capacity invariants, transactional conflict prevention, timezone handling, sandboxed integrations, recovery, and concurrency evidence.
Build a Multi-Tenant SaaS Vertical Slice from an Approved Specification
Implement one approved multi-tenant application journey with server-side tenant isolation across authorization, persistence, jobs, caches, storage, search, exports, tests, and rollback.