Amo.ng curated workflow
Verify an AI-Generated Software Change for Release
Reconcile a coding agent’s instructions and completion claims against the actual change set, verify dependency and API assertions, close test-evidence gaps, and prepare controlled release gates.
# Verify an AI-Generated Software Change for Release Workflow ID: AMO-W-000014 Workflow URL: https://amo.ng/workflows/verify-ai-generated-software-change-for-release ## Outcome A release-review package containing change attribution, unsupported completion claims, verified or unresolved dependency and API assertions, a risk-based test plan, deployment and rollback controls, and an authorized merge or release recommendation. ## Before you begin - Original task, repository instructions, scope, acceptance criteria, and protected behavior - Coding-agent transcript, completion notes, diff or commit range, and changed files - Dependency manifests, lockfiles, API or framework claims, and relevant source or documentation - Existing tests, CI output, runtime evidence, deployment topology, observability, and rollback controls - Authorized inspection, edit, test, merge, and release boundaries ## Step 1 — Attribute changes and test completion claims **Prompt** Coding-Agent Change Attribution and Test Evidence Review **Instructions** Inspect the task, instruction hierarchy, transcript, diff, files, and available test evidence. Classify material changes, identify uninstructed or unexplained behavior, and decide which completion claims are supported. **Input for this step** Supply repository instructions, original task, agent transcript, change set, changed files, protected behavior, acceptance criteria, and actual test or command output. **Carry forward** Carry the change-attribution ledger, supported and unsupported claims, unexplained changes, suspected defects, test evidence gaps, and protected behavior into technical-claim verification. **Review note** The engineering reviewer confirms the review scope and resolves any unexplained, unauthorized, security-sensitive, or data-affecting change before release planning. **Prompt ID** AMO-P-000276 **Prompt URL** https://amo.ng/prompts/coding-agent-change-attribution-and-test-evidence-review **Prompt content** Review the coding-agent change set and run evidence for provenance, instruction compliance, and support for completion claims. Focus on what can be observed from the supplied repository, diff, transcript, and test evidence. Do not perform a generic PR review unless it is necessary to attribute a material change or assess whether a completion claim is supported. Context to provide: - Repository scope and task: [Repository scope and task] - Relevant files and instructions: [Relevant files and instructions] - Observed evidence: [Observed evidence] - Constraints and authorized changes: [Constraints and authorized changes] - Environment details without secrets: [Environment details without secrets] - Verification commands and acceptance criteria: [Verification commands and acceptance criteria] Review rules: 1. Inspect the relevant files, diffs, transcript, and test evidence first before drawing conclusions. 2. Distinguish observation from inference. Mark unsupported assumptions as missing information. 3. Do not claim that a file, command, test, approval, system, or external source was inspected or completed unless there is evidence in the provided materials or you actually inspected or ran it in the available environment. 4. Attribute each material change to one of these categories: - Directly instructed - Reasonably necessary to satisfy the instruction - Incidental but explainable - Unexplained drift - Potentially harmful or out of scope 5. Treat a change as material if it affects behavior, public API, data model, security posture, dependency surface, build/test configuration, generated artifacts, migrations, operational behavior, or protected behavior listed by the release owner. 6. Preserve existing behavior as the default expectation. Flag behavior changes that are not explicitly instructed or clearly necessary. 7. Avoid broad rewrites and style-only judgments unless they obscure attribution, create risk, or conflict with project conventions. 8. If you identify a defect and decide to edit code, first identify the likely root cause, then apply the smallest safe change that preserves existing behavior. Do not perform broad rewrites. Run syntax checks and tests where available, then summarize files changed and verification results. If editing is not requested or not safe, provide proposed changes only. 9. Run or recommend syntax checks and tests where available and proportionate. If you cannot run them, state exactly what evidence is missing and what the test owner or release owner should run. 10. Do not recommend merge solely because the agent said the work was complete. Completion claims must be reconciled against observable diff and test evidence. Deliverable: ## 1. Review Scope and Evidence Used List the materials actually inspected: - Instructions reviewed - Transcript or completion notes reviewed - Files or diffs reviewed - Tests, logs, or command outputs reviewed - Repository context used - Evidence not provided or not inspectable ## 2. Agent Completion Claims Create a table with these columns: - Claim made by agent - Evidence offered by agent - Evidence independently visible in supplied materials or environment - Supported, partially supported, unsupported, or contradicted - Notes for release owner or test owner ## 3. Instruction-to-Diff Trace Map the original instruction to the observed changes. Use this table: - Instruction requirement - Related files or hunks - How the change satisfies the requirement - Evidence level: direct, inferred, weak, or missing - Compliance assessment Call out any instruction requirement that appears unimplemented, only partially implemented, or implemented through an unexpected approach. ## 4. Change Attribution Ledger Create a ledger for each material changed area. Use this table: - File or component - Material change observed - Attribution category - Evidence supporting attribution - Behavior or interface impact - Risk level: low, medium, high - Owner who should verify: release owner, test owner, security reviewer, data owner, product owner, or other specific accountable role ## 5. Unexplained-Change Register List all changes that cannot be clearly tied to the instruction or necessary implementation path. For each item include: - File or hunk - What changed - Why attribution is unclear - Potential consequence - What evidence would resolve it - Recommended handling: accept with owner verification, revert, isolate into separate change, or investigate before merge ## 6. Test Evidence Reconciliation Assess whether the supplied tests and logs support the completion claims. Include: - Commands claimed to have run - Commands evidenced by logs or terminal output - Pass/fail status shown by evidence - Coverage relevance to the changed behavior - Gaps, skipped tests, stale outputs, or ambiguous timestamps - Additional checks the test owner should run before merge If you run any commands, list: - Command - Purpose - Result - Relevant output summary - Any limitations If no commands were run, say so explicitly. ## 7. Risk and Protected Behavior Review Evaluate only the risks that arise from attribution, instruction compliance, or evidence gaps. Cover applicable areas: - Behavior drift - Public API or contract change - Data migration or persistence risk - Authentication, authorization, or secrets handling - Dependency or build-system changes - Generated files or lockfiles - Test configuration changes - Operational or deployment implications ## 8. Merge Recommendation Choose one recommendation: - Merge: evidence supports the agent’s claims and material changes are attributable - Merge after owner verification: residual gaps are narrow and assigned to accountable owners - Do not merge yet: unsupported claims, unexplained drift, insufficient tests, or high-risk uncertainty remain - Rework required: changes are out of scope, unsafe, or not traceable to the instruction Include: - Recommendation - Primary reasons - Required pre-merge checks - Owners responsible for verification - Conditions that would change the recommendation ## 9. Completion Check Before finalizing, confirm: - Relevant files and evidence were inspected first - Material changes were attributed - Intended edits were separated from incidental or unexplained drift - Agent claims were reconciled with test evidence - Missing evidence and uncertainty were explicitly stated - Any commands run are reported with results, or lack of execution is stated - Any files changed by you are summarized, with syntax/test verification results where available ## Step 2 — Verify dependency and API claims when applicable **Prompt** AI-Generated Dependency and API Claim Verification **Instructions** Run this step when the change adds, removes, upgrades, or relies on packages, APIs, framework behavior, commands, configuration, or version-specific features. Otherwise record Not applicable. Classify each assertion as verified, contradicted, unresolved, or unsupported. **Input for this step** Provide the attribution findings, manifests and lockfiles, imports and calls, runtime and framework versions, local source or stubs, official documentation where available, and the agent’s technical claims. **Carry forward** Carry the technical-claim ledger, incompatible or hallucinated dependencies, unsupported APIs, correction options, and verification commands into the test plan. **Review note** The code owner or maintainer approves dependency corrections and any contract, compatibility, security, or licensing consequence. **Prompt ID** AMO-P-000277 **Prompt URL** https://amo.ng/prompts/ai-generated-dependency-and-api-claim-verification **Prompt content** Verify dependency, package, framework, and API claims introduced by AI-generated code before merge. Context to provide: - Repository scope and task: [Repository scope and task] - Relevant files and instructions: [Relevant files and instructions] - Observed evidence: [Observed evidence] - Constraints and authorized changes: [Constraints and authorized changes] - Environment details without secrets: [Environment details without secrets] - Verification commands and acceptance criteria: [Verification commands and acceptance criteria] Objective: Determine whether generated code relies on packages, versions, imports, methods, configuration keys, framework behavior, runtime features, or API signatures that are unsupported by the repository’s installed dependency set or by authoritative documentation. Produce a defensible review artifact for the release owner or package maintainer before merge. Scope boundaries: - Focus only on dependency, package, framework, runtime, type, and API claims introduced or materially affected by the generated code. - Do not perform a broad dependency upgrade, architectural rewrite, style review, or generic code review. - Do not assume an API exists because it appears plausible. - Do not claim documentation, commands, tests, approvals, or files were inspected unless you actually inspected them. - Separate direct observations from inferences and mark anything unverified. - If authoritative documentation is unavailable, use installed package source, generated types, local docs, lockfiles, manifests, and compiler or test output where available; otherwise classify the claim as Unresolved and state what evidence is needed. - Use exactly these claim outcomes: - Verified: admissible evidence confirms the claim for the installed or target version. - Contradicted: admissible evidence shows the claim is false or incompatible. - Unresolved: relevant evidence is missing, inaccessible, incomplete, or conflicting, so no conclusion is supportable yet. - Unsupported: no admissible evidence supports the assertion after the available repository and authoritative sources are checked. Do not collapse Unresolved and Unsupported into a generic warning. Preserve uncertainty and identify the evidence needed to resolve each open claim. Missing-input gate: - Treat the generated change, applicable repository instructions, dependency manifests or lockfiles, target runtime or package version, and relevant API evidence as blocking when their absence or conflict prevents a claim from being scoped. Request all blocking items in one consolidated clarification and leave the affected claim Unresolved until they are supplied. - Continue with non-blocking gaps only when each is recorded as Unknown or Unresolved, with the evidence needed and the consequence for merge or release confidence. Required process: 1. Inspect relevant files first. - Review the generated diff or branch. - Inspect dependency manifests, lockfiles, package manager configuration, runtime configuration, framework configuration, relevant imports, generated or installed type definitions, and nearby usage patterns. - Identify every dependency or API claim introduced by the generated code before deciding whether edits are needed. 2. Build a claim inventory. Include claims such as: - Package or framework is available. - A package version supports a named API, export, method, hook, decorator, CLI option, configuration key, schema field, or runtime behavior. - Import paths, module formats, peer dependencies, plugins, adapters, or provider names are valid. - Type signatures, return values, error shapes, async behavior, or environment requirements match the generated code. 3. Verify each claim against evidence. Use the strongest available evidence in this order where practical: - Repository manifests and lockfiles. - Installed package source or type definitions. - Existing repository usage and tests. - Package manager, compiler, typechecker, linter, or framework diagnostics. - Authoritative documentation or release notes supplied or accessible in the environment. 4. Identify root cause before editing. For each Contradicted, Unresolved, or Unsupported claim, determine whether the issue is caused by hallucinated API usage, wrong package name, incompatible installed version, missing peer dependency, incorrect import path, runtime mismatch, stale documentation, incomplete local install, or insufficient evidence. 5. Correction policy. - Prefer no code edits unless a minimal correction is clearly justified by evidence. - If editing is necessary and within scope, apply the smallest safe change. - Preserve existing behavior and public interfaces unless the merge or release context explicitly authorizes a change. - Avoid broad rewrites, opportunistic refactors, dependency upgrades, and speculative migrations. - If the safest fix requires a dependency upgrade or product decision, do not perform it silently; document the decision required from the release owner, package maintainer, or security reviewer. 6. Verification. - Run syntax checks, type checks, targeted tests, or package manager inspection commands where available and appropriate. - If a command cannot be run, state why and list the verification gap. - Mention verification results exactly: command, outcome, and relevant error excerpt or confirmation. Required deliverable: A. Claim inventory Provide a table with: - Claim ID - Generated-code location - Claim being made - Dependency, framework, runtime, or API involved - Why the claim matters for merge safety B. Version-and-source verification matrix Provide a table with: - Claim ID - Installed or resolved version observed - Evidence source inspected, with file path, lockfile entry, type definition, package source, command output, or documentation reference - Claim status: Verified, Contradicted, Unresolved, Unsupported, or Not applicable - Notes distinguishing observation from inference C. Contradicted, unresolved, or unsupported claims For each Contradicted, Unresolved, or Unsupported claim, include: - Finding title - Severity for merge: Blocker, High, Medium, or Low - Direct evidence - Root cause - Expected failure mode - Confidence level - Missing evidence, if any D. Minimal correction plan For each finding, provide: - Smallest safe correction - Whether code change, dependency decision, documentation check, or owner decision is needed - Files likely affected - Behavior expected to remain unchanged - Risk of the correction E. Changes made, if any - List every file changed. - Summarize the exact purpose of each change. - If no files were changed, state: No files changed during this verification pass. F. Reproducible verification record Include: - Files inspected - Commands run, if any - Tests or checks run, if any - Results observed - Checks not run and why - Open questions for the accountable owner named in [Merge or release context and accountable owner] Completion criteria: - Every generated dependency or API claim in scope is inventoried. - Each claim is classified as Verified, Contradicted, Unresolved, Unsupported, or Not applicable using the stated evidence rules. - Every Unresolved or Unsupported claim identifies the missing evidence, responsible evidence source or owner where known, and the consequence of proceeding without resolution. - Unsupported claims have root cause and expected failure mode. - No broad upgrades or rewrites are proposed as the default fix. - Verification record is sufficient for the release owner, package maintainer, or security reviewer to reproduce or challenge the conclusion. ## Step 3 — Build the risk-based verification plan **Prompt** Production Test and Verification Plan Prompt **Instructions** Translate the change and claim findings into proportional automated tests, manual checks, CI gates, environment checks, observability expectations, rollback criteria, and an evidence ledger. **Input for this step** Supply the reviewed change set, defects and unresolved claims, acceptance criteria, relevant test suites and CI configuration, runtime context, data or migration risks, and available execution evidence. **Carry forward** Carry the test matrix, actual results, unrun checks, coverage gaps, required fixes, monitoring signals, and release confidence into deployment-safety review. **Review note** The test owner and engineering reviewer confirm that material risks have an observable check and that missing execution evidence remains blocking where appropriate. **Prompt ID** AMO-P-000010 **Prompt URL** https://amo.ng/prompts/test-and-verification-prompt **Prompt content** Build a risk-based test and verification plan for the following software change or release. Inputs - Change or release under test: [Change or release under test] - Repository and relevant files: [Repository and relevant files] - System and runtime context: [System and runtime context] - Acceptance criteria: [Acceptance criteria] - Test and deployment constraints: [Test and deployment constraints] - Available evidence: [Available evidence] - Authorized actions and environment: [Authorized actions and environment] - CI/CD and rollback context: [CI/CD and rollback context] Codex operating boundaries - Use Codex to inspect supplied repository content, diffs, configuration, test suites, CI definitions, logs, and command output that are actually available in the session. - Run tests or read additional files only when the environment provides that capability and the authorized-actions input permits it. Prefer targeted, read-only inspection before expensive or state-changing commands. - Do not deploy, merge, approve a release, alter production, access undeclared systems, expose secrets, create real customer data, disable safeguards, or run destructive commands. Treat migrations, load tests, security probes, external API calls, and commands that write or delete data as approval-gated. - Stop before an action if its target, blast radius, data handling, cost, reversibility, or authorization is unclear. Record the blocked action, required approval, and a safe alternative. - Never imply that a command ran merely because it was proposed. Never claim that code is fixed, tests passed, coverage improved, a release was approved, a rollback works, or a deployment completed without corresponding execution evidence. Input and evidence rules 1. Treat the change target, acceptance criteria, repository or equivalent technical artifacts, runtime context, and authority scope as prerequisites for an execution-backed assessment. If one is missing, ask only the questions necessary to unblock it. 2. If execution is blocked but supplied artifacts are sufficient, produce a bounded plan and mark execution-dependent conclusions unverified. If the change boundary or acceptance criteria cannot be established, do not issue a release-confidence recommendation. 3. Maintain an evidence ledger that distinguishes supplied facts, direct Codex observations, command execution evidence, assumptions, hypotheses, conflicts, and unknowns. Cite file paths, symbols, diff locations, log excerpts, CI job names, test identifiers, commands, exit codes, or artifact locations where available. 4. Do not resolve conflicting documentation, code behavior, logs, or requirements by guessing. Describe the conflict, its verification impact, and who must resolve it. 5. Do not infer test success from the existence of test files, infer production behavior solely from mocks, or equate code coverage with behavioral correctness. Assessment workflow 1. Establish scope and baseline - Identify changed components, interfaces, dependencies, data stores, feature flags, configuration, infrastructure, schemas, jobs, and user journeys. - Determine the comparison baseline and whether generated files, lockfiles, migrations, API contracts, or deployment manifests changed. - Record exclusions and distinguish intentional scope limits from unavailable evidence. 2. Perform change-impact and risk analysis - Trace affected call paths, consumers, upstream and downstream integrations, shared libraries, background work, cache behavior, concurrency boundaries, and compatibility requirements. - Rate each material risk by likelihood and impact. Include regression, data integrity, authorization, privacy, availability, performance, observability, backward compatibility, migration, retry or idempotency, and rollback risks when relevant. - Prioritize tests by risk reduction rather than test count. 3. Build acceptance traceability - Decompose each acceptance criterion into observable behavior. - Map it to one or more unit, component, integration, contract, end-to-end, migration, security, performance, resilience, or manual checks as appropriate. - Define setup, fixtures or test data, action, expected result, required evidence, cleanup, and ownership for every check. - Include negative paths and boundaries such as empty, null, malformed, duplicate, maximum-size, timeout, partial-failure, retry, race, permission-denied, stale-cache, and dependency-unavailable conditions where applicable. 4. Evaluate existing verification assets - Identify relevant tests and assess whether their assertions prove the required behavior rather than merely execute code. - Detect missing assertions, over-mocking, nondeterministic time or randomness, shared-state leakage, order dependence, brittle snapshots, unsafe fixtures, hidden network access, and flaky retries. - Review CI triggers, path filters, matrices, service dependencies, caches, artifacts, timeouts, required checks, branch protections, and failure propagation for gaps that could produce false confidence. 5. Specify the verification sequence - Order checks from fast and isolated to broad and operational: static checks, targeted unit tests, component or integration tests, contracts, migrations, end-to-end paths, non-functional checks, and manual exploration. - Provide exact commands only when supported by repository evidence. Otherwise label commands as proposed and identify what must be confirmed. - Separate blocking release gates from advisory checks. Define retry policy, flaky-test handling, artifact retention, test-data cleanup, and ownership of failures. 6. Execute only authorized checks - Before each command, state its purpose, environment, expected side effects, and why it is within authority. - Capture the exact command, working directory, relevant environment details with secrets redacted, start and finish state, exit code, actual observation, and artifact reference. - Do not silently rewrite code or tests to make checks pass. If modification is expressly authorized, present the proposed patch and its rationale separately, then verify it with fresh evidence. - Classify each check as passed, failed, blocked, not run, or inconclusive. A zero exit code is not sufficient when assertions, logs, skipped-test counts, or produced artifacts contradict success. 7. Assess deployment and recovery readiness - Verify pre-deployment prerequisites, configuration compatibility, secret references without revealing values, migration ordering, backward and forward compatibility, feature-flag behavior, health checks, and capacity assumptions where relevant. - Define post-deployment smoke tests and observability signals with query or dashboard source, baseline, threshold, observation window, and owner. Cover errors, latency, saturation, queue lag, data reconciliation, and key business behavior as applicable. - Specify rollback or roll-forward triggers, decision owner, procedure reference, data consequences, compatibility limits, recovery verification, and cases where rollback is unsafe, such as irreversible schema or data transformations. 8. Reconcile evidence and determine confidence - Reconcile every acceptance criterion, risk, test result, defect, skipped check, and conflicting observation. - Recommend exactly one state: Ready, Conditionally ready, Not ready, or Unassessed. This is a technical recommendation, not release approval. - Ready requires all blocking criteria to have passing evidence and no unresolved release-blocking defect or unknown. Conditionally ready requires explicit conditions, owners, and deadlines. Not ready requires named blockers. Unassessed applies when evidence is insufficient to support a conclusion. Required deliverable A. Scope and evidence ledger - Change boundary, baseline, affected systems, exclusions, authority scope, and environment. - Evidence table with ID, classification, source or artifact, observation, reliability limitation, and related conclusion. - Assumptions, unknowns, and conflicts, each with impact and resolution owner. B. Change-impact and risk register - Component or behavior, change mechanism, dependent systems, failure mode, likelihood, impact, detectability, risk priority, proposed control, and residual risk. C. Acceptance-to-test matrix - Criterion ID, observable behavior, risk covered, test level, setup and data, procedure or command, expected observation, required evidence, cleanup, owner, priority, and status. D. Existing test and CI assessment - Relevant test or job, what it proves, identified gap, flakiness or isolation concern, CI gate status, and recommended correction. E. Execution record - Check ID, proposed or executed state, exact command or manual procedure, environment, expected observation, actual observation, exit code when applicable, duration when known, evidence reference, and result classification. F. Defect and unresolved-work register - Defect or gap ID, reproduction evidence, affected criterion, severity, release impact, workaround, owner, and retest requirement. Keep proposed fixes separate from applied changes. G. Deployment, observability, and recovery checks - Pre-deployment gates, smoke tests, monitored signals, baselines and thresholds, observation windows, rollback or roll-forward triggers, recovery procedure references, data reconciliation, and responsible approvers. H. Verification verdict - Recommended state, evidence-backed rationale, passed blocking gates, failed or missing gates, residual risks, approval still required, and the smallest safe next action. Use concise technical language. Preserve unresolved states and make every consequential conclusion traceable to evidence. ## Step 4 — Prepare deployment, rollback, and release gates **Prompt** CI/CD Deployment Safety Checklist Generator **Instructions** Review the supplied pipeline, release, migration, observability, and recovery evidence. Produce the smallest release-specific checklist with stop conditions, rollback triggers, and post-release verification. **Input for this step** Provide the verified change package, CI evidence, deployment scripts or workflow, environment topology, configuration and stateful changes, monitoring, backup or rollback capability, and release constraints. **Carry forward** Produce the final merge and release recommendation, changed-file and evidence ledger, blocking gaps, approved conditions, deployment checklist, rollback plan, post-release checks, and unresolved risks. **Review note** The code owner controls merge authority; the release owner makes the deploy, delay, or rollback decision after reviewing the evidence package. **Prompt ID** AMO-P-000116 **Prompt URL** https://amo.ng/prompts/ci-cd-deployment-safety-checklist-generator **Prompt content** Review the supplied release materials and produce an evidence-traceable CI/CD deployment safety assessment. Use Codex to inspect the repository and only files, text, command output, and repository context that are actually supplied or available in the current session. Do not imply access to a repository, CI provider, cloud account, secrets store, database, monitoring system, or production environment unless that access is demonstrably available. Inputs Repository and release scope: [Repository and release scope] Pipeline and deployment artifacts: [Pipeline and deployment artifacts] Platform and environment topology: [Platform and environment topology] Migration and stateful workload details: [Migration and stateful workload details] Verification and observability evidence: [Verification and observability evidence] Rollback and governance requirements: [Rollback and governance requirements] Input expectations The repository and release scope should identify the change set, affected services, release reference, critical user flows, external dependencies, and known high-risk changes such as billing, authentication, authorization, data deletion, or infrastructure changes. Pipeline and deployment artifacts should include relevant workflow files, reusable workflows, deployment scripts, manifests, infrastructure definitions, build configuration, test commands, and release instructions. Platform and environment topology should describe environments, promotion flow, deployment strategy, runtime components, regions, traffic routing, queues, caches, scheduled jobs, and secret or identity mechanisms without exposing secret values. Migration and stateful workload details should cover schema and data migrations, compatibility assumptions, expected duration, locking risk, backups, restoration, and interactions with workers or older application versions. Verification and observability evidence should provide health checks, smoke tests, dashboards, alerts, logs, service-level indicators, prior command output, and acceptance thresholds. Rollback and governance requirements should identify rollback or roll-forward procedures, approval owners, change windows, incident contacts, communication requirements, and the release definition of done. Input and evidence rules 1. Create an input ledger before drawing conclusions. Classify each needed item as supplied, observed in an accessible artifact, conflicting, missing, or not applicable. Cite file paths and line ranges when available; otherwise cite the supplied input section or evidence item. 2. Never invent workflow behavior, provider settings, branch protection, environment rules, test outcomes, secret values, migration reversibility, backup validity, monitoring coverage, approvals, or production state. 3. If inputs conflict, record both claims, identify their sources, explain the safety consequence, and request the authoritative source. Do not silently choose one. 4. If a critical fact is missing, mark the affected conclusion unverified and make the release disposition Blocked when safe deployment depends on that fact. Noncritical gaps may receive a clearly labeled conservative hypothesis, but a hypothesis is not evidence. 5. Treat documentation as evidence of an intended process, not proof that a control ran. Treat configuration as evidence of a configured control, not proof of successful execution. Treat logs, CI run records, signed approvals, artifact metadata, command output, or monitoring observations as execution evidence only when their source and release relevance are supplied. 6. Use these work-state labels consistently: Requested for work the user asked for; Proposed for changes or commands not applied; Executed only for an action actually performed in the current session; Unavailable when access or capability is absent; Unverified when evidence is insufficient. Every claim that something was tested, fixed, deployed, rolled back, approved, or verified must include execution evidence. Otherwise label it Proposed or Unverified. 7. Bind every material piece of evidence to the exact release under review. A passing test, approval, artifact, log entry, monitoring observation, or prior deployment from another commit, branch, artifact digest, environment, configuration state, or execution window is not evidence for this release unless a traceable relationship is supplied. Record the commit, release reference, artifact identity, target environment, and evidence timestamp where available. Authority and safeguards Unless [Rollback and governance requirements] expressly restrict access, permit read-only repository inspection and non-mutating diagnostics within the workspace actually available to Codex. Treat file edits, mutating commands, pipeline or configuration changes, database writes or migrations, secret rotation, infrastructure changes, deployment, rollback, production access, and external side effects as unauthorized unless expressly approved. Do not deploy, merge, approve, rotate secrets, alter infrastructure, run migrations, modify production data, disable controls, or trigger rollback. If a read-only check against a production target is expressly authorized and Codex has demonstrable access, limit it to a clearly non-mutating command against the stated target. Record the exact command, target, exit status, relevant output, time, and limitations. Never run destructive, state-changing, costly, financially consequential, or irreversibly production-affecting commands within this prompt. Otherwise provide commands as Proposed and do not fabricate output. Do not reproduce secret values, tokens, credentials, private keys, customer data, or sensitive log content. Refer to secret names or redacted identifiers only. Flag excessive permissions, untrusted code paths with secret access, unsafe pull-request triggers, command injection surfaces, unpinned third-party actions, mutable artifacts, and credential persistence. Human approval remains mandatory for production release decisions and for changes involving billing, identity, permissions, security controls, destructive data operations, non-backward-compatible migrations, or infrastructure replacement. Focused review workflow 1. Trace the failure modes and map the delivery path from source trigger to production: event and branch or tag filters, pull-request trust boundary, build, tests, artifact creation, provenance or digest handling, promotion, environment selection, deployment, verification, and rollback. Identify reusable workflows and dependencies that can alter this path. 2. Inspect trigger and concurrency safety. Check accidental production triggers, skipped required jobs, path-filter blind spots, duplicate deployments, cancellation behavior, race conditions, environment locks, release serialization, and whether the deployed commit or artifact is uniquely identified. 3. Inspect identity, permissions, and supply-chain controls. Check least-privilege workflow permissions, OIDC or credential scope where evidenced, secret availability by event and environment, masking and log exposure, dependency or action pinning, artifact integrity, provenance, retention, and separation between build and deploy authority. 4. Inspect build and test gates. Trace dependency installation, lockfile enforcement, deterministic builds, static checks, unit and integration tests, security checks where required, failure propagation, retry behavior, test exclusions, coverage of critical flows, and whether the exact promoted artifact passed the cited checks. 5. Inspect environment and deployment correctness. Check staging-to-production parity, configuration validation, immutable artifact promotion, deployment strategy, traffic shifting, readiness versus liveness semantics, timeout behavior, partial failure across services or regions, infrastructure ordering, external API dependencies, maintenance requirements, and idempotency of repeated deployment attempts. 6. Inspect migration and stateful-component safety. Evaluate expand-and-contract compatibility, application and migration order, mixed-version operation, transaction and lock behavior, table rewrites, long-running backfills, retry and resume behavior, data validation, queue payload compatibility, worker draining, cron overlap, cache-key or serialization changes, backup freshness, restore evidence, and whether rollback would leave code and schema compatible. Treat an unproven destructive or irreversible migration as a blocking risk. 7. Inspect observability and release control. Check that health endpoints test meaningful dependencies without leaking data; smoke tests cover critical user journeys; dashboards and alerts identify error rate, latency, saturation, queue lag, failed jobs, database health, and business-critical signals; thresholds, observation windows, owners, and escalation paths are defined. 8. Build rollback and roll-forward logic. Define measurable triggers, decision owner, last known good artifact, code and configuration restoration, schema mitigation, traffic restoration, queue and cache handling, external side-effect reconciliation, user communication, and post-recovery verification. Do not call rollback viable without evidence that required artifacts, procedures, permissions, and schema compatibility exist. 9. Prioritize findings using impact and likelihood rated Low, Medium, High, or Critical. Distinguish release blockers from required follow-ups and optional hardening. Prefer the smallest control that materially reduces the identified risk; do not recommend broad platform rewrites without evidence that they are necessary. Base impact and likelihood on release-specific evidence. Do not infer likelihood solely from generic industry experience or the theoretical existence of a failure mode. When the available evidence cannot support a defensible likelihood rating, mark likelihood Unverified, explain the uncertainty, and state what evidence is needed. Output contract: required CI/CD safety deliverable Produce the following task-specific sections in markdown. A. Review basis and evidence ledger Provide a table with Evidence ID, item or artifact, source locator, relevance to this release, evidence class, and status. Evidence class must distinguish intended process, static configuration, and execution evidence. Follow it with missing and conflicting inputs, their consequences, and the exact evidence needed to resolve each one. B. Delivery-path map Describe the evidenced path from trigger to production in order. For every stage list trigger or input, responsible workflow or script, output artifact or state transition, environment, controlling gate, and evidence ID. Mark inferred or unknown transitions explicitly. C. Risk register Provide Finding ID, delivery stage, failure mode, supporting evidence IDs, impact, likelihood, severity, affected environment or service, release consequence, required mitigation, owner or approver if supplied, and state. Include concrete findings for triggers, permissions, secrets, artifact integrity, tests, environment drift, deployment ordering, migrations, stateful workers, health checks, monitoring, and rollback when relevant. Do not create findings unsupported by the supplied architecture; record missing evidence instead. D. Release gate checklist Create ordered Pre-deployment, Deployment, and Post-deployment gates. Each checklist row must contain Gate ID, check, reason, execution target, method or proposed command, expected observation, supplied actual observation, evidence ID, pass criterion, stop or pause condition, responsible human, and state. Leave actual observation as Not supplied unless real output exists. Commands must identify assumptions and must not expose secrets or mutate production. Include, where applicable, confirmation of the exact commit and immutable artifact; required CI results; configuration-key presence without values; environment and identity target; backup and restoration evidence; backward-compatible migration sequence; worker, queue, cache, and scheduler coordination; approval and communication gates; deployment progress; health and readiness; critical API and user-flow smoke tests; error, latency, saturation, queue, database, and business-signal thresholds; and an observation window. E. Migration and stateful-workload decision record State the proposed sequence for application versions, schema changes, backfills, workers, queues, caches, and scheduled jobs. Document compatibility across old code, new code, old schema, and new schema; lock and duration concerns; abort criteria; backup or restoration prerequisites; data-integrity reconciliation; and rollback versus roll-forward constraints. For each conclusion cite evidence or mark it Unverified. F. Rollback readiness record Provide rollback trigger, decision owner, code or artifact action, configuration action, database mitigation, traffic action, queue and cache handling, external side-effect reconciliation, communications, verification check, expected observation, and evidence. Identify the point after which rollback becomes unsafe and a roll-forward is required. Mark readiness Unverified if no tested procedure or equivalent execution evidence is supplied. G. Verification plan and evidence requirements For each proposed verification, give the exact non-destructive command or manual action, target environment, prerequisite, expected observation, acceptance threshold, failure interpretation, evidence to retain, and current work state. Reconcile the deployed release identity with the reviewed commit and artifact digest. Reconcile migration version and data checks with the expected release state. Reconcile health and smoke-test results with monitoring over the stated observation window. Never populate actual results unless they were supplied or executed with recorded evidence. H. Release disposition Choose exactly one disposition: Blocked, Conditional candidate for human approval, or Ready for human approval. This is advice, not approval or authorization to deploy. List the decisive evidence, unresolved blockers, conditions that must be satisfied, required human gates, monitoring obligations, and safest next action. A disposition of Ready for human approval requires traceable evidence that required tests passed for the reviewed release artifact, the deployment target is identified, migration and configuration prerequisites are satisfied, meaningful health and smoke checks have acceptance thresholds, observability and escalation are active, and rollback or roll-forward is operationally credible. If any required evidence is missing, use Blocked or Conditional candidate for human approval. Keep every section concise and proportional to the release’s actual scope and risk. Do not repeat the same evidence across multiple sections unnecessarily. Where a section or control area is genuinely not applicable, retain the heading, state Not applicable, and explain briefly why using the supplied release evidence. Never omit the evidence ledger, risk register, release gates, release disposition, or completion-integrity distinctions. Final integrity check Before returning the deliverable, confirm that every material conclusion cites evidence or is marked Unverified; every proposed command has a target and expected observation; every completion claim has execution evidence; no secret value appears; migration, stateful components, artifact identity, monitoring, and rollback were addressed when applicable; and the disposition does not exceed the available evidence or human authority. ## Completion criteria The workflow is complete when: - Every material change and completion claim is reconciled to observable diff, transcript, and test evidence. - Dependency and API assertions are verified, contradicted, unresolved, or unsupported; inapplicable checks are explicitly bounded. - Risk-based automated, manual, CI, observability, and rollback checks have owners and expected results. - Merge, deployment, and rollback authority remain with the responsible owners. - Unrun tests and unavailable systems remain clearly identified rather than represented as passed. # Verify an AI-Generated Software Change for Release Workflow ID: AMO-W-000014 Workflow URL: https://amo.ng/workflows/verify-ai-generated-software-change-for-release Use this Amo.ng workflow with your preferred AI tool. Complete the steps in order and carry the specified output forward. Outcome: A release-review package containing change attribution, unsupported completion claims, verified or unresolved dependency and API assertions, a risk-based test plan, deployment and rollback controls, and an authorized merge or release recommendation. Required inputs: - Original task, repository instructions, scope, acceptance criteria, and protected behavior - Coding-agent transcript, completion notes, diff or commit range, and changed files - Dependency manifests, lockfiles, API or framework claims, and relevant source or documentation - Existing tests, CI output, runtime evidence, deployment topology, observability, and rollback controls - Authorized inspection, edit, test, merge, and release boundaries ## Step 1 — Attribute changes and test completion claims **Instructions** Inspect the task, instruction hierarchy, transcript, diff, files, and available test evidence. Classify material changes, identify uninstructed or unexplained behavior, and decide which completion claims are supported. **Input for this step** Supply repository instructions, original task, agent transcript, change set, changed files, protected behavior, acceptance criteria, and actual test or command output. **Carry forward** Carry the change-attribution ledger, supported and unsupported claims, unexplained changes, suspected defects, test evidence gaps, and protected behavior into technical-claim verification. **Review note** The engineering reviewer confirms the review scope and resolves any unexplained, unauthorized, security-sensitive, or data-affecting change before release planning. **Prompt** Coding-Agent Change Attribution and Test Evidence Review **Prompt ID** AMO-P-000276 **Prompt URL** https://amo.ng/prompts/coding-agent-change-attribution-and-test-evidence-review ## Step 2 — Verify dependency and API claims when applicable **Instructions** Run this step when the change adds, removes, upgrades, or relies on packages, APIs, framework behavior, commands, configuration, or version-specific features. Otherwise record Not applicable. Classify each assertion as verified, contradicted, unresolved, or unsupported. **Input for this step** Provide the attribution findings, manifests and lockfiles, imports and calls, runtime and framework versions, local source or stubs, official documentation where available, and the agent’s technical claims. **Carry forward** Carry the technical-claim ledger, incompatible or hallucinated dependencies, unsupported APIs, correction options, and verification commands into the test plan. **Review note** The code owner or maintainer approves dependency corrections and any contract, compatibility, security, or licensing consequence. **Prompt** AI-Generated Dependency and API Claim Verification **Prompt ID** AMO-P-000277 **Prompt URL** https://amo.ng/prompts/ai-generated-dependency-and-api-claim-verification ## Step 3 — Build the risk-based verification plan **Instructions** Translate the change and claim findings into proportional automated tests, manual checks, CI gates, environment checks, observability expectations, rollback criteria, and an evidence ledger. **Input for this step** Supply the reviewed change set, defects and unresolved claims, acceptance criteria, relevant test suites and CI configuration, runtime context, data or migration risks, and available execution evidence. **Carry forward** Carry the test matrix, actual results, unrun checks, coverage gaps, required fixes, monitoring signals, and release confidence into deployment-safety review. **Review note** The test owner and engineering reviewer confirm that material risks have an observable check and that missing execution evidence remains blocking where appropriate. **Prompt** Production Test and Verification Plan Prompt **Prompt ID** AMO-P-000010 **Prompt URL** https://amo.ng/prompts/test-and-verification-prompt ## Step 4 — Prepare deployment, rollback, and release gates **Instructions** Review the supplied pipeline, release, migration, observability, and recovery evidence. Produce the smallest release-specific checklist with stop conditions, rollback triggers, and post-release verification. **Input for this step** Provide the verified change package, CI evidence, deployment scripts or workflow, environment topology, configuration and stateful changes, monitoring, backup or rollback capability, and release constraints. **Carry forward** Produce the final merge and release recommendation, changed-file and evidence ledger, blocking gaps, approved conditions, deployment checklist, rollback plan, post-release checks, and unresolved risks. **Review note** The code owner controls merge authority; the release owner makes the deploy, delay, or rollback decision after reviewing the evidence package. **Prompt** CI/CD Deployment Safety Checklist Generator **Prompt ID** AMO-P-000116 **Prompt URL** https://amo.ng/prompts/ci-cd-deployment-safety-checklist-generator Completion criteria: The workflow is complete when: - Every material change and completion claim is reconciled to observable diff, transcript, and test evidence. - Dependency and API assertions are verified, contradicted, unresolved, or unsupported; inapplicable checks are explicitly bounded. - Risk-based automated, manual, CI, observability, and rollback checks have owners and expected results. - Merge, deployment, and rollback authority remain with the responsible owners. - Unrun tests and unavailable systems remain clearly identified rather than represented as passed.Copy workflow includes every step and the full linked Prompt content. Use with AI copies a shorter guide with Prompt links; neither action runs the Workflow.
Outcome
A release-review package containing change attribution, unsupported completion claims, verified or unresolved dependency and API assertions, a risk-based test plan, deployment and rollback controls, and an authorized merge or release recommendation.
Before you begin
Have all or some of the following available before you start. The more relevant context you can provide, the stronger the workflow output will be.
- Original task, repository instructions, scope, acceptance criteria, and protected behavior
- Coding-agent transcript, completion notes, diff or commit range, and changed files
- Dependency manifests, lockfiles, API or framework claims, and relevant source or documentation
- Existing tests, CI output, runtime evidence, deployment topology, observability, and rollback controls
- Authorized inspection, edit, test, merge, and release boundaries
Ordered sequence
Workflow steps
Complete the steps in order. For each step, provide the listed context, carry its result into the next step, and pause wherever a review note is shown.
-
Step 1 Attribute changes and test completion claims
Inspect the task, instruction hierarchy, transcript, diff, files, and available test evidence. Classify material changes, identify uninstructed or unexplained behavior, and decide which completion claims are supported.
Prompt: Coding-Agent Change Attribution and Test Evidence ReviewReview the coding-agent change set and run evidence for provenance, instruction compliance, and support for completion claims. Focus on what can be observed from the supplied repository, diff, transcript, and test evidence. Do not perform a generic PR review unless it is necessary to attribute a material change or assess whether a completion claim is supported. Context to provide: - Repository scope and task: [Repository scope and task] - Relevant files and instructions: [Relevant files and instructions] - Observed evidence: [Observed evidence] - Constraints and authorized changes: [Constraints and authorized changes] - Environment details without secrets: [Environment details without secrets] - Verification commands and acceptance criteria: [Verification commands and acceptance criteria] Review rules: 1. Inspect the relevant files, diffs, transcript, and test evidence first before drawing conclusions. 2. Distinguish observation from inference. Mark unsupported assumptions as missing information. 3. Do not claim that a file, command, test, approval, system, or external source was inspected or completed unless there is evidence in the provided materials or you actually inspected or ran it in the available environment. 4. Attribute each material change to one of these categories: - Directly instructed - Reasonably necessary to satisfy the instruction - Incidental but explainable - Unexplained drift - Potentially harmful or out of scope 5. Treat a change as material if it affects behavior, public API, data model, security posture, dependency surface, build/test configuration, generated artifacts, migrations, operational behavior, or protected behavior listed by the release owner. 6. Preserve existing behavior as the default expectation. Flag behavior changes that are not explicitly instructed or clearly necessary. 7. Avoid broad rewrites and style-only judgments unless they obscure attribution, create risk, or conflict with project conventions. 8. If you identify a defect and decide to edit code, first identify the likely root cause, then apply the smallest safe change that preserves existing behavior. Do not perform broad rewrites. Run syntax checks and tests where available, then summarize files changed and verification results. If editing is not requested or not safe, provide proposed changes only. 9. Run or recommend syntax checks and tests where available and proportionate. If you cannot run them, state exactly what evidence is missing and what the test owner or release owner should run. 10. Do not recommend merge solely because the agent said the work was complete. Completion claims must be reconciled against observable diff and test evidence. Deliverable: ## 1. Review Scope and Evidence Used List the materials actually inspected: - Instructions reviewed - Transcript or completion notes reviewed - Files or diffs reviewed - Tests, logs, or command outputs reviewed - Repository context used - Evidence not provided or not inspectable ## 2. Agent Completion Claims Create a table with these columns: - Claim made by agent - Evidence offered by agent - Evidence independently visible in supplied materials or environment - Supported, partially supported, unsupported, or contradicted - Notes for release owner or test owner ## 3. Instruction-to-Diff Trace Map the original instruction to the observed changes. Use this table: - Instruction requirement - Related files or hunks - How the change satisfies the requirement - Evidence level: direct, inferred, weak, or missing - Compliance assessment Call out any instruction requirement that appears unimplemented, only partially implemented, or implemented through an unexpected approach. ## 4. Change Attribution Ledger Create a ledger for each material changed area. Use this table: - File or component - Material change observed - Attribution category - Evidence supporting attribution - Behavior or interface impact - Risk level: low, medium, high - Owner who should verify: release owner, test owner, security reviewer, data owner, product owner, or other specific accountable role ## 5. Unexplained-Change Register List all changes that cannot be clearly tied to the instruction or necessary implementation path. For each item include: - File or hunk - What changed - Why attribution is unclear - Potential consequence - What evidence would resolve it - Recommended handling: accept with owner verification, revert, isolate into separate change, or investigate before merge ## 6. Test Evidence Reconciliation Assess whether the supplied tests and logs support the completion claims. Include: - Commands claimed to have run - Commands evidenced by logs or terminal output - Pass/fail status shown by evidence - Coverage relevance to the changed behavior - Gaps, skipped tests, stale outputs, or ambiguous timestamps - Additional checks the test owner should run before merge If you run any commands, list: - Command - Purpose - Result - Relevant output summary - Any limitations If no commands were run, say so explicitly. ## 7. Risk and Protected Behavior Review Evaluate only the risks that arise from attribution, instruction compliance, or evidence gaps. Cover applicable areas: - Behavior drift - Public API or contract change - Data migration or persistence risk - Authentication, authorization, or secrets handling - Dependency or build-system changes - Generated files or lockfiles - Test configuration changes - Operational or deployment implications ## 8. Merge Recommendation Choose one recommendation: - Merge: evidence supports the agent’s claims and material changes are attributable - Merge after owner verification: residual gaps are narrow and assigned to accountable owners - Do not merge yet: unsupported claims, unexplained drift, insufficient tests, or high-risk uncertainty remain - Rework required: changes are out of scope, unsafe, or not traceable to the instruction Include: - Recommendation - Primary reasons - Required pre-merge checks - Owners responsible for verification - Conditions that would change the recommendation ## 9. Completion Check Before finalizing, confirm: - Relevant files and evidence were inspected first - Material changes were attributed - Intended edits were separated from incidental or unexplained drift - Agent claims were reconciled with test evidence - Missing evidence and uncertainty were explicitly stated - Any commands run are reported with results, or lack of execution is stated - Any files changed by you are summarized, with syntax/test verification results where availableInput for this step
Supply repository instructions, original task, agent transcript, change set, changed files, protected behavior, acceptance criteria, and actual test or command output.
Carry forward
Carry the change-attribution ledger, supported and unsupported claims, unexplained changes, suspected defects, test evidence gaps, and protected behavior into technical-claim verification.
Review note
The engineering reviewer confirms the review scope and resolves any unexplained, unauthorized, security-sensitive, or data-affecting change before release planning.
-
Step 2 Verify dependency and API claims when applicable
Run this step when the change adds, removes, upgrades, or relies on packages, APIs, framework behavior, commands, configuration, or version-specific features. Otherwise record Not applicable. Classify each assertion as verified, contradicted, unresolved, or unsupported.
Prompt: AI-Generated Dependency and API Claim VerificationVerify dependency, package, framework, and API claims introduced by AI-generated code before merge. Context to provide: - Repository scope and task: [Repository scope and task] - Relevant files and instructions: [Relevant files and instructions] - Observed evidence: [Observed evidence] - Constraints and authorized changes: [Constraints and authorized changes] - Environment details without secrets: [Environment details without secrets] - Verification commands and acceptance criteria: [Verification commands and acceptance criteria] Objective: Determine whether generated code relies on packages, versions, imports, methods, configuration keys, framework behavior, runtime features, or API signatures that are unsupported by the repository’s installed dependency set or by authoritative documentation. Produce a defensible review artifact for the release owner or package maintainer before merge. Scope boundaries: - Focus only on dependency, package, framework, runtime, type, and API claims introduced or materially affected by the generated code. - Do not perform a broad dependency upgrade, architectural rewrite, style review, or generic code review. - Do not assume an API exists because it appears plausible. - Do not claim documentation, commands, tests, approvals, or files were inspected unless you actually inspected them. - Separate direct observations from inferences and mark anything unverified. - If authoritative documentation is unavailable, use installed package source, generated types, local docs, lockfiles, manifests, and compiler or test output where available; otherwise classify the claim as Unresolved and state what evidence is needed. - Use exactly these claim outcomes: - Verified: admissible evidence confirms the claim for the installed or target version. - Contradicted: admissible evidence shows the claim is false or incompatible. - Unresolved: relevant evidence is missing, inaccessible, incomplete, or conflicting, so no conclusion is supportable yet. - Unsupported: no admissible evidence supports the assertion after the available repository and authoritative sources are checked. Do not collapse Unresolved and Unsupported into a generic warning. Preserve uncertainty and identify the evidence needed to resolve each open claim. Missing-input gate: - Treat the generated change, applicable repository instructions, dependency manifests or lockfiles, target runtime or package version, and relevant API evidence as blocking when their absence or conflict prevents a claim from being scoped. Request all blocking items in one consolidated clarification and leave the affected claim Unresolved until they are supplied. - Continue with non-blocking gaps only when each is recorded as Unknown or Unresolved, with the evidence needed and the consequence for merge or release confidence. Required process: 1. Inspect relevant files first. - Review the generated diff or branch. - Inspect dependency manifests, lockfiles, package manager configuration, runtime configuration, framework configuration, relevant imports, generated or installed type definitions, and nearby usage patterns. - Identify every dependency or API claim introduced by the generated code before deciding whether edits are needed. 2. Build a claim inventory. Include claims such as: - Package or framework is available. - A package version supports a named API, export, method, hook, decorator, CLI option, configuration key, schema field, or runtime behavior. - Import paths, module formats, peer dependencies, plugins, adapters, or provider names are valid. - Type signatures, return values, error shapes, async behavior, or environment requirements match the generated code. 3. Verify each claim against evidence. Use the strongest available evidence in this order where practical: - Repository manifests and lockfiles. - Installed package source or type definitions. - Existing repository usage and tests. - Package manager, compiler, typechecker, linter, or framework diagnostics. - Authoritative documentation or release notes supplied or accessible in the environment. 4. Identify root cause before editing. For each Contradicted, Unresolved, or Unsupported claim, determine whether the issue is caused by hallucinated API usage, wrong package name, incompatible installed version, missing peer dependency, incorrect import path, runtime mismatch, stale documentation, incomplete local install, or insufficient evidence. 5. Correction policy. - Prefer no code edits unless a minimal correction is clearly justified by evidence. - If editing is necessary and within scope, apply the smallest safe change. - Preserve existing behavior and public interfaces unless the merge or release context explicitly authorizes a change. - Avoid broad rewrites, opportunistic refactors, dependency upgrades, and speculative migrations. - If the safest fix requires a dependency upgrade or product decision, do not perform it silently; document the decision required from the release owner, package maintainer, or security reviewer. 6. Verification. - Run syntax checks, type checks, targeted tests, or package manager inspection commands where available and appropriate. - If a command cannot be run, state why and list the verification gap. - Mention verification results exactly: command, outcome, and relevant error excerpt or confirmation. Required deliverable: A. Claim inventory Provide a table with: - Claim ID - Generated-code location - Claim being made - Dependency, framework, runtime, or API involved - Why the claim matters for merge safety B. Version-and-source verification matrix Provide a table with: - Claim ID - Installed or resolved version observed - Evidence source inspected, with file path, lockfile entry, type definition, package source, command output, or documentation reference - Claim status: Verified, Contradicted, Unresolved, Unsupported, or Not applicable - Notes distinguishing observation from inference C. Contradicted, unresolved, or unsupported claims For each Contradicted, Unresolved, or Unsupported claim, include: - Finding title - Severity for merge: Blocker, High, Medium, or Low - Direct evidence - Root cause - Expected failure mode - Confidence level - Missing evidence, if any D. Minimal correction plan For each finding, provide: - Smallest safe correction - Whether code change, dependency decision, documentation check, or owner decision is needed - Files likely affected - Behavior expected to remain unchanged - Risk of the correction E. Changes made, if any - List every file changed. - Summarize the exact purpose of each change. - If no files were changed, state: No files changed during this verification pass. F. Reproducible verification record Include: - Files inspected - Commands run, if any - Tests or checks run, if any - Results observed - Checks not run and why - Open questions for the accountable owner named in [Merge or release context and accountable owner] Completion criteria: - Every generated dependency or API claim in scope is inventoried. - Each claim is classified as Verified, Contradicted, Unresolved, Unsupported, or Not applicable using the stated evidence rules. - Every Unresolved or Unsupported claim identifies the missing evidence, responsible evidence source or owner where known, and the consequence of proceeding without resolution. - Unsupported claims have root cause and expected failure mode. - No broad upgrades or rewrites are proposed as the default fix. - Verification record is sufficient for the release owner, package maintainer, or security reviewer to reproduce or challenge the conclusion.Input for this step
Provide the attribution findings, manifests and lockfiles, imports and calls, runtime and framework versions, local source or stubs, official documentation where available, and the agent’s technical claims.
Carry forward
Carry the technical-claim ledger, incompatible or hallucinated dependencies, unsupported APIs, correction options, and verification commands into the test plan.
Review note
The code owner or maintainer approves dependency corrections and any contract, compatibility, security, or licensing consequence.
-
Step 3 Build the risk-based verification plan
Translate the change and claim findings into proportional automated tests, manual checks, CI gates, environment checks, observability expectations, rollback criteria, and an evidence ledger.
Prompt: Production Test and Verification Plan PromptBuild a risk-based test and verification plan for the following software change or release. Inputs - Change or release under test: [Change or release under test] - Repository and relevant files: [Repository and relevant files] - System and runtime context: [System and runtime context] - Acceptance criteria: [Acceptance criteria] - Test and deployment constraints: [Test and deployment constraints] - Available evidence: [Available evidence] - Authorized actions and environment: [Authorized actions and environment] - CI/CD and rollback context: [CI/CD and rollback context] Codex operating boundaries - Use Codex to inspect supplied repository content, diffs, configuration, test suites, CI definitions, logs, and command output that are actually available in the session. - Run tests or read additional files only when the environment provides that capability and the authorized-actions input permits it. Prefer targeted, read-only inspection before expensive or state-changing commands. - Do not deploy, merge, approve a release, alter production, access undeclared systems, expose secrets, create real customer data, disable safeguards, or run destructive commands. Treat migrations, load tests, security probes, external API calls, and commands that write or delete data as approval-gated. - Stop before an action if its target, blast radius, data handling, cost, reversibility, or authorization is unclear. Record the blocked action, required approval, and a safe alternative. - Never imply that a command ran merely because it was proposed. Never claim that code is fixed, tests passed, coverage improved, a release was approved, a rollback works, or a deployment completed without corresponding execution evidence. Input and evidence rules 1. Treat the change target, acceptance criteria, repository or equivalent technical artifacts, runtime context, and authority scope as prerequisites for an execution-backed assessment. If one is missing, ask only the questions necessary to unblock it. 2. If execution is blocked but supplied artifacts are sufficient, produce a bounded plan and mark execution-dependent conclusions unverified. If the change boundary or acceptance criteria cannot be established, do not issue a release-confidence recommendation. 3. Maintain an evidence ledger that distinguishes supplied facts, direct Codex observations, command execution evidence, assumptions, hypotheses, conflicts, and unknowns. Cite file paths, symbols, diff locations, log excerpts, CI job names, test identifiers, commands, exit codes, or artifact locations where available. 4. Do not resolve conflicting documentation, code behavior, logs, or requirements by guessing. Describe the conflict, its verification impact, and who must resolve it. 5. Do not infer test success from the existence of test files, infer production behavior solely from mocks, or equate code coverage with behavioral correctness. Assessment workflow 1. Establish scope and baseline - Identify changed components, interfaces, dependencies, data stores, feature flags, configuration, infrastructure, schemas, jobs, and user journeys. - Determine the comparison baseline and whether generated files, lockfiles, migrations, API contracts, or deployment manifests changed. - Record exclusions and distinguish intentional scope limits from unavailable evidence. 2. Perform change-impact and risk analysis - Trace affected call paths, consumers, upstream and downstream integrations, shared libraries, background work, cache behavior, concurrency boundaries, and compatibility requirements. - Rate each material risk by likelihood and impact. Include regression, data integrity, authorization, privacy, availability, performance, observability, backward compatibility, migration, retry or idempotency, and rollback risks when relevant. - Prioritize tests by risk reduction rather than test count. 3. Build acceptance traceability - Decompose each acceptance criterion into observable behavior. - Map it to one or more unit, component, integration, contract, end-to-end, migration, security, performance, resilience, or manual checks as appropriate. - Define setup, fixtures or test data, action, expected result, required evidence, cleanup, and ownership for every check. - Include negative paths and boundaries such as empty, null, malformed, duplicate, maximum-size, timeout, partial-failure, retry, race, permission-denied, stale-cache, and dependency-unavailable conditions where applicable. 4. Evaluate existing verification assets - Identify relevant tests and assess whether their assertions prove the required behavior rather than merely execute code. - Detect missing assertions, over-mocking, nondeterministic time or randomness, shared-state leakage, order dependence, brittle snapshots, unsafe fixtures, hidden network access, and flaky retries. - Review CI triggers, path filters, matrices, service dependencies, caches, artifacts, timeouts, required checks, branch protections, and failure propagation for gaps that could produce false confidence. 5. Specify the verification sequence - Order checks from fast and isolated to broad and operational: static checks, targeted unit tests, component or integration tests, contracts, migrations, end-to-end paths, non-functional checks, and manual exploration. - Provide exact commands only when supported by repository evidence. Otherwise label commands as proposed and identify what must be confirmed. - Separate blocking release gates from advisory checks. Define retry policy, flaky-test handling, artifact retention, test-data cleanup, and ownership of failures. 6. Execute only authorized checks - Before each command, state its purpose, environment, expected side effects, and why it is within authority. - Capture the exact command, working directory, relevant environment details with secrets redacted, start and finish state, exit code, actual observation, and artifact reference. - Do not silently rewrite code or tests to make checks pass. If modification is expressly authorized, present the proposed patch and its rationale separately, then verify it with fresh evidence. - Classify each check as passed, failed, blocked, not run, or inconclusive. A zero exit code is not sufficient when assertions, logs, skipped-test counts, or produced artifacts contradict success. 7. Assess deployment and recovery readiness - Verify pre-deployment prerequisites, configuration compatibility, secret references without revealing values, migration ordering, backward and forward compatibility, feature-flag behavior, health checks, and capacity assumptions where relevant. - Define post-deployment smoke tests and observability signals with query or dashboard source, baseline, threshold, observation window, and owner. Cover errors, latency, saturation, queue lag, data reconciliation, and key business behavior as applicable. - Specify rollback or roll-forward triggers, decision owner, procedure reference, data consequences, compatibility limits, recovery verification, and cases where rollback is unsafe, such as irreversible schema or data transformations. 8. Reconcile evidence and determine confidence - Reconcile every acceptance criterion, risk, test result, defect, skipped check, and conflicting observation. - Recommend exactly one state: Ready, Conditionally ready, Not ready, or Unassessed. This is a technical recommendation, not release approval. - Ready requires all blocking criteria to have passing evidence and no unresolved release-blocking defect or unknown. Conditionally ready requires explicit conditions, owners, and deadlines. Not ready requires named blockers. Unassessed applies when evidence is insufficient to support a conclusion. Required deliverable A. Scope and evidence ledger - Change boundary, baseline, affected systems, exclusions, authority scope, and environment. - Evidence table with ID, classification, source or artifact, observation, reliability limitation, and related conclusion. - Assumptions, unknowns, and conflicts, each with impact and resolution owner. B. Change-impact and risk register - Component or behavior, change mechanism, dependent systems, failure mode, likelihood, impact, detectability, risk priority, proposed control, and residual risk. C. Acceptance-to-test matrix - Criterion ID, observable behavior, risk covered, test level, setup and data, procedure or command, expected observation, required evidence, cleanup, owner, priority, and status. D. Existing test and CI assessment - Relevant test or job, what it proves, identified gap, flakiness or isolation concern, CI gate status, and recommended correction. E. Execution record - Check ID, proposed or executed state, exact command or manual procedure, environment, expected observation, actual observation, exit code when applicable, duration when known, evidence reference, and result classification. F. Defect and unresolved-work register - Defect or gap ID, reproduction evidence, affected criterion, severity, release impact, workaround, owner, and retest requirement. Keep proposed fixes separate from applied changes. G. Deployment, observability, and recovery checks - Pre-deployment gates, smoke tests, monitored signals, baselines and thresholds, observation windows, rollback or roll-forward triggers, recovery procedure references, data reconciliation, and responsible approvers. H. Verification verdict - Recommended state, evidence-backed rationale, passed blocking gates, failed or missing gates, residual risks, approval still required, and the smallest safe next action. Use concise technical language. Preserve unresolved states and make every consequential conclusion traceable to evidence.Input for this step
Supply the reviewed change set, defects and unresolved claims, acceptance criteria, relevant test suites and CI configuration, runtime context, data or migration risks, and available execution evidence.
Carry forward
Carry the test matrix, actual results, unrun checks, coverage gaps, required fixes, monitoring signals, and release confidence into deployment-safety review.
Review note
The test owner and engineering reviewer confirm that material risks have an observable check and that missing execution evidence remains blocking where appropriate.
-
Step 4 Prepare deployment, rollback, and release gates
Review the supplied pipeline, release, migration, observability, and recovery evidence. Produce the smallest release-specific checklist with stop conditions, rollback triggers, and post-release verification.
Prompt: CI/CD Deployment Safety Checklist GeneratorReview the supplied release materials and produce an evidence-traceable CI/CD deployment safety assessment. Use Codex to inspect the repository and only files, text, command output, and repository context that are actually supplied or available in the current session. Do not imply access to a repository, CI provider, cloud account, secrets store, database, monitoring system, or production environment unless that access is demonstrably available. Inputs Repository and release scope: [Repository and release scope] Pipeline and deployment artifacts: [Pipeline and deployment artifacts] Platform and environment topology: [Platform and environment topology] Migration and stateful workload details: [Migration and stateful workload details] Verification and observability evidence: [Verification and observability evidence] Rollback and governance requirements: [Rollback and governance requirements] Input expectations The repository and release scope should identify the change set, affected services, release reference, critical user flows, external dependencies, and known high-risk changes such as billing, authentication, authorization, data deletion, or infrastructure changes. Pipeline and deployment artifacts should include relevant workflow files, reusable workflows, deployment scripts, manifests, infrastructure definitions, build configuration, test commands, and release instructions. Platform and environment topology should describe environments, promotion flow, deployment strategy, runtime components, regions, traffic routing, queues, caches, scheduled jobs, and secret or identity mechanisms without exposing secret values. Migration and stateful workload details should cover schema and data migrations, compatibility assumptions, expected duration, locking risk, backups, restoration, and interactions with workers or older application versions. Verification and observability evidence should provide health checks, smoke tests, dashboards, alerts, logs, service-level indicators, prior command output, and acceptance thresholds. Rollback and governance requirements should identify rollback or roll-forward procedures, approval owners, change windows, incident contacts, communication requirements, and the release definition of done. Input and evidence rules 1. Create an input ledger before drawing conclusions. Classify each needed item as supplied, observed in an accessible artifact, conflicting, missing, or not applicable. Cite file paths and line ranges when available; otherwise cite the supplied input section or evidence item. 2. Never invent workflow behavior, provider settings, branch protection, environment rules, test outcomes, secret values, migration reversibility, backup validity, monitoring coverage, approvals, or production state. 3. If inputs conflict, record both claims, identify their sources, explain the safety consequence, and request the authoritative source. Do not silently choose one. 4. If a critical fact is missing, mark the affected conclusion unverified and make the release disposition Blocked when safe deployment depends on that fact. Noncritical gaps may receive a clearly labeled conservative hypothesis, but a hypothesis is not evidence. 5. Treat documentation as evidence of an intended process, not proof that a control ran. Treat configuration as evidence of a configured control, not proof of successful execution. Treat logs, CI run records, signed approvals, artifact metadata, command output, or monitoring observations as execution evidence only when their source and release relevance are supplied. 6. Use these work-state labels consistently: Requested for work the user asked for; Proposed for changes or commands not applied; Executed only for an action actually performed in the current session; Unavailable when access or capability is absent; Unverified when evidence is insufficient. Every claim that something was tested, fixed, deployed, rolled back, approved, or verified must include execution evidence. Otherwise label it Proposed or Unverified. 7. Bind every material piece of evidence to the exact release under review. A passing test, approval, artifact, log entry, monitoring observation, or prior deployment from another commit, branch, artifact digest, environment, configuration state, or execution window is not evidence for this release unless a traceable relationship is supplied. Record the commit, release reference, artifact identity, target environment, and evidence timestamp where available. Authority and safeguards Unless [Rollback and governance requirements] expressly restrict access, permit read-only repository inspection and non-mutating diagnostics within the workspace actually available to Codex. Treat file edits, mutating commands, pipeline or configuration changes, database writes or migrations, secret rotation, infrastructure changes, deployment, rollback, production access, and external side effects as unauthorized unless expressly approved. Do not deploy, merge, approve, rotate secrets, alter infrastructure, run migrations, modify production data, disable controls, or trigger rollback. If a read-only check against a production target is expressly authorized and Codex has demonstrable access, limit it to a clearly non-mutating command against the stated target. Record the exact command, target, exit status, relevant output, time, and limitations. Never run destructive, state-changing, costly, financially consequential, or irreversibly production-affecting commands within this prompt. Otherwise provide commands as Proposed and do not fabricate output. Do not reproduce secret values, tokens, credentials, private keys, customer data, or sensitive log content. Refer to secret names or redacted identifiers only. Flag excessive permissions, untrusted code paths with secret access, unsafe pull-request triggers, command injection surfaces, unpinned third-party actions, mutable artifacts, and credential persistence. Human approval remains mandatory for production release decisions and for changes involving billing, identity, permissions, security controls, destructive data operations, non-backward-compatible migrations, or infrastructure replacement. Focused review workflow 1. Trace the failure modes and map the delivery path from source trigger to production: event and branch or tag filters, pull-request trust boundary, build, tests, artifact creation, provenance or digest handling, promotion, environment selection, deployment, verification, and rollback. Identify reusable workflows and dependencies that can alter this path. 2. Inspect trigger and concurrency safety. Check accidental production triggers, skipped required jobs, path-filter blind spots, duplicate deployments, cancellation behavior, race conditions, environment locks, release serialization, and whether the deployed commit or artifact is uniquely identified. 3. Inspect identity, permissions, and supply-chain controls. Check least-privilege workflow permissions, OIDC or credential scope where evidenced, secret availability by event and environment, masking and log exposure, dependency or action pinning, artifact integrity, provenance, retention, and separation between build and deploy authority. 4. Inspect build and test gates. Trace dependency installation, lockfile enforcement, deterministic builds, static checks, unit and integration tests, security checks where required, failure propagation, retry behavior, test exclusions, coverage of critical flows, and whether the exact promoted artifact passed the cited checks. 5. Inspect environment and deployment correctness. Check staging-to-production parity, configuration validation, immutable artifact promotion, deployment strategy, traffic shifting, readiness versus liveness semantics, timeout behavior, partial failure across services or regions, infrastructure ordering, external API dependencies, maintenance requirements, and idempotency of repeated deployment attempts. 6. Inspect migration and stateful-component safety. Evaluate expand-and-contract compatibility, application and migration order, mixed-version operation, transaction and lock behavior, table rewrites, long-running backfills, retry and resume behavior, data validation, queue payload compatibility, worker draining, cron overlap, cache-key or serialization changes, backup freshness, restore evidence, and whether rollback would leave code and schema compatible. Treat an unproven destructive or irreversible migration as a blocking risk. 7. Inspect observability and release control. Check that health endpoints test meaningful dependencies without leaking data; smoke tests cover critical user journeys; dashboards and alerts identify error rate, latency, saturation, queue lag, failed jobs, database health, and business-critical signals; thresholds, observation windows, owners, and escalation paths are defined. 8. Build rollback and roll-forward logic. Define measurable triggers, decision owner, last known good artifact, code and configuration restoration, schema mitigation, traffic restoration, queue and cache handling, external side-effect reconciliation, user communication, and post-recovery verification. Do not call rollback viable without evidence that required artifacts, procedures, permissions, and schema compatibility exist. 9. Prioritize findings using impact and likelihood rated Low, Medium, High, or Critical. Distinguish release blockers from required follow-ups and optional hardening. Prefer the smallest control that materially reduces the identified risk; do not recommend broad platform rewrites without evidence that they are necessary. Base impact and likelihood on release-specific evidence. Do not infer likelihood solely from generic industry experience or the theoretical existence of a failure mode. When the available evidence cannot support a defensible likelihood rating, mark likelihood Unverified, explain the uncertainty, and state what evidence is needed. Output contract: required CI/CD safety deliverable Produce the following task-specific sections in markdown. A. Review basis and evidence ledger Provide a table with Evidence ID, item or artifact, source locator, relevance to this release, evidence class, and status. Evidence class must distinguish intended process, static configuration, and execution evidence. Follow it with missing and conflicting inputs, their consequences, and the exact evidence needed to resolve each one. B. Delivery-path map Describe the evidenced path from trigger to production in order. For every stage list trigger or input, responsible workflow or script, output artifact or state transition, environment, controlling gate, and evidence ID. Mark inferred or unknown transitions explicitly. C. Risk register Provide Finding ID, delivery stage, failure mode, supporting evidence IDs, impact, likelihood, severity, affected environment or service, release consequence, required mitigation, owner or approver if supplied, and state. Include concrete findings for triggers, permissions, secrets, artifact integrity, tests, environment drift, deployment ordering, migrations, stateful workers, health checks, monitoring, and rollback when relevant. Do not create findings unsupported by the supplied architecture; record missing evidence instead. D. Release gate checklist Create ordered Pre-deployment, Deployment, and Post-deployment gates. Each checklist row must contain Gate ID, check, reason, execution target, method or proposed command, expected observation, supplied actual observation, evidence ID, pass criterion, stop or pause condition, responsible human, and state. Leave actual observation as Not supplied unless real output exists. Commands must identify assumptions and must not expose secrets or mutate production. Include, where applicable, confirmation of the exact commit and immutable artifact; required CI results; configuration-key presence without values; environment and identity target; backup and restoration evidence; backward-compatible migration sequence; worker, queue, cache, and scheduler coordination; approval and communication gates; deployment progress; health and readiness; critical API and user-flow smoke tests; error, latency, saturation, queue, database, and business-signal thresholds; and an observation window. E. Migration and stateful-workload decision record State the proposed sequence for application versions, schema changes, backfills, workers, queues, caches, and scheduled jobs. Document compatibility across old code, new code, old schema, and new schema; lock and duration concerns; abort criteria; backup or restoration prerequisites; data-integrity reconciliation; and rollback versus roll-forward constraints. For each conclusion cite evidence or mark it Unverified. F. Rollback readiness record Provide rollback trigger, decision owner, code or artifact action, configuration action, database mitigation, traffic action, queue and cache handling, external side-effect reconciliation, communications, verification check, expected observation, and evidence. Identify the point after which rollback becomes unsafe and a roll-forward is required. Mark readiness Unverified if no tested procedure or equivalent execution evidence is supplied. G. Verification plan and evidence requirements For each proposed verification, give the exact non-destructive command or manual action, target environment, prerequisite, expected observation, acceptance threshold, failure interpretation, evidence to retain, and current work state. Reconcile the deployed release identity with the reviewed commit and artifact digest. Reconcile migration version and data checks with the expected release state. Reconcile health and smoke-test results with monitoring over the stated observation window. Never populate actual results unless they were supplied or executed with recorded evidence. H. Release disposition Choose exactly one disposition: Blocked, Conditional candidate for human approval, or Ready for human approval. This is advice, not approval or authorization to deploy. List the decisive evidence, unresolved blockers, conditions that must be satisfied, required human gates, monitoring obligations, and safest next action. A disposition of Ready for human approval requires traceable evidence that required tests passed for the reviewed release artifact, the deployment target is identified, migration and configuration prerequisites are satisfied, meaningful health and smoke checks have acceptance thresholds, observability and escalation are active, and rollback or roll-forward is operationally credible. If any required evidence is missing, use Blocked or Conditional candidate for human approval. Keep every section concise and proportional to the release’s actual scope and risk. Do not repeat the same evidence across multiple sections unnecessarily. Where a section or control area is genuinely not applicable, retain the heading, state Not applicable, and explain briefly why using the supplied release evidence. Never omit the evidence ledger, risk register, release gates, release disposition, or completion-integrity distinctions. Final integrity check Before returning the deliverable, confirm that every material conclusion cites evidence or is marked Unverified; every proposed command has a target and expected observation; every completion claim has execution evidence; no secret value appears; migration, stateful components, artifact identity, monitoring, and rollback were addressed when applicable; and the disposition does not exceed the available evidence or human authority.Input for this step
Provide the verified change package, CI evidence, deployment scripts or workflow, environment topology, configuration and stateful changes, monitoring, backup or rollback capability, and release constraints.
Carry forward
Produce the final merge and release recommendation, changed-file and evidence ledger, blocking gaps, approved conditions, deployment checklist, rollback plan, post-release checks, and unresolved risks.
Review note
The code owner controls merge authority; the release owner makes the deploy, delay, or rollback decision after reviewing the evidence package.
Completion criteria
The workflow is complete when:
- Every material change and completion claim is reconciled to observable diff, transcript, and test evidence.
- Dependency and API assertions are verified, contradicted, unresolved, or unsupported; inapplicable checks are explicitly bounded.
- Risk-based automated, manual, CI, observability, and rollback checks have owners and expected results.
- Merge, deployment, and rollback authority remain with the responsible owners.
- Unrun tests and unavailable systems remain clearly identified rather than represented as passed.
Related Workflows
Browse WorkflowsProduction Incident to Safe Patch and Prevention Plan
Turn production logs and repository evidence into a minimal patch proposal, verification plan, independent review, deployment controls, and a blameless prevention backlog.
Plan and Review a Complex Laravel Feature for Safe Release
Turn an evidence-supported product opportunity into a phased Laravel implementation plan, conditionally review migration safety, and—after separately authorized implementation produces a real change set—review the pull request and prepare a risk-based release gate.
Laravel Incident Investigation and Controlled Correction
Use a compact three-step path to diagnose a Laravel production incident, make only an authorized minimal correction, independently review the change, and prepare a controlled release.
Was this useful?