Webhook Failure Debug Prompt

## Debugging objective

Investigate the webhook failure described below and produce an evidence-backed diagnosis, remediation package, and verification plan.

### Incident inputs
- Webhook provider and event: [Webhook provider and event]
- Failure symptom and time window: [Failure symptom and time window]
- Endpoint and environment: [Endpoint and environment]
- Evidence bundle: [Evidence bundle]
- Signature verification configuration: [Signature verification configuration]
- Retry and delivery policy: [Retry and delivery policy]
- Idempotency design: [Idempotency design]
- Constraints and authorized actions: [Constraints and authorized actions]
- Acceptance criteria: [Acceptance criteria]

Treat the webhook provider, failure symptom, endpoint or environment, time window, and at least one relevant evidence artifact as minimum inputs. Useful evidence includes redacted delivery records, immutable event or delivery IDs, attempt numbers, timestamps with time zones, HTTP status codes, response bodies, latency, request headers, payload digests, endpoint logs, traces, queue records, database records, configuration, relevant source files, dependency versions, and existing test commands.

If a minimum input is missing or conflicting, ask only the questions that block a reliable investigation. Otherwise, continue with bounded analysis and mark the limitation. Do not invent payloads, logs, provider behavior, configuration, command output, dashboard observations, or test results.

## Codex operating boundaries

Use Codex to inspect the supplied workspace, trace relevant request-handling paths, compare configuration with code, prepare focused diffs, and run permitted local diagnostic or test commands when those capabilities and files are actually available. State which files and commands were inspected or executed.

Do not imply that Codex accessed a provider dashboard, production host, secret store, observability platform, network endpoint, database, or deployment system unless that access was explicitly provided and successfully used. If execution is unavailable, provide commands as proposed commands rather than reporting their results.

Default to read-only inspection. Do not deploy, rotate secrets, modify production data, disable signature checks, acknowledge or discard queued events, contact third parties, or replay deliveries without explicit human authorization. A code or configuration change may be prepared only within the stated permissions; production application requires review and approval.

Protect webhook secrets, authorization headers, personal data, payment data, and full production payloads. Use redacted samples or secure local files. When byte-exact payload analysis is necessary for signature verification, inspect the authorized local artifact without reproducing sensitive contents; report a digest and relevant byte-level characteristics instead. Stop and request human review if the proposed action could cause financial transactions, customer notifications, destructive mutations, privilege changes, uncontrolled traffic, or duplicate business operations.

## Investigation workflow

1. **Establish scope and evidence states.** Record the incident window, environment, affected event types, expected behavior, observed behavior, and business impact. Classify each material statement as supplied fact, direct observation, execution evidence, assumption, hypothesis, conflict, or unknown. Preserve conflicting timestamps, statuses, and IDs instead of silently choosing one.

2. **Build an attempt timeline.** Correlate provider event ID, delivery ID, request or trace ID, idempotency key, attempt number, provider send time, endpoint receive time, response time, HTTP status, latency, queue transition, handler outcome, and downstream side effect. Normalize time zones while retaining original timestamps. Identify gaps that prevent end-to-end correlation.

3. **Locate the failing stage.** Trace the path from provider event creation through DNS, TLS, routing or gateway, request-size limits, framework middleware, raw-body capture, signature verification, parsing, handler dispatch, queueing, downstream calls, persistence, and final acknowledgement. Distinguish transport failure, timeout, non-2xx response, malformed payload, unsupported event version, signature rejection, handler exception, resource exhaustion, downstream failure, and observability gaps.

4. **Audit signature verification.** Check that verification uses the exact raw request bytes before decoding or mutation; confirm the expected signature algorithm, signed components, header parsing, timestamp tolerance, clock skew, constant-time comparison, encoding, secret selection, environment, secret rotation overlap, and multiple-signature handling. Never recommend bypassing verification as a fix. If raw bytes or authoritative provider documentation are unavailable, state that the signature conclusion is limited.

5. **Audit acknowledgement and retry behavior.** Determine when the endpoint emits its response relative to durable persistence and side effects. Compare actual statuses and latency with the provider's success criteria, timeout, backoff schedule, retry window, maximum attempts, ordering guarantees, and manual-redelivery behavior. Identify ambiguous outcomes where the provider timed out but processing may have completed.

6. **Audit idempotency and concurrency.** Verify that deduplication uses a stable provider event identifier rather than a delivery-attempt ID. Examine atomicity, unique constraints, transaction boundaries, inbox or outbox records, processing states, lock behavior, retention period relative to the retry window, crash recovery, and concurrent duplicate delivery. Check whether marking an event complete can occur before all required side effects succeed and whether failed events can be safely resumed.

7. **Rank root-cause hypotheses.** For each hypothesis, give supporting evidence, contradicting evidence, confidence, missing evidence, and a falsification check. Separate the primary incident cause from contributing weaknesses such as inadequate logging, unsafe retries, or missing deduplication. Do not collapse correlation into causation.

8. **Prepare the smallest safe remediation.** Identify the exact files, configuration, data structures, or operational procedures that should change. Explain behavior before and after, compatibility impact, security consequences, duplicate-processing trade-offs, migration needs, observability additions, rollout sequence, rollback trigger, and recovery path. When authorized and feasible, prepare a minimal diff and tests; otherwise provide a patch plan or illustrative diff clearly marked as unexecuted.

9. **Design controlled replay or recovery.** Reconcile candidate events against durable processing records and downstream side effects before proposing replay. Specify event selection, dry-run behavior, rate limits, ordering, idempotency preconditions, monitoring, abort thresholds, audit records, and approval owner. Quarantine uncertain events rather than replaying them blindly. Never recommend bulk replay solely because the provider reports failure.

10. **Verify against failure modes.** Define checks for a valid signed event, byte-altered payload, missing or malformed signature, wrong secret, rotated secret, stale timestamp, sequential duplicate, concurrent duplicate, endpoint timeout after commit, retry after 5xx, out-of-order event, handler crash, downstream partial failure, unsupported event type, and recovery after restart. Include expected observation, actual observation when a test was run, evidence reference, and disposition.

11. **Determine handoff state.** Use only these states: `diagnosed`, `probable cause`, `blocked`, `change proposed`, `change prepared`, `locally tested`, `ready for authorized rollout`, or `unverified`. Claims such as fixed, tested, replayed, deployed, or verified require corresponding execution evidence. A prepared diff is not a deployed fix, a passing unit test is not production verification, and a proposed replay is not an executed replay.

## Required deliverable

Return the following task-specific sections:

1. **Scope and capability record** — incident scope, supplied artifacts, Codex access actually available, actions authorized, actions not performed, blocking gaps, and data-redaction notes.
2. **Webhook attempt timeline** — a table with correlation IDs, attempts, normalized and original timestamps, HTTP outcomes, latency, processing outcomes, side effects, and evidence references.
3. **Evidence ledger** — numbered evidence items with source, evidence state, relevance, reliability limitation, and any conflict.
4. **Failure-stage assessment** — each delivery stage marked supported, failed, bypassed, unknown, or not applicable, with evidence.
5. **Root-cause hypothesis matrix** — ranked hypotheses with supporting and contradicting evidence, confidence, falsification check, and result or unverified status.
6. **Signature, retry, and idempotency findings** — concrete implementation findings, affected files or configuration, operational consequence, and severity.
7. **Remediation package** — smallest safe change, proposed or prepared diff summary, migration and compatibility notes, rollout controls, rollback conditions, and unresolved decisions.
8. **Replay and recovery runbook** — eligibility query or selection logic, reconciliation method, dry run, approval gate, rate limit, monitoring, abort threshold, rollback or quarantine path, and audit evidence to retain. State `not required` when justified.
9. **Verification matrix** — test case, setup, expected observation, actual observation, evidence reference, pass, fail, blocked, or not run status, and follow-up owner.
10. **Acceptance reconciliation** — map every supplied acceptance criterion to evidence and mark it met, not met, blocked, or unverified.
11. **Handoff** — current handoff state, residual risks, approvals required, exact next safe action, and the evidence still needed before any stronger completion claim.