Reusable AI capability

Recover Failed Kubernetes Rollouts Safely

Apply a repeatable evidence and decision method to isolate a failed Kubernetes rollout, choose the smallest safe recovery path, and verify restored service before closing the incident.

This Skill is a reusable set of instructions based on the linked Amo.ng Prompt or Workflow. It tells you what to provide, how to apply the method and how to know when the task is complete. It does not run automatically.

Skill ID
AMO-S-000017
Based on
Prompt
Published

Copy Skill copies the full Skill instructions. Use with AI copies those instructions with a short guide for your chosen AI tool. Both actions only copy text—they do not send anything to an AI provider or run the Skill.

Purpose

Give platform and service teams a reusable rollout-recovery capability that separates diagnosis, recovery authorization, execution evidence, and service verification across different Kubernetes workloads and release mechanisms.

Required inputs

Have these details available before following the usage instructions.

  • Affected environment, namespace, workloads, services, and incident window
  • Release, manifest, image, configuration, secret, ingress, service, policy, or infrastructure changes
  • Available rollout status, events, logs, probes, endpoints, routing, metrics, alerts, and application evidence
  • User impact, SLO constraints, recovery window, rollback and forward-fix options
  • Platform, service, security, incident, and release owners plus authorization boundaries

How to use this Skill

When to use:
- A Kubernetes rollout is failed, stalled, or degraded and the team needs an evidence-based recovery decision.
- The same recovery discipline must be applied consistently across Deployments, StatefulSets, Services, ingress, probes, configuration, or dependencies.

When not to use:
- Designing a new cluster architecture or generic Kubernetes hardening.
- Executing production changes without the relevant platform and release authority.
- Declaring recovery when workload and application evidence are unavailable.

Reusable method:
1. Freeze the incident boundary, latest known-good release, impact, recent changes, and inspectable evidence.
2. Build a timeline and classify observations, inferences, assumptions, missing evidence, and actions already taken.
3. Localize failure across artifact, scheduling, configuration, secret, lifecycle, probe, service discovery, ingress, network, dependency, resource, and autoscaling layers.
4. Rank competing causes and define the next discriminating observation for unresolved hypotheses.
5. Compare rollback, configuration reversal, image correction, probe correction, scaling, traffic diversion, dependency recovery, and continue-investigation paths.
6. Select the smallest safe path with prerequisites, expected effect, blast radius, stop condition, rollback or compensation option, and authorized owner.
7. Verify workload availability, pod stability, probes, endpoints, routing, application transactions, metrics, logs, and customer-impact indicators over a defined monitoring window.

Expected output:
A recovery decision record containing evidence timeline, affected resources, layered hypothesis register, chosen and rejected recovery paths, authorization gates, execution checklist, service-verification matrix, monitoring window, unresolved evidence, and owner handoff.

Boundaries:
Do not invent cluster state, command output, changes, approvals, or recovery. Platform and service owners authorize configuration and workload actions; the security reviewer dispositions material security impact; the incident and release owners control production recovery and closure. AMO-P-000235 is the grounding source, not a prerequisite to invoke the Skill.

Based on an Amo.ng Prompt

Kubernetes Deployment Recovery Runbook

Open the linked Prompt to read the instructions this Skill is based on.

Open prompt

Completion criteria

Complete when:
- Every material cause finding links to supplied evidence and unresolved causes have a discriminating check.
- The chosen path has prerequisites, owner, expected effect, risk, stop condition, and fallback.
- Production execution and observed results are distinguished from proposed commands.
- Workload and application verification have observable results or remain explicitly pending.
- The status is Ready for authorized recovery, Conditionally ready, Blocked, or Recovered with evidence.

Was this useful?

Browse Prompts