Kubernetes incidents become easier to resolve when troubleshooting follows a consistent path instead of a collection of remembered commands. This runbook provides a reusable, symptom-based process for checking cluster context, workloads, scheduling, resources, services, ingress, networking, and rollouts. Use it during an incident, then revisit the tracking sections monthly or quarterly to turn repeated fixes into documented Kubernetes best practices.
Overview
Start by defining the symptom in observable terms. “The application is down” is a useful alert, but it is not yet a diagnosis. Establish whether requests fail at the ingress, service, or pod layer; whether the problem affects one workload or the wider cluster; and whether the failure began after a deployment, configuration change, node event, or dependency outage.
Before changing anything, confirm that your commands target the intended cluster and namespace. A surprising number of operational mistakes come from inspecting one context and modifying another.
kubectl config current-context
kubectl config get-contexts
kubectl get namespaces
kubectl get pods -AFor a focused investigation, set a namespace explicitly on each command or use a shell variable:
NS=production
kubectl get pods -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestampCapture the initial state before taking corrective action. Save pod status, recent events, deployment history, and relevant logs. This record helps distinguish a successful fix from a temporary recovery and gives the team material for a later runbook update.
What to track
Pods and containers
For a pod that is pending, restarting, or failing readiness checks, inspect its status and description first.
kubectl get pods -n "$NS" -o wide
kubectl describe pod <pod-name> -n "$NS"
kubectl logs <pod-name> -n "$NS" --all-containers
kubectl logs <pod-name> -n "$NS" --previousThe --previous option is especially useful when a container has restarted and the current container has not produced the failure message. Track the restart count, container state, exit reason, readiness condition, node assignment, image reference, and recent events.
For CrashLoopBackOff, group the investigation into five questions: did the process exit by itself, was it killed, did it fail a probe, did it lack configuration or a secret, or did it fail while connecting to a dependency? Check the command and arguments, environment references, mounted volumes, resource limits, and application logs before repeatedly deleting the pod. A detailed guide to this symptom is available in the Kubernetes CrashLoopBackOff troubleshooting guide.
Deployments and rollouts
When a release is incomplete or traffic still reaches an older version, compare the desired, updated, available, and ready replica counts.
kubectl get deployment <deployment-name> -n "$NS"
kubectl describe deployment <deployment-name> -n "$NS"
kubectl rollout status deployment/<deployment-name> -n "$NS"
kubectl rollout history deployment/<deployment-name> -n "$NS"Look for unavailable replicas, failed scheduling, image-pull errors, invalid configuration, failed readiness probes, and progress-deadline events. If the previous version is known to be safer and the deployment history is available, a rollback may be appropriate:
kubectl rollout undo deployment/<deployment-name> -n "$NS"
kubectl rollout status deployment/<deployment-name> -n "$NS"Record why the rollout was reversed. A rollback restores service, but it does not explain the defect. Preserve the failed image tag, configuration revision, and relevant logs for follow-up.
Services and endpoints
When a Kubernetes service is not working, verify the service selector before investigating more complex networking layers.
kubectl get service <service-name> -n "$NS" -o yaml
kubectl get endpoints <service-name> -n "$NS"
kubectl get endpointslice -n "$NS" -l kubernetes.io/service-name=<service-name>
kubectl get pods -n "$NS" --show-labelsA service can exist and still have no usable backends if its selector does not match pod labels, or if matching pods are not ready. Compare the selector with actual labels, then confirm that the target port matches the application’s listening port. Test from inside the cluster when possible, using a temporary diagnostic container approved by your operating procedures. This separates cluster DNS or routing problems from application-level failures.
Ingress and external routing
For ingress troubleshooting, move from the outside inward: DNS, load balancer or ingress controller, ingress resource, service, endpoints, and finally the application container.
kubectl get ingress -n "$NS"
kubectl describe ingress <ingress-name> -n "$NS"
kubectl get pods -A | grep -i ingress
kubectl get events -A --sort-by=.lastTimestampCheck the hostname, path rules, TLS secret reference, ingress class, backend service name, and backend port. Controller logs often reveal rejected configuration, certificate errors, or an inability to reach the service. If the request reaches the service but returns an application error, stop changing ingress resources and continue at the pod and application layers.
Scheduling, nodes, and resources
Pending pods commonly indicate a scheduling constraint rather than an application failure. Inspect events for insufficient CPU or memory, taints, affinity rules, node selectors, volume constraints, or unavailable resources.
kubectl describe pod <pod-name> -n "$NS"
kubectl get nodes
kubectl describe node <node-name>
kubectl top pods -n "$NS"
kubectl top nodesWhere metrics are available, compare requested resources with observed usage and look for persistent pressure. Avoid increasing limits as an automatic fix: an oversized request can prevent scheduling, while an undersized limit can lead to throttling or termination. Track resource changes alongside deployment changes so cost and reliability effects remain visible. The Kubernetes cost optimization checklist can support this review.
Cadence and checkpoints
Use this runbook at three different speeds. During an incident, follow the symptom path and capture evidence before changing resources. After recovery, hold a short review while the details are still available. On a monthly or quarterly cadence, inspect recurring signals across workloads rather than waiting for another outage.
- Per incident: record the first symptom, affected namespace and workload, timeline, commands used, evidence collected, mitigation, and follow-up owner.
- Monthly: review restart counts, pending-pod events, failed rollouts, image-pull failures, probe failures, node pressure, and service or ingress incidents.
- Quarterly: test whether the documented commands still match your cluster access model, namespaces, ingress implementation, deployment process, and observability coverage.
- After platform changes: update the runbook when Kubernetes versions, ingress controllers, networking, admission policies, secret management, or deployment tooling changes.
Keep a team-owned table for recurring fixes. Useful columns include date, symptom, affected service, likely cause, confirming evidence, mitigation, permanent fix, alert or dashboard link, and runbook owner. This converts individual troubleshooting knowledge into a shared resource for the DevOps community and on-call team.
How to interpret changes
Look for patterns, not isolated values. A single restart may be harmless during a rollout, but a steadily increasing restart count after deployment suggests a container, probe, configuration, or dependency problem. A rise in pending pods across several namespaces points more toward capacity or scheduling constraints than one faulty application.
Correlate Kubernetes events with release timestamps, node changes, dependency alerts, and application logs. A new image followed by readiness failures is a different investigation from readiness failures that began after a network policy change. Similarly, a service with no endpoints requires a selector or readiness investigation, while a service with healthy endpoints but failed external requests points toward ingress, load balancing, DNS, or TLS.
Use a simple decision path:
- No pod is running: inspect deployment status, scheduling events, image pulls, and resource constraints.
- Pods run but are not ready: inspect readiness probes, application startup, ports, configuration, and dependency access.
- Pods are ready but service traffic fails: verify selectors, endpoints, ports, DNS, and in-cluster connectivity.
- Internal traffic works but external traffic fails: inspect ingress rules, controller logs, DNS, TLS, and the external load-balancing path.
- The issue began after a release: compare the current revision with the previous one and consider a controlled rollback while preserving evidence.
When to revisit
Revisit this Kubernetes troubleshooting guide whenever the cluster’s failure modes change. Schedule a monthly review for active production environments and a quarterly review for teams with fewer operational changes. Update it immediately after a new recurring incident, a major platform change, or a remediation that required undocumented commands.
At each review, remove commands that no longer apply, add the exact labels and namespaces your team uses, verify that diagnostic access is safe and authorized, and link to the dashboards or logs responders actually need. Test one representative path—such as a failed rollout, unavailable service, or pending pod—in a non-production environment when practical.
Finish with one concrete action: choose the most repeated symptom from the last review and add its confirming evidence, mitigation, and permanent fix to the team runbook. Then assign an owner and a review date. Small, regular updates make Kubernetes operations more predictable than a large rewrite after the next incident.
