Kubernetes Troubleshooting Guide: Diagnose Pods, Services, Ingress, and Deployments

Overview

A Kubernetes troubleshooting session should answer three questions in order: what is failing, where is it failing, and what evidence supports the next change? Start with the workload and its namespace, then move outward through scheduling, containers, service discovery, ingress, and external dependencies.

Before making changes, record the affected namespace, resource names, approximate start time, recent deployments, and the user-visible symptom. This context prevents a common failure mode: fixing a nearby warning while overlooking the original cause. If the issue began after a release, compare the current manifest and image with the last known-good version. The CI/CD pipeline templates guide can help connect deployment changes to the operational investigation.

Use a read-first approach wherever possible. Commands such as kubectl get, kubectl describe, kubectl logs, and kubectl get events expose state without altering it. Apply a change only after forming a testable hypothesis, and verify the result using the same signal that demonstrated the failure.

Step-by-step workflow

1. Confirm the scope and current state

Begin by checking whether the problem affects one pod, one workload, one namespace, or multiple applications.

kubectl get pods -A
kubectl get deploy,rs,svc,ingress -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Look for a pattern: many pods pending may indicate capacity or scheduling constraints; one failing replica may indicate an application or image problem; a healthy workload with failed requests points toward service, ingress, policy, or dependency configuration.

2. Inspect the workload and pod conditions

For a deployment, inspect its status and the replica set it created.

kubectl describe deployment <deployment> -n <namespace>
kubectl get rs -n <namespace>
kubectl describe pod <pod> -n <namespace>

Pay attention to conditions, scheduling messages, image pull errors, readiness failures, mounted-volume errors, and container termination details. A deployment can exist while its desired number of ready replicas remains unmet, so do not treat the existence of the resource as proof that the application is available.

3. Investigate common pod symptoms

Pending: Check events for insufficient resources, node selectors, taints, tolerations, affinity rules, or unbound persistent volume claims. Compare the pod's requests with available node capacity, and verify that scheduling constraints match the nodes intended to run the workload.

ImagePullBackOff or ErrImagePull: Confirm the image name and tag, registry reachability, and image-pull secret configuration. Check whether the workload references the intended registry and whether a recent change altered the tag or credentials.

CrashLoopBackOff: Inspect current and previous logs, then compare the container's command, arguments, environment variables, mounted files, and dependency endpoints with the application's expectations.

kubectl logs <pod> -n <namespace> -c <container>
kubectl logs <pod> -n <namespace> -c <container> --previous
kubectl get pod <pod> -n <namespace> -o yaml

A pod may restart because the process exits, a liveness probe fails, or the container is terminated for resource pressure. Review the termination reason and exit code before changing probe thresholds or resource settings. For a focused reference, see the CrashLoopBackOff troubleshooting guide.

Running but not ready: A readiness probe failure normally means Kubernetes should not send traffic to the pod. Test the configured path, port, scheme, and expected response from inside the relevant network context. Confirm that the application is listening on the address and port named by the probe.

4. Test service discovery and routing

When a Kubernetes service is not working, inspect its selector, ports, and endpoints.

kubectl get svc <service> -n <namespace> -o yaml
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>
kubectl get pods -n <namespace> --show-labels

Compare the service selector with pod labels. A selector mismatch can leave a service without usable endpoints even though the pods appear healthy. Then compare the service port and target port with the container port and application listener. Test the service from a temporary diagnostic pod or an existing pod with suitable network tools, while remembering that network policies may make results differ between namespaces.

5. Troubleshoot ingress and external access

Separate ingress problems into three paths: DNS and load-balancer reachability, ingress-controller routing, and backend service connectivity. Confirm that the hostname resolves to the expected entry point, that the ingress resource uses the intended class, and that its rules reference an existing service and port.

kubectl get ingress <ingress> -n <namespace> -o yaml
kubectl describe ingress <ingress> -n <namespace>
kubectl get pods -n <ingress-controller-namespace>

Review ingress-controller logs for rejected configuration, TLS errors, or upstream connection failures. If the controller receives the request but returns an upstream error, return to service endpoints and pod readiness. If the request never reaches the controller, investigate DNS, the load balancer, firewall rules, or network policy.

6. Check dependencies and recent changes

Once Kubernetes objects appear healthy, investigate databases, queues, identity providers, configuration stores, and external APIs. Verify that secrets and configuration values are present, mounted, and compatible with the deployed application. Correlate the incident with image changes, configuration updates, node maintenance, certificate rotation, or autoscaling activity. Avoid deleting pods as a first response; a restart may hide the evidence without removing the underlying fault.

Tools and handoffs

kubectl is the primary inspection tool, but a reliable handoff also includes application logs, metrics, traces, deployment history, and cloud-provider or load-balancer logs where applicable. Capture commands run, timestamps, observed conditions, hypotheses tested, and changes made. Redact tokens, passwords, private keys, and sensitive payloads before sharing output.

For recurring failures, convert the successful investigation into a runbook with a symptom, entry conditions, commands, expected evidence, escalation boundary, and rollback step. A service catalog can link that runbook to an owning team and deployment metadata; see the comparison of service catalog tools for planning considerations. During an active incident, use the on-call handoff checklist so that the next responder receives context rather than a list of unexplained commands.

Observability should support the same diagnostic layers. Metrics can show readiness, restarts, saturation, and request errors; logs explain process behavior; traces help locate latency or failures across services. Keep dashboards and alerts tied to actionable symptoms instead of creating alerts for every available Kubernetes field.

Quality checks

• Confirm the namespace and cluster context before running commands that can change resources.
• Check events, pod conditions, logs, and resource status before editing manifests.
• Form one explicit hypothesis at a time and record the evidence that supports or disproves it.
• Test both the direct path and the user path: pod, service, ingress, DNS, and external dependency as appropriate.
• After remediation, verify readiness, error rate, latency, restart count, and replica availability over a suitable observation period.
• Document the root cause, contributing conditions, temporary workaround, and permanent follow-up.
• Review resource requests, limits, and scheduling choices as part of prevention; the Kubernetes cost optimization checklist provides a separate review path.

When to revisit

Revisit this Kubernetes troubleshooting guide whenever the cluster version, ingress controller, service-mesh layer, networking implementation, observability stack, or deployment process changes. Update command examples when resource APIs or controller behavior changes, and add a new symptom when an incident exposes a gap in the runbook.

Make the review practical: choose one recent incident, follow the workflow from symptom to verification, and remove steps that no longer produce useful evidence. Then test the runbook in a non-production environment, confirm that permissions are sufficient but not excessive, and link it from the owning service's catalog entry. A short, current playbook is more useful during an outage than a comprehensive document that no responder trusts.