When a Kubernetes pod is not starting or becoming ready, the status alone rarely identifies the real cause. This Kubernetes troubleshooting guide provides a repeatable workflow for examining scheduling, image pulls, container exits, probes, resources, events, and service connectivity. Use it during an incident, then return to the tracking and review checklists after deployments or cluster changes.
Overview
A pod status is a symptom, not a diagnosis. Kubernetes reports conditions such as Pending, CrashLoopBackOff, and ImagePullBackOff based on what the control plane and node agent currently know. The underlying issue may be a missing secret, an unsatisfied scheduling constraint, an invalid image reference, an application configuration error, or a health check that does not match the application’s startup behavior.
Start with the workload rather than immediately deleting the pod. Identify whether the pod belongs to a Deployment, StatefulSet, Job, or another controller, because a manual fix to one pod may disappear when the controller creates a replacement. Also confirm the namespace and cluster context before running commands that could affect production.
kubectl config current-context
kubectl get pods -A
kubectl get pod <pod-name> -n <namespace> -o wide
kubectl get deploy,statefulset,job -n <namespace>The broad diagnostic sequence is:
- Confirm the exact pod, namespace, owner, node, and current phase.
- Read the pod description and recent events.
- Inspect container state, exit codes, and logs.
- Check configuration, secrets, image access, resources, and scheduling.
- Evaluate startup, readiness, and liveness probes separately.
- Test networking only after the container is running, unless the failure is clearly related to initialization or DNS.
- Apply the smallest reversible correction and watch the resulting rollout.
Avoid treating kubectl delete pod as a diagnosis. It can temporarily hide a symptom while leaving the deployment, image, configuration, or node problem unchanged.
What to track
Symptom-to-cause comparison
| Observed state | Common areas to inspect | Useful first checks |
|---|---|---|
Pending | Scheduling, resources, node health, affinity, taints, storage | kubectl describe pod, pod events, node capacity, PVC status |
CrashLoopBackOff | Application startup, configuration, dependencies, signals, probes | Current and previous logs, container exit code, environment and mounted files |
ImagePullBackOff | Image name or tag, registry access, credentials, network, architecture | Pod events, image field, image pull secret, registry response |
| Running but not ready | Readiness probe, listening address, service port, dependency health | Probe configuration, endpoints, container port, application logs |
| Terminated or failed | Exit code, resource limit, command, signal, Job behavior | Container status, previous logs, node and kubelet events |
Describe the pod and read events
The description combines pod conditions, container states, volumes, probes, scheduling details, and recent events. It is usually the highest-value first command after identifying the pod.
kubectl describe pod <pod-name> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestampEvent output varies by Kubernetes distribution, retention settings, and command-line version, so treat it as a time-sensitive clue rather than a permanent audit trail. Record important events during an incident. Look for messages about failed scheduling, rejected admission, mount failures, image authorization, probe failures, back-off delays, or node pressure.
Inspect logs and container state
For a container that starts and exits, compare current logs with logs from the previous instance:
kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*]}'The previous-log option is especially useful with CrashLoopBackOff, where the current container may have produced little output before restarting. Check the termination reason, exit code, signal, restart count, and whether the container was killed by a probe or resource limit.
Track scheduling and resource facts
A pod can remain pending because its requests cannot fit on any eligible node, even when the cluster appears to have unused capacity. Requests are evaluated against allocatable capacity and scheduling rules. Inspect the pod’s requests, limits, node selectors, affinity, tolerations, and priority alongside node conditions.
kubectl get pod <pod-name> -n <namespace> -o yaml
kubectl get nodes
kubectl describe node <node-name>Review resource sizing separately from this incident workflow when needed; requests that are too large can prevent scheduling, while limits that are too small can cause runtime failures. A resource change should be tested against the workload’s normal and peak behavior rather than made solely to clear a status.
Track image and configuration dependencies
For ImagePullBackOff, verify the complete image reference, including registry, repository, and tag. Then check whether the referenced image exists and whether the pod’s service account or image pull secret has permission to retrieve it. Confirm that the node can reach the registry and that the image architecture is compatible with the node pool.
For startup failures, compare the rendered Deployment or StatefulSet with the intended configuration. Check ConfigMaps, Secrets, mounted paths, command and argument overrides, and required environment variables. A secret can exist in the cluster but still be unavailable if it is in the wrong namespace or is referenced under the wrong key.
Cadence and checkpoints
Troubleshooting is most effective when operational checks happen before an incident. Use the following cadence as a practical baseline, adapting it to the change rate and criticality of each workload.
At every deployment
- Watch the rollout rather than stopping at a successful image push:
kubectl rollout status deployment/<name> -n <namespace>. - Check pod readiness, restart counts, recent events, and application logs.
- Confirm that the Service has the expected endpoints and that the new pods are selected by the intended labels.
- Record image digests, configuration changes, probe changes, and resource changes with the deployment record.
After a node, registry, or configuration change
- Inspect nodes for pressure, readiness changes, taints, and unexpected capacity differences.
- Test a representative image pull and verify registry credentials without exposing secret values.
- Compare the rendered workload manifest with the version that previously worked.
- Review storage mounts, DNS resolution, and network policies if startup depends on external services.
Monthly or quarterly review
Maintain a small troubleshooting record for recurring variables: restart counts, pending duration, probe failures, failed rollouts, image-pull errors, node pressure events, and the most common remediation. Review whether alerts identify the affected workload and whether logs contain a useful startup marker. Recheck runbooks after Kubernetes, the container runtime, a managed cluster platform, or a major ingress, storage, or networking component changes.
How to interpret changes
Pending versus running but unready
A pending pod has not completed scheduling or initialization. Focus first on events, nodes, requests, constraints, storage, and admission. A running but unready pod has been placed and started, but its readiness condition is false. Focus on the readiness probe, application listener, required dependencies, and Service selection. Do not increase resource limits or restart nodes without evidence that those areas are involved.
CrashLoopBackOff versus probe failure
CrashLoopBackOff means the container is repeatedly failing and Kubernetes is spacing out restarts. The application may be exiting on its own, receiving a signal, failing to load configuration, or being killed by the node. A liveness probe can also restart a process that is running but considered unhealthy. Use container state and previous logs to distinguish an application exit from a probe-triggered restart.
Readiness is not liveness
Readiness controls whether a pod should receive traffic. Liveness determines whether Kubernetes should restart a container. Startup probes can give slow-starting applications time to initialize before liveness and readiness checks take effect. Probe paths, ports, schemes, initial delays, timeouts, and failure thresholds must match the application’s actual behavior. A probe that checks a dependency too aggressively can create a restart cycle, while a probe that checks only process existence may allow a broken instance to receive traffic.
Use networking checks at the right layer
First establish whether the container is listening on the expected interface and port. Then verify the Service selector and endpoints, followed by DNS and network policy behavior. A Service port and target port can be syntactically valid while still pointing to the wrong container port. Test from an appropriate pod or approved debugging environment, and avoid installing tools into production images as a first response.
kubectl get svc <service-name> -n <namespace> -o yaml
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name>
kubectl get networkpolicy -n <namespace>When deeper inspection is necessary, use an ephemeral or purpose-built debugging container if the cluster policy and Kubernetes version support it. Keep the session temporary, document commands, and avoid copying sensitive environment variables into logs or tickets.
When to revisit
Revisit this guide immediately after a failed deployment, repeated pod restart, node replacement, registry incident, certificate or secret rotation, probe change, resource-policy change, or networking modification. These events alter the assumptions behind a previously successful workload.
For routine maintenance, review the workflow monthly or quarterly. Check whether command output still matches the Kubernetes and managed-platform versions your teams operate, whether event retention is sufficient for incident review, and whether workload APIs or container-runtime behavior have changed. Validate commands in a non-production namespace before adding them to an automated runbook.
Incident-ready checklist
- Confirm context, namespace, workload owner, pod phase, node, and deployment revision.
- Capture
kubectl describe pod, relevant events, current logs, previous logs, and container status. - Classify the failure as scheduling, image, configuration, application exit, probe, storage, or networking.
- Compare the failing revision with the last known healthy revision.
- Make one reversible change, then watch pod conditions, events, logs, and rollout status.
- Document the root cause, evidence, remediation, and a preventive check for the next review.
Keep this checklist beside your deployment and on-call documentation. A consistent evidence trail turns Kubernetes troubleshooting from trial-and-error into a controlled diagnostic process.
