Kubernetes Troubleshooting Guide: A Repeatable Workflow for Pods, Services, and Deployments

Overview

Kubernetes troubleshooting works best when you move from the outside of the system toward the failing component. Start by confirming the cluster context and namespace, then establish whether the problem affects one pod, a workload, a service, or the wider cluster. This prevents a common mistake: changing application configuration before confirming that the request is reaching the application at all.

A practical diagnostic order is:

1. Confirm the active cluster and namespace.
2. Inspect workload status and recent events.
3. Check pod states, container restarts, and scheduling details.
4. Review logs and health probe results.
5. Test service discovery and endpoint selection.
6. Compare resource usage with requests, limits, and node capacity.
7. Review deployment history before rolling back or changing configuration.

Use read-only commands first. Capture the output that explains the symptom before making a corrective change. That record helps the team distinguish between a confirmed cause, a working hypothesis, and an unrelated warning.

kubectl config current-context
kubectl get namespaces
kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Replace the placeholders with values from your environment, and be especially careful with commands that can delete resources, restart workloads, or alter traffic. A troubleshooting runbook should make safe investigation easy without hiding the operational impact of later actions.

What to track

Context and scope

Record the cluster context, namespace, workload name, deployment revision, and approximate start time of the problem. A command run against the wrong context can produce valid-looking output from the wrong environment. Begin each investigation with:

kubectl config current-context
kubectl get deployment -n <namespace>
kubectl get pods -n <namespace> -o wide

Then define the scope. Is one replica failing, are all replicas affected, or do multiple namespaces show the same symptom? Scope often separates an application issue from a node, networking, admission, or cluster-capacity issue.

Pod states and events

Pod status is a starting point, not a diagnosis. For a failing pod, inspect its full description and events:

kubectl describe pod <pod-name> -n <namespace>
kubectl get pod <pod-name> -n <namespace> -o yaml

Look for states such as Pending, CrashLoopBackOff, ImagePullBackOff, and Terminating. In Pending, scheduling events may point to insufficient resources, node selectors, taints, affinity rules, or unfulfilled volume requirements. Image-related states suggest checking the image reference, registry access, pull credentials, and the node's ability to retrieve the image. Repeated restarts require both the current container state and the previous container's logs.

Logs and container behavior

Read logs for the specific container when a pod has more than one:

kubectl logs <pod-name> -n <namespace> -c <container-name>
kubectl logs <pod-name> -n <namespace> -c <container-name> --previous
kubectl logs <pod-name> -n <namespace> --tail=200

Compare application errors with restart times and Kubernetes events. A log line alone is not proof of the root cause; establish whether it occurs before the failure, during it, or after recovery. If logs are structured as JSON, formatting the payload can make nested error fields easier to inspect. The online JSON formatter and validator guide can help when an event or API response is difficult to read.

Deployments, replicas, and rollouts

For deployment troubleshooting, compare the desired state with the available state:

kubectl get deployment <deployment-name> -n <namespace>
kubectl describe deployment <deployment-name> -n <namespace>
kubectl rollout status deployment/<deployment-name> -n <namespace>
kubectl rollout history deployment/<deployment-name> -n <namespace>

Check whether new replicas are ready, whether old replicas are still serving traffic, and whether a rollout is paused or blocked. Review the deployed image, environment variables, mounted configuration, and probe definitions. If the failure began immediately after a change, preserve the rollout evidence before considering a rollback:

kubectl rollout undo deployment/<deployment-name> -n <namespace>

Use rollback only when it matches your release process and the team understands the consequences. It may restore an earlier application version while leaving a configuration, data, or dependency problem unresolved.

Services, endpoints, and networking

When a service is unreachable, verify each layer separately:

kubectl get service <service-name> -n <namespace>
kubectl describe service <service-name> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service-name>

Confirm that the service selector matches pod labels and that the endpoint list contains ready backends. A service can exist and resolve correctly while sending traffic to no usable pods. If endpoints are present, test name resolution and connectivity from an appropriate diagnostic pod or from a client pod in the same network context. Check the target port, container port, network policies, ingress configuration, and application listener address. This layered approach is more reliable than changing service fields until traffic happens to work.

Probes and resources

Readiness, liveness, and startup probes have different operational purposes. A readiness failure usually removes a pod from service traffic, while a liveness failure can trigger a restart. A startup probe can give a slow-starting application time to initialize before other checks become active. Inspect probe paths, ports, schemes, timeouts, delays, and thresholds alongside application logs.

For resource-related symptoms, inspect requests and limits in the workload specification and compare them with available node capacity. If metrics are available in the cluster, use them as supporting evidence rather than as a substitute for events and application behavior:

kubectl top pods -n <namespace>
kubectl top nodes

Resource pressure can appear as scheduling failures, evictions, throttling, slow startup, or repeated restarts. Track the symptom, the affected node or workload, and whether the pattern is isolated or recurring.

Cadence and checkpoints

Use this workflow during incidents, but also review the signals before they become incidents. A useful monthly checkpoint is a short review of recurring pod restarts, failed rollouts, warning events, probe failures, pending pods, and service endpoint changes. Look for workloads that repeatedly recover without an obvious explanation; silent recovery can hide a capacity, dependency, or configuration problem.

Use a quarterly checkpoint for broader operational maintenance:

• Test that team members can identify the correct context and namespace safely.
• Review whether runbook commands still match the cluster's access controls and tooling.
• Remove obsolete dashboards, alerts, namespaces, and deployment references.
• Inspect probe settings and resource requests after meaningful application changes.
• Review deployment and rollback procedures with the people who respond to incidents.
• Check whether logs and events retain enough context for diagnosis.

Automate collection where it is safe to do so. Scheduled checks can report restart counts, pending workloads, failed jobs, or deployments that remain unavailable. Avoid treating every warning as an incident; the purpose of automation is to surface patterns for review. The DevOps automation workflow guide covers ways to combine scheduled checks with CI/CD and controlled validation.

How to interpret changes

Interpret Kubernetes signals as relationships rather than isolated values. For example, a restart becomes more meaningful when it aligns with an out-of-memory termination, a failed liveness probe, a deployment revision, or a node condition. A service error becomes more specific when endpoint slices are empty, selectors do not match labels, or only one backend fails its readiness check.

Use a simple evidence table during an investigation:

Do not assume that the newest event is the root cause. Events can be symptoms, retries, or consequences of an earlier failure. Establish a timeline using deployment changes, pod creation, probe failures, restarts, and application logs. If the issue involves multiple systems, document the boundary where evidence stops: for example, the service selects a ready endpoint, but the application does not complete the request.

When to revisit

Revisit this guide after a Kubernetes version change, a new ingress or service-mesh layer, a change to cluster networking, a migration to a different container registry, or a substantial revision to deployment and observability tooling. These changes can alter command output, available metrics, event details, access requirements, or the meaning of a familiar symptom.

Update the runbook after every incident that exposes a missing diagnostic step. Add the exact symptom, the evidence that confirmed the cause, the safe commands used, and the corrective action. Remove steps that no longer apply. Keep environment-specific values out of the general guide; link to an internal inventory or service catalog instead.

For the next incident, use this short action list:

1. Confirm context, namespace, scope, and incident time.
2. Capture workload status, pod descriptions, events, and relevant logs.
3. Trace traffic from service selector to endpoint to container port.
4. Check probes, resource conditions, node health, and recent rollout history.
5. State the evidence-based hypothesis before changing anything.
6. Record the result and update the runbook during the follow-up review.

Keep the workflow visible to developers, platform engineers, and incident responders. Consistent evidence collection reduces guesswork, makes handoffs clearer, and gives your team a practical way to improve Kubernetes operations over time.