Kubernetes Deployment Troubleshooting Runbook: From Failed Pods to Unhealthy Services

Overview

Kubernetes reports deployment problems through several layers. A Deployment may exist while its ReplicaSet cannot create healthy Pods. Pods may be scheduled but fail during startup. Containers may run but fail readiness checks, leaving a Service with no usable endpoints. A Service may have endpoints but still be unreachable because of a selector, port, network policy, or application-level issue.

A useful Kubernetes troubleshooting guide therefore follows the path of a request and the lifecycle of a workload:

1. Confirm the affected namespace and workload.
2. Inspect the Deployment, ReplicaSet, and Pod states.
3. Read events and container logs before changing configuration.
4. Check scheduling, images, configuration, probes, and resources.
5. Trace traffic from the Service to its selected Pods.
6. Record the cause, fix, and verification steps for the next incident.

Run commands against the correct context and namespace. A simple starting point is:

kubectl config current-context
kubectl get namespaces
kubectl get deploy,rs,pods -n <namespace>

If the environment is shared, verify the target context explicitly before using commands that modify resources. Treat production and preproduction as separate operating environments, with access and change controls appropriate to each. For broader environment practices, see the staging, preproduction, and production environment strategy checklist.

Step-by-step workflow

1. Establish the exact failure

Start with the workload name, namespace, release version, and time the problem began. These details prevent an investigation from mixing an old failed ReplicaSet with the current deployment.

kubectl get deploy <deployment> -n <namespace>
kubectl describe deploy <deployment> -n <namespace>
kubectl get rs,pods -l app=<label> -n <namespace> -o wide

Compare desired, current, ready, and available replicas. A Deployment with the expected number of replicas but no ready Pods points to a different problem than a Deployment that has not created a new ReplicaSet.

2. Inspect Pod status and events

Pod status is a direction, not a diagnosis. List Pods with their readiness and restart counts, then inspect the affected Pod:

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Events often identify scheduling failures, rejected images, missing configuration objects, failed mounts, probe failures, or resource constraints. Check the Events section at the bottom of describe output and note whether the message belongs to the current Pod attempt.

3. Resolve CrashLoopBackOff methodically

CrashLoopBackOff means a container is repeatedly starting and stopping with increasing delays between attempts. It does not identify the reason for the crash. Check the current and previous container logs:

kubectl logs <pod> -n <namespace> -c <container>
kubectl logs <pod> -n <namespace> -c <container> --previous
kubectl describe pod <pod> -n <namespace>

Look for an invalid command or argument, an application startup error, a missing environment variable, an unavailable dependency, a permissions issue, or a process that exits immediately by design. The previous log is especially useful when the current container has not stayed alive long enough to produce output.

Confirm the container command and arguments in the rendered Pod specification rather than assuming they match the image defaults. If the application requires a database, queue, or external API at startup, verify the dependency and distinguish a temporary dependency failure from a bad application configuration.

4. Resolve ImagePullBackOff and ErrImagePull

For ImagePullBackOff or ErrImagePull, inspect the exact image reference and the event message:

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

Check for a misspelled repository or tag, an image that has not been published, registry connectivity, and credentials required by a private registry. Confirm that the intended image was promoted to the target environment; a CI/CD pipeline can succeed while deployment fails if the referenced artifact is unavailable. The container image promotion workflow provides useful context for separating build, promotion, and deployment checks.

5. Investigate Pending Pods

A Pending Pod has not reached a runnable state. The scheduler event usually gives the first actionable clue. Common categories include insufficient CPU or memory, node selectors or affinity rules that match no nodes, taints without tolerations, unavailable persistent volumes, and quota limits.

kubectl describe pod <pod> -n <namespace>
kubectl get nodes
kubectl describe nodes
kubectl get resourcequota,limitrange -n <namespace>

Compare the Pod's resource requests with allocatable capacity and check whether recent placement rules made the workload unschedulable. Avoid simply reducing requests to force scheduling; first confirm the application's operational requirements and the capacity plan.

6. Check probes and application health

A container can be running while remaining unavailable. A failed liveness probe may restart a healthy-but-slow application, while a failed readiness probe keeps it out of Service endpoints. Review the probe path, port, scheme, headers, timeout, initial delay, and failure thresholds against the application actually running in the container.

kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> -c <container>
kubectl get pod <pod> -n <namespace> -o yaml

Test the health endpoint from an appropriate location, such as an ephemeral debugging container or a temporary diagnostic Pod. A probe should represent the condition it is intended to measure: readiness should indicate whether the Pod can receive traffic, and liveness should be reserved for a condition where restarting the process is a reasonable recovery action.

7. Trace Service routing

When Pods are ready but the application is unreachable, follow the routing chain: Service selector, Endpoints or EndpointSlices, Service port, target port, and application listener.

kubectl get svc <service> -n <namespace> -o yaml
kubectl get endpoints,endpointslices -n <namespace>
kubectl get pods --show-labels -n <namespace>

A selector mismatch can leave a Service with no endpoints. A port mismatch can connect to the wrong container port or to no listener. If the Service has endpoints, test connectivity from inside the cluster and then continue outward through ingress, load balancing, DNS, or network policy. This separates Kubernetes routing issues from application and external network issues.

8. Check configuration and resource exhaustion

Missing ConfigMaps, Secrets, volume mounts, or environment variables can prevent startup or cause subtle runtime failures. Confirm that referenced objects exist in the same namespace and that the rendered Pod specification contains the expected references. Do not print sensitive Secret values into tickets or chat.

For suspected resource exhaustion, inspect restarts, termination reasons, node conditions, and workload metrics where available. An OOMKilled termination suggests that the container exceeded its memory limit or the node experienced memory pressure. CPU throttling, disk pressure, and ephemeral-storage limits can produce different symptoms. Record the observed evidence before changing limits, requests, or autoscaling settings.

Tools and handoffs

kubectl is the primary inspection tool, but reliable handoffs depend on combining several sources:

• Cluster state: Deployments, Pods, Services, nodes, events, and rendered manifests.
• Application evidence: Current and previous logs, exit codes, stack traces, and startup timing.
• Observability: Request failures, latency, saturation, restart counts, and resource trends.
• Delivery records: Commit, image digest or tag, manifest change, rollout status, and rollback activity.
• Ownership context: The service owner, dependency owners, incident channel, and change record.

Useful handoff notes answer four questions: what changed, what is failing, what evidence supports the current hypothesis, and what action is safe next. Include namespace, workload, timestamps, relevant event messages, and the exact command output needed by the next responder. Keep secrets, tokens, and unnecessary personal data out of shared diagnostics.

Automated delivery should catch preventable errors before they reach a cluster. Add manifest validation, image availability checks, rollout status checks, and smoke tests to the CI/CD pipeline quality gates. If infrastructure changes are involved, review them alongside the Terraform workflow and troubleshooting guide.

Quality checks

Before declaring a Kubernetes deployment recovered, verify more than the absence of an error message:

1. The intended image and configuration are running in the intended namespace.
2. Desired and ready replica counts remain stable through more than one check.
3. Pods pass readiness checks without an unusual restart increase.
4. The Service has the expected endpoints and routes to the correct port.
5. A representative request succeeds from the relevant network location.
6. Logs and metrics show no continuing error, saturation, or dependency failure.
7. The deployment, rollback, or configuration change is recorded for audit and learning.

Prefer a controlled rollback when the failure is linked to a recent release and recovery is time-sensitive. After service is stable, reproduce the failure in a safe environment where possible and update the deployment or pipeline checks so the same class of error is easier to detect earlier.

When to revisit

Review this runbook whenever the cluster platform, ingress path, observability stack, deployment controller, or organizational ownership model changes. Update command examples when resource types, output formats, or standard debugging tools change. Revisit it after a significant incident, especially if responders needed undocumented access, a missing diagnostic command, or a workaround that should become a normal quality gate.

Keep a short service-specific appendix for non-obvious details: required dependencies, expected startup time, health endpoints, safe rollback steps, resource assumptions, and escalation contacts. As a practical next step, choose one production or preproduction workload and walk through this workflow without making changes. Confirm that you can identify its Pods, events, logs, probes, Service endpoints, image reference, and resource settings. Any missing answer is a concrete improvement for the runbook, dashboard, or deployment pipeline.