Kubernetes Preproduction Troubleshooting Guide: Pods, Services, Ingress, and Config Failures
preproductionkubernetescontainerstroubleshootingdeployment-debugging

Kubernetes Preproduction Troubleshooting Guide: Pods, Services, Ingress, and Config Failures

PPreprod Cloud Editorial Team
2026-11-15
8 min read

A reusable Kubernetes checklist for diagnosing preproduction failures involving Pods, images, probes, Services, ingress, configuration, and storage.

A Kubernetes deployment can work locally and fail in preproduction because the environments differ in scheduling capacity, image access, configuration, networking, permissions, or dependent services. This guide provides a repeatable troubleshooting workflow, a symptom-to-command reference, and a practical checklist for isolating the failure before changing multiple variables at once.

Overview

Start with the deployment’s observable state, then move outward through the Kubernetes control plane, workload, network, and external dependencies. The goal is to identify the first failing layer rather than treating every downstream error as a separate incident.

  1. Confirm the target: Verify the cluster context, namespace, release, image tag, and deployment revision. A surprisingly common cause of confusion is inspecting one namespace while the CI/CD pipeline deployed to another.
  2. Inspect workload state: Check whether Pods are pending, starting, running but unready, restarting, or terminating. Each state suggests a different investigation.
  3. Read events and logs: Kubernetes events often explain scheduling, mounting, probing, and image errors. Container logs show application startup failures that events cannot explain.
  4. Trace dependencies: Validate ConfigMaps, Secrets, Services, DNS, ingress routing, persistent volumes, and external endpoints in that order where relevant.
  5. Change one variable at a time: Record the hypothesis, command output, remediation, and verification result. This preserves a useful timeline and reduces accidental fixes that cannot be reproduced.

Use a namespace-scoped workflow whenever possible. Replace <namespace>, <deployment>, and <pod> in the examples below with the values for your environment.

kubectl config current-context
kubectl get deploy,pods,svc,ingress -n <namespace>
kubectl describe deployment <deployment> -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Checklist by scenario

Pods are pending or never start

A Pending Pod has not been placed on a node or cannot complete a required admission or volume step. Inspect the Pod and its events first.

kubectl get pods -n <namespace> -o wide
kubectl describe pod <pod> -n <namespace>
kubectl get nodes
kubectl describe node <node>
  • Look for insufficient CPU or memory, taints, node selectors, affinity rules, or an unavailable topology constraint.
  • Compare the preproduction resource requests with available node capacity. Local Docker or a local Kubernetes cluster may provide a very different resource profile.
  • Check whether a PersistentVolumeClaim is still pending or cannot bind to a compatible volume.
  • Confirm that the workload’s service account and admission requirements are accepted by the cluster.

Images fail to pull or the wrong image runs

ImagePullBackOff, ErrImagePull, and similar states usually point to a registry, tag, credentials, or network problem.

kubectl describe pod <pod> -n <namespace>
kubectl get pod <pod> -n <namespace> -o jsonpath='{.spec.containers[*].image}'
kubectl get serviceaccount <service-account> -n <namespace> -o yaml
  • Confirm that the image tag or digest exists and that the deployment references the intended build.
  • Check image-pull credentials and whether the correct Secret is attached to the Pod or service account.
  • Verify that preproduction nodes can reach the registry through their network path.
  • Do not assume that a locally available image is available to the cluster. Local images often bypass the registry entirely.

Pods run but remain unready

A running container is not necessarily ready to receive traffic. Readiness probes can fail because the application binds to the wrong port, starts slowly, requires missing configuration, or cannot reach a dependency.

kubectl get pod <pod> -n <namespace> -o yaml
kubectl describe pod <pod> -n <namespace>
kubectl logs <pod> -n <namespace> --all-containers=true
kubectl logs <pod> -n <namespace> --previous
  • Compare the probe path, port, scheme, initial delay, timeout, and failure threshold with the application’s actual startup behavior.
  • Test the endpoint from inside the Pod when an appropriate shell or diagnostic container is available.
  • Separate startup failures from dependency failures. A health endpoint that checks a database may remain unready when the web process itself is functioning.
  • Inspect restart counts and termination reasons for crash loops or out-of-memory terminations.

Services do not route traffic

When a Service has no working backend, inspect selectors and endpoints before debugging ingress. A selector mismatch can leave a perfectly healthy Pod unreachable.

kubectl get svc <service> -n <namespace> -o yaml
kubectl get endpoints <service> -n <namespace>
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>
kubectl get pods -n <namespace> --show-labels
  • Compare the Service selector with the labels on the intended Pods.
  • Confirm that only ready Pods are being selected when readiness gating is expected.
  • Check the Service port, targetPort, and named-port mapping. The target port must match the container’s listening port or named port.
  • Test DNS and the Service name from another Pod in the same namespace or from the expected client namespace.

Ingress returns 404, 502, timeout, or TLS errors

Ingress troubleshooting has two paths: the request may not match a rule, or the ingress controller may match the rule but fail to reach the backend.

kubectl get ingress -n <namespace>
kubectl describe ingress <ingress> -n <namespace>
kubectl get ingressclass
kubectl get svc <service> -n <namespace>
# Use the controller's namespace and label for your installation:
kubectl logs -n <controller-namespace> -l app.kubernetes.io/component=controller
  • Verify the host header, path, path type, ingress class, and backend Service name.
  • Confirm that DNS points to the intended preproduction entry point and that the request uses the expected hostname.
  • For TLS failures, check the referenced Secret, certificate hostname, and controller events.
  • For 502 or timeout responses, test the backend Service directly before changing ingress rules.

Configuration, Secrets, or volumes fail

Preproduction commonly differs from local development in Secret names, environment variables, identity permissions, storage classes, and external service endpoints.

kubectl get configmap -n <namespace>
kubectl get secret -n <namespace>
kubectl describe pod <pod> -n <namespace>
kubectl get pvc -n <namespace>
kubectl describe pvc <claim> -n <namespace>
  • Check that referenced keys exist without printing Secret values into tickets or chat.
  • Confirm that the rendered manifest uses the intended namespace and configuration overlay.
  • For volumes, inspect claim status, storage class, access mode, capacity, and node or zone constraints.
  • Verify file permissions and mount paths inside the container when the application reports a missing or unwritable file.

Symptom-to-command reference

SymptomFirst commandsWhat to look for
Pod Pendingkubectl describe pod, kubectl get eventsCapacity, taints, affinity, PVC, admission messages
ImagePullBackOffkubectl describe podTag, registry access, pull credentials
CrashLoopBackOffkubectl logs, kubectl logs --previousStartup error, exit code, missing configuration
NotReadykubectl describe pod, kubectl get pod -o yamlProbe path, port, timing, dependency status
Service unavailablekubectl get endpoints, kubectl get pods --show-labelsSelector mismatch, no ready endpoints, wrong port
Ingress 404/502kubectl describe ingress, controller logsRule match, ingress class, backend reachability
PVC Pendingkubectl describe pvc, kubectl get storageclassBinding, access mode, storage class, topology

What to double-check

Before applying a fix, compare the preproduction deployment with the known-good local or development configuration, but do not assume that copying local settings is safe. Review the rendered manifests produced by the release process, not only the source templates.

  • Context and namespace: Run kubectl config current-context and explicitly include -n in commands.
  • Release revision: Confirm the deployment’s image, annotations, rollout history, and commit or build identifier.
  • Environment variables: Check names, required keys, URLs, ports, feature flags, and formatting. Avoid exposing Secret values.
  • Identity and permissions: Verify the service account, cloud identity mapping where applicable, and permissions required by the application.
  • Resource behavior: Compare requests, limits, node capacity, and observed usage. A limit change should be tested deliberately rather than used as a blanket remedy.
  • Network assumptions: Validate DNS, egress rules, network policies, firewall paths, and dependency allowlists.
  • Observability: Correlate Kubernetes events with application logs, metrics, traces, and deployment signals. The preproduction observability guide provides a broader method for doing this.

Use the secrets and configuration workflow when configuration is involved, and compare environment assumptions with the guidance on staging, preproduction, and production boundaries.

A troubleshooting record that remains useful

Incident or rollout:
Time and cluster context:
Namespace and workload:
Observed symptom:
First failing layer:
Hypothesis:
Commands and relevant output:
Change applied:
Verification result:
Rollback or follow-up needed:

Common mistakes

  • Restarting before collecting evidence: A restart may remove the state needed to understand a crash or probe failure. Capture describe output, events, logs, and the current revision first.
  • Debugging ingress before the Service: Test the Service and its endpoints directly. Ingress cannot repair a selector mismatch or an application that is not listening.
  • Using localhost for a dependency: Inside a Pod, localhost refers to that Pod’s network namespace, not another Kubernetes Service.
  • Changing several manifests simultaneously: Multiple edits make verification ambiguous and can conceal a second failure.
  • Ignoring generated configuration: Templates, Helm values, Kustomize overlays, admission mutation, and CI/CD substitutions can change the final workload.
  • Printing sensitive data while debugging: Use key names, Secret metadata, and redacted output. Do not paste tokens, passwords, private keys, or full environment dumps into shared channels.
  • Assuming a successful rollout means a successful release: A rollout can complete while traffic, background jobs, migrations, or downstream calls remain broken. Validate the user journey and relevant dependencies.

When to revisit

Keep this Kubernetes troubleshooting checklist close to the release workflow and revisit it whenever the underlying inputs change. Update command examples and local operating notes when the cluster version, kubectl behavior, ingress controller, storage implementation, or managed-cluster defaults change. Controller label selectors and provider-specific diagnostics should always be checked against the installation in use rather than copied blindly from another cluster.

Run a short review before seasonal planning cycles, after a platform upgrade, when onboarding a new service, or when a deployment pattern changes from a shared environment to an ephemeral one. Also revisit the guide after a significant incident: add the first useful signal, the command that exposed it, and the verification step that proved the fix. The preproduction environment checklist and release readiness template can help turn those lessons into repeatable checks.

Before your next rollout: verify the context and namespace, inspect the rendered manifest, confirm image access, review probes and resources, test Service endpoints, validate ingress rules, check configuration and volumes, and define the rollback or recovery path. If the rollout fails, record evidence before changing it, isolate the first failing layer, make one controlled adjustment, and verify the complete request path afterward.

Related Topics

#preproduction#kubernetes#containers#troubleshooting#deployment-debugging
P

Preprod Cloud Editorial Team

DevOps and Platform Engineering Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.