Engineering problem

Kubernetes Troubleshooting: Pods, Services and Rollouts

Classify a failed Kubernetes workload using status, previous logs, endpoint membership, and a read-only incident timeline.

Editorial update

Decision in brief

Scope the incident to the affected cluster, namespace, workload, and release before changing anything. Use Pod state to locate scheduling and startup failures, then use ready endpoints and a known-good comparison to locate traffic failures. Revert a recent application change through the agreed rollout procedure when it fits the evidence and the service objective.

Triage a rollout: Scope the incident. Check Pod state. Check endpoints.
Compare with a known-good release before deciding whether to roll back.

Start with the observable symptom

From an authorized context, inspect the relevant deployment, Pods, and recent events with kubectl get and kubectl describe. Collect prior logs with kubectl logs POD -c CONTAINER --previous where available. Correlate timestamps with deployment changes and dependency symptoms before issuing a restart. Keep logs and event exports scoped and scrub sensitive values.

If Pods are ready but requests fail, check Service selector labels, EndpointSlices, target ports, and the client path. Investigate DNS and ingress separately if the internal Service path works. Treat a displayed status such as CrashLoopBackOff as a pointer to earlier termination evidence and probe events.

Observed symptomFirst evidenceWhy
Pending PodScheduler events and volume claimsDistinguishes resource constraints, placement rules, and unbound storage
ImagePullBackOffEvent reason, image reference, and approved registry pathSeparates missing tags and credentials from network or policy issues
CrashLoopBackOffPrevious container log, exit state, and probe eventsKeeps first failure evidence separate from backoff behavior
Ready Pods but failed requestsEndpointSlices, label selector, and traffic pathSeparates missing backends from downstream DNS or ingress problems
Progress deadline exceededDeployment conditions and new ReplicaSet eventsHelps decide whether to pause or revert through the release system

Make incident evidence available after the failure

In the Radar launch thread, creator nadaverell described deriving a Kubernetes interface from internal debugging tooling. Test whether the responder can reach the relevant Pod, logs, and owner through whatever interface the team actually uses.

In a separate Show HN, simalapo introduced Kube-insight to retain Kubernetes evidence for incident investigations. Keep a sanitized record of the first failing event, the deployed image digest, and the roll-forward or rollback outcome. Verify that the record remains available after a Pod is replaced.

Exercise three failure classes in a test namespace

  1. Record a known-good deployment, image digest, visible endpoints, and request trace in a disposable namespace. Check that the responder can collect logs and events with the approved role.
  2. Inject one benign fixture at a time, such as a missing image tag, a wrong readiness path, and a Service selector mismatch. Let the responder classify each while preserving previous logs and relevant events.
  3. Restore the approved configuration through the normal delivery path and verify rollout completion, ready endpoints, and successful synthetic traffic. Record an explanation that another responder can reproduce from the timeline.
  4. Pass if all three failures are classified and recovered within the agreed incident target, such as 15 minutes each for this drill. Stop if the proposed fix would require cluster-wide changes before the affected scope is understood.

Related articles