CrashLoopBackOff is one of the first pod states every Kubernetes operator learns to dread, and also one of the most misunderstood. It is not an error in itself — it is Kubernetes telling you that a container keeps starting, crashing, and being restarted, and that the kubelet is now deliberately waiting longer between each restart. This playbook walks through exactly how to read that signal, find the real root cause, and fix the most common ones, with the commands you actually run.
The key mindset shift: CrashLoopBackOff is a symptom, never a diagnosis. Your job is to get from the status to the crashed container's own logs and exit code, because that is where the truth lives.
What CrashLoopBackOff actually means
When a container exits, the kubelet restarts it according to the pod's restartPolicy (Always by default for Deployments). If the container keeps exiting, the kubelet applies an exponential back-off delay so it is not hammering a broken process: 10s, then 20s, 40s, 80s, and so on, capped at 5 minutes. While the pod is sitting in that penalty box, its status reads CrashLoopBackOff.
So the loop is: container starts → container exits (crash) → kubelet waits (back-off) → restarts → repeat. The RESTARTS count climbs, and the delay grows. Fix the reason the container exits and the loop ends on its own.
Prerequisites
kubectlconfigured against the target cluster and the right namespace (kubectl config current-context).- Read access to pods, logs, and events in that namespace.
- The pod name — grab it from
kubectl get pods.
Step 1: Confirm the state and the restart count
Start broad. List the pods and look at the STATUS and RESTARTS columns:
kubectl get pods -n my-namespace
kubectl get pod my-pod -n my-namespace -o wideA high, steadily-climbing RESTARTS value confirms the loop. The -o wide view also shows the node — useful later if the problem turns out to be node-level (memory pressure, disk).
Step 2: Read the crashed container's logs
This is the single most important step, and the one people skip. A crashed container's current logs are often empty because it just restarted. You almost always want the previous instance's logs:
kubectl logs my-pod -n my-namespace
kubectl logs my-pod -n my-namespace --previousIf the pod has more than one container, target the crashing one explicitly with -c:
kubectl logs my-pod -n my-namespace -c app-container --previousNine times out of ten the application prints the reason it died — a stack trace, a "connection refused", a "config file not found", a missing environment variable — right here.
Step 3: Describe the pod for events and exit codes
If the logs are silent (the process died before it could log, or the crash is outside the app), describe the pod. Read the Last State block and the Events at the bottom:
kubectl describe pod my-pod -n my-namespaceLook for a section like Last State: Terminated with a Reason and an Exit Code. You can pull the exit code directly:
kubectl get pod my-pod -n my-namespace -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'And scan recent cluster events, newest last:
kubectl get events -n my-namespace --sort-by=.lastTimestampExit codes worth memorising
| Exit code | Usual meaning |
|---|---|
| 0 | Clean exit. If a long-running service exits 0, its command probably finished instead of staying up. |
| 1 | General application error — check the app logs. |
| 126 | Command found but not executable (permissions / wrong binary format). |
| 127 | Command not found — bad command/args or missing binary in the image. |
| 137 | Killed by SIGKILL (128+9). Almost always OOMKilled — out of memory. |
| 139 | SIGSEGV (128+11) — segmentation fault in the process. |
| 143 | SIGTERM (128+15) — graceful termination that the app did not handle in time. |
CrashLoopBackOff vs other failing states
It is easy to waste time chasing the wrong problem because several pod states look similar in kubectl get pods. Knowing which one you have narrows the search immediately:
| Status | What it really means |
|---|---|
| CrashLoopBackOff | The container starts but keeps exiting; the kubelet is backing off between restarts. The image is fine — the process dies. Look at logs and exit code. |
| Error | The container exited non-zero on its most recent run and has not yet entered back-off. Same investigation as CrashLoopBackOff, just earlier in the loop. |
| ImagePullBackOff / ErrImagePull | Kubernetes cannot even pull the image — wrong tag, private registry without credentials, or a typo. Nothing to do with your code. |
| CreateContainerConfigError | A referenced ConfigMap or Secret key is missing, so the container never starts. Fix the reference, not the app. |
| RunContainerError | The container runtime failed to start it — often a bad mount, device, or securityContext. |
The practical rule: if the container runs and then dies, it is a CrashLoopBackOff investigation (logs + exit code). If it never runs, it is a pull or config problem, and the fix is in the manifest or the registry, not the application.
Common root causes and how to fix each
1. The application errors on startup
The most common cause. A bad database URL, an unreachable dependency, a malformed config, a failed migration — the app throws on boot and exits non-zero. The fix is driven entirely by the log line from Step 2. Reproduce locally with the same image and env if the message is cryptic.
2. A missing or wrong ConfigMap / Secret / env var
If the container needs a value that is not mounted or is empty, it often crashes immediately. Confirm the referenced objects actually exist:
kubectl get configmap,secret -n my-namespace
kubectl describe pod my-pod -n my-namespace | grep -A3 -i "env\|mount"A CreateContainerConfigError in describe is a dead giveaway that a referenced ConfigMap/Secret key is missing.
3. Exit code 137 — OOMKilled
The container exceeded its memory limit and the kernel killed it. describe shows Reason: OOMKilled. Either the limit is too low or the app has a leak. Raise the limit (and request) sensibly:
resources:
requests:
memory: "256Mi"
limits:
memory: "512Mi"Then confirm the node actually has headroom with kubectl describe node <node> before assuming the limit alone is the fix.
4. A failing liveness probe restarts a healthy app
An over-aggressive livenessProbe can kill an app that is simply slow to start, producing a crash loop where the app itself is fine. Give it time to boot and be realistic about the endpoint:
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3Symptom to watch for in describe: Liveness probe failed events immediately before each restart, with the app logs otherwise clean.
5. A bad command, args, or entrypoint
Exit code 127 (command not found) or 126 (not executable) points at the command/args in the manifest or a broken image entrypoint. Verify the binary exists and the path is right; a quick kubectl run debug pod with the same image and a shell entrypoint lets you poke around:
kubectl run debug --rm -it --image=my-image:tag --restart=Never -- sh6. A dependency is not ready yet
If the app crashes because the database or a downstream service is not up, the honest fix is application-level ret/back-off, not fighting Kubernetes. As a stopgap, an initContainer that waits for the dependency, or a readiness gate on the dependency, keeps the loop from starting.
7. Permissions and read-only filesystems
Containers running as non-root (or with readOnlyRootFilesystem: true) crash when they try to write where they cannot. Check the securityContext and mount a writable emptyDir for scratch paths the app insists on writing.
Step 4: Verify the fix
After you change the manifest, roll it out and watch the pod stabilise — the RESTARTS count should stop climbing and the status should settle on Running:
kubectl apply -f deployment.yaml
kubectl rollout status deployment/my-app -n my-namespace
kubectl get pods -n my-namespace -wGive it a couple of minutes past the longest back-off you saw. A pod that survives well beyond its previous crash interval is genuinely fixed, not just early in the next back-off window.
Key Takeaways
- CrashLoopBackOff is a symptom. The real answer is always in the crashed container's logs or its exit code.
- Always use
kubectl logs --previous— the current instance's logs are usually empty right after a restart. - Exit code 137 = OOMKilled. Fix the memory limit or the leak; 127/126 mean a bad command or permissions.
- An aggressive liveness probe can crash a healthy app. Tune
initialDelaySecondsandfailureThreshold. - Verify past the back-off window. Watch
RESTARTSstop climbing before you call it fixed.
Frequently Asked Questions
What does CrashLoopBackOff mean in Kubernetes?
It means a container in the pod keeps starting and then exiting, and the kubelet is now waiting progressively longer (up to 5 minutes) between restart attempts. It is a status describing the restart loop, not the underlying cause of the crash.
How do I see the logs of a container that already crashed?
Use kubectl logs <pod> --previous to read the previous (crashed) instance. Add -c <container> for a specific container in a multi-container pod.
What causes exit code 137 in a pod?
Exit code 137 is the process being killed by SIGKILL (128 + 9). In Kubernetes this is almost always OOMKilled — the container exceeded its memory limit. Raise the memory limit or fix the memory leak.
Why does my healthy app keep restarting?
Usually a failing livenessProbe. If the app is slow to start or the probe path/port is wrong, Kubernetes kills a perfectly healthy container. Increase initialDelaySeconds/failureThreshold and confirm the probe endpoint actually responds.
What is the difference between CrashLoopBackOff and ImagePullBackOff?
They fail at different stages. ImagePullBackOff means Kubernetes cannot pull the container image at all (bad tag, missing registry credentials), so the container never starts. CrashLoopBackOff means the image pulled fine and the container ran, but the process keeps exiting. The first is a manifest/registry fix; the second is a logs-and-exit-code investigation.
Disclosure: some links above are affiliate links — if you buy through them I may earn a small commission at no extra cost to you. Thanks for supporting the channel!