CrashLoopBackOff

CrashLoopBackOff means a container in your Pod keeps starting and exiting, and the kubelet is waiting before it tries again. Each wait doubles: 10 seconds, then 20, then 40, up to a cap of five minutes, and the timer resets once the container has run for 10 minutes without trouble. It's a symptom. The cause is whatever made the process exit, and three commands find it: kubectl get pods to spot the Pod, kubectl describe pod <name> for the exit code and events, and kubectl logs <name> --previous for what the crashed run printed before it died.

This chapter breaks a Deployment on purpose, reads each of those outputs, maps the exit code to a cause, and then shows how to stop the loop cleanly. You'll need a cluster to follow along; the KinD chapter gets one running on a laptop in a few minutes.

What the back-off does, and what changed in 1.32 to 1.37#

A Pod's restartPolicy can be Always, OnFailure or Never, and it defaults to Always. A Deployment only accepts Always. So when the container in a Deployment's Pod exits, for any reason and with any exit code, the kubelet restarts it in place on the same node. The first restart is immediate. After that comes the exponential delay, and while the kubelet is sitting out that delay, kubectl shows CrashLoopBackOff in the STATUS column.

That STATUS value isn't the Pod's phase. The phase stays Running through the whole loop, which is why a crash-looping Pod can look healthy to a script that only checks phase.

Two feature gates touch the timing, and both appeared in recent releases:

• KubeletCrashLoopBackOffMax arrived as alpha in 1.32 and has been beta, on by default, since 1.35. It lets a cluster admin lower the five-minute cap per node with crashLoopBackOff.maxContainerRestartPeriod in the kubelet configuration, anywhere from "1s" to "300s".
• ReduceDefaultCrashLoopBackOffDecay is still alpha in 1.37 and off by default. With it switched on, delays start at 1 second and stop growing at 60.

Neither changes anything in your Pod spec. Unless someone has edited the kubelet config on your nodes, the 10-second start and 300-second cap above are what you'll see.

Make a Pod crash on purpose#

A failure you caused yourself is the easiest one to read. This Deployment runs busybox with the command false, which exits with code 1 the moment it starts:

kubectl create deployment crasher --image=busybox:1.28 -- false
kubectl get pods --watch

Within a minute or so the output looks like this (names and timings will differ):

NAME                       READY   STATUS             RESTARTS      AGE
crasher-6d8f7b9c4d-x2k9p   0/1     Error              1 (3s ago)    5s
crasher-6d8f7b9c4d-x2k9p   0/1     CrashLoopBackOff   1 (5s ago)    7s
crasher-6d8f7b9c4d-x2k9p   0/1     Error              2 (17s ago)   19s

Watch the STATUS flip between Error and CrashLoopBackOff while RESTARTS climbs. Press Ctrl+C to stop watching. If you haven't met Deployments yet, the Kubernetes Deployment chapter explains what kubectl create deployment just built.

Read the container's last state with describe#

Copy the Pod name from your own output and describe it:

kubectl describe pod crasher-6d8f7b9c4d-x2k9p

Skip the top of the output. The part that matters sits under Containers:, and it looks like this:

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      ...
      Finished:     ...
    Ready:          False
    Restart Count:  4
...
Events:
  Type     Reason   Age   From     Message
  ----     ------   ----  ----     -------
  Warning  BackOff  12s   kubelet  Back-off restarting failed container busybox in pod crasher-6d8f7b9c4d-x2k9p_default(...)

State tells you only that the kubelet is waiting. Last State is the run that died, and it carries the two facts you came for: a Reason and an Exit Code. With containerd, Reason reads Completed for exit 0, Error for any other code, or OOMKilled when the kernel's out-of-memory killer ended the process. Started and Finished tell you how long it lived. A container that dies within a second of starting has a startup problem (command, config, permissions). One that runs happily for several minutes and only then dies points somewhere else entirely: memory, a probe, or a dependency that went away while it was running.

Events at the bottom add context the container status can't hold: probe failures, mount errors, image pulls. The same events are available on their own with kubectl events --for pod/<name>.

Read the logs of the run that crashed#

Plain kubectl logs shows the current container, which in a crash loop is often brand new or not started yet. Add --previous (short form -p) to get the instance that exited:

kubectl logs crasher-6d8f7b9c4d-x2k9p --previous

For crasher the result is empty, and that's a finding in itself. The process exited before it wrote a single line, so the application never got as far as its own error handling. Empty previous logs plus a fast exit usually mean the container was told to run the wrong thing, or couldn't read something it needed at the very start.

In a Pod with more than one container, name the one you want with -c <container>. That includes init containers: when one of them is failing, the Pod's STATUS reads Init:CrashLoopBackOff, and kubectl logs <pod> -c <init-container> --previous is where the reason is.

Some applications write their last words to a file in place of stdout. Kubernetes reads /dev/termination-log into the container status by default, and you can print it straight from the API:

kubectl get pod crasher-6d8f7b9c4d-x2k9p \
  -o go-template='{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}'

If your app never writes that file, set terminationMessagePolicy: FallbackToLogsOnError on the container. Kubernetes then stores the tail of the log (up to 2048 bytes or 80 lines) as the message whenever the container exits with an error, so the reason survives even after the logs rotate away.

What the exit code tells you#

Exit codes above 128 follow the shell convention of 128 plus the number of the signal that ended the process. The common ones in a crash loop:

Read the code together with Reason. A 137 with OOMKilled is a memory problem; a 137 with Error is something else that sent a hard kill, so don't raise the memory limit on the strength of the number alone.

Causes, and the fix for each#

Once you have the exit code, Reason, lifetime and logs, the cause usually falls into one of the groups below.

The application exits on an error#

Exit code 1, a stack trace or a one-line complaint in --previous. Typical lines: a required environment variable is unset, a config file path doesn't exist, the database refused the connection. Fix the configuration and roll out again. When the dependency is merely slow to come up, teach the application to retry; the Pod will otherwise keep crashing until the database happens to win the race.

The command or args are wrong#

In a Pod spec, command replaces the image's ENTRYPOINT and args replaces its CMD. Set only args and the image's own entrypoint receives them. Plenty of 127s and instant exits start here: a command copied from a Dockerfile, a shell-only construct with no sh -c in front of it, or arguments meant for one binary handed to another. Compare what you wrote with the image's documented entrypoint, then correct the Deployment.

A ConfigMap or Secret is missing (a different error)#

If the Pod references a ConfigMap or Secret that doesn't exist, the container never starts at all, so there's nothing to crash. With an environment-variable reference, the container waits with Reason CreateContainerConfigError, and the message names the missing object or key. Create the object (the ConfigMaps chapter walks through it), or mark the reference optional: true if the app can run without it. A CrashLoopBackOff that involves config means the object exists but holds a value the app rejects.

The liveness probe kills a healthy container#

A failing liveness probe makes the kubelet kill and restart the container, and enough of those restarts put it into the back-off like any other crash. Events show it plainly:

Warning  Unhealthy  10s (x3 over 20s)  kubelet  Liveness probe failed: ...
Normal   Killing    10s                kubelet  Container app failed liveness probe, will be restarted

Usually the app is fine and slow. It hasn't finished starting when the probe begins, so it's killed before it ever serves a request. Give it a startupProbe: liveness checks are held back until the startup probe succeeds, and the startup probe allows failureThreshold × periodSeconds for that to happen.

        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          periodSeconds: 10
        startupProbe:
          httpGet:
            path: /healthz
            port: 8080
          failureThreshold: 30
          periodSeconds: 10    # 30 x 10 = up to 300s to start

A readiness probe can't cause this. When readiness fails, the Pod stops receiving Service traffic, and nothing restarts.

The container runs out of memory#

Reason OOMKilled, exit code 137, often after the container has run for a while. Either the memory limit is too low for the workload or the app leaks. It has its own chapter: OOMKilled covers requests, limits and how to tell which of the two you're facing.

Permissions or a read-only filesystem#

Logs say "permission denied" or "read-only file system" and the exit code is 1 or 126. Two settings are the usual suspects. readOnlyRootFilesystem: true in the container's securityContext makes everything outside mounted volumes read-only, and an app that writes a PID file, cache or temp file to its image's filesystem will fail. Mount an emptyDir volume at the path it writes to. The second is runAsUser (or runAsNonRoot) pointing at a user that doesn't own the files the image ships; fix the ownership in the image, or set fsGroup so mounted volumes are group-writable for that user.

A one-shot task exits 0 under a Deployment#

Last State says Completed with exit code 0, and the Pod still loops. Nothing is broken: the task did its work. A Deployment exists to keep a process running, so under restartPolicy: Always it restarts a container that exits cleanly too. Try it with kubectl create deployment once --image=busybox:1.28 -- date, which prints the date and exits. Work like this belongs in a Job, whose Pods use OnFailure or Never. Then delete the Deployment.

Get inside with kubectl debug#

You can't kubectl exec into a container that lives for half a second. kubectl debug gets around that by making a copy of the Pod with the crashing command replaced by a shell. First find the container's name:

kubectl get pod crasher-6d8f7b9c4d-x2k9p -o jsonpath='{.spec.containers[*].name}'

Then copy the Pod, override that container's command, and attach:

kubectl debug crasher-6d8f7b9c4d-x2k9p -it --copy-to=crasher-debug --container=busybox -- sh

Now you're in a shell with the same image, environment and volumes. Check that the files you expect are there, print the variables with env, and run the real start command by hand to watch it fail. The --container flag matters: leave it out, and kubectl adds a new container running sh beside the broken one, which keeps crashing.

The copy is a standalone Pod. By default it doesn't keep the original's labels or its liveness, readiness and startup probes (the --keep-labels and --keep-liveness family of flags turns them back on). No Service selects it, and no probe will kill your shell. Delete it when you're done:

kubectl delete pod crasher-debug

For an image with no shell at all, swap the image in the copy with --set-image=*=ubuntu (or name a single container in place of *). The other mode, an ephemeral container added to the running Pod with kubectl debug -it <pod> --image=busybox:1.28 --target=<container>, has been stable since v1.25. It shares the target's process namespace, which helps when the container survives for a while before it dies. During the back-off there's no process left to look at, so for a crash at startup the copy is the better tool.

Stop the loop#

Deleting the Pod feels like the obvious move. It doesn't help. The Pod belongs to a ReplicaSet, which belongs to the Deployment, and the ReplicaSet replaces a deleted Pod straight away with a fresh one whose restart count starts again at zero. Ask who owns it before deleting anything:

kubectl get pod crasher-6d8f7b9c4d-x2k9p -o jsonpath='{.metadata.ownerReferences[0].kind}'
kubectl get deployments

Then pick the option that matches what you want:

• Fixed the cause? Apply the corrected manifest, or change the image with kubectl set image deployment/crasher busybox=<new-image>, and the rollout replaces the crashing Pods.
• If the loop started with your last change, kubectl rollout undo deployment/crasher goes back to the previous revision.
• kubectl scale deployment/crasher --replicas=0 parks it. The Deployment and its history stay, no Pods run, and you can investigate without a restart counter ticking.
• To be rid of it for good: kubectl delete deployment crasher. Its ReplicaSet and Pods go with it.

For this chapter, clean up both practice Deployments:

kubectl delete deployment crasher once
kubectl get pods

Where this fits in the book#

The other error you'll meet early is ImagePullBackOff, where the container never starts because its image can't be pulled; it uses the same back-off and the same describe-first habit. If you'd like restart counts, events and previous logs in one live screen, the k9s chapter sets up a terminal UI over the same data.

The official pages behind this chapter: Pod Lifecycle (restart policy and back-off), Debug Running Pods (logs and kubectl debug), Configure Liveness, Readiness and Startup Probes, Determine the Reason for Pod Failure and the Bash manual on exit status.

Further watching#

• Video by Anais Urlichs (the original 100 Days of Kubernetes series)