OOMKilled

OOMKilled means the Linux kernel stopped your container because it tried to use more memory than its resources.limits.memory. The container exits with code 137, which is 128 plus 9, the number of SIGKILL. The kubelet then restarts it according to the Pod's restartPolicy, and if it keeps dying the same way you'll see the Pod settle into CrashLoopBackOff. The fastest fix has two steps. Confirm the reason with kubectl describe pod, then either raise the limit to the memory the app needs at its peak, or find out why it needs that much (a leak, or a runtime heap sized for a bigger machine).

The rest of this chapter makes it happen on purpose, so you know what each signal looks like before you meet one in production.

What the kernel is doing#

When the kubelet starts a container, it hands the container's requests and limits to the runtime, and on Linux the runtime turns the memory limit into a cgroup setting. On cgroup v2 that's memory.max. Once usage reaches it and the kernel can't reclaim anything, the OOM killer runs inside that cgroup and kills a process.

Two details from the Kubernetes docs are worth holding on to. Memory limits are enforced reactively: a container can briefly sit above its limit, and it gets killed once the kernel steps in. CPU is different. Go over a CPU limit and you're throttled, never killed, so a CPU limit can't produce OOMKilled.

Which process dies depends on the cgroup version. On cgroup v2 nodes the kubelet sets memory.oom.group by default, which tells the kernel to kill every process in the container together. The kubelet setting singleProcessOOMKill: true switches that off and goes back to the cgroup v1 behaviour, where only one process is killed. That matters for apps that fork workers. If the victim isn't the container's PID 1, the container can keep running with one worker fewer, and nothing in kubectl get pods tells you. Kubernetes has deprecated cgroup v1 since 1.35, and the kubelet refuses to start on a v1 node by default, so on a current cluster you'll usually be on v2. You can check on a node with stat -fc %T /sys/fs/cgroup/; cgroup2fs means v2.

Confirm it with kubectl#

Start with the Pod's description. The current state can say Running, because the container has already been restarted; the evidence sits under Last State.

kubectl describe pod <pod-name>

The part you care about looks like this (illustrative output, trimmed):

    Limits:
      memory:  100Mi
    Requests:
      memory:  50Mi
    State:          Running
      Started:      ...
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      ...
      Finished:     ...
    Ready:          True
    Restart Count:  3

For a script, or a Pod with several containers, ask the API for the last termination of each container directly:

kubectl get pod <pod-name> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

Output looks like this:

app      OOMKilled   137
sidecar

An empty column means there's no recorded termination for that container. The node keeps its own record too. kubectl describe nodes lists a warning event with the reason OOMKilling and the kernel's message, which names the process it killed. That's handy when you want to know which binary inside the container was the one growing.

If the STATUS column in kubectl get pods flips between OOMKilled, Running and CrashLoopBackOff, you're watching the restart back-off at work. That loop has its own chapter, CrashLoopBackOff. Here the back-off is only the symptom.

Three ways memory ends a Pod#

OOMKilled is one of three memory failures, and they get mixed up constantly because all three end with something dead. They differ in who acts and what survives.

Eviction is the kubelet protecting the node. It compares the node's memory working set against its thresholds, ranks Pods, and terminates whole Pods, starting with those using more than they requested. With a hard threshold there's no grace period at all, and PodDisruptionBudgets aren't consulted. The node-pressure eviction page lists every signal and flag.

The third case happens when memory climbs so fast that the kubelet, which polls on an interval, never sees the pressure. The kernel's global OOM killer then picks a victim by score, and Kubernetes has already nudged those scores through oom_score_adj, set per container from the Pod's QoS class. So a container well under its own limit can still die here, because another Pod spent the node's memory.

Requests, limits and QoS classes#

The request and the limit do different jobs. The scheduler places a Pod using requests only: a node qualifies if it has enough unreserved memory for the sum of the containers' requests. The limit is the ceiling the kernel enforces once the container runs. Set a limit with no request and Kubernetes copies the limit into the request, unless an admission default (such as a namespace LimitRange) supplies one first. Leave the limit off with no namespace default to fill it in, and the container has no ceiling at all.

From those two numbers Kubernetes gives every Pod a QoS class, fixed at creation:

• Guaranteed: every container has CPU and memory requests and limits, and each limit equals its request.
• Burstable covers any Pod that doesn't qualify as Guaranteed but has at least one request or limit somewhere.
• BestEffort, with no CPU or memory requests or limits on any container. It's first in line when the node runs short.

The class feeds straight into the node-wide OOM case. Per the eviction docs, the kubelet sets oom_score_adj to -997 for Guaranteed containers and 1000 for BestEffort. Burstable containers get a value between 2 and 999 that shrinks as their memory request grows relative to the node's capacity. A container with a large request is harder to pick; one that requested little and uses a lot is the kernel's favourite target.

You can read a Pod's class with kubectl get pod <pod-name> -o jsonpath='{.status.qosClass}'. Run it on the Pods from the Kubernetes Pods chapter: if they had no resources block, they're BestEffort.

Reproduce it on your cluster#

This walkthrough uses the manifests from the Kubernetes task Assign Memory Resources to Containers and Pods. Both run the polinux/stress image, which allocates however much memory you tell it to and holds it.

You need a cluster and kubectl talking to it. A local one from the KinD or K3s chapter is enough, and each node needs at least 300 MiB free. The kubectl top step also needs metrics-server. K3s deploys it as a packaged add-on, so it's already there. On kind you install it yourself:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl get apiservices | grep metrics.k8s.io

If its Pod logs show certificate errors against the kubelets, the metrics-server README documents the --kubelet-insecure-tls flag, for test clusters only. Version 0.9.x supports Kubernetes 1.34 and newer.

First, a container that behaves. It requests 100 MiB, is limited to 200 MiB, and allocates 150 MiB:

kubectl create namespace mem-example
kubectl apply -f https://k8s.io/examples/pods/resource/memory-request-limit.yaml --namespace=mem-example
kubectl top pod memory-demo --namespace=mem-example

Usage lands around 150 MiB, above the request and under the limit. That's allowed. A container may use more than it requested as long as the node has room.

Now the one that fails. Here's the second manifest, with the limit at 100 MiB and the workload asking for 250 MB:

apiVersion: v1
kind: Pod
metadata:
  name: memory-demo-2
  namespace: mem-example
spec:
  containers:
  - name: memory-demo-2-ctr
    image: polinux/stress
    resources:
      requests:
        memory: "50Mi"
      limits:
        memory: "100Mi"
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "250M", "--vm-hang", "1"]

kubectl apply -f https://k8s.io/examples/pods/resource/memory-request-limit-2.yaml --namespace=mem-example
kubectl get pod memory-demo-2 --namespace=mem-example --watch

After a few restarts the output looks like this (illustrative):

NAME            READY   STATUS      RESTARTS   AGE
memory-demo-2   0/1     OOMKilled   1          24s
memory-demo-2   1/1     Running     2          40s

Stop the watch with Ctrl+C. Then:

1. Run the jsonpath command from earlier against memory-demo-2. You'll get OOMKilled and 137.
2. kubectl describe pod memory-demo-2 --namespace=mem-example shows the events, ending in Back-off restarting failed container once the restarts pile up.
3. Check kubectl describe nodes for the OOMKilling warning naming stress.
4. Last, compare the QoS class of both Pods. Each has a request lower than its limit, so both are Burstable.

The task page has a third manifest that requests 1000 GiB. It never gets OOMKilled; it stays Pending with Insufficient memory, because the scheduler finds no node big enough. Try it if you want to see the request side fail.

Clean up when you're done:

kubectl delete namespace mem-example

Find the right limit#

Raising the number until the restarts stop works, briefly. You want the app's real peak, measured over a stretch that includes its busiest hour.

kubectl top gives a quick reading per container:

kubectl top pod --containers --sort-by=memory --namespace=<namespace>

It's a snapshot, though, and metrics-server's own README says it isn't meant to be an accurate source of usage metrics or a feed for monitoring. For history you want Prometheus scraping the kubelet's cAdvisor metrics, set up in the Prometheus chapter. Three series cover this error:

• container_memory_working_set_bytes, the working set. The kubelet's eviction signal is built from working set too, so it's the figure that lines up with how Kubernetes sees memory.
• container_spec_memory_limit_bytes: the limit itself.
• container_oom_events_total counts OOM events per container, which also catches the silent single-process kills described above.

A peak over the last week, against the limit, for one container:

max_over_time(container_memory_working_set_bytes{namespace="shop", container="api"}[7d])
  / on(namespace, pod, container)
container_spec_memory_limit_bytes{namespace="shop", container="api"}

Swap in your own namespace and container labels. A value creeping toward 1 is an OOMKilled waiting to happen, and a ratio that rises steadily between deploys and never falls back looks more like a leak than a limit that's too small. Leave headroom above the peak. The Memory QoS section of the QoS page tells you to size limits with room for page cache, because a workload with a large cache can otherwise be OOM killed.

One more place memory hides: an emptyDir volume with medium: Memory is tmpfs, and the kubelet counts what's written there as the container's memory use. Temp files there count against the limit, even when the process itself stays small.

When the runtime is the problem#

Garbage-collected runtimes size their heap from what they think the machine has. If that guess and the container limit disagree, raising the limit only moves the point where it dies.

Java. On Linux the JVM detects container limits by default (-XX:+UseContainerSupport is on), and the Kubernetes cgroup page lists the OpenJDK builds with full cgroup v2 support: jdk8u372, 11.0.16, and 15 or later. The heap ceiling is then -XX:MaxRAMPercentage of that memory, and per the java command reference it defaults to 25 percent. That default is cautious. Raise it with something like -XX:MaxRAMPercentage=75, and keep in mind the percentage only covers the heap, while the limit has to fit the whole JVM process.

Node.js reads cgroup v2 memory limits from v20.3.0 onward, according to the Kubernetes cgroup v2 page. Older lines, v18 included, may size the heap from the host's memory and get OOMKilled for it. On those, set the old-space size in MiB with --max-old-space-size. It's on the list of V8 options allowed in NODE_OPTIONS, so you can set it from the Pod spec without touching the image:

    env:
    - name: NODE_OPTIONS
      value: "--max-old-space-size=384"

Pick a value below the container limit; buffers and native memory live outside that heap.

Raise the limit without guessing#

For anything run by a controller, change the Pod template. Edit resources in the manifest from the Kubernetes Deployment chapter and kubectl apply it; the Deployment rolls out new Pods with the new numbers.

In-place resize has been stable since Kubernetes 1.35. It changes a running Pod's CPU and memory through the resize subresource, and it needs kubectl v1.32 or newer:

kubectl patch pod <pod-name> --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"memory":"300Mi"}, "limits":{"memory":"300Mi"}}}]}}'

A few rules from the resize task catch people out. The QoS class can't change, so a Guaranteed Pod has to keep request equal to limit. Many runtimes can't grow into new memory while running, which is why you'd set resizePolicy with restartPolicy: RestartContainer for memory. And if the node can't fit the new size, the Pod gets a PodResizePending condition with reason Infeasible or Deferred. Keep in mind that a resized Pod still belongs to its Deployment. The next rollout or replacement uses the template again, so fix the template as well.

If you'd rather not size by hand at all, the Vertical Pod Autoscaler watches usage over time and sets requests for you, keeping the ratio between requests and limits you started with. It's installed separately, and its README notes that it doesn't yet support workloads with pod-level resources.

Mistakes that bring it back#

• Writing 100M when you meant 100Mi. M is decimal and Mi is binary, so 129M and 123Mi come out about the same.
• Deleting limits to make OOMKilled go away. The container becomes Burstable or BestEffort, the node-wide OOM killer can still find it, and it now takes neighbours down with it.
• Sizing from one kubectl top reading taken on a quiet afternoon.
• Doubling the limit on a leak. That buys time, and the restart count tells you how much.
• Checking only the main container. A sidecar with a tight limit gets OOMKilled too, and the jsonpath command above shows every container.

Common questions#

These come up once the walkthrough works.

Is exit code 137 always OOMKilled?#

No. 137 only says the process received SIGKILL. A container that ignores SIGTERM during shutdown also gets SIGKILL once its grace period runs out, and that's a 137 with no memory problem involved. Trust the Reason field.

Can a container be OOM killed under its limit?#

Yes, when the whole node runs out. Look for MemoryPressure and the OOMKilling events in kubectl describe nodes, then at the Pods on that node that requested little and use a lot.

Does a CPU limit matter here?#

Only indirectly. CPU limits throttle and never kill; a throttled app can fall behind, queue more work in memory and then hit its memory limit, but the kill itself is always about memory.

Next#

If the Pod is stuck restarting, go on to CrashLoopBackOff. To keep watching memory over weeks, set up Prometheus and alert on container_oom_events_total.