EC
Pending (Taint Toleration)
KubernetesWARNINGCriticalSchedulingHIGH confidence

Pod cannot be scheduled due to taints

Production Risk

If misconfigured, taints and tolerations can prevent any new pods from being scheduled, leading to failed deployments and an inability to recover from pod failures.

What this means

The Kubernetes scheduler cannot place the pod on any node because all available nodes have 'taints' that the pod does not 'tolerate'. Taints are used to repel pods from certain nodes unless the pod has an explicit toleration.

Why it happens
  1. 1The pod is missing a required toleration for a taint applied to all available nodes
  2. 2Nodes are tainted to reserve them for specific workloads (e.g., GPU jobs, critical system daemons)
  3. 3A node is in a 'NotReady' or 'Unschedulable' state, which is often managed by taints
How to reproduce

A pod remains pending, and describing it reveals that all nodes failed the taint/toleration check.

trigger — this will error
trigger — this will error
kubectl describe pod my-pod

expected output

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  5m (x20 over 10m)     default-scheduler  0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.

Fix 1

Check node taints

WHEN To see what taints are applied to the nodes

Check node taints
kubectl get nodes -o custom-columns=NODE:.metadata.name,TAINTS:.spec.taints

Why this works

This command lists all nodes and their corresponding taints, allowing you to see what toleration is required.

Fix 2

Add a toleration to the pod spec

WHEN The pod needs to be scheduled on a tainted node

Add a toleration to the pod spec
kubectl patch deployment my-app --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "my-taint-key", "operator": "Exists", "effect": "NoSchedule"}]}]'

Why this works

This patches the deployment to add a toleration to the pod spec, allowing the scheduler to place it on nodes that have the matching taint.

Fix 3

Remove the taint from a node

WHEN The taint is no longer needed and nodes should be generally available

Remove the taint from a node
kubectl taint nodes my-node my-taint-key:NoSchedule-

Why this works

Adding a hyphen to the end of the taint definition removes that taint from the node, making it available for scheduling any pod.

What not to do

✕ Add tolerations for all taints to all pods

This defeats the purpose of taints, which are meant to control scheduling and reserve nodes for specific workloads. Only add tolerations that are explicitly needed.

Sources
Official documentation ↗

k8s.io/kubernetes/pkg/scheduler/framework/plugins/tainttoleration/taint_toleration.go

Content generated with AI assistance and reviewed for accuracy. Found an error? hello@errcodes.dev

← All Kubernetes errors