Karpenter Troubleshooting: Pending Pods, Failed Launches, and Nodes That Never Join

Karpenter changes what a Pending pod means. On a fixed set of nodes, Pending is a question about the nodes you have: nothing had room, or nothing matched. On a Karpenter cluster there is a second question underneath it - whether Karpenter considered provisioning a node and declined, or tried and failed - and the scheduler's message speaks only to the first.

The evidence exists. It is just spread across NodePool specs, NodeClaim conditions, NodeClass conditions, controller logs, and an event stream that ages out while you read the logs.

Karpenter is not silent about this. It emits scheduling events that name the NodePools it ruled out and, often, the constraint that ruled them out. What no event does is connect that verdict to the state of the pool that should have worked. The event tells you nothing matched. It does not tell you that one of those pools has been failing to launch nodes since Tuesday because its NodeClass cannot resolve a subnet - a fact that lives three objects away, on a condition nobody reads, and that was true long before your pod existed.

This is a guide to those failures: the exact strings Kubernetes and Karpenter emit for each, and where the answer actually lives. The screenshots come from Radar's Capacity view, which assembles all of it from the cluster itself. Capacity only diagnoses - it never mutates a NodePool - while fixing one is a YAML edit in the resource view, within whatever your RBAC allows.

Karpenter pending pods: symptom, cause, where to look

Scheduler messages arrive as 0/N nodes are available: followed by a reason. The reason is what matters:

The Overview is the posture version of that table. The operational signals list is prioritized and each row links to its diagnosis, and a cascade collapses to its root cause, so a broken NodeClass shows up as one incident rather than a warning on every NodePool that references it.

"didn't match Pod's node affinity/selector"

This is the most common Karpenter misconfiguration, and it is decidable from configuration alone. Karpenter only applies labels that come from a NodePool's requirements or its template labels. If a pod requires a custom label that no NodePool declares, no node Karpenter can build will ever carry it, and the pod waits forever.

Radar's Demand screen groups pending pods by scheduling signature - canonicalized selectors, required affinity, tolerations, and per-pod requests - so five hundred pending replicas read as one group instead of five hundred rows. Each group is evaluated against every NodePool, with per-predicate evidence. Here both pools are Declared incompatible and the group headline names the dominant reason.

The selector in this screenshot is a test label, chosen so the failure is unambiguous. In production this is usually a label that used to match: a workload-type that outlived the migration which renamed it, or a nodeSelector copied from another cluster's manifests.

The scheduler's own reason is kept as evidence rather than replaced - NodeAffinitySelector, 2 events, read from pod.status.conditions.PodScheduled. That is the same data kubectl describe pod prints.

"had untolerated taint" on a Karpenter cluster

Karpenter taints nodes from spec.template.spec.taints on the NodePool. A pod without a matching toleration cannot land on anything that pool provisions, so the pool is ruled out before provisioning is ever considered.

Radar treats permanent taints against tolerations as one of the evaluated predicates, alongside readiness, selector feasibility, configured limits, minValues, and observed member shapes. When a pool fails on taints, that predicate is what the evaluation names.

The distinction that matters here is permanent versus transient. Startup taints and the well-known unreachable/not-ready taints are not evidence of misconfiguration, and treating them as disqualifying would rule out every pool on a cluster that is merely mid-rollout.

LaunchFailed, VCPULimitExceeded, and InsufficientInstanceCapacity

When Karpenter creates a NodeClaim and the cloud provider refuses it, the failure surfaces on the NodeClaim's lifecycle conditions rather than on the pod. Karpenter keeps failing lifecycle stages at status=Unknown with a failure reason, and reserves False for hard invariants on karpenter.sh/v1. The reasons include Karpenter's own LaunchFailed and cloud-provider vocabulary passed straight through:

• VCPULimitExceeded - the account's vCPU quota for that instance family is exhausted. Nothing about the cluster is wrong; the fix is a quota increase or a different instance family in the NodePool requirements.
• InsufficientInstanceCapacity - the region or zone has no capacity for that instance type right now. Common on spot, and common on the newest GPU families.
• Unauthorized - the instance profile or IAM configuration is wrong.

The trap is that these are transient by design. Karpenter deletes timed-out claims, and a deleted claim takes its conditions with it, so the failure that woke you can be gone by the time you look.

Radar's Activity screen keeps the trace. A claim that records a failure signal closes its provisioning episode as failed the moment the signal lands, so a later deletion cannot soften it - an ICE storm reads as a run of failed episodes rather than as nothing at all. A claim deleted before reaching Ready with no recorded cause closes as ended, not failed, because no failure was observed. Manual deletions and cascading NodePool deletions land in that second category rather than being reported as provisioning failures.

Why aren't my Karpenter nodes joining the cluster?

This is the failure that costs the most time, because the NodePool can look healthy while it happens.

Karpenter reports failed node registration on the NodePool's NodeRegistrationHealthy condition, and that condition sits outside the pool's Ready condition. A NodePool can therefore report Ready=True while every launch it attempts fails: instances come up, miss the registration timeout, get deleted, and the cycle repeats. Anything that reads Ready and stops shows green throughout.

Registration failures usually trace back to the NodeClass. Here EC2NodeClass/gpu-optimized is not ready:

ValidationSucceeded=False
SecurityGroupsReady=False
SubnetsReady=False
InstanceProfileReady=Unknown

which puts the NodeClass Ready condition at False with reason UnhealthyDependents. The usual causes are subnet or security-group selectors that match nothing, a missing instance profile, or user data that fails before the kubelet registers.

Radar leads this page with the diagnosis rather than the ledger when a pool is broken, and each finding carries a cause, a next step, and a link into the NodeClass. The raw controller conditions stay available behind a toggle.

Am I about to hit a NodePool limit?

Karpenter stops provisioning when a NodePool's spec.limits are reached, and the pods that would have triggered provisioning stay Pending with no NodeClaim behind them. It does say so. The scheduler records one of these against the pool and publishes it as a FailedScheduling event on the pod:

node limits have been exhausted for nodepool
all available instance types exceed limits for nodepool

That event is the fastest confirmation you will get, and it names the pool.

What it does not give you is the margin. Knowing you are at the limit is not the same as knowing how close the other pools are.

The number Karpenter checks is status.resources, which already includes launching claims. So headroom is limit - provisioned, not limit - allocatable. A pool can be at its limit with barely any registered nodes if claims are in flight or if nodes launched and failed to register.

Radar surfaces limit pressure as an operational signal on the Overview, and the NodePool ledger shows configured limit, provisioned, and limit headroom as separate columns. In the screenshot above, gpu-workloads reads 48 cores configured, 0 provisioned, 48 of headroom - the pool is not limit-blocked, it is broken, which are different problems with the same symptom.

Provisioned, allocatable, requests, usage: which number means what

Four numbers get conflated constantly, and on a Karpenter cluster they answer different questions:

Karpenter schedules on requests, so usage is never scheduler headroom. A node at 20% CPU utilization with all its CPU requested is full as far as scheduling is concerned. This is why Radar's ledger keeps eight columns apart instead of blending them into a utilization percentage, and why the capacity bar never turns red for being full - tight packing is the goal, not an incident.

One more distinction worth internalising: unallocated is aggregate subtraction, not a bin-packing result. It does not prove another pod fits, because a pod needs its resources on a single node.

What does DisruptionBlocked mean in Karpenter?

DisruptionBlocked and Unconsolidatable both mean disruption was prevented. Read either as disruption happening and the timeline reads backwards.

The usual causes are a disruption budget that is exhausted, a PodDisruptionBudget that cannot be satisfied, a karpenter.sh/do-not-disrupt annotation, or - as here - a NodeClaim with no associated node to disrupt in the first place.

Radar classifies these from Karpenter's exact event vocabulary into provisioning, disruption, interruption, termination, and config-change episodes, and keeps the raw event as evidence:

DisruptionBlocked: Nodeclaim does not have an associated node

Heuristic matches are labeled inferred rather than presented as fact. The window is bounded and the header states its range and its 10,000-event cap, so the timeline says how far back it can see instead of implying it saw everything.

Why a pod evaluates as "unknown" instead of incompatible

An evaluation returns declared compatible, incompatible, or unknown, and the boundaries are deliberate:

• A required custom label the NodePool never declares is incompatible. Configuration alone settles it.
• Provider labels - zone, instance-type, capacity-type, arch - are unknown when undeclared, because the offering catalogue can supply them. They are compatible when the pool's own In requirement bounds the values and the pod's need intersects them, so a normal capacity-type: spot setup evaluates as compatible rather than unknown.
• Requests that fit no observed member shape degrade to unknown. Shapes compare as whole vectors, so a pool whose largest CPU and largest memory live on different instance types cannot fabricate a composite machine.
• Unevaluable constraints - exotic toleration operators, unsupported affinity fields - degrade to unknown rather than to a false verdict in either direction.

Radar does not simulate provider offerings or bin-packing, and never claims a pod will schedule. Declared compatibility means the declarations agree, which is where Karpenter's own provisioning and the cloud's real inventory take over. During an incident I would rather return unknown than be confidently wrong.

Evaluations also cover Karpenter NodePools only. On a mixed cluster, blocked means no NodePool can take the pod, not that no node can - capacity Karpenter does not manage may still satisfy it.

Why some numbers show ≥ or ?

Every quantity carries its own certainty, and a glyph appears only when the value is not exact:

The rule underneath is that unavailable is never rendered as zero, and partial is never rendered as exact. An RBAC-denied source reads "Unavailable". Metrics sampled on 3 of 100 nodes come back as a lower bound with the sample share attached. A resource the fleet does not have at all reads "0 requested of 0 allocatable, N pending" instead of vanishing from the page - which is the missing-GPU-pool case stated in one line.

The same applies to access. Capacity needs cluster-wide node visibility, and the Karpenter screens additionally need NodePools listable by your identity. Deny that and the Overview still renders its cluster-only shape behind a coverage banner while the Karpenter screens fail closed. The 403 names what remains visible without the grant and includes a ClusterRole you can paste.

What this doesn't cover

• No scheduling simulation. Declared contracts only - no offering catalogues, no spot availability forecasting, no bin-packing.
• DRA demand is invisible to the requests ledger. Dynamic Resource Allocation ResourceClaims, GA in Kubernetes 1.34, carry accelerator demand outside container requests. Those pods degrade to a labeled unknown rather than a guess.
• No trends. Current state plus the bounded activity window.
• Single cluster. Same as the rest of Radar OSS.

Clusters without Karpenter still get the node group inventory, built from node labels and the cluster-autoscaler-status ConfigMap across GKE node pools, EKS managed node groups, AKS agent pools, and kops/DOKS/ACK/Linode/Scaleway. There is no Demand or Activity, because there are no NodePools to evaluate against.

Running it

Radar is Apache-2.0, a single binary, with nothing installed in the cluster:

brew install skyhook-io/tap/radar
radar

Capacity needs v1.9.0 or newer. Open it from the sidebar, or jump straight from a stuck pod: the pod drawer carries an "Evaluate against Karpenter NodePools" link that lands on Demand filtered to that workload.

If it reaches a verdict you can prove wrong, that's worth an issue on GitHub.

Related reading: Five Questions You Can't Answer With kubectl, Everything Is Green and Nothing Works: Network Reachability in Radar, and Kubernetes 1.37 Breaking Changes.