# Karpenter Not Scaling Up Nodes: Causes and Fixes (2026) When Karpenter is not scaling up nodes, check four gates in order: the Kubernetes scheduler never marked your pod unschedulable, your NodePool or EC2NodeClass is not Ready, a NodePool limit is exceeded, or no instance type satisfies both the pod's and the NodePool's requirements. ## TL;DR Karpenter is not a general-purpose "add capacity" controller. Its own documentation is precise about the trigger: "Karpenter focuses on scheduling pods that the Kubernetes scheduler has marked as unschedulable."[^faq] Everything downstream of that follows a short, checkable chain — a NodePool that is Ready, an EC2NodeClass it references that is also Ready, limits that have not been hit, and at least one instance type whose offering satisfies the intersection of the pod's requirements and the NodePool's. Break any link and pods sit Pending with no new nodes, sometimes with nothing on the pod that points at the reason. This guide walks the chain in the order that resolves fastest, using the behaviour documented for the Karpenter v1.14 docs line.[^trouble] One scoping note before you start: karpenter.sh is the AWS provider's documentation site, so EC2NodeClass, EC2 Fleet behaviour, and the VPC CNI specifics below are AWS-only. NodePools, NodeClaims, and the scheduling model are common to every Karpenter provider. ## What you'll learn - How to decide quickly whether Karpenter or `kube-scheduler` owns your problem — and why both write `FailedScheduling` events on the same pod - How to tell "Karpenter is slow" from "Karpenter is stuck", and why a few seconds of nothing is expected - How to check NodePool and EC2NodeClass readiness, and why an unready NodeClass disables every NodePool that references it - What happens when `spec.limits` is exceeded, and how to see current usage - What `no instance type met the scheduling requirements or had a required offering` actually means - Why a pod can be "incompatible" with a NodePool it looks compatible with - Why topology spread constraints deadlock even when the cluster has spare zones - Why Karpenter sometimes launches nodes that stay empty and get reaped in a loop - Why a launched node can still leave your pod Pending or stuck in `ContainerCreating` - Why a full ResourceQuota produces no scale-up at all - Which metrics and log settings to reach for first ## Is it Karpenter's problem, or the Kubernetes scheduler's? Start here, because a pod stuck Pending is not by itself evidence that Karpenter did anything wrong. Karpenter reacts to pods that `kube-scheduler` has already declared unschedulable[^faq] — in practice, pods carrying a `PodScheduled` condition with reason `Unschedulable`. If nothing has marked your pod that way, Karpenter has nothing to react to. (Worth knowing if you run a second scheduler: one that reports a different reason will not trigger Karpenter at all.) ```bash kubectl describe pod -n | sed -n '/Events/,$p' ``` Read that output carefully, because **two different controllers write `FailedScheduling` events onto the same pod**, and it is easy to read one and stop. `kube-scheduler` writes the familiar `0/N nodes are available…` message explaining why every *existing* node was rejected. Karpenter writes its own `FailedScheduling` event explaining why it declined to build a *new* one, in the form `Failed to schedule pod, ` — where the reason is a string like `nodepool requirements filtered out all available instance types`, `no instance type met the scheduling requirements or had a required offering`, `incompatible requirements`, `did not tolerate taint`, `node limits have been exhausted for nodepool`, or `all available instance types exceed limits for nodepool`. Several of these append structured detail in parentheses — the taint that was not tolerated, or the NodePool that ran out of room — so grep for the phrase, not the whole line. That second message is the one to look for, and it is already on screen from the command above. If it is there, it has told you the answer and you can skip most of what follows. The sections below exist for when it is absent, terse, or you want to understand what it means. (These strings are internal to the controller and have been reworded across releases, so match on the shape of the message rather than the exact text — the list above is the v1.14 wording.) | What you see | Who owns it | Where to go next | |---|---|---| | The pod was never created at all, or is `Pending` for a non-scheduling reason | Kubernetes admission | Check `kubectl get events -n ` and `kubectl describe replicaset` — then see the ResourceQuota section | | `Failed to schedule pod, …` from `karpenter` | Karpenter, and it has already diagnosed itself | Read the reason, then jump to the matching section below | | `FailedScheduling` from `default-scheduler` only, no Karpenter event | Karpenter never acted — or has not acted *yet*; see the batching section before assuming failure | Continue down this list from the top | | Pod is `ContainerCreating` on a fresh node | Node bootstrap / CNI | Jump to the node-launched-but-still-Pending section | The scheduler's message doubles as a specification: it enumerates why each existing node was rejected — untolerated taint, insufficient CPU, unmatched selector — and Karpenter's job is to build a node that would not draw those rejections. One caveat on relying on events at all: Kubernetes events are garbage-collected after the API server's `--event-ttl`, which defaults to one hour, so absence of a `FailedScheduling` event is not evidence that nothing failed — particularly for the `kube-scheduler` side. The pod's `PodScheduled` condition is the durable signal, and it is what Karpenter actually keys on. And Karpenter writes events onto the NodePool as well as the pod — a `NoCompatibleInstanceTypes` warning lands on the NodePool object, never on the pod — so check both: ```bash kubectl describe nodepool | sed -n '/Events/,$p' ``` ## Is Karpenter slow rather than stuck? Worth ruling out early, because it is indistinguishable from a failure if you are watching a terminal. Karpenter deliberately batches pending pods before provisioning: `BATCH_IDLE_DURATION` (default `1s`) extends the batching window each time a new pending pod arrives, and `BATCH_MAX_DURATION` (default `10s`) caps how long that extension can continue.[^settings] A steady arrival of pending pods will hold the window open to the maximum. So a scale-up that has not happened after five seconds is not yet evidence of anything. Give it past the batch ceiling, plus EC2 launch and node-registration time, before concluding Karpenter is not scaling up nodes at all. ## Is your NodePool actually Ready? A NodePool that is not Ready is invisible to scheduling, and the pod's own events will not name it as the cause — you have to go and look. The docs are unambiguous: "If a NodePool is not ready, it will not be considered for scheduling."[^nodepools] ```bash kubectl get nodepool -o wide kubectl describe nodepool | sed -n '/Conditions/,$p' ``` `kubectl get nodepool -o wide` prints the weight, resource totals and — usefully for the next section — the `NodeClass` each pool references, so read the real names from there rather than assuming everything is called `default`. NodePools carry four status conditions:[^nodepools] | Condition | Meaning | |---|---| | `NodeClassReady` | The underlying NodeClass is ready | | `ValidationSucceeded` | NodePool CRD validation succeeded | | `NodeRegistrationHealthy` | Reports whether a misconfiguration is preventing launched nodes from registering successfully, and requires manual investigation | | `Ready` | "Top level condition that indicates if the nodePool is ready. This condition will not be true until all the other conditions on nodePool are true" | That last row needs a caveat, because the docs contradict themselves about it. The table says `Ready` "will not be true until all the other conditions on nodePool are true," but the paragraph immediately below that table states that `NodeRegistrationHealthy` is "informational and does not affect the top-level `Ready` condition."[^nodepools] The second statement is the operative one: a NodePool can report `Ready=True` while `NodeRegistrationHealthy=False` is separately telling you a misconfiguration is stopping launched nodes from registering. Read both conditions rather than trusting `Ready` alone. And the degenerate case, which is easy to hit in a fresh cluster or after a botched Helm upgrade: "Karpenter won't do anything if there is not at least one NodePool configured."[^nodepools] An empty `kubectl get nodepool` is a complete explanation. ## Is your EC2NodeClass Ready? Check this even when the NodePool looks fine, because an unready NodeClass disables its NodePools without touching the NodePool's own configuration. Per the NodeClass docs: "If a NodeClass is not ready, NodePools that reference it through their `nodeClassRef` will not be considered for scheduling."[^nodeclasses] ```bash kubectl get ec2nodeclass kubectl describe ec2nodeclass | sed -n '/Conditions/,$p' ``` The condition set is granular, which is what makes it useful — each one names a discovery or validation step that can independently fail:[^nodeclasses] | Condition | What it covers | |---|---| | `SubnetsReady` | Subnets are discovered | | `SecurityGroupsReady` | Security Groups are discovered | | `InstanceProfileReady` | Instance Profile is discovered | | `AMIsReady` | AMIs are discovered | | `ValidationSucceeded` | EC2NodeClass validation succeeded | | `PlacementGroupReady` | Referenced placement groups are discovered | | `CapacityReservationsReady` | Referenced capacity reservations are discovered — present only when the capacity reservation feature is enabled | | `Ready` | Top-level; false if any of the above is false | You do not have to guess which dependency broke: when `Ready` is false, "`Message` on the condition indicates the dependency that was not resolved."[^nodeclasses] A false `SubnetsReady` or `AMIsReady` points directly at selector terms whose tags no longer match anything — for example a subnet retagged during a VPC migration. If you have just changed IAM permissions and the NodeClass still reports stale validation, Karpenter caches validation results; the documented way to force a refresh is to add any annotation to the EC2NodeClass.[^trouble] ## Have you hit the NodePool limits? `spec.limits` is a hard stop. Unlike the readiness gates above, this one does announce itself on the pod — the `node limits have been exhausted for nodepool` and `all available instance types exceed limits for nodepool` messages from the triage step both come from here, with the offending NodePool named in the structured detail. If you have one of those, you already know the cause and only need the numbers. The documented behaviour: "If a limit has been exceeded, nodes provisioning is prevented until some nodes have been terminated."[^nodepools] ```bash kubectl get nodepool -o custom-columns=\ NAME:.metadata.name,LIMITS:.spec.limits,USED:.status.resources,NODES:.status.nodes ``` The docs' own one-liner for this is `kubectl get nodepool -o=jsonpath='{.items[0].status}'`,[^nodepools] which is fine for a single-pool cluster but reads one arbitrary NodePool and prints no limits to compare against — not what you want on a cluster with several pools. Either way you are comparing `status.resources` — the CPU, memory and ephemeral-storage the pool has actually provisioned — plus `status.nodes` against `spec.limits`. Two adjacent gotchas: - `limits.nodes` is separate from the resource limits and "constrains the maximum number of nodes during scaling operations or drift replacement."[^nodepools] A pool can be far under its CPU limit and still be node-capped. - Limit enforcement is not transactional. The docs state plainly that "limit checking is eventually consistent, which can result in overrun during rapid scale outs."[^nodepools] If you are debugging why a pool went slightly *over* its limit, that is expected behaviour, not a bug. If `spec.limits` is unset there is "no default limitation on resource allocation," and your ceiling becomes your cloud provider's own quotas instead.[^nodepools] That relocates the same symptom to a different console — worth checking EC2 vCPU quotas before concluding Karpenter is at fault. For Prometheus users, `karpenter_nodepools_limit` and `karpenter_nodepools_usage` expose both sides of this comparison, labelled by nodepool name and resource type.[^metrics] ## What does "no instance type met the scheduling requirements or had a required offering" mean? This log line means Karpenter simulated the launch and found nothing to launch. The docs split it into two distinct failures wearing one message.[^trouble] The first half — no instance type met the scheduling requirements — is a sizing or filtering problem. A pod may have resource requests that necessitate a minimum instance size, and if the NodePool is confined to a particular instance family and size, nothing may fit. Critically, "resource requests from daemonsets are considered when determining if an instance type is compatible with the pod."[^trouble] That is the detail that breaks a hand-checked calculation: a pod whose requests fit an allowed instance type on paper can still be rejected, because the DaemonSets that will land on that node are subtracted first. The second half — *or had a required offering* — is about availability rather than shape. Karpenter's FAQ defines an offering as "a combination of zone and capacity type" for a given instance type.[^faq] If a pod is pinned to one availability zone, the instance type must exist *there*. The docs give the canonical example: a StatefulSet pod with an attached EBS volume whose subnet changed can end up in a different availability zone from the volume it needs, producing a required-offering error.[^trouble] Two capacity behaviours shape what you see next. Karpenter prioritises `reserved` capacity, then `spot`, then `on-demand`, falling back "generally within milliseconds" when a higher-priority type is unavailable.[^nodepools] And when the Fleet API reports insufficient capacity, "Karpenter caches that result across all attempts to provision EC2 capacity for that instance type and zone for the next 3 minutes."[^nodepools] That three-minute window is worth knowing about before you start changing configuration: if a retry succeeds a few minutes later, what expired may simply have been a cached unavailability rather than anything you fixed. Watch `karpenter_cloudprovider_instance_launch_failures_total`, which breaks CreateFleet failures down by availability zone, zone ID, capacity type, and launch failure reason.[^metrics] One AWS-account-level prerequisite applies to new accounts and looks exactly like a capacity problem: unless the account has already onboarded to EC2 Spot, the spot service-linked role will not exist and every spot launch fails with `AuthFailure.ServiceLinkedRoleCreationNotPermitted`. The fix is a single command, `aws iam create-service-linked-role --aws-service-name spot.amazonaws.com`.[^trouble] On a NodePool restricted to spot, there is no on-demand fallback to mask it. ## Why is my pod "incompatible" with a NodePool that looks compatible? Compatibility is an intersection, not a similarity. Nodes are chosen using both the NodePool's and the pod's requirements, and the NodePool docs are blunt about the consequence: "If there is no overlap, nodes will not be launched."[^nodepools] Or, as the same page puts it, a pod's requirements must be within the NodePool's. Three things narrow that intersection — and then a fourth failure that wears the same costume without being an intersection problem at all. **Taints.** "If Karpenter encounters a taint in the NodePool that is not tolerated by a Pod, Karpenter won't use that NodePool to provision the pod."[^nodepools] That pool is quietly dropped from consideration for this pod; if no other pool can take it either, the pod's `Failed to schedule pod` event ends with `did not tolerate taint` and the offending taint in the structured detail. To confirm and fix, read the pool's taints and then decide which side to change: ```bash kubectl get nodepool -o jsonpath='{.spec.template.spec.taints}' ``` Either add a matching `tolerations` entry to the pod spec, or relax `spec.template.spec.taints` on the NodePool. Which one is correct depends on intent: taints on a NodePool exist to reserve that capacity for particular workloads — the docs' own example taints a GPU pool so that "in order for a pod to run on a node defined in this NodePool, it must tolerate `nvidia.com/gpu` in its pod spec."[^nodepools] If the reservation is deliberate, tolerate it on the pod. If the taint is vestigial, remove it from the pool — remembering that `taints` is one of the `spec.template` fields that feeds the drift hash, so editing it rolls that pool's existing nodes. **Requirements that exclude the pod's nodeSelector.** The docs give the direct example: if a pod requests an instance type via `nodeSelector` and that type is not in the NodePool's instance-type requirements, "Karpenter will not create a node or schedule the pod."[^nodepools] **`minValues` under the default policy.** If a requirement sets `minValues` and Karpenter cannot meet that flexibility minimum, the behaviour depends on `--min-values-policy` (or `MIN_VALUES_POLICY`): under `Strict` it fails the scheduling loop for that NodePool, falling back to another NodePool or failing the pod outright; under `BestEffort` it relaxes `minValues` until they can be met.[^nodepools] The trap is that **`Strict` is the default**,[^settings] so this applies to you whether or not you configured it — an aggressive `minValues` is a quiet way to disqualify your only viable pool. Note the corollary: `karpenter_nodeclaims_created_total` carries a label for "if min values was relaxed for this nodeclaim,"[^metrics] but under `Strict` relaxation never happens, so that label only carries signal once you have switched to `BestEffort`. **An empty instance-type catalogue.** This one is not a pod-versus-NodePool problem at all: the NodePool's own requirements can filter out *every* instance type the provider offers, by stacking constraints such as `instance-family`, `instance-generation`, `instance-category` and `capacity-type` until nothing satisfies all of them at once. Karpenter reports this per NodePool, as a `NoCompatibleInstanceTypes` warning event whose message reads "NodePool requirements filtered out all compatible available instance types" — or, when `minValues` was the culprit, the same message with "due to minValues incompatibility" appended. Crucially that event is attached to the NodePool object, not to the pod, so the object it hangs off is what identifies which pool came back empty. That attachment is why this cause is so easy to miss. The pod only picks up a matching `nodepool requirements filtered out all available instance types` reason when *every* NodePool has been filtered down to nothing. In a cluster with several pools, one pool can empty out completely while another still works, and the pod's events will say nothing about it at all — the only trace is the warning sitting on the NodePool. Check `kubectl describe nodepool` for each pool you expect to be serving the workload, not just the pod. When several pools match, the docs are careful to say "there is no ordering guarantee",[^faq] while also stating that Karpenter uses the NodePool with the highest `weight` if more than one matches.[^nodepools] Read together: `weight` is the documented tiebreak when the weights differ, and where they do not, the docs decline to promise anything about which pool wins. The implementation does order equal-weight pools deterministically, but that is not a documented contract — which is why the docs recommend making NodePools mutually exclusive rather than relying on selection order at all.[^nodepools] To pin a workload deliberately, use the node selector `karpenter.sh/nodepool: my-nodepool`.[^faq] One thing that is *not* a scheduling-time incompatibility, despite looking like one: the cap of "a limit of 100 on the total number of requirements on both the NodePool and the NodeClaim," which `spec.template.metadata.labels` also count toward.[^nodepools] That ceiling is enforced by the CRD schema, so you hit it as a rejected `kubectl apply` or a failed NodeClaim creation — not as a running NodePool quietly declining your pod. ## Why won't Karpenter scale up for my topology spread constraint? Because pods do not inherit the requirements of the NodePools that could serve them. Karpenter derives a pod's eligible domains from the pod's own requirements, not from the pool that will actually serve it.[^trouble] Reading the documented example closely, the *universe* those domains are drawn from spans every NodePool in the cluster — plus zones already represented on existing nodes — rather than only the matching one. The deadlock appears whenever that universe is wider than the set of zones the pool actually serving the pod can reach. The documented example produces it with two NodePools: a permissive `default` with `topology.kubernetes.io/zone: Exists`, able to launch in all three availability zones, alongside `np-zonal-constraint`, pinned to two. A Deployment with a `nodeSelector` that only `np-zonal-constraint` satisfies, and with a zonal `topologySpreadConstraint` but no zonal `nodeAffinity`, sees all three zones as eligible domains — because the `default` pool put the third one in the universe — while the only pool that can serve it covers two.[^trouble] The result has a distinctive shape: the first two replicas launch fine, and the third never provisions, because Karpenter "can't provision capacity in the third domain."[^trouble] The fix is to make the pod's view match the serving NodePool's by adding matching zonal `nodeAffinity` to the pod spec, or to widen the NodePool.[^trouble] This is worth internalising because the naive reading — "my NodePool is missing a zone" — is not by itself the failure. A restricted pool in a cluster with nothing else in that third zone contributes no third domain, so nothing deadlocks. Something else has to be putting that zone into the universe. When you hit this, go looking for what: another NodePool that reaches the zone your serving pool cannot, or existing nodes already sitting there — a managed node group, for instance — since zones represented on current nodes count too. A related but distinct complaint is spread constraints that provision correctly and then *distribute* wrongly. That one is a `kube-scheduler` behaviour: if Karpenter launches nodes that can each hold more than the required number of pods and they become Ready at different times, the scheduler may overfill the first Ready node. The documented preferred solution is the `minDomains` field in `topologySpreadConstraints`, "enabled by default starting in Kubernetes 1.27."[^faq] ## Why does Karpenter keep launching nodes that stay empty? This is the startup-taint loop, and it costs real money for as long as it runs. Something — a DaemonSet, a `userData` script, a networking agent — applies a taint *after* the node is provisioned. Karpenter then sees that the pending pod cannot schedule to the node it just launched, and provisions another.[^faq] Usually the taint clears and the extra node is reaped by consolidation. But if it does not clear fast enough, the docs describe the pathological outcome directly: "an infinite loop of nodes being provisioned and consolidated without the pending pod ever scheduling."[^faq] The NodePool docs put the same warning from the other direction — "Failure to provide accurate `startupTaints` can result in Karpenter continually provisioning new nodes."[^nodepools] First, find the taint that is actually landing, rather than guessing: ```bash kubectl get node -o jsonpath='{.spec.taints}' ``` Then declare it in `startupTaints` **on the NodePool that is doing the looping**. This is the step to get right: adding a new, separate NodePool with the right `startupTaints` does not help, because the original pool still exists and still matches the pod, so it can keep being selected and keep looping. Edit the offending pool. Two details that decide whether this works. Startup taints are matched on **key and effect**, so the effect you declare must match the effect actually applied — declaring `NoExecute` against a taint applied as `NoSchedule` will not match, and you stay in the loop. (Karpenter's own docs are a live example of the confusion: the NodePools page uses `NoExecute` for the Cilium taint while the FAQ uses `NoSchedule` for the same one.[^nodepools][^faq]) And `startupTaints` is one of the `spec.template` fields that feeds Karpenter's drift hash, so editing it on a live NodePool **drifts every NodeClaim that pool owns** — expect a rolling replacement of its nodes as the price of the fix. (Not every field behaves this way: changing `requirements` is treated as a special case and does not by itself drift nodes whose existing values remain compatible.[^disruption]) A complete, applyable NodePool — note that `nodeClassRef` and `requirements` are both required by the CRD, so the fragment shown in the docs will be rejected on its own:[^nodepools] ```yaml apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: nodeClassRef: group: karpenter.k8s.aws kind: EC2NodeClass name: default requirements: - key: kubernetes.io/arch operator: In values: ["amd64"] startupTaints: - key: node.cilium.io/agent-not-ready value: "true" effect: NoExecute ``` Karpenter applies startup taints to nodes it provisions, but does not require pods to tolerate them — it treats them as temporary and expects another system to remove them.[^nodepools] The corollary matters for the next section: a node is not considered initialized until every startup taint declared in `.spec.template.spec.startupTaints` has been removed from the node's `.spec.taints`.[^trouble] ## Karpenter launched a node — why is my pod still Pending? At this point the autoscaler has done its job and the problem has moved to node bootstrap or networking. Do not reason this out from scratch — the NodeClaim's own status conditions name the blocking step, with the offending taint or resource interpolated into the message: ```bash kubectl get nodeclaim kubectl get nodeclaim -o jsonpath='{.status.conditions}' ``` Two conditions matter here and they fail for different reasons. `Registered` covers the earlier failure: the instance launched but never joined the cluster at all. The troubleshooting docs give three reasons a node can fail to join — permissions, security groups, and networking — and cover the diagnosis under Node NotReady, including how to pull kubelet logs over SSM.[^trouble] `Initialized` covers what happens after it joins. For initialization specifically, the docs name three factors: the node's `Ready` condition is `True`, all expected resources have registered to a non-zero quantity in `.status.allocatable`, and NodePool startup taints have been removed.[^trouble] Two resources commonly fail to register: `nvidia.com/gpu`, when a GPU instance launches but the device-plugin DaemonSet that advertises the resource is absent, and `vpc.amazonaws.com/pod-eni`, when Karpenter expects it but `ENABLE_POD_ENI` is `false` in the VPC CNI.[^trouble] Read the condition rather than working through the list by hand — the `reason` field tells you which of them tripped. If the pod has moved to `ContainerCreating` rather than `Pending`, check IP allocation first — the aws-cni error reads `failed to assign an IP address to container`.[^trouble] Two documented causes: - **`maxPods` exceeds the instance's supported pod density.** If the `maxPods` you configured in the EC2NodeClass kubelet configuration is larger than the IPs the instance type supports, the CNI cannot assign one. The documented fixes are enabling prefix delegation, lowering `maxPods`, removing `maxPods` entirely to fall back on Karpenter and EKS AMI defaults, or setting `RESERVED_ENIS=1` when using Security Groups for Pods.[^trouble] - **Subnet IP exhaustion.** EC2 launches the instance successfully because the subnet has room for an ENI, but there are no IPs left for pods. Documented fixes: topology spread on `topology.kubernetes.io/zone`, a larger subnet CIDR, custom networking, or an IPv6 cluster.[^trouble] One more that looks like a hang but is not: with Security Groups for Pods, pods requesting `vpc.amazonaws.com/pod-eni` can sit in `ContainerCreating` "for up to 30 minutes before transitioning to `Running`," an interaction with the amazon-vpc-resource-controller. The documented workaround is the `vpc.amazonaws.com/has-trunk-attached: "false"` label on the NodePool, with instance-type requirements restricted to types that support ENI trunking.[^trouble] Persistent volumes add their own scale-up failure. Karpenter does not support in-tree storage plugins, so a StorageClass provisioned by something like `kubernetes.io/aws-ebs` leaves Karpenter unable to discover volume-attachment limits — it logs that "Scale-ups may fail because Karpenter will not discover driver limits" and may believe a node has room it does not have.[^trouble] Migrating to the CSI driver is the fix. Separately, a known Kubernetes race between the scheduler and CSINode registration can let the scheduler assume a node supports more volume attachments than it does; the aws-ebs-csi-driver and aws-efs-csi-driver both support a startup taint to eliminate it, configured through `startupTaints` on the NodePool.[^trouble] ## Why doesn't Karpenter scale up when a ResourceQuota is full? Because there is no Pending pod. When a namespace's ResourceQuota is exhausted, Kubernetes rejects the *creation* request: "If creating or updating a resource violates a quota constraint, the control plane rejects that request with HTTP status code `403 Forbidden`."[^quota] The pod object is never created, so `kube-scheduler` never marks anything unschedulable, and Karpenter — which only acts on unschedulable pods[^faq] — has nothing to observe. The symptom is distinctive: your Deployment's replica count never rises, and because the rejection lands on the controller doing the creating, the failure surfaces as a `FailedCreate` event on the ReplicaSet rather than on any pod. This one is worth stating plainly rather than treating as a bug to be fixed: the behaviour is correct. A namespace quota is an administrative ceiling, and an autoscaler that provisioned past it would be defeating the control. There is a long-open request against the AWS provider asking Karpenter to scale in this situation, still open and filed against a pre-v1 version, but its stated "expected behaviour" amounts to asking the autoscaler to provision past the quota.[^quota6737] Treat quota headroom as a prerequisite for autoscaling, not something autoscaling can resolve. ## Why won't Karpenter scale up for a new DaemonSet — or for itself? These are two documented refusals, and both are deliberate — neither is a bug to be worked around so much as a behaviour to design for. **DaemonSets.** "Karpenter will not scale-up more capacity for an additional DaemonSet on its own," because the only pod that would land on the new node is the DaemonSet pod itself — capacity consumed for no benefit. DaemonSets are counted as overhead when sizing scale-ups for workload pods, but never as the reason for one. The documented workaround is to give DaemonSet pods a high priority with `preemptionPolicy: PreemptLowerPriority`, so they preempt lower-priority pods on existing nodes and push those pods into the Pending state that *does* trigger Karpenter.[^faq] **Karpenter itself.** "Karpenter won't launch capacity to run itself," which surfaces as a log line about the `karpenter.sh/nodepool DoesNotExist requirement`. Since version 0.16.0 the default replica count is 2, so a cluster with only enough non-Karpenter capacity for one controller pod leaves the second permanently Pending. The documented fixes are to reduce replicas back to 1, or to ensure there is enough capacity not managed by Karpenter to run both pods — on AWS, by raising the `minimum` and `desired` parameters on the node group's autoscaling group.[^trouble] A third case worth knowing before you file a bug: extra nodes that appear during a rollout and vanish shortly after may simply be `maxSurge`. Consolidation packs nodes tightly, so with a default 25% `maxSurge` the surge pods may have nowhere to run; Karpenter launches a node for them and removes it once it is no longer needed.[^faq] ## Why did Karpenter pick a node that turned out too small? Because Karpenter models allocatable memory before the node exists, and the model has a deliberate fudge factor. For a new AMI and instance-type pair, it reduces instance memory by `VM_MEMORY_OVERHEAD_PERCENT`, whose default is 7.5% — "tuned to closely match reality for the majority of instance types while not overestimating."[^trouble] After the first launch of that pair, Karpenter caches the observed capacity and uses the real number for subsequent nodes. The direction of the error is the useful part: Karpenter "will typically underestimate the memory available on a node for a given instance type".[^trouble] Underestimating is safe. Tuning the value down to tighten the bound is the risky direction — a value that causes overestimation "can result in Karpenter launching nodes which are too small for your workload."[^trouble] To detect it rather than infer it, watch the NodeClaim's `ConsistentStateFound` condition, which flips to `False` with reason `ConsistencyCheckFailed`:[^trouble] ```bash kubectl get nodeclaim $NODECLAIM_NAME \ -o jsonpath='{.status.conditions[?(@.type=="ConsistentStateFound")]}' ``` The troubleshooting page gives `operator_status_condition_count{type="ConsistentStateFound",kind="NodeClaim",status="False"}` as the metric to monitor for this.[^trouble] Note the tension with the metrics reference: that un-prefixed family is marked DEPRECATED, and the BETA per-kind equivalent is `operator_nodeclaim_status_condition_count`.[^metrics] Alert on the deprecated name if that is what your build emits, but expect to migrate. The adjacent cause is under-specified pods rather than mis-modelled nodes. Karpenter bin-packs on resource *requests*, so pods with very low or missing requests get packed too densely, "resulting in the pods getting CPU throttled or terminated due to the OOM killer." The documented mitigation is per-namespace LimitRanges to enforce minimum request sizes.[^trouble] This is not Karpenter-specific — `kube-scheduler` does the same thing with inaccurate requests — but Karpenter amplifies it by sizing the node to the same wrong number. If you are still deciding what those requests should be, in-place pod resizing gives you a way to [change CPU and memory requests without restarting the pod](/resize-kubernetes-pods-without-restart) while you measure. ## Which metrics and logs should I check first? Read the controller's ordinary logs first. Scheduling failures and NodePool exclusions are already written at info and error level, so you do not need to change any configuration to see them: ```bash kubectl logs -n "${KARPENTER_NAMESPACE:-kube-system}" \ -l app.kubernetes.io/name=karpenter -c controller --tail=200 ``` Debug logging — the `LOG_LEVEL` environment variable, or `--set logLevel=debug` at install time[^trouble] — is genuinely useful, but treat it as a second step rather than a reflex. Changing it requires restarting the deployment, and Karpenter provisions nothing until its cluster-state cache has re-synced after a restart. Restarting the autoscaler on a cluster that is already failing to scale is not where you want to begin. Karpenter exposes Prometheus metrics at `karpenter.kube-system.svc.cluster.local:8080/metrics`, configurable via `METRICS_PORT`.[^metrics] Six carry most of the signal for scale-up investigations:[^metrics] | Metric | What it tells you | Stability | |---|---|---| | `karpenter_scheduler_unschedulable_pods_count` | The number of unschedulable Pods | ALPHA | | `karpenter_scheduler_queue_depth` | The number of pods currently waiting to be scheduled | BETA | | `karpenter_scheduler_pending_pods_by_effective_zone_count` | Pending pods by effective zone constraint — reports a zone name, `flexible`, or `none` for no valid intersection | ALPHA | | `karpenter_nodepools_limit` | Limits configured on the nodepool, by resource type | ALPHA | | `karpenter_nodepools_usage` | Resources actually provisioned for the nodepool | ALPHA | | `karpenter_cluster_state_synced` | 1 if Karpenter's cluster state matches the API server, 0 otherwise | STABLE | Do not skip that last row. Cluster-state sync is a gate, not a curiosity: while it reads 0, Karpenter is not making provisioning decisions at all, so a cluster that has just restarted the controller or is churning heavily can look like every other failure on this list. The `none` value on `karpenter_scheduler_pending_pods_by_effective_zone_count` is the one to alert on: it means the intersection of pod-level zone signals, PVC volume topology, and topology constraints is empty.[^metrics] That is a zonal contradiction inside the pod's own spec, and it turns a category of guesswork — "is this a zone problem?" — into a number you can graph. Two stability notes so you do not build a dashboard on sand. Most of the useful scheduler metrics are labelled ALPHA rather than STABLE, and the metrics reference publishes a stability level for every metric precisely so you can tell which ones are safe to depend on.[^metrics] And the un-prefixed `operator_status_condition_*` family is marked DEPRECATED, while the per-kind variants — `operator_nodepool_status_condition_count`, `operator_nodeclaim_status_condition_count`, `operator_ec2nodeclass_status_condition_count` — are BETA.[^metrics] ## How is this different from Cluster Autoscaler? The difference that matters for debugging is where the constraint lives. Rather than scaling a group you pre-defined, Karpenter derives the instance from the pending pods — which is why its failures present as requirement-intersection problems rather than a node group sitting at its maximum. It bin-packs the pending batch onto the smallest instance type that fits, then adds 59 larger types and passes all 60 options to EC2 Fleet, which selects using the Price Capacity Optimized allocation strategy.[^faq] A NodePool that permits only a handful of instance types starves that mechanism, so a required-offering error is worth checking against your own requirements before you conclude the capacity was not there. Two operational differences follow. Karpenter "is not tied to a specific Kubernetes version, as the Cluster Autoscaler is," so you upgrade it on its own cadence[^faq] — though the compatibility matrix still sets a floor: Kubernetes 1.36 requires Karpenter 1.13 or later, 1.35 requires 1.9 or later, and 1.34 requires 1.6 or later.[^compat] And these are not either/or choices: "Karpenter can work alongside Cluster Autoscaler," and NodePools are "designed to work alongside static capacity management solutions like EKS Managed Node Groups and EC2 Auto Scaling Groups."[^faq] If you are mid-migration, confirm which controller owns the pods you are debugging before assuming Karpenter is the one ignoring them. ## FAQ **Why is Karpenter not scaling up nodes when pods are Pending?** Work four gates in order: `kube-scheduler` never marked the pod unschedulable, the NodePool or its EC2NodeClass is not Ready, a NodePool limit is exceeded, or no instance type satisfies the intersection of the pod's and the NodePool's requirements. If all four are open and pods are still stuck, the cause is one of the deliberate refusals (a DaemonSet-only scale-up, or Karpenter's own pods) or a post-provisioning failure — see the DaemonSet, startup-taint, node-initialization and node-sizing sections. **How do I know whether Karpenter or the Kubernetes scheduler is at fault?** Run `kubectl describe pod` and read the events. Both controllers write `FailedScheduling`: `kube-scheduler`'s message explains why existing nodes were rejected, while Karpenter's explains why it declined to launch a new one. A Karpenter event naming a nodepool or instance types often gives you the cause outright. If there is no `FailedScheduling` event at all, do not conclude anything yet — events expire, so check the pod's `PodScheduled` condition, which is the durable signal Karpenter keys on. **Is Karpenter not scaling up, or just slow?** Karpenter batches pending pods before provisioning: `BATCH_IDLE_DURATION` defaults to `1s` and extends the window as new pods arrive, up to `BATCH_MAX_DURATION`, which defaults to `10s`.[^settings] Wait past that ceiling plus instance launch and node registration before treating it as a failure. **Why does my NodePool look healthy but never launch anything?** Its requirements may exclude every available instance type — an over-tight combination of instance family, generation, category and capacity type. Karpenter reports this as a `NoCompatibleInstanceTypes` warning event attached to the NodePool object itself. If other pools are still working, the pod's events will not mention it, so run `kubectl describe nodepool ` against each pool you expect to serve the workload. **Why is my Karpenter NodePool not ready?** Check `ValidationSucceeded` and `NodeClassReady` on the NodePool. An unready NodePool "will not be considered for scheduling."[^nodepools] Note that `NodeRegistrationHealthy` is informational and does not gate the top-level `Ready` condition — the docs' own conditions table appears to say otherwise, which is why the NodePool section above walks through the discrepancy. **Why is my EC2NodeClass not ready?** One of `SubnetsReady`, `SecurityGroupsReady`, `InstanceProfileReady`, `AMIsReady`, `ValidationSucceeded`, `PlacementGroupReady`, or `CapacityReservationsReady` is false. The `Ready` condition's message names the unresolved dependency.[^nodeclasses] NodePools referencing an unready NodeClass are excluded from scheduling. **What happens when a Karpenter NodePool hits its limits?** Provisioning stops: "If a limit has been exceeded, nodes provisioning is prevented until some nodes have been terminated."[^nodepools] Compare each pool's `status.resources` against its `spec.limits` — the `custom-columns` command above does this across every NodePool at once — and remember `limits.nodes` is a separate cap from CPU and memory. **What does "no instance type met the scheduling requirements or had a required offering" mean?** Either no instance type is large enough or permitted enough to fit the pod plus its DaemonSets, or the type that would fit is not available in the zone or capacity type the pod is pinned to.[^trouble] **Why does Karpenter say my pod is incompatible with the NodePool?** An untolerated NodePool taint, a `nodeSelector` outside the NodePool's requirements, `minValues` failing under the default `Strict` min-values policy, or NodePool requirements so tight they filter out every available instance type.[^nodepools] The separate 100-requirement cap is not a scheduling-time incompatibility — it is enforced by the CRD schema, so it surfaces as a rejected apply. **Why did Karpenter launch a node but my pod is still Pending?** The node has not initialized. Karpenter requires `Ready=True`, all expected resources registered in `.status.allocatable`, and all NodePool startup taints removed.[^trouble] If the pod moved to `ContainerCreating`, suspect CNI IP allocation instead. **Why doesn't Karpenter scale up when a ResourceQuota is full?** Kubernetes rejects the pod creation with `403 Forbidden`, so no pod exists to be Pending and Karpenter has nothing to react to.[^quota] It is tracked as an open issue against the AWS provider.[^quota6737] **Why does Karpenter keep launching nodes that stay empty?** A taint is being applied after launch that Karpenter does not know about, so it considers the pending pod unschedulable on the node it just created and launches another — potentially in a loop. Declare it in `startupTaints`.[^faq] **Does Karpenter scale up for a new DaemonSet?** No — "Karpenter will not scale-up more capacity for an additional DaemonSet on its own."[^faq] Give DaemonSet pods high priority with `preemptionPolicy: PreemptLowerPriority` so they preempt existing pods and create the pending workload pods that do trigger scale-up. **Why is a Karpenter-provisioned node smaller than my workload needs?** Karpenter models allocatable memory with `VM_MEMORY_OVERHEAD_PERCENT` (default 7.5%) until it has observed a real node for that AMI and instance-type pair. Lowering that value risks overestimation and undersized nodes.[^trouble] ## Bottom line Karpenter not scaling up nodes becomes tractable once you stop looking at the autoscaler and start looking at the gates in front of it. Run the sequence: start by reading the pod's events for Karpenter's own `Failed to schedule pod` message, which often names the cause outright and lets you skip the rest. If it is absent or unclear, work the gates in order — `kubectl get nodepool` and `kubectl get ec2nodeclass` for readiness, then `status` versus `spec.limits`, then the controller logs for the requirement-intersection message. Each of those either clears a gate or names the cause, which is why working them in order beats guessing. Two habits prevent repeats. Declare every taint that lands on a node after launch as a `startupTaint`, because the alternative is a provisioning loop that costs real money. And keep NodePool requirements as wide as your workload genuinely tolerates — the docs recommend leaving instance-type requirements undefined because it "maximizes choices,"[^nodepools] and separately note that the best defence against running out of spot capacity is to allow as many distinct instance types as possible.[^faq] For the deployment side of the same problem, our guide to [zero-downtime deployments on Kubernetes](/kubernetes-zero-downtime-deployments-tutorial) covers the readiness and disruption settings that determine whether pods actually move onto the capacity Karpenter provisions, and the walkthrough of [making `helm upgrade --install` idempotent](/helm-upgrade-install-idempotent-deploy) is worth reading before your next Karpenter chart upgrade, since a partially applied release can leave NodePools unready. If you are debugging a different Kubernetes object stuck in a pending state, the same condition-reading discipline applies in our writeup on [a cert-manager Certificate stuck in Pending](/cert-manager-certificate-stuck-pending). [^trouble]: Karpenter, "Troubleshooting," karpenter.sh documentation (v1.14), last modified 12 August 2026. [^nodepools]: Karpenter, "NodePools," karpenter.sh documentation (v1.14), last modified 12 August 2026. [^nodeclasses]: Karpenter, "NodeClasses," karpenter.sh documentation (v1.14), last modified 12 August 2026. [^faq]: Karpenter, "FAQs," karpenter.sh documentation (v1.14), last modified 12 August 2026. [^metrics]: Karpenter, "Metrics," karpenter.sh documentation (v1.14), last modified 12 August 2026. [^compat]: Karpenter, "Compatibility," karpenter.sh documentation (v1.14), last modified 12 August 2026. [^disruption]: Karpenter, "Disruption," karpenter.sh documentation (v1.14), last modified 12 August 2026 — see "Special Cases on Drift," which lists `spec.template.spec.requirements` as a field where a CRD change does not necessarily drift a NodeClaim whose existing value remains compatible. [^settings]: Karpenter, "Settings," karpenter.sh documentation (v1.14) — environment variables and CLI flags, including `MIN_VALUES_POLICY` (default `Strict`), `BATCH_IDLE_DURATION` (default `1s`) and `BATCH_MAX_DURATION` (default `10s`). [^quota]: Kubernetes, "Resource Quotas," kubernetes.io documentation. [^quota6737]: citiatish, "Karpenter not scaling nodes when ResourceQuota in namespace is fully utilized," issue #6737, aws/karpenter-provider-aws, opened 14 August 2024; open and labelled `bug` / `triage/needs-investigation` as of 16 August 2026.

#eks

Karpenter لا يقوم بزيادة عدد العقد: الأسباب والحلول (2026)