Autoscaling Configuration to Reduce Idle Compute Costs

Three failure modes appear repeatedly, and they compound each other in ways that make the cumulative cost invisible until someone finally sits down to measure it.

Oversized minimum replicas. Minimum replica counts are almost always set during initial deployment, based on a gut feeling about worst-case traffic rather than any measured observation of actual idle baseline. The conversation goes something like: "What if traffic spikes overnight?" The floor gets set at a number that feels safe, and it stays there. Nobody revisits it after the service has six months of traffic data sitting in a dashboard nobody opens.

Scale-down thresholds set too high to fire. A CPU utilization target of 80% or higher means the scaler has almost no incentive to reduce replica count. If your pods are running at 30% utilization and your scale-down threshold is 80%, the autoscaler looks at that gap and concludes everything is fine. It does nothing. AWS data supports this: EC2 instances averaging below 40% CPU and memory utilization over four weeks are typically oversized, and teams holding 80% thresholds never accumulate enough signal to trigger action.

Cooldown periods set too conservatively. Cooldowns exist to prevent thrashing, and that is a legitimate concern. But a cooldown window of twenty or thirty minutes means that after a traffic spike subsides, the infrastructure sits over-provisioned for the entire duration before scale-down is even eligible to evaluate. Conservative cooldowns made sense as a defensive default during initial setup. Leaving them unchanged for years is a different decision, and most teams never consciously made it.

The reason idle compute persists is the interaction between all three. A high minimum means you start from an inflated floor. A high threshold means the scaler rarely identifies a reason to descend from it. A long cooldown means that even when a scale-down signal finally arrives, execution is deferred. The autoscaler is active in the technical sense; in any practical sense, it is not doing much.

No single vendor is responsible for this. These are configuration decisions, made or quietly deferred during initial setup, and rarely revisited because the infrastructure continues to function. It just costs more than it needs to.

How to Set Minimum Replicas and Scale-Down Thresholds That Actually Fire

The corrective principle for minimum replicas is straightforward: the floor should reflect measured idle demand, not perceived safety. Pull your actual traffic data for the quietest period in a typical week, identify the replica count that handled that load with reasonable headroom, and set that as your minimum. If your off-hours baseline genuinely requires two replicas and you have been running six, that gap is pure waste. Four replicas paying to do nothing, every night, indefinitely.

Scale-down thresholds work better when they define a utilization band rather than a ceiling. Framing it as "scale down when utilization is below X" is less useful than "keep utilization between Y and Z." A lower bound gives the scaler permission to act when the cluster is genuinely underloaded. Without one, the default behavior is to preserve capacity until something breaks.

There is an asymmetry in how teams approach this, and it matters more than most people acknowledge. Scale-up gets tuned aggressively because scale-up failures are visible: latency degrades, errors surface, users notice. Scale-down failures are invisible. The bill is slightly higher than it should be, no alarm fires, and the waste accumulates in silence. Configuration effort gets applied systematically in one direction, and idle compute builds up in the other.

On cooldowns: the practical question is not "how long is safe" but "what is the actual tradeoff between thrashing risk and idle cost at my workload's hourly rate." A twenty-minute cooldown after a spike carries very different cost implications for a GPU inference pod versus a small CPU service. The right cooldown is the shortest one that does not introduce instability, calibrated to the real characteristics of your traffic, not inherited from a template someone copied three years ago and nobody has touched since.

The four settings worth auditing first, in order: minimum replica count, scale-down utilization threshold, scale-down cooldown period, and stabilization window. Get these right before touching anything else.

When HPA's CPU-Based Model Isn't Measuring the Right Thing

The Horizontal Pod Autoscaler is the right tool for stateless APIs under steady, CPU-correlated traffic. Request volume goes up, CPU goes up, HPA adds replicas. Volume drops, CPU drops, HPA removes them. The model is clean and works well within those conditions.

Outside those conditions, it breaks in specific and predictable ways.

Architecturally, HPA enforces a minimum of one replica. Services with genuine idle periods, background processors that sit dormant between jobs, webhook handlers waiting for inbound events, cannot scale to zero under HPA regardless of how well everything else is configured. That one baseline replica runs around the clock, every day, whether or not anyone needs it.

The second problem is signal quality. Event-driven and batch workloads frequently exhibit low CPU utilization even when they are doing real work. A queue consumer processing messages at high throughput will barely register on CPU metrics depending on how the workload is structured. HPA reads idle. The system is not idle. Nothing happens. This is not a threshold problem; it is a measurement problem, and adjusting a threshold on a bad metric produces a more aggressively misconfigured system, not a better-tuned one.

The diagnostic worth paying attention to: if scale-down events rarely fire despite consistently low observed utilization, the metric is wrong. That conclusion feels counterintuitive, but it is usually correct. The signal is not reflecting what the workload is actually doing.

This is where KEDA becomes relevant, not as a wholesale replacement for HPA, but as a targeted correction for workloads where CPU is a lagging or irrelevant proxy for actual demand.

Using KEDA to Scale on the Signals That Actually Reflect Demand

KEDA, the Kubernetes Event-Driven Autoscaler, graduated from the CNCF in August 2023. Created by Microsoft and Red Hat, it is now deployed in production at meaningful scale across a wide range of organizations. It is not experimental infrastructure.

The core mechanism: KEDA scales pods on external event signals rather than internal resource metrics. Queue depth, Kafka consumer lag, Prometheus query results, and a large catalog of additional sources can all serve as the scaling signal. When the queue is empty, KEDA scales the consumer to zero. When messages arrive, it scales up in response to actual load. This is a fundamentally different relationship between signal and action than anything HPA offers.

Scale-to-zero is the primary cost lever HPA cannot provide. For workloads with genuine idle periods, the difference between one replica running continuously and zero replicas during off-hours compounds over a month into real spend. Background job processors, scheduled pipelines, webhook handlers, async summarization services: these are exactly the workloads where scale-to-zero translates directly into a smaller bill.

The production-ready pattern is not KEDA replacing HPA. It is both coexisting, each applied to the workloads where it measures the right thing. Event-driven workloads scale on event signals via KEDA. CPU-correlated APIs scale on resource metrics via HPA. A node provisioner like Karpenter handles the infrastructure layer beneath both. These three layers compound; each one addresses a failure mode the others cannot.

The decision rule is practical. Use HPA for steady, CPU-correlated APIs. Use KEDA for anything event-driven, bursty, or intermittently idle. Your workload's behavior tells you which category you are in.

Right-Sizing Resource Requests With VPA So Pod Scaling Starts From an Honest Baseline

Resource requests are the numbers your scheduler and autoscalers believe your pods need. If those numbers are inflated, which Cast AI's 2026 data suggests is the norm rather than the exception across production clusters, the cluster's view of capacity is systematically distorted. Pods look full when they are not. Horizontal scaling fires before it needs to. Nodes fill up on paper while running at a fraction of real utilization.

The Vertical Pod Autoscaler addresses this. VPA observes actual resource consumption over time and adjusts requests and limits to match reality. Once requests are honest, the scheduler bins pods more efficiently, HPA's utilization calculations are grounded in actual load, and every threshold decision made at the signal layer has an accurate denominator.

One conflict worth understanding clearly: running VPA and HPA simultaneously against the same CPU metric creates instability. HPA adds replicas while VPA adjusts per-pod resource requests, and the two can work against each other in ways that are genuinely difficult to debug. The safe pattern is HPA on custom or external metrics via KEDA for event-driven workloads, while VPA handles request right-sizing. They operate on different dimensions and coexist without conflict when separated this way.

VPA in recommendation-only mode is the right starting point for most teams. Let it run, observe what it recommends, compare those recommendations against your current requests, and apply changes incrementally. The output is frequently revealing. Teams running CPU requests at ten times observed utilization are not unusual, and that inflation is the hidden multiplier making every other configuration decision less effective than it should be.

Node-Level Autoscaling: What Karpenter Does That Cluster Autoscaler Does Not

Pod autoscaling is necessary but not sufficient. A KEDA scaler can reduce a workload to zero replicas, and the node those pods were running on will remain active, billed by the hour, until the cluster autoscaler decides to remove it. Empty nodes are one of the most persistent sources of idle compute cost, precisely because they require a separate layer of tooling to address.

The Cluster Autoscaler has handled this job for years. Its limitation is structural: it works with pre-defined node groups and scales within the constraints of those groups. It cannot select a smaller, more appropriate instance type for a reduced workload. It cannot proactively consolidate pods onto fewer nodes to create empty ones eligible for removal. It reacts to unschedulable pods; it does not optimize for idle capacity.

Karpenter, which reached a stable v1.0 API in late 2024, addresses these limitations and has become the standard recommendation for new EKS deployments. The specific behaviors that reduce idle waste: it selects node type and count based on the actual resource requirements of pending pods rather than pre-defined group configurations, provisions nodes in under sixty seconds, and proactively consolidates underutilized nodes to create removal candidates without waiting for pods to become unschedulable. AWS EKS Auto Mode deployments using Karpenter have documented infrastructure cost reductions of 60 to 70% in validated cases.

GCP GKE Autopilot and Azure AKS Node Auto-Provisioning offer comparable node-level efficiency for teams on those clouds. The mechanics differ; the principle converges: pay for pod resources, not idle node capacity.

The compounding effect of the full stack is worth stating plainly. KEDA scales a workload to zero replicas. Karpenter identifies the now-empty node, finds no pending workloads that require it, and removes it. Each layer enables the next.

GPU Autoscaling Requires Different Signals and Different Floors Than CPU Workloads

Inference now consumes more than half of total AI infrastructure spending as of 2026, and unlike training workloads that run in discrete bounded windows, inference runs continuously against live user traffic. The economics of idle GPU compute are not analogous to idle CPU compute. The hourly rate is an order of magnitude higher, and the cost of misconfigured autoscaling is proportionally more punishing. Getting this wrong is expensive in a way that CPU misconfiguration simply is not.

The fundamental problem is that GPU utilization does not correlate with CPU utilization. A GPU-accelerated inference service can be handling significant request volume while its CPU metrics look unremarkable. CPU-based autoscaling is functionally blind to the actual load. HPA with a CPU target is the wrong tool, full stop, for GPU inference workloads.

The signals that actually reflect demand for inference workloads are pending request queue depth, inference latency, and GPU utilization directly. A reasonable scale-out trigger is queue depth per GPU exceeding a threshold your SLA cannot sustain; the right threshold is specific to your workload and latency requirements. The scale-down signal is GPU utilization dropping into a sustained idle band, well below the healthy operating range of roughly 70 to 85%. A single-point reading is not the signal; a sustained period below the floor is.

Cooldown misconfiguration is disproportionately expensive here. Tune cooldowns for GPU workloads with explicit attention to the cost per minute of idle capacity, not with the same defaults you use across everything else.

The spot versus on-demand decision is also workload-specific. Batch inference pipelines, embeddings generation, async summarization: these can absorb interruption and should run on spot where available. Synchronous inference APIs with latency SLAs cannot. The workload's tolerance for interruption, not its resource profile, is the right decision criterion.

KEDA with the NVIDIA GPU Operator enables scale-to-zero for GPU pods during genuinely idle periods. The same event-driven scaling pattern described earlier applies; the event sources are GPU-specific, but the architecture is identical.

Scheduled Scaling for Workloads With Predictable Demand Patterns

Reactive autoscaling is always a trailing indicator. The metric changes, the scaler evaluates, the decision executes. For workloads with predictable demand patterns, that lag produces two distinct forms of waste: the cold-start period when capacity is scrambling to meet known demand, and the off-peak period when reactive autoscaling is slow to scale down from a floor that no longer reflects reality.

Scheduled scaling solves this by setting minimum capacity ahead of known demand. If your traffic rises predictably with business hours, or a nightly batch job runs on a fixed schedule, the minimum replica count for that window can be pre-configured to arrive before the load does. The scale-down schedule can be set aggressively for confirmed quiet windows, operating below what a reactive system would conservatively maintain.

GCP's predictive autoscaling uses machine learning to anticipate demand curves for workloads with consistent but irregular patterns, useful when the shape of demand is recognizable but not trivially schedulable. Azure Databricks autoscaling offers a concrete reference point: switching from fixed-size clusters to properly configured autoscaling with schedules cuts Databricks compute costs by 40 to 60%.

The combination of scheduled minimums, reactive autoscaling for variance within the window, and spot instances for the bulk of compute is particularly effective for batch AI workloads. Schedule capacity for the nightly processing window, let reactive autoscaling handle variance within it, and use spot for compute that can tolerate interruption. Each layer addresses the failure mode the others cannot.

If your last thirty days of traffic has a recognizable shape, scheduled scaling belongs in the stack.

What Datadog's Internal Autoscaling Migration Shows About Configuration at Scale

Datadog's platform team, operating under the name Rapid, supported more than 1,800 services and over 20,000 deployments. At that scale, per-service negotiation over autoscaling configuration is not an operational model; it is a reliable mechanism for producing inconsistency at volume. The team's migration produced more than three million dollars in annualized idle compute savings.

The mechanism was not new tooling. The savings came from applying consistent, correct configuration to existing infrastructure at scale. The enabler was guardrails: predefined safe configurations that allowed the team to apply autoscaling across 3,000 deployments in a single day without hand-tuning each one. Opinionated defaults, applied uniformly, produced results that years of individual service configuration had not.

The configuration principles are identical regardless of scale. The organizational unlock is the same too: a repeatable, policy-driven approach to threshold selection and signal configuration yields compounding returns as services proliferate. Per-service negotiation does not scale; a shared, well-reasoned policy does.

How Platforms Like Porter Remove the Configuration Burden Without Removing Control

The gap this article describes is not an ignorance problem. Most engineers working on Kubernetes infrastructure understand, at some level, that their minimum replicas are too high, their thresholds misconfigured, their cooldowns too conservative. The constraint is operational: correctly tuning and maintaining these settings across dozens of services requires continuous attention that product teams rarely have the bandwidth to sustain.

Porter deploys production-ready environments directly into a customer's own AWS, GCP, or Azure account, handling cluster autoscaling, cost optimization, and resource configuration at the platform level rather than delegating it to individual service owners. Because the infrastructure runs in the customer's own cloud account, cost savings from autoscaling flow directly to the customer's bill rather than disappearing into a shared-tenant pricing abstraction. Platforms like Heroku or Render operate on dyno and container slot billing, which obscures the relationship between autoscaling behavior and actual compute costs; the bring-your-own-cloud model makes that relationship transparent and direct.

The configuration decisions covered here, thresholds, signals, cooldowns, node provisioning, VPA baselines, GPU-specific scaling floors, are decisions a well-designed platform should encode as defaults. When they are encoded correctly, the product team does not have to revisit them. The autoscaler fires when it should. Nodes deprovision when the pods are gone. The bill reflects actual demand. You either do this work carefully and continuously yourself, or you use infrastructure that has already done it by design.