Companion slides: Optimizing GPU Utilization on AWS deck, for walking your team through the classes, metrics, and actions.
GPUs are among the most expensive and hardest-to-acquire resources an organization runs, and the methods used to optimize CPU-based EC2 instances do not carry over to them. The default GPU utilization metric records only whether any work ran, so it can read high while most of the chip sits idle. Efficient use depends on the workload: training can be queued, but real-time inference must stay available to meet latency targets. And because capacity is scarce, teams that fear losing access have good reason to hold GPUs they are not using.
FinOps Practitioners and the Engineering, Finance, and Product Personas who run GPU workloads often read the same utilization data differently. A shared method turns “this GPU is 20 percent utilized” into a specific diagnosis and action. Peak utilization and hourly max place each GPU in one of four classes, read against its workload profile: training, real-time inference, batch inference, or development. Streaming multiprocessor (SM) activity confirms high readings, and supporting metrics such as time to first token identify the cause. Each combination maps to Usage Optimization, Rate Optimization, and Architecting & Workload Placement actions. Because GPU families have few sizes, stepping down usually means changing the instance family, sharing the GPU, or changing the model’s parallelism. The method also separates reserved GPU-hours from productive GPU-hours. The gap is the price of guaranteed availability, which the organization can then manage as a deliberate decision instead of hiding it in a low utilization number.
The goals are lower cost for the same output, reserved capacity whose cost is visible against productive GPU-hours, AI unit costs that teams can track and compare, and service levels that hold as costs fall. For inference, that means the lowest cost per 1M tokens that still meets the SLO. Examples use Amazon EC2, AWS capacity and pricing options, and NVIDIA tooling. Teams on other cloud providers or on-premises clusters will recognize the supporting metrics, but the named services are specific to AWS.
The standard GPU utilization metric tells you whether the GPU was doing anything, not how much of it was working. Two terms help read the metric correctly:
| Metric | Question it answers | CPU analogy |
|---|---|---|
| GPU utilization | Was at least one kernel running during the sample? | “Was any core busy?” |
| SM activity | What share of time were the SMs active, averaged across all SMs? | CPU utilization averaged across all cores |
AWS defines the GPU Utilization CloudWatch metric as the percentage of time over the sample period during which one or more kernels on the GPU was running. NVIDIA defines SM activity as the share of time each SM had any work assigned, averaged over all multiprocessors.
NVIDIA’s own worked example shows how the two diverge. A kernel that uses one-fifth of the SMs for the whole interval has SM activity of 0.2, but GPU utilization near 100 percent. A kernel that uses every SM for one-fifth of the interval also has SM activity of 0.2, with GPU utilization about 20 percent.
GPU tools usually offer two memory metrics with similar names. One is the percentage of time device memory was being read or written, which measures how busy memory is; the other is memory used against total, which measures how full it is. Check which one a dashboard plots. On a GPU with a model loaded but idle, bandwidth reads near zero while capacity stays high. Because inference servers reserve most memory at startup, ask the owning team how much of that reservation their server actually uses before calling a GPU too big or too small.
On CPU-based EC2 instances, CPU utilization is a fair proxy for how much of an instance is used. On GPU instances, GPU utilization is only part of the picture, and the economics differ on almost every dimension.
| Dimension | CPU-based EC2 instances | GPU-accelerated EC2 instances (P and G families) |
|---|---|---|
| What “utilization” measures | Share of vCPU capacity consumed | Share of the sample period in which at least one kernel was running. A small kernel can read 100% while most of the chip sits unused |
| Metric availability | Published to Amazon CloudWatch by default | Not published by default. Requires the CloudWatch agent GPU plugin or the NVIDIA Data Center GPU Manager (DCGM) exporter |
| Rightsizing granularity | Many sizes per family; step down one size | Single-GPU G sizes share one GPU and differ in vCPU and memory; most current-generation P families have one 8-GPU size. Stepping down often means changing GPU type or partitioning the GPU |
| Main bottlenecks | CPU, memory, network | SM compute, GPU memory capacity, memory bandwidth, GPU interconnect, data pipeline, storage |
| Capacity | Generally available on demand | Constrained. Releasing an instance risks not getting it back |
| Elasticity | Starts in seconds to minutes; stateless scale-out | Model weights take minutes to load; multi-node training needs all nodes at once |
| Optimization levers | Rightsizing, Savings Plans, Reserved Instances, Spot Instances | Batching, GPU sharing, AWS Inferentia and AWS Trainium, quantization, scheduling, Spot Instances, Savings Plans, capacity reservations |
| Measure of value | Requests served per instance | Tokens per second, training steps, cost per 1M tokens, model FLOPs utilization (MFU) |
Three implications follow:
On GPU instances, the next size down often changes only the vCPU and memory around the GPU, or does not exist. Know which case applies before recommending a resize.
Most G families have several sizes with exactly one GPU. A g7.8xlarge has the same GPU as a g7.4xlarge, with 32 vCPUs and 128 GiB of memory instead of 16 vCPUs and 64 GiB (EC2 specifications). Size names do not track GPU count: g5.12xlarge has four GPUs, while the larger g5.16xlarge has one.
Within these sizes, the only in-family lever is resizing the vCPU and memory around the same GPU:
Several GPU families come in one size, or in sizes so far apart that stepping down removes most of the GPUs a workload uses.
| Family | Sizes | What that means for rightsizing |
|---|---|---|
| P4d, P4de | 24xlarge only (8 GPUs) | No smaller size. Consolidate jobs, partition GPUs with MIG, change family, or release the instance |
| P5 | 4xlarge (1 GPU) or 48xlarge (8 GPUs) | One step: a job that uses one H100 can move to p5.4xlarge |
| P5e, P5en, P6-B200, P6-B300 | 48xlarge only (8 GPUs) | No smaller size. Same options as P4d |
| G5, G6, G6e | 1-GPU sizes; 12xlarge and 24xlarge (4 GPUs); 48xlarge (8 GPUs) | No 2-GPU size. Moving from 4 GPUs to 1 means re-sharding the model, not just resizing |
| G7, G7e | 1-GPU sizes; 12xlarge (2 GPUs); 24xlarge (4 GPUs); 48xlarge (8 GPUs) | Moving from 4 GPUs to 2 or 1 still means re-sharding the model |
Compute Optimizer can recommend only instance types that exist. With so few GPU sizes, it rarely suggests a smaller GPU, so most GPU savings come from scheduling, consolidation, and family changes. At the time of writing:
Treat its output as a starting point, since it relies on GPU utilization and the limits described above.
Good utilization depends on whether the workload can be scheduled. Training can run at a set time; real-time inference must be available whenever a user sends a request.
| Training (pre-training, large fine-tuning) | Real-time inference | Batch inference | Development and notebooks | |
|---|---|---|---|---|
| Access pattern | Planned, long-running, often multi-node | Ad hoc, user-driven, bursty | Offline jobs over a dataset | Interactive, human-paced |
| Schedulable? | Yes: queued, time-boxed, checkpointed | No: must stay up to meet latency targets | Yes | Partly |
| Bottlenecks | Compute, GPU interconnect, data pipeline | Prefill: compute. Decode: memory bandwidth | Same as real-time inference | The person using it; the GPU waits between runs |
| What fills GPU memory | Model weights, gradients, and optimizer state; the most of any profile | Model weights plus KV cache; servers reserve most memory at startup | Same as real-time inference | Varies; often far below capacity |
| What “good” looks like | High sustained compute during the job; near-zero GPU-hours between jobs | Latency SLO met at the largest batch the SLO allows | Maximum throughput per GPU-hour | GPUs released when the person stops working |
| Primary metrics | SM activity, tensor activity, MFU, data-loader wait, checkpoint time | Time to first token (TTFT), inter-token latency, throughput, cost per 1M SLO-compliant tokens | Tokens or items per second, cost per 1M tokens | Idle hours, session length |
Why inference stays up. Requests arrive unpredictably, and interactive users expect responses in seconds. A cold start means provisioning an instance, pulling a container, and loading model weights. Weights range from gigabytes to hundreds of gigabytes, so this takes minutes. Teams keep a minimum number of replicas warm and hold headroom for peaks. Some low average utilization in real-time inference is structural: it is the price of the latency SLO.
Why training can be scheduled. A training job has a defined start and end, and checkpoints let it stop and resume. It can wait in a queue for free capacity and release that capacity when done. GPUs held idle between training jobs buy no service level, so they are avoidable cost. Queue time has a cost too, since researchers waiting for GPUs lose productivity. Size a shared pool for acceptable wait times as well as high utilization.
Inside inference, two phases behave differently. Prefill processes the prompt in parallel and is compute-bound. Decode generates one token at a time and is bound by memory bandwidth. A decode-heavy workload can show high GPU utilization while tensor cores sit mostly idle, which is why the supporting metrics include bandwidth and throughput.
Teams often call it hoarding, but holding idle GPUs is frequently a rational response to scarcity. When releasing an instance risks losing it for weeks, teams keep it running, and utilization targets alone will not change that.
Common behaviors driven by scarcity:
The last behavior matters for measurement design. If GPU utilization alone is the target, it is easy to inflate. Pairing it with SM activity and a business output metric (tokens served, training steps completed) exposes placeholder work.
Levers that address capacity supply:
| Lever | What it does | Best fit | FinOps Framework Capability |
|---|---|---|---|
| Amazon EC2 On-Demand Capacity Reservations (ODCRs) | Holds specific capacity as long as needed, billed at On-Demand rates whether used or not. Savings Plans and Regional Reserved Instances apply to the reservation | Steady inference base load | Planning & Estimating; Rate Optimization |
| Amazon EC2 Capacity Blocks for ML | Reserves GPU capacity for a fixed future window | Planned training runs | Planning & Estimating |
| Internal GPU pool with a scheduler (e.g., Slurm, or Kueue on Amazon EKS) | Teams submit jobs to a shared pool instead of owning instances | Training, batch inference, fine-tuning | Usage Optimization |
| Reacquisition guarantee | Released capacity returns to the team within an agreed time when needed | Reducing defensive provisioning | Governance, Policy & Risk |
| Showback of reserved-but-idle GPU-hours | Makes the cost of holding capacity visible per team | All profiles | Allocation; Reporting & Analytics |
A useful reporting split is reserved versus productive GPU-hours. The gap is the price of availability, which the organization can then manage as an explicit decision instead of hiding it in a low utilization number. FinOps Open Cost and Usage Specification (FOCUS) billing data shows part of this gap directly: the CapacityReservationStatus column marks ODCR charges as Used or Unused. Idle time on running instances still requires GPU metrics.
Illustrative example. An organization reserves 20 eight-GPU instances for a 30-day month, or 115,200 GPU-hours. At an illustrative $5 per GPU-hour, the reservation costs $576,000.
| Reserved GPU-hours | Share | GPU-hours | Cost | Instance equivalent |
|---|---|---|---|---|
| Productive | 60% | 69,120 | $345,600 | 12 |
| Required availability headroom | 25% | 28,800 | $144,000 | 5 |
| Avoidable idle capacity | 15% | 17,280 | $86,400 | 3 |
| Total | 100% | 115,200 | $576,000 | 20 |
Nonproductive capacity costs $230,400, but only the avoidable idle portion, $86,400, is an optimization opportunity; the headroom is availability the organization chose to pay for. Because reservations are released in whole instances, capturing it means consolidating onto 17 instances and releasing 3. Before counting that as savings, confirm how quickly the capacity could be reacquired, and apply the checks in Measuring Financial Outcomes.
Two metrics over an evaluation window place each GPU in one of four classes. Idle is defined exactly; the other thresholds are placeholders to calibrate against the fleet’s own distribution.
The two metrics:
Peak shows whether the GPU is ever used heavily; hourly max shows how often it is used at all. Hourly max can never exceed peak, so the classes sit on or below a diagonal.
| Class | Peak utilization | Hourly max (typical hour) | Reads as | Likely causes |
|---|---|---|---|---|
| Idle | 0% | 0% in every hour | No kernel ran during the window | Abandoned instance, forgotten notebook, capacity held for fear of losing it |
| Low-Spiky | High, e.g., ≥ [60]% | Low, e.g., ≤ [10]% | Capacity sized for occasional bursts | Ad hoc inference traffic, short experiments, jobs run once a day |
| Low-Persistent | Low, e.g., < [30]% | Low but non-zero most hours | Always on, never pushed | Model too small for the GPU, batch size too small, data pipeline starving the GPU |
| High-Sustained | High, e.g., ≥ [60]% | High, e.g., ≥ [40]% | Used heavily and often | Well-sized training or batch inference; busy shared endpoint. Also small kernels, memory-bound decode, or placeholder work inflating the reading |
Measurement notes:
The classification shows whether a GPU is underused; supporting metrics show why. They fall into four groups: hardware activity, inference service, training efficiency, and data and storage.
| Metric | DCGM field | What it adds |
|---|---|---|
| GPU utilization | DCGM_FI_DEV_GPU_UTIL |
Time any kernel ran. The input to the classification, but coarse |
| SM activity | DCGM_FI_PROF_SM_ACTIVE |
Share of streaming multiprocessors actually busy. Exposes “100% utilized but mostly empty” |
| Tensor core activity | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE |
Use of the matrix units that make GPUs worth their price |
| Memory bandwidth | DCGM_FI_PROF_DRAM_ACTIVE |
The real limit for decode-heavy inference |
| GPU memory used | DCGM_FI_DEV_FB_USED |
Capacity headroom. Inference servers such as vLLM pre-allocate about 90% of memory by default, so high memory use does not mean the GPU is busy |
| Interconnect | NVLink and Elastic Fabric Adapter (EFA) counters | Communication stalls in multi-node training |
Formula: Cost per 1M tokens = effective GPU cost per hour ÷ (tokens per second × 3,600) × 1,000,000
Effective cost includes amortized Savings Plan or Reserved Instance coverage and attributable storage, consistent with FOCUS EffectiveCost.
For real-time inference, count only SLO-compliant tokens: output tokens from requests that met both the TTFT and ITL thresholds. Compliance must come from request telemetry, because GPU utilization says nothing about whether a request met its latency target.
For example, an endpoint costing an illustrative $10 per hour produces 648,000 output tokens per hour, or $15.43 per 1M tokens. If 518,400 of those tokens come from requests that met the SLO, the cost rises to $19.29 per 1M SLO-compliant tokens, because the 20 percent that missed its latency target was still paid for. For real-time inference, the target is the lowest cost per 1M SLO-compliant tokens; raw GPU utilization is secondary.
Storage affects GPU utilization and is a cost line of its own.
The same class calls for different actions depending on the profile. Scheduling addresses most avoidable training cost; batching, GPU sharing, and smaller hardware address most avoidable inference cost.
| Class | Training and fine-tuning | Real-time inference | Batch inference | Development and notebooks |
|---|---|---|---|---|
| Idle | Release to a shared pool; submit future work through a job queue | Confirm whether it is a deliberate warm standby. If not, scale to minimum, or to zero where cold start fits the SLO | Stop; run on a schedule | Auto-stop after a set idle period |
| Low-Spiky | Consolidate short jobs under a scheduler; use Capacity Blocks for ML for planned bursts; run checkpointed jobs on Spot Instances | Autoscale on concurrency or queue depth; size minimum replicas to base load; host several models per endpoint | Pack jobs into fewer, fuller windows | Move to a shared pool with on-demand checkout |
| Low-Persistent | Check data-loader wait and storage throughput first; then raise batch size or move to a smaller GPU or AWS Trainium | Enable continuous batching; share the GPU (MIG, time-slicing); quantize; move to a smaller GPU or AWS Inferentia | Raise batch size; check the data pipeline | Use a GPU slice or a smaller GPU |
| High-Sustained | Track MFU and training goodput; cover with Savings Plans | Track cost per 1M SLO-compliant tokens; cover base load with Savings Plans | Track cost per 1M tokens; use Spot Instances for interruptible jobs | Confirm with SM activity |
These actions fall under three FinOps Capabilities: Usage Optimization (scheduling, batching, auto-stop), Architecting & Workload Placement (instance family, GPU sharing, AWS Inferentia and AWS Trainium), and Rate Optimization (Savings Plans, Spot Instances).
Before any action on a real-time endpoint, check TTFT and ITL percentiles. A change that lowers cost but breaches the SLO moves cost to the business, not out of it.
Validate the savings. Higher utilization lowers the bill only if the organization then runs, reserves, or commits to less. Larger batches or consolidated jobs free capacity, but the bill does not change until instances stop or reservations are released, and Savings Plan and Reserved Instance commitments keep billing regardless. Measure the result as described in Measuring Financial Outcomes.
Utilization metrics show how GPUs are used. Four financial measures help FinOps Practitioners, Engineering, Product, and Finance judge whether that use is worth its cost and whether an optimization saved money.
Effective cost per GPU-hour. The amortized cost of each provisioned GPU-hour, including Savings Plan, Reserved Instance, and capacity reservation coverage, plus attributable storage and networking. It feeds the cost per 1M tokens formula in Supporting Metrics. Track it separately from marginal cost: releasing a GPU covered by a commitment or reservation saves little unless the commitment can be used elsewhere.
Productive GPU-hour ratio. Productive GPU-hours divided by provisioned GPU-hours. Define “productive” for each workload profile, such as SM activity above an agreed threshold for training, or traffic served within the SLO for inference. Report deliberate headroom, such as warm inference replicas, separately from avoidable idle capacity.
Unit cost by workload. For inference, track cost per 1M SLO-compliant tokens (see Supporting Metrics), reporting input and output tokens separately, since prefill and decode load the GPU differently. For training, track cost per completed run or per effective training hour.
Realized savings. Before each optimization, record the baseline cost, estimated savings, implementation effort, and expected effect on the workload, then measure the savings afterward on the bill. Account for existing commitments, reservations, migration effort, and any change in service levels, so the result reflects money the organization actually stopped spending.
These measures fall under the Unit Economics and KPIs & Benchmarking Capabilities. They give teams a common basis for comparing GPU efficiency across workloads and choosing which optimizations to pursue first. They show whether GPUs are used efficiently. Whether the output is worth producing is a question for product owners and executives, and it belongs to the broader AI Economics (Tokenomics) conversation.
A GPU utilization number alone says little about whether a GPU is well used. Read against the workload profile and supporting metrics, it points to a specific cause: an abandoned notebook, a data pipeline starving the GPU, or an inference endpoint sized for its latency target. Once FinOps Practitioners and Engineering teams agree on the cause, the right action follows.
Most teams can start with four steps:
Some idle GPU capacity is worth paying for. Warm inference replicas and reserved training capacity protect service levels and keep researchers productive while GPUs are scarce. The goal is for the organization to choose how much availability to buy, with the cost visible to the teams that hold it.
The FinOps Foundation thanks the following contributors: