Tokenomicon + FinOps X 2027 San Diego · Jun 7-10 & London · Feb 8-9 Apply to attend
Assets
This work is licensed under CC BY 4.0 - Read how use or adaptation requires attribution

Optimizing GPU Utilization on AWS

At a Glance

  • The default metric overstates use. GPU utilization can read 100 percent while most of the chip sits idle; confirm with SM activity, tensor core activity, and memory bandwidth.
  • Judge each GPU by its workload. Idle GPUs between training jobs are avoidable cost, while warm inference replicas often protect the latency service-level objective (SLO).
  • Classify each GPU, not each instance. Peak utilization and hourly max sort GPUs into four classes that point to the right action.
  • Treat idle capacity as a scarcity problem. Capacity reservations, shared pools, and reacquisition guarantees address the cause; reporting reserved against productive GPU-hours shows what availability costs.
  • Price inference against latency targets. Aim for the lowest cost per 1M tokens that meets time to first token and inter-token latency targets, and check those before changing any real-time endpoint.

Companion slides: Optimizing GPU Utilization on AWS deck, for walking your team through the classes, metrics, and actions.

Executive Summary

GPUs are among the most expensive and hardest-to-acquire resources an organization runs, and the methods used to optimize CPU-based EC2 instances do not carry over to them. The default GPU utilization metric records only whether any work ran, so it can read high while most of the chip sits idle. Efficient use depends on the workload: training can be queued, but real-time inference must stay available to meet latency targets. And because capacity is scarce, teams that fear losing access have good reason to hold GPUs they are not using.

FinOps Practitioners and the Engineering, Finance, and Product Personas who run GPU workloads often read the same utilization data differently. A shared method turns “this GPU is 20 percent utilized” into a specific diagnosis and action. Peak utilization and hourly max place each GPU in one of four classes, read against its workload profile: training, real-time inference, batch inference, or development. Streaming multiprocessor (SM) activity confirms high readings, and supporting metrics such as time to first token identify the cause. Each combination maps to Usage Optimization, Rate Optimization, and Architecting & Workload Placement actions. Because GPU families have few sizes, stepping down usually means changing the instance family, sharing the GPU, or changing the model’s parallelism. The method also separates reserved GPU-hours from productive GPU-hours. The gap is the price of guaranteed availability, which the organization can then manage as a deliberate decision instead of hiding it in a low utilization number.

The goals are lower cost for the same output, reserved capacity whose cost is visible against productive GPU-hours, AI unit costs that teams can track and compare, and service levels that hold as costs fall. For inference, that means the lowest cost per 1M tokens that still meets the SLO. Examples use Amazon EC2, AWS capacity and pricing options, and NVIDIA tooling. Teams on other cloud providers or on-premises clusters will recognize the supporting metrics, but the named services are specific to AWS.

What “GPU Utilization” Actually Measures

The standard GPU utilization metric tells you whether the GPU was doing anything, not how much of it was working. Two terms help read the metric correctly:

  • Streaming multiprocessor (SM): the GPU’s compute unit. A data center GPU has many SMs, roughly like cores on a CPU. An SM counts as active even when it does very little work.
  • Kernel: a function the application launches on the GPU, such as one matrix multiply. It runs as blocks of threads across some or all SMs.

Two Metrics, Two Different Questions

Metric Question it answers CPU analogy
GPU utilization Was at least one kernel running during the sample? “Was any core busy?”
SM activity What share of time were the SMs active, averaged across all SMs? CPU utilization averaged across all cores

AWS defines the GPU Utilization CloudWatch metric as the percentage of time over the sample period during which one or more kernels on the GPU was running. NVIDIA defines SM activity as the share of time each SM had any work assigned, averaged over all multiprocessors.

NVIDIA’s own worked example shows how the two diverge. A kernel that uses one-fifth of the SMs for the whole interval has SM activity of 0.2, but GPU utilization near 100 percent. A kernel that uses every SM for one-fifth of the interval also has SM activity of 0.2, with GPU utilization about 20 percent.

How to Read the Default Metric

  • Low GPU utilization is reliable. If no kernel ran, no SM was busy, so low readings can be acted on.
  • High GPU utilization is ambiguous. A 60 percent reading could mean a few SMs or all of them. Confirm with SM activity before calling a GPU well used.
  • SM activity is necessary, not sufficient. NVIDIA says 0.8 or greater is necessary, but not sufficient, for effective use, and below 0.5 likely indicates ineffective use. Work waiting on memory still counts as active, so an active SM is not always computing.
  • Classify per GPU, then roll up. One busy GPU and seven idle ones on an 8-GPU instance average 12.5 percent, which hides the seven idle GPUs. Query CloudWatch per GPU to see each one’s utilization.

The Memory Metric Trap

GPU tools usually offer two memory metrics with similar names. One is the percentage of time device memory was being read or written, which measures how busy memory is; the other is memory used against total, which measures how full it is. Check which one a dashboard plots. On a GPU with a model loaded but idle, bandwidth reads near zero while capacity stays high. Because inference servers reserve most memory at startup, ask the owning team how much of that reservation their server actually uses before calling a GPU too big or too small.

Why EC2 Rightsizing Practices Do Not Fit GPU Instances

On CPU-based EC2 instances, CPU utilization is a fair proxy for how much of an instance is used. On GPU instances, GPU utilization is only part of the picture, and the economics differ on almost every dimension.

Dimension CPU-based EC2 instances GPU-accelerated EC2 instances (P and G families)
What “utilization” measures Share of vCPU capacity consumed Share of the sample period in which at least one kernel was running. A small kernel can read 100% while most of the chip sits unused
Metric availability Published to Amazon CloudWatch by default Not published by default. Requires the CloudWatch agent GPU plugin or the NVIDIA Data Center GPU Manager (DCGM) exporter
Rightsizing granularity Many sizes per family; step down one size Single-GPU G sizes share one GPU and differ in vCPU and memory; most current-generation P families have one 8-GPU size. Stepping down often means changing GPU type or partitioning the GPU
Main bottlenecks CPU, memory, network SM compute, GPU memory capacity, memory bandwidth, GPU interconnect, data pipeline, storage
Capacity Generally available on demand Constrained. Releasing an instance risks not getting it back
Elasticity Starts in seconds to minutes; stateless scale-out Model weights take minutes to load; multi-node training needs all nodes at once
Optimization levers Rightsizing, Savings Plans, Reserved Instances, Spot Instances Batching, GPU sharing, AWS Inferentia and AWS Trainium, quantization, scheduling, Spot Instances, Savings Plans, capacity reservations
Measure of value Requests served per instance Tokens per second, training steps, cost per 1M tokens, model FLOPs utilization (MFU)

Three implications follow:

  1. A single utilization number needs companion metrics to be trusted.
  2. The same number means different things for training and inference.
  3. Utilization targets that ignore capacity risk will be resisted or gamed.

Rightsizing Constraints Specific to GPU Instances

On GPU instances, the next size down often changes only the vCPU and memory around the GPU, or does not exist. Know which case applies before recommending a resize.

Single-GPU Sizes Differ Only in vCPU and Memory

Most G families have several sizes with exactly one GPU. A g7.8xlarge has the same GPU as a g7.4xlarge, with 32 vCPUs and 128 GiB of memory instead of 16 vCPUs and 64 GiB (EC2 specifications). Size names do not track GPU count: g5.12xlarge has four GPUs, while the larger g5.16xlarge has one.

Within these sizes, the only in-family lever is resizing the vCPU and memory around the same GPU:

  • GPU busy, CPU and host memory low: step down within the single-GPU sizes. The GPU stays the same, and the team stops paying for unused vCPU and memory.
  • GPU starved, CPU saturated: step up within the single-GPU sizes. Data loading and preprocessing run on the CPU, so a CPU bottleneck shows up as low GPU utilization.
  • GPU itself underused: no in-family size helps. Move to a smaller GPU family or consolidate workloads.

Some GPU Families Have Only One or Two Sizes

Several GPU families come in one size, or in sizes so far apart that stepping down removes most of the GPUs a workload uses.

Family Sizes What that means for rightsizing
P4d, P4de 24xlarge only (8 GPUs) No smaller size. Consolidate jobs, partition GPUs with MIG, change family, or release the instance
P5 4xlarge (1 GPU) or 48xlarge (8 GPUs) One step: a job that uses one H100 can move to p5.4xlarge
P5e, P5en, P6-B200, P6-B300 48xlarge only (8 GPUs) No smaller size. Same options as P4d
G5, G6, G6e 1-GPU sizes; 12xlarge and 24xlarge (4 GPUs); 48xlarge (8 GPUs) No 2-GPU size. Moving from 4 GPUs to 1 means re-sharding the model, not just resizing
G7, G7e 1-GPU sizes; 12xlarge (2 GPUs); 24xlarge (4 GPUs); 48xlarge (8 GPUs) Moving from 4 GPUs to 2 or 1 still means re-sharding the model

Getting a Smaller GPU When No Smaller Size Exists

  • Change family or GPU model: for example, move from the L40S (G6e) to the L4 (G6) if the model fits in the L4’s 24 GB of GPU memory. Benchmark first, since the result must still meet the latency target.
  • Multi-Instance GPU (MIG): splits one GPU into isolated instances, each with its own memory and SMs. GPUs in P4 and later P families support up to seven. Among G families, only the RTX PRO GPUs in G7 and G7e appear on NVIDIA’s supported list, with two and four instances. Confirm MIG is available on the instance before planning around it.
  • Time-slicing and shared endpoints: let several low-traffic models share one GPU without hard isolation.

AWS Compute Optimizer Recommendations

Compute Optimizer can recommend only instance types that exist. With so few GPU sizes, it rarely suggests a smaller GPU, so most GPU savings come from scheduling, consolidation, and family changes. At the time of writing:

Treat its output as a starting point, since it relies on GPU utilization and the limits described above.

Workload Profiles

Good utilization depends on whether the workload can be scheduled. Training can run at a set time; real-time inference must be available whenever a user sends a request.

Training (pre-training, large fine-tuning) Real-time inference Batch inference Development and notebooks
Access pattern Planned, long-running, often multi-node Ad hoc, user-driven, bursty Offline jobs over a dataset Interactive, human-paced
Schedulable? Yes: queued, time-boxed, checkpointed No: must stay up to meet latency targets Yes Partly
Bottlenecks Compute, GPU interconnect, data pipeline Prefill: compute. Decode: memory bandwidth Same as real-time inference The person using it; the GPU waits between runs
What fills GPU memory Model weights, gradients, and optimizer state; the most of any profile Model weights plus KV cache; servers reserve most memory at startup Same as real-time inference Varies; often far below capacity
What “good” looks like High sustained compute during the job; near-zero GPU-hours between jobs Latency SLO met at the largest batch the SLO allows Maximum throughput per GPU-hour GPUs released when the person stops working
Primary metrics SM activity, tensor activity, MFU, data-loader wait, checkpoint time Time to first token (TTFT), inter-token latency, throughput, cost per 1M SLO-compliant tokens Tokens or items per second, cost per 1M tokens Idle hours, session length

Why inference stays up. Requests arrive unpredictably, and interactive users expect responses in seconds. A cold start means provisioning an instance, pulling a container, and loading model weights. Weights range from gigabytes to hundreds of gigabytes, so this takes minutes. Teams keep a minimum number of replicas warm and hold headroom for peaks. Some low average utilization in real-time inference is structural: it is the price of the latency SLO.

Why training can be scheduled. A training job has a defined start and end, and checkpoints let it stop and resume. It can wait in a queue for free capacity and release that capacity when done. GPUs held idle between training jobs buy no service level, so they are avoidable cost. Queue time has a cost too, since researchers waiting for GPUs lose productivity. Size a shared pool for acceptable wait times as well as high utilization.

Inside inference, two phases behave differently. Prefill processes the prompt in parallel and is compute-bound. Decode generates one token at a time and is bound by memory bandwidth. A decode-heavy workload can show high GPU utilization while tensor cores sit mostly idle, which is why the supporting metrics include bandwidth and throughput.

Capacity Scarcity and Defensive Provisioning

Teams often call it hoarding, but holding idle GPUs is frequently a rational response to scarcity. When releasing an instance risks losing it for weeks, teams keep it running, and utilization targets alone will not change that.

Common behaviors driven by scarcity:

  • Keeping instances or reservations running between jobs so they are not lost
  • Requesting more GPUs than the next job needs, as a buffer
  • Running placeholder work to avoid being flagged as idle, once utilization becomes a tracked KPI

The last behavior matters for measurement design. If GPU utilization alone is the target, it is easy to inflate. Pairing it with SM activity and a business output metric (tokens served, training steps completed) exposes placeholder work.

Levers that address capacity supply:

Lever What it does Best fit FinOps Framework Capability
Amazon EC2 On-Demand Capacity Reservations (ODCRs) Holds specific capacity as long as needed, billed at On-Demand rates whether used or not. Savings Plans and Regional Reserved Instances apply to the reservation Steady inference base load Planning & Estimating; Rate Optimization
Amazon EC2 Capacity Blocks for ML Reserves GPU capacity for a fixed future window Planned training runs Planning & Estimating
Internal GPU pool with a scheduler (e.g., Slurm, or Kueue on Amazon EKS) Teams submit jobs to a shared pool instead of owning instances Training, batch inference, fine-tuning Usage Optimization
Reacquisition guarantee Released capacity returns to the team within an agreed time when needed Reducing defensive provisioning Governance, Policy & Risk
Showback of reserved-but-idle GPU-hours Makes the cost of holding capacity visible per team All profiles Allocation; Reporting & Analytics

A useful reporting split is reserved versus productive GPU-hours. The gap is the price of availability, which the organization can then manage as an explicit decision instead of hiding it in a low utilization number. FinOps Open Cost and Usage Specification (FOCUS) billing data shows part of this gap directly: the CapacityReservationStatus column marks ODCR charges as Used or Unused. Idle time on running instances still requires GPU metrics.

Illustrative example. An organization reserves 20 eight-GPU instances for a 30-day month, or 115,200 GPU-hours. At an illustrative $5 per GPU-hour, the reservation costs $576,000.

Reserved GPU-hours Share GPU-hours Cost Instance equivalent
Productive 60% 69,120 $345,600 12
Required availability headroom 25% 28,800 $144,000 5
Avoidable idle capacity 15% 17,280 $86,400 3
Total 100% 115,200 $576,000 20

Nonproductive capacity costs $230,400, but only the avoidable idle portion, $86,400, is an optimization opportunity; the headroom is availability the organization chose to pay for. Because reservations are released in whole instances, capturing it means consolidating onto 17 instances and releasing 3. Before counting that as savings, confirm how quickly the capacity could be reacquired, and apply the checks in Measuring Financial Outcomes.

GPU Utilization Classes

Two metrics over an evaluation window place each GPU in one of four classes. Idle is defined exactly; the other thresholds are placeholders to calibrate against the fleet’s own distribution.

The two metrics:

  1. Peak utilization: the maximum GPU utilization observed in the window (for example, 14 or 30 days).
  2. Hourly max: the maximum GPU utilization within each hour, summarized across the window (for example, the median hour, or the share of hours above a threshold).

Peak shows whether the GPU is ever used heavily; hourly max shows how often it is used at all. Hourly max can never exceed peak, so the classes sit on or below a diagonal.

Class Peak utilization Hourly max (typical hour) Reads as Likely causes
Idle 0% 0% in every hour No kernel ran during the window Abandoned instance, forgotten notebook, capacity held for fear of losing it
Low-Spiky High, e.g., ≥ [60]% Low, e.g., ≤ [10]% Capacity sized for occasional bursts Ad hoc inference traffic, short experiments, jobs run once a day
Low-Persistent Low, e.g., < [30]% Low but non-zero most hours Always on, never pushed Model too small for the GPU, batch size too small, data pipeline starving the GPU
High-Sustained High, e.g., ≥ [60]% High, e.g., ≥ [40]% Used heavily and often Well-sized training or batch inference; busy shared endpoint. Also small kernels, memory-bound decode, or placeholder work inflating the reading

Measurement notes:

  • Standardize the measurement settings. Use one sampling interval, one evaluation window, and one set of thresholds across the fleet, and record them with the results. An hourly max from 1-minute averages behaves differently from one from 10-second samples, so GPUs measured differently cannot be compared. Recalibrate thresholds against the fleet’s distribution when the workload mix changes.
  • The class is a symptom, not a verdict. Read it with the workload profile and supporting metrics before acting. A training GPU and a real-time inference GPU can both be Low-Spiky (peak above 60 percent, typical hour at or below 10 percent). On the training GPU, the quiet hours are capacity held between jobs and usually avoidable; on the inference GPU, they may be the headroom that keeps TTFT within its SLO at peak. Check latency percentiles and business requirements before treating either as an opportunity.
  • High-Sustained does not mean well used. It can read high while few SMs are busy, while decode waits on memory bandwidth, or while placeholder work keeps the number up. Confirm with SM activity and throughput. Then check MFU and goodput for training, or cost per 1M tokens and latency percentiles for inference, before treating the GPU as optimized.

Supporting Metrics

The classification shows whether a GPU is underused; supporting metrics show why. They fall into four groups: hardware activity, inference service, training efficiency, and data and storage.

Hardware Activity (from NVIDIA DCGM)

Metric DCGM field What it adds
GPU utilization DCGM_FI_DEV_GPU_UTIL Time any kernel ran. The input to the classification, but coarse
SM activity DCGM_FI_PROF_SM_ACTIVE Share of streaming multiprocessors actually busy. Exposes “100% utilized but mostly empty”
Tensor core activity DCGM_FI_PROF_PIPE_TENSOR_ACTIVE Use of the matrix units that make GPUs worth their price
Memory bandwidth DCGM_FI_PROF_DRAM_ACTIVE The real limit for decode-heavy inference
GPU memory used DCGM_FI_DEV_FB_USED Capacity headroom. Inference servers such as vLLM pre-allocate about 90% of memory by default, so high memory use does not mean the GPU is busy
Interconnect NVLink and Elastic Fabric Adapter (EFA) counters Communication stalls in multi-node training

Inference Service Metrics

  • Time to first token (TTFT): time from request to first output token, driven by queueing and prefill. It is the main latency SLO for interactive use, usually tracked at p50, p95, and p99.
  • Inter-token latency (ITL), also called time per output token: the decode speed a user perceives while text streams.
  • Throughput: output tokens per second per GPU, and requests per second.
  • Goodput: throughput from requests that met the SLO, measured per request from inference telemetry against TTFT and ITL thresholds. Larger batches raise throughput until requests start breaching the SLO; goodput shows where that happens.
  • Cost per 1M tokens: the Unit Economics metric that ties utilization to spend.

Formula: Cost per 1M tokens = effective GPU cost per hour ÷ (tokens per second × 3,600) × 1,000,000

Effective cost includes amortized Savings Plan or Reserved Instance coverage and attributable storage, consistent with FOCUS EffectiveCost.

For real-time inference, count only SLO-compliant tokens: output tokens from requests that met both the TTFT and ITL thresholds. Compliance must come from request telemetry, because GPU utilization says nothing about whether a request met its latency target.

For example, an endpoint costing an illustrative $10 per hour produces 648,000 output tokens per hour, or $15.43 per 1M tokens. If 518,400 of those tokens come from requests that met the SLO, the cost rises to $19.29 per 1M SLO-compliant tokens, because the 20 percent that missed its latency target was still paid for. For real-time inference, the target is the lowest cost per 1M SLO-compliant tokens; raw GPU utilization is secondary.

Training Efficiency

  • Model FLOPs utilization (MFU): achieved FLOPs as a share of the hardware’s theoretical peak. Published large language model runs report roughly 35–45 percent.
  • Step time and training goodput: the share of wall-clock time spent on productive steps, excluding restarts, checkpoint stalls, and failures.

Data and Storage

Storage affects GPU utilization and is a cost line of its own.

  • Data-loader wait: time the GPU waits for the next batch. A common cause of Low-Persistent training GPUs, driven by storage throughput (for example, Amazon FSx for Lustre or Amazon S3 reads), CPU preprocessing, and network.
  • Checkpoint write time and size: a full checkpoint with optimizer state is roughly 12–16 bytes per parameter, or about 1 TB for a 70B-parameter model. Slow writes stall GPUs, and retained checkpoints add storage cost.
  • Model load time (cold start): decides whether scaling inference to zero can stay within the SLO.
  • Storage cost per GPU-hour: file systems provisioned for training often stay up after the cluster is released.

The same class calls for different actions depending on the profile. Scheduling addresses most avoidable training cost; batching, GPU sharing, and smaller hardware address most avoidable inference cost.

Class Training and fine-tuning Real-time inference Batch inference Development and notebooks
Idle Release to a shared pool; submit future work through a job queue Confirm whether it is a deliberate warm standby. If not, scale to minimum, or to zero where cold start fits the SLO Stop; run on a schedule Auto-stop after a set idle period
Low-Spiky Consolidate short jobs under a scheduler; use Capacity Blocks for ML for planned bursts; run checkpointed jobs on Spot Instances Autoscale on concurrency or queue depth; size minimum replicas to base load; host several models per endpoint Pack jobs into fewer, fuller windows Move to a shared pool with on-demand checkout
Low-Persistent Check data-loader wait and storage throughput first; then raise batch size or move to a smaller GPU or AWS Trainium Enable continuous batching; share the GPU (MIG, time-slicing); quantize; move to a smaller GPU or AWS Inferentia Raise batch size; check the data pipeline Use a GPU slice or a smaller GPU
High-Sustained Track MFU and training goodput; cover with Savings Plans Track cost per 1M SLO-compliant tokens; cover base load with Savings Plans Track cost per 1M tokens; use Spot Instances for interruptible jobs Confirm with SM activity

These actions fall under three FinOps Capabilities: Usage Optimization (scheduling, batching, auto-stop), Architecting & Workload Placement (instance family, GPU sharing, AWS Inferentia and AWS Trainium), and Rate Optimization (Savings Plans, Spot Instances).

Before any action on a real-time endpoint, check TTFT and ITL percentiles. A change that lowers cost but breaches the SLO moves cost to the business, not out of it.

Validate the savings. Higher utilization lowers the bill only if the organization then runs, reserves, or commits to less. Larger batches or consolidated jobs free capacity, but the bill does not change until instances stop or reservations are released, and Savings Plan and Reserved Instance commitments keep billing regardless. Measure the result as described in Measuring Financial Outcomes.

Measuring Financial Outcomes

Utilization metrics show how GPUs are used. Four financial measures help FinOps Practitioners, Engineering, Product, and Finance judge whether that use is worth its cost and whether an optimization saved money.

Effective cost per GPU-hour. The amortized cost of each provisioned GPU-hour, including Savings Plan, Reserved Instance, and capacity reservation coverage, plus attributable storage and networking. It feeds the cost per 1M tokens formula in Supporting Metrics. Track it separately from marginal cost: releasing a GPU covered by a commitment or reservation saves little unless the commitment can be used elsewhere.

Productive GPU-hour ratio. Productive GPU-hours divided by provisioned GPU-hours. Define “productive” for each workload profile, such as SM activity above an agreed threshold for training, or traffic served within the SLO for inference. Report deliberate headroom, such as warm inference replicas, separately from avoidable idle capacity.

Unit cost by workload. For inference, track cost per 1M SLO-compliant tokens (see Supporting Metrics), reporting input and output tokens separately, since prefill and decode load the GPU differently. For training, track cost per completed run or per effective training hour.

Realized savings. Before each optimization, record the baseline cost, estimated savings, implementation effort, and expected effect on the workload, then measure the savings afterward on the bill. Account for existing commitments, reservations, migration effort, and any change in service levels, so the result reflects money the organization actually stopped spending.

These measures fall under the Unit Economics and KPIs & Benchmarking Capabilities. They give teams a common basis for comparing GPU efficiency across workloads and choosing which optimizations to pursue first. They show whether GPUs are used efficiently. Whether the output is worth producing is a question for product owners and executives, and it belongs to the broader AI Economics (Tokenomics) conversation.

Conclusion

A GPU utilization number alone says little about whether a GPU is well used. Read against the workload profile and supporting metrics, it points to a specific cause: an abandoned notebook, a data pipeline starving the GPU, or an inference endpoint sized for its latency target. Once FinOps Practitioners and Engineering teams agree on the cause, the right action follows.

Most teams can start with four steps:

  1. Collect DCGM profiling metrics alongside GPU utilization, since SM and tensor activity are not always enabled.
  2. Classify the fleet per GPU, and calibrate class thresholds against its own distribution.
  3. Report reserved against productive GPU-hours by team.
  4. Before changing any real-time endpoint, agree on its latency targets with the team that owns it.

Some idle GPU capacity is worth paying for. Warm inference replicas and reserved training capacity protect service levels and keep researchers productive while GPUs are scarce. The goal is for the organization to choose how much availability to buy, with the cost visible to the teams that hold it.

Acknowledgments

The FinOps Foundation thanks the following contributors: