The Case of too Many EBS Volumes

AWS
by Joe Daly, Nationwide

Forgetting to delete EBS volumes is extremely common for engineers just starting their cloud journey. At a small scale, they accumulate slowly and most third party cloud cost management platforms can identify and even terminate them for you. However, all it takes is one buggy autoscaling group that doesn’t properly cleanup after itself after it scales down and you could be spending thousands of dollars a day in abandoned EBS volume waste.

We identified this problem through the use of our third party cloud cost management tool, which identified unattached EBS volumes and recommended we terminate them. We quickly incurred a problem with keeping up with all of the unattached EBS volumes.

Our cloud engineering team wrote a script that identified abandoned EBS volumes and tagged them with the date they were abandoned. Then 5 or 14 days later, depending on what environment they are in, a snapshot is taken and they are terminated.

One of the benefits of this method is that it is easy to create a visualization to track wasteful spend due to the abandoned tag date. This also makes it easy to determine if the script stops working or if there are any unanticipated spikes in abandoned EBS volumes. When abandoned EBS costs seem like they are running high, I run a report on their detach date in order to check the script is still working

Resources opt-in to this script by default. In the rare case that abandoned EBS volumes are a feature rather than waste (it happens), the app teams have to opt-out. They can tag their resources to not be included in the script.

It is important to note: when abandoned EBS volumes are growing exponentially, it is a symptom of a problem. Make sure you treat the symptom so you stop your cloud bill from bleeding, but it’s important to identify the problem causing the unattached EBS volumes. Is it an autoscaling group gone wild? Is it a faulty AMI? Lean in and treat the source.

Related Member Stories

A Guide for Adopting FinOps in Your Organization

AWS
Azure
GCP
Industry: Internet
Persona: FinOps Practitioner
by F2 Working Group, FinOps Foundation

One of the biggest challenges in starting a FinOps practice is getting broad executive support and buy-in to dedicate the time and resources needed for the cultural change.

Read more

Removing AMI Snapshots

AWS
by Stephanie Gooch, AWS

Upon reviewing the amount of snapshots a customer had we found a large proportion of them were created from AMIs. This was found by listing all available amis in and connecting them back to the snapshot using the description. However, many of the AMIs that created them had been released....

Read more

Architecting Cloud Workloads for Financial Reporting

AWS
Azure
GCP
Industry: Information Technology & Services
Persona: FinOps Practitioner
by Rich Hoyer, SADA

A list of best practices for cloud architects to design systems to optimize FinOps.

Read more