For Week 8 of our Member Calls to dig into best practices, we heard from Bhupendra Hirani at YouView about building cost awareness with your engineering team.
(A full video of the presentation is available for FinOps Foundation Members)
Bhupendra “Bhups” Hirani is the Optimisation Manager and Systems Administrator at YouView, a content delivery platform with content from some of the biggest names in media. Similar to Darek Gajewski who presented last week, Bhups spent some time in the capacity management trenches before taking a more FinOps path in the cloud, making him familiar with both technology and finance sides of the conversation.
This understanding is crucial to his FinOps efforts, since it allows him to effectively bridge the gap between the engineering and finance teams. In our member call this week, he shared some of his lessons and insights into spreading cost awareness among engineers — a crucial step in any organization’s FinOps journey.
Lesson #1: Find the Motivation for Cost Awareness
On the surface, everyone agrees that cost awareness is a good thing, but you have to go a little deeper if you want the concept to really dig in enough to cause a long-term change in behavior. So before you start talking to the engineering team, get an understanding of how they work and what incentives they may have. In essence, you need to answer the question, “Why do I need to care about cost awareness?”
There a number of reasons you might find to answer that question. Engineers might get bonuses if their team is under budget at the end of the year, or keeping costs low might open up funding for their pet projects and new engineers. (A few weeks ago, Anders Hagman from Spotify also had some good insights on putting cost in terms engineers understand.)
At the same time, remember that every engineer is different. They have their favorite services and vendors, their preferred working style, and they might have different employment grounds, such as contractors. Your conversations will have to adapt to these differences. For example, conversations about bonuses might not mean much if the engineer isn’t eligible for them, and a focus on optimizing RIs might be less effective for someone always experimenting with new services.
Come at the conversation on their terms and you’ll get better results.
Lesson #2: Educate Beyond the Cost Basics…but Not Too Much
If you got a bunch of engineers together and ask them what generates costs, they’d probably instantly spit out the most common cost centers, like AWS EC2, AWS S3, etc. But there’s so much more that can generate costs in a cloud architecture, and they’re not going to become aware of the costs if they’re not aware of what can generate them.
So make sure to go a little deeper into how costs are generated and influenced by the cloud architecture. Go over things like data transfer fees, service licenses, monitoring, logging, alerting, backups, API calls, security services and more. By being aware of these, they can start making better choices.
At the same time, you don’t have to go too far into the weeds. Things like special volume pricing, Reserved Instances/Committed Use Discounts, true up costs or currency conversions can distract engineers. Try to find the balance that allows them to take control of their costs without getting mired in the cost weeds.
When you educate, your goal should be to enable them as much as possible to do their job better.
Lesson #3: Find a Way to Compare Apples to Apples Between Teams
Gamification is a great way to motivate engineering teams. Competing with each other for an optimization prize (even if it’s just bragging rights!) makes the process more fun and more effective. But it’s important to make sure it’s a level playing field and that you’re playing by the same rules. That’s why it’s not a good idea to compare raw currency alone, but instead to find a way to compare everyone on the same grounds.
Think of it this way: If Team A has a budget of $100,000 per month and drops their costs by 5%, then they’ve saved $5,000. Team B might drop their costs by an impressive 20%, but with a budget of only $25,000, it comes out to the same amount, even though it probably took a lot more cleverness and effort for Team B — effort that should be rewarded. On the flip side, Team B might look at their spend of $25,000 each month and decide not to do anything since they’re doing so much better than Team A and its huge $100,000 spend.
In either of these scenarios, a number of factors are getting missed, including the relative size of the teams, relative complexity of their architecture and more. It’s better to find more universal metrics, like how well they’re doing on rightsizing, tagging coverage or spot usage for certain workloads.
Additional Lessons from the Q&A
After Bhups finished his presentation, there was a general discussion about the subject, including about what to do when engineers make costly mistakes. There were various approaches, and here are some key insights:
- Publicly applaud good behavior, but address mistakes privately.
- Try turning the mistake into an educational opportunity, such as by writing an internal blog post or talking about the problem (and its solution!) without pointing fingers.
- In a large organization, try going through team leaders and let them work it out with the engineer instead of putting the engineer on blast in front of the whole company.
If you want to hear the whole discussions, along with real-life examples, then join us in the FinOps Foundation!
All in all, it was great to see how different FinOps practitioners tackle the monumental task of bringing all of the engineers into the conversation. And even better to hear how successful they’ve been.