For week #6 of the push into FinOps best practices for the upcoming O’Reilly book, we dug into various spend allocation strategies ranging from showback, full chargeback to “shameback” with Ally Anderson, business operations manager and FinOps army of one at Neustar.
Ally has been at Neustar for over 6 years and for 5 of those has been doing IT Financial Management, more recently dipping into the world of FinOps. She sits within a governance group that handles both sides of the tech world: she does on-prem IT Asset Management as well as public cloud management. Neustar is an American technology company that provides real-time information and analytics for the Internet, risk, digital performance and defense, telecommunications, entertainment, and marketing industries.
Ally shared their spend allocation journey, which is complex at Neustar given the organic growth their cloud usage has taken. Years ago, they started with the classic shadow IT model: various engineering teams who wanted to try cloud. These teams would often start with their own AWS account using a credit card. Eventually, spending grew to a point where the decision was made to pull everything together into a master payer account over all the various teams.
Somewhere along the way, a central best practice team decided to take a multi-tenant approach and opened multi-tenant master production and staging accounts. Instead of having each team manage their own accounts, teams were encouraged to move into one of those two accounts. Though this team has since disbanded, they still have a number of application teams using those centralized accounts. Neustar has since evolved into the direction recommended by the FinOps Foundation which is toward distributed teams owning and managing their own accounts to encourage accountability for spending.
They’ve also grown their cloud spend via mergers and acquisitions which has contributed a mix of different account structures. Some opening one account per product, others have opened an account per environment, others remaining multi-tenant. As such, allocation is complex to execute.
Focus of FinOps spend allocation
She has three key approaches to allocation:
- Account naming. Each time there is a request for a new account, it must be named for the product that it will be supporting and, if it’s environment specific, the environment must be represented in the name. Ally is the central check-point to ensure this process is followed and requests for accounts must go through her. In AWS this would be take the form of linked accounts, in GCP this would be in the form of projects or folders.
- Tagging. Because not all of their legacy accounts were opened under the new account naming convention, they also make heavy use of tagging especially in the legacy multi-tenant accounts. Their primary tag key is called “Department” and helps them carve out spending in those accounts. The legacy multi-tenant accounts require that tag to be applied to all resources using various levels proactive enforcement. But because not all spending is taggable and not all end-users are compliant, there is still often a chunk of untagged spend which is reported out to management.
- Shared corporate costs across accounts. Ally spoke on the common problem of how to allocate central costs in the cloud world. In her management reporting, she splits and spread out costs like volume discounts, enterprise support and reserved instance pre-payment amortization.
Ally shared graphs of how they’ve reduced untagged spending over time. In the early days, when they were using the multi-tenant account approach, tagging coverage was not as good with most of spend not being tagged. They have focused on reducing their ratio of spend to untagged over time. On an account by account basis, they’ve given the leaders in charge of particular products a granular view into the services that comprised their individual products. Near real-time reporting is provided to increase visibility into spend over the month and quarter, as well as to close allocation gaps.
Following a typical maturity curve in FinOps, she originally kept cloud volume discounts, enterprise support charges and pre-payments like RI amortization out of the reporting that end-teams used. This year they began factoring all those costs into their spend reporting so teams now see the properly adjusted prices factoring in all of the above. Teams are now able to look at their fully adjusted spending with amortizations and accurate volume rates baked into the numbers giving them a more accurate picture of actual cost.
Chargeback, Showback and Shameback
I’ve seen an ongoing conversation over the years around whether to do chargeback or showback. It’s been my perception that this was more of a function of an enterprise’s FinOps maturity: less mature you do showback, more mature you do chargeback. What’s the difference between the two? Chargeback happens when the actual cost being incurred is allocated back to the budget or P&L of an individual team. Showback happens when you show teams what they are spending, but the dollars are allocated internally to a central budget(s).
Ally dispelled the notion of chargeback as a maturity model and later during the discussion, the Foundation members further supported her assertion. The assertion was that enterprises will choose showback vs chargeback due to an organizational bent toward or away from departmental P&Ls. A finance leader or CFO may be against the concept of P&L across the enterprise or it may not make sense for the business, in this case the organization will lean toward showback and not do full chargeback.
One member shared that they do a combination of chargeback and showback in their organization: when it’s a customer facing SaaS workload they showback spend while centrally managing the budget. However, for specific R&D workloads, they do a chargeback to the specific team’s P&L.
A tipping point of cloud spend
As is all too often the case, different teams were running in cloud with little oversight until the spend got to be high enough that it received the attention of leadership. While there was pushback against chargeback — and, to a certain degree, to showback initially — they did want to have questions answered around who was spending what.
During a here at AWS Sydney Summit in 2015, I shared a typical maturity curve showing the point at which organizations start to optimize their cloud spend. The illustrative graph culminated in a very specific point: the moment at which spending crosses an invisible threshold where the exec team starts to really care about it. That’s the point at which the tech organization is whiplashed into caring about — and optimizing — cloud spend. Ally showed a similar story in her own organization indicating there have been 3 points over time where there was a “spend panic” of sorts that resulted in a push for more accountability.
The first one drove a need for questions being answered. This got finance onboard with the idea of showing teams what they were spending. The spending increased again and drove a period of “shameback”: a light was shone on teams who were increasing their cloud costs. This brought in showback for a spell until another threshold was crossed that resulting in another period of “shameback” in an attempt to try to get teams to reduce spend.
It wasn’t until a 3rd period of spending panic that the light bulb went off: “The bill going up isn’t necessarily a bad thing if our business is growing.” They began to reframe the question to, “Are we spending as efficiently as possible? The conversation moved to ‘how do we optimize?’ Instead of mandating a cost reduction target”. This reminded me of a conversation I had with a large cloud spender in Europe last year about the 5 levels of budgeting. Teams often start with 1) no budgets, before 2) setting arbitrary ones, then 3) defining fixed cost savings goals, before ultimately 4) moving to the nirvana of unit economics and 5) activity-based costing.
Mapping cloud spend to the business
Ally showed an anonymized version of the complexity they manage when allocating spend between teams. They map each of their dozens of disparate linked accounts against the relevant products and cost centers that the business needs to report against. She drew upon a common theme we’ve seen play out time and again over the years: the enterprise re-org. She highlighted that after a big restructure that teams don’t tear down cloud resources and rebuild them in different accounts to match the new organizational structure. Instead they need to take existing usage of cloud resources and reattribute them to a different team or VP.
This separation is a common challenge that FinOps teams encounter. The way that engineers tag resources don’t necessarily align to the way the business needs to report cloud spend. That spending often needs to be re-mapped to match the cost centers or products that are the rightful ‘owners’ of cloud spend.
A cloud showback model
So what does showback look like in their organization? For each team, they show actual spend against forecasted spend, including the variance between the two and the comparison to what was budgeted. This is shown to teams on both a monthly and quarterly basis as well as regular, more granular reporting into cost drivers and anomalies.
The CTO has a monthly operating review with each of the engineering VPs showing their spend. If there are not any glaring variances, things go quickly. Attention is focused where there are large variances. What Ally is able to do within her real-time FinOps processes is give the VP’s a heads-up if they will overshoot their budget as early as possible. She highlights the issues and the cost drivers that drove the variance and then the VP works with his/her team to implement a fix before the monthly management review happens. It’s a great example of real-time decision making coupled with a breaking down of silos.
Ally closed with a few key points that she might share with someone new to this journey:
- Leadership buy-in. Successful FinOps processes require executive involvement. If you have shared accounts, it’s important to have the support from the leader who owns those accounts to support your allocation and optimization initiatives.
- Don’t go it alone. In a nice nod to our fledgling foundation, Ally mentioned how great it is “knowing that I’m not alone. This organization is super helpful, I wish it had come earlier. I’ve been stumbling along and trying to figure it out, in the short time [of the FinOps Foundation] it’s been great to have a community around me.”
- Get the Right Tools. Doing this all with spreadsheets would have been painful to say the least. Ally shared that although they still have manual processes, her FinOps tools have helped a lot.
- Have a designated team to do the work. Having a multi-functional team to carry out things like automation work (as Sarah Hussainy from ThermoFishers showed a few weeks back) are super helpful.
Closing discussion on tags
One of the members from a major US retailer asked this closing question:
“What are your top points of advice to get resources tagged?” Ally made it clear that she works closely with the the technical contacts in each team on this topic.
“If there’s a large amount coming in as untagged, I makes it visible to them via a report.” She found that if she gets the data consistently in front of the team leader, then she can get tags applied. As part of her executive reporting, she has made untagged spending a core part of ongoing reporting to encourage the executive to push their own team to minimize the unallocated number.
Another member shared the idea that “tag-on-create” is increasingly a strategy for tag compliance. The cloud providers keep adding more and more support to require certain tags to be set to launch infrastructure. Taking advantage of that technique can be a huge help in ensuring tag coverage.
On that topic, this week we have Darek Gajewski from Ancestry.com speaking on increasing and automating tag coverage. Ancestry has been very effective in increasing their tag coverage over a multi-year journey and Darek will share their story and their journey to get there.
Hope to see you tomorrow, keep breaking down the silos…