Data ingestion and normalization in the context of FinOps represents the set of functional activities involved with processing/transforming data sets to create a queryable common repository for your cloud cost management needs. In this context, data ingestion and normalization occurs when bringing together cloud billing data, cloud usage data, cloud utilization and performance data, on-premises CMDB or ITAM data, business-specific data, and other data points from a variety of cloud providers and IT data repositories to create a collection of cost and usage information which can be queried to support and enable the FinOps Capabilities.
Effective FinOps practice requires access to regular streams of detailed usage, utilization and cost data, which can be categorized and analyzed to drive decision making. Unlike the world of on-premises data centers, there is no shortage of data on cloud usage. Cloud vendors produce massive amounts of very granular usage and cost data on which to base a FinOps practice. Monitoring platforms, security platforms, and business operations applications can also provide data that will inform on utilization, location, value and usage, oftentimes at similar levels of volume and granularity.
So while the issue many data centers experienced was a lack of detailed data, the challenge faced by cloud users is oftentimes that there is too much. An effective data ingestion and normalization strategy should strive to provide the FinOps team with the right combination of data from:
- the right source systems
- at the right timeliness to support the cadence of decision making
- at the right level of granularity to support aggregate reporting and drill down investigation
- with the appropriate standardization, augmentation, normalization, etc.
The strategy for data ingestion will be driven largely by the needs of the reporting, cost allocation and optimization reporting needs. Data required to make decisions at a business unit level will not be in need of as much detailed or granular information.
FinOps teams which manage or allocate costs at a resource level may require multiple sources of data to gather resource information for some cloud providers which don’t natively provide it.
The data ingestion and normalization challenge grows as the complexity and diversity of the data increases. The level of granularity, the consistency of cost and rate metrics, mismatches between similar or analogous services, application of various types of discounts, the application of metadata to resources and hierarchies, and many other differences between the billing and usage data provided by the cloud providers can all provide challenges, even before considering the regular change and huge volume of data provided.
a collection of real world examples, stories and “how to” for this Capability; based on FinOps community member experiences; information here may:
- apply to one or more cloud providers
- include specific types of cloud services used) (compute, storage, database, etc…)
- describe a combination of tooling, platform or vendor
- describe the industry the organization belongs to
- describe the complexity of the organization (global, enterprise, etc…)
- include the FinOps personas involved and any other organizational roles
Get involved and contribute to the community by sharing your real world experiences related to this Capability in the form of a story or providing a playbook for how you have implemented best practices in your organization. Your real world experiences can be provided in the context of:
- one or more cloud providers
- the types of cloud services used (compute, storage, database, etc…)
- describe a combination of tooling, platform or vendor, and processes including KPIs
- the industry the organization belongs to
- the complexity of the organization (global enterprise, start-up, etc…)
- the FinOps personas involved / organizational roles
Join the conversation about this Capability in Slack . You can submit stories, how-tos and suggest improvements using one of the options for contributing here.