Framework / Domains / Understand Cloud Usage & Cost / Data Ingestion
Collect, transfer, store, and normalize data from various sources, with a goal of creating a complete, contextual dataset of cloud usage and cost data available for analysis.
Manage data sources
Ensure data quality
Maintain data timeliness and availability
Data Ingestion involves gathering, transferring, processing, transforming, and correlating various datasets to create a queryable, contextualized repository, at the appropriate level of granularity, accessibility, and completeness to support activities in all of the FinOps Capabilities across all FinOps Personas.
Data Ingestion needs can vary greatly depending on how each organization conducts its FinOps practice:
As such, this Capability requires continuous development as an organization matures and its needs for data to perform other Capabilities change.
Data ingested to support FinOps activities must include cloud cost and usage data from cloud service providers or other metered service providers, but may also include:
FinOps Capabilities such as Allocation, Reporting & Analytics, or Unit Economics will provide requirements that identify the specific sources, granularity, degree of normalization, correlation, and manner of storing cloud data needed at any given time.
Effective FinOps practice requires access to regular, iterative, frequently-updated streams of detailed usage, utilization, and cost data, which can be categorized, contextualized, and analyzed to drive decision making.
Data Ingestion can be an early challenge for organizations that are new to cloud. Cloud service provider cost and usage datasets are massive and complex, and each provider has traditionally used proprietary schemas and data structures. The complexity, size, inconsistency, and latency of cloud data has created barriers to using standard business intelligence (BI) tools or building custom tools. The size and scale of the data makes it difficult to effectively analyze without higher level technical or big data skills.
The FinOps Open Cost & Usage Specification (FOCUS) project delivers consistency and standardization to cloud cost data, and will eventually extend to SaaS provider data, sustainability data, license publisher data, private clouds, observability provider data, and other metered service provider data sources. Organizations will benefit through the interoperability of common and custom tools as the FOCUS specification is adopted by vendors and data providers.
Observability platforms, security platforms, carbon usage platforms, and business operations applications can also provide very large datasets that may need to be correlated to cloud data. Metadata created as part of tagging or allocation strategies, managed in the Allocation Capability, can provide important keys for correlating, contextualizing, and summarizing all of these datasets together. Data Ingestion ensures that tags or labels created in the cloud platforms are gathered, and mapped to internal allocation metadata as well.
Data Ingestion as a Capability will identify or establish the cloud data Reporting & Analytics data sources. Organizations may create common data repositories for cloud data, or may use existing ones, depending on the complexity of the data, the needs of the organization and the desire to connect the data to other data sources.
The goal of Data Ingestion is not to amass the largest and most granular dataset available in real-time, but to gather and integrate the data that provides value to the organization at its current maturity. Over time, the data that will be required by an organization will evolve as the organization gets more mature in its analytical needs, as the types of services it uses become more varied, as additional clouds or SaaS products are used, or as internal policies and usage changes.
Action in this Capability will be triggered by the need to build or add data sources, retrieve more granular data, contextualize with metadata, normalize into common specifications, build custom tooling, or make data more quickly or readily accessible. This work can be done iteratively as an organization grows in FinOps maturity and realizes value from investing in these steps.
As someone in the FinOps team role, I will…
As someone in a Product role, I will…
As someone in a Finance role, I will…
As someone in a Procurement role, I will…
As someone in an Engineering role, I will…
As someone in a Leadership role, I will…
As someone in an Allied Persona role, I will…