Manage Version Sharding with Multiple FOCUS Data Generator Exports

The FinOps Open Cost and Usage Specification (FOCUS) unifies cost and usage data from many data generators into one consistent dataset. Data generators are the cloud, SaaS, and other technology providers that produce billing data.

FOCUS is also an open specification that grows with its community and data generator adoption, and a new version is ratified roughly every six months. Since each data generator upgrades on its own schedule, most practitioners will eventually hold FOCUS data from several versions in the same data warehouse.

We call this version sharding.

What Is FOCUS Version Sharding?

Version sharding means practitioners having to build and manage one unified view, combining data generator exports that use different versions of the specification. It can look like the undoing of a specification meant to simplify, with different columns and rules in each dataset and more time spent reconciling data than doing FinOps.

Managing version sharding isn’t as complicated as it seems. FOCUS versions nest, and changes between them are classified and documented. FinOps practitioners can learn and use the following approach: record the data generator and FOCUS version on every row, query the columns all generators share, add the rest on your own schedule, and document a decision for every gap.

How FOCUS Versions Nest

A common assumption is that FOCUS versions behave like operating system releases: different schemas, potentially major changes, and a reconciliation project between you and various data generator exports. Instead, each FOCUS version contains nearly everything in the one before it, like the layers of an onion expanding outward.

• Version 1.0 began with 43 columns and 1 dataset
• Version 1.1 added 7 columns
• Version 1.2 added 7 more columns
• Version 1.3 added 20 columns and an additional dataset
• Version 1.4 added 32 columns, removed the 2 columns deprecated in v1.3, and added 2 datasets

Of the 43 columns defined in v1.0, 41 are still present as of v1.4. Both of the removals were deprecated a full release before v1.4 so that users could plan ahead.

Changes to FOCUS Are Bounded and Classified

Here are three features of how FOCUS manages change to keep its shared core stable.

Every Change Is Classified by Its Impact on Queries

FOCUS contributors classify each change in each release into one of three categories:

• Compatible: Existing queries keep working.
• Migration Compatible: Queries need a defined, documented change.
• Incompatible: Queries break with no migration path.

The project defined this classification in v1.2 as a governance commitment and first applied it in the v1.3 changelog. It lives in the project’s specification change guidelines and is applied release by release. Both the v1.3 and v1.4 changelogs record no Incompatible changes. The record is short, and anyone can check it.

The classification also shows where to focus: Migration Compatible changes. Compatible changes can still require work, as the section on definitions below explains.

The Shared Core Is the Whole Original Specification

No column was removed between v1.0 and v1.3, so all 43 columns defined in v1.0 are present in every version through v1.3. Version 1.4 is the first to remove any, and it removes two.

Column Removal Came with a Full Cycle of Notice

Version 1.4 removed ProviderName and PublisherName because their definitions conflicted with each other. Both were formally deprecated in v1.3, using the Deprecated and PreviousColumnName metadata properties added in v1.2. Their use cases moved to ServiceProviderName, HostProviderName, InvoiceIssuerName, and DataGenerator metadata. Generators had a full release cycle, roughly six months, with old and new columns present together and a documented mapping between them. The removal affects practitioners only when a data generator upgrades to v1.4.

The project settled its vocabulary up to and including v1.0. Since that baseline, the column record has been additive, with one deprecate-then-remove cycle across four releases. Version 1.4 also retired three attributes and two glossary entries without a deprecation cycle, all classified Migration Compatible.

Version Sharding Visualized

The walkthrough below follows three data generators, each exporting FOCUS datasets on a different version. The counter under the grid tracks the columns you can query across all three. It holds steady as the generators drift four releases apart, drops once when the v1.4 removal lands, and then flags two columns that kept their names but changed their meaning.

Start Here: Record the Generator and Version on Every Row

If you want to begin managing multiple versions of FOCUS across your data generators, the highest-leverage step costs two lines in your ingest process.

FOCUS rows do not carry version information, so hard-code Generator and FocusVersion at ingest time for each generator and each export. Take the version from the generator’s published FOCUS metadata where it exists. Once these are ordinary columns, each of the following questions becomes a simple GROUP BY:

• Which generators are on which version?
• Which columns are safe to aggregate across version boundaries?
• Which caveats belong on which dashboard?
• Which backfills do you still owe?

Before: Versions never recorded

-- Ingest
INSERT INTO focus_costs
SELECT * FROM read_parquet('s3://billing/gen_a/cost_and_usage/*.parquet');

-- What is my cross-generator spend?
SELECT SUM(EffectiveCost)
FROM focus_costs;
-- 4,182,930.11: one number, with no way to know it spans
-- two different specification versions

-- Which generators are on which version?
-- Not answerable. Go ask each vendor.

After: Two columns hard-coded on every row

-- Ingest (hard-coded values for this generator and export)
INSERT INTO focus_costs
SELECT d.*,
       'gen_a' AS Generator,   -- hard-coded
       '1.3'   AS FocusVersion -- hard-coded
FROM read_parquet('s3://billing/gen_a/cost_and_usage/*.parquet') d;

-- Repeat this pattern for every other dataset the generator emits

-- What is my cross-generator spend?
SELECT FocusVersion, SUM(EffectiveCost)
FROM focus_costs
GROUP BY 1;
-- 1.2    3,004,117.55
-- 1.3    1,178,812.56
-- Same total, with the version boundary now visible

-- Which generators are on which version?
SELECT FocusVersion, Generator, COUNT(*)
FROM focus_costs
GROUP BY 1, 2;

Next: Climb FOCUS Versions Deliberately

Follow each step one at a time and apply them to one question: how much of the version difference does your data warehouse absorb before a query runs?

Step 1: Query the Floor

Find the lowest FOCUS version across your generators, then subtract any columns that a generator on a higher version has removed. The result is your floor. Run every cross-generator query against it.

Because no column was removed between v1.0 and v1.3, the subtraction matters only when a dataset is on v1.4. Within the floor, you get a defensible cross-generator total from day one, with known exclusions.

Step 2: Build Two Tiers

Use the identified floor columns to aggregate across generators, and use ceiling columns, the full column set of each generator’s version, to go deeper on one generator at a time. Label the two dashboard tiers clearly so each is explicit about its scope.

Step 3: Normalize Up to the Ceiling

Bring older-version data up to newer versions. Declare a backfill policy for each column, using the four outcomes in the next section.

When generators begin delivering recency metadata, DatasetInstanceComplete and TimeSectorComplete will show whether a period is final. Until then, record your own load dates so data consumers can tell a true zero from data that has not arrived yet.

Decide What to Do about History, Column by Column

When a data generator moves up a version, you inherit new columns with no history behind them. Each column has one of four outcomes. Choose one explicitly for each column and record the decision.

1. Derived. The specification defines the arithmetic, and you already hold every input. ListCost must equal ListUnitPrice multiplied by PricingQuantity when both are non-null, and the same relationship holds for ContractedCost and ContractedUnitPrice. This category is smaller than it looks. All five of these columns date to v1.0, so they arrive together rather than at a version boundary. That makes derivation the path for a generator that starts emitting FOCUS partway through your history, not for one that moves up a version. Watch for the near miss: PricingCurrencyEffectiveCost must be the PricingCurrency-denominated equivalent of EffectiveCost. That reads like arithmetic, but the specification carries no exchange rate, so this column belongs in the next category.
2. Re-derived from the native export. You kept the generator’s proprietary data, and the new FOCUS column maps from a field the generator always had. This is how a source that was not FOCUS-native gains history retroactively. The generator’s Column Mapping Reference often already names the native field behind each FOCUS column.
3. Re-requested. Only the generator can produce the history, by re-emitting it in the new version. Usually it will not. Ask anyway and record the answer, because the answer tells you whether the gap is permanent.
4. Unrecoverable. No one ever computed the underlying fact. For example, commitment eligibility details for a period before the generator computed eligibility cannot be recovered by any amount of engineering. Record a documented null and scope the business question to the period where data exists.

A mature FOCUS practice still has gaps. It knows exactly which cells are empty and why.

Check Definitions, Not Just Column Names

A column can keep its name across a version boundary and change what it measures. That makes a version move a definition change as well as a schema change, and a diagram of column names cannot show it.

Version 1.4 substantially revised the normative requirements for BilledCost (see the v1.3 and v1.4 definitions) and EffectiveCost (see the v1.3 and v1.4 definitions), including rules for amortizing covering charges and a cross-record sum validation. A covering charge is a purchase whose cost offsets other charges, such as a commitment discount purchase. As a result, SUM(EffectiveCost) across generators on either side of the boundary compares two differently defined quantities, even though both datasets use the same column name.

Watch null rates too. InvoiceId moved from Recommended to Conditional in v1.4, a change the specification classifies as Compatible. Your queries keep parsing, but your alerts may not stay quiet: on a dashboard, a shift in population rate looks identical to a data quality regression.

When a generator moves versions, read the changed-columns section of the FOCUS changelog for every version you are crossing, alongside that generator’s Conformance Gap Report. This matters most as the first v1.4 generators ship. With a written policy in place, you can safely ingest data from generators on mixed versions. To query the specification without reading the full changelog, use the FOCUS Model Context Protocol (MCP) server with your AI assistant.

Four Questions for Your Data Generator’s Account Team

1. Do you publish FOCUS metadata, and where? Metadata confirms which version you are receiving.
2. When you move versions, do you re-emit history at the new version, or does the change apply from a cutover date forward?
3. Where are your Conformance Gap Report and Column Mapping Reference published?
4. Which FOCUS version are you certified for, which datasets did that certification cover, and is that the version you are emitting?

The last question is easy to skip. FOCUS certification applies per version and per generator, and under the program rules, “No, those are different versions” is a legitimate answer. It just needs to be stated. Coverage is a separate question from version: the currently certifiable version defines a single dataset, so a certificate says nothing about a generator’s Contract Commitment, Invoice Detail, or Billing Period data.

Get Ahead of Version Sharding (and Its Costs)

The fear was that six-month releases would leave practitioners reconciling versions instead of doing FinOps. The record shows two columns removed since v1.0, both with a full cycle of notice and named successors, and no Incompatible changes recorded since the classification began.

Carrying several specification versions still costs more than carrying one, but the cost is bounded, visible, and documented. Before FOCUS, the alternative was maintaining hand-built bridges between many vendors’ private schemas and finding out one had broken from a board deck.

Get started now by stamping the generator and version on every row. Then compute your floor, query against it, and climb on your own schedule.