Solutions

Data platform modernization when nobody trusts the numbers

The symptom is always the same meeting: two teams present two numbers for the same thing and the discussion becomes about the data instead of the decision. That is rarely a pipeline failure. It is the absence of one governed layer that both reporting and AI read from.

“Two teams brought two revenue figures to the same meeting”

The version we hear most often has a specific texture. There is a monthly pack that senior people rely on, and it is assembled from exports. One analyst maintains a spreadsheet that is genuinely load-bearing, and everyone knows this is a risk and nobody has time to fix it. There is a nightly job that has failed silently more than once, discovered only because a figure looked wrong.

Underneath, there are usually two or three definitions of the same measure, each of them defensible. Finance counts revenue at invoice, operations counts at delivery, the customer success team counts at contract signature. None of them is wrong. Nobody wrote down which one the board pack uses, so the argument recurs monthly and gets attributed to the data being bad.

Then an AI project arrives, needs to ground on that same data, and stalls in month two. That is the point where the reporting problem stops being an inconvenience and becomes a blocker on something with executive attention.

Why the usual answer fails

Buying the platform and moving the warehouse. The most common program is a migration: lift the existing schema into Microsoft Fabric or a new warehouse, repoint the reports, declare modernization. What arrives on the other side is the same conflicting definitions, the same undocumented transformations and the same ownership vacuum, now running on a capacity you pay for monthly. The storage engine was not the constraint. Reproducing the semantics faithfully is exactly the thing you did not want to do.

Self-service as a strategy. Give every team Power BI, a workspace and a training day. It produces a real improvement in speed and a serious deterioration in agreement, because forty datasets built independently on the same source will encode forty slightly different opinions. Self-service without certified semantic models underneath is decentralised disagreement with better charts. The fix is not to take the tool away; it is to give people a certified model to build on so the fast path and the correct path are the same path.

Governance as documentation. A data catalog is bought, a data dictionary is written, a stewardship committee meets. Six months later the catalog reflects the estate as it was at the time of writing. Governance that is not enforced where data is consumed, through certification, lineage, classification and access control, decays into a wiki nobody opens.

Boiling the ocean. An enterprise data model covering every domain, designed before anything ships. Eighteen months of modeling, no delivered decision, and a sponsor who has moved on. The alternative is not to abandon the model; it is to build it one domain at a time against a decision someone is waiting on.

Capacity bought before workload. An F-SKU chosen from a sizing table rather than from measurement. Too small and refreshes throttle in the first month, which costs the platform its credibility. Too large and it becomes the line item finance points at. Neither is recoverable by argument.

How we build the governed layer

Start with one contested number. We pick the measure that causes the recurring argument, trace it to source, and get the definition agreed in writing with a named owner. This is unglamorous and it is the work. A semantic model encodes a decision, and if the decision has not been made, the model will make it silently.

One lakehouse, layered. Microsoft Fabric with a medallion structure: raw landing, conformed, and a curated layer that reporting and AI both read. Data Factory pipelines with tested contracts, failure alerting and a retry policy, replacing the nightly job that fails without telling anyone.

Certified semantic models, not exports. Direct Lake models over the curated layer, with row-level and object-level security so one model can serve audiences that must not see the same rows. Certified datasets and deployment pipelines, so the path of least resistance for an analyst is the governed one.

Catalog and classification that are enforced. Microsoft Purview for lineage, classification and sensitivity labels, applied to the data that carries obligations rather than to everything. Where those obligations are formal, the retention and residency rules come from the compliance architecture and are enforced by policy at the platform level, not left to convention. Clinical and patient data carries additional segmentation, covered in our healthcare architecture work.

One layer for reporting and for AI. The same curated layer grounds retrieval, which is what makes an assistant answer with numbers the organization already accepts. Building a second stack for AI is how you end up with a fourth definition of revenue. This is the handoff into AI engineering on Azure AI Foundry, and it is why an agent project that stalls on data quality often belongs here first.

Capacity governance from the start. Workspace strategy, capacity monitoring, and F-SKU sizing decided at the end of the build on measured refresh and query behavior. Reservation purchased against evidence, using the same discipline as our cloud cost optimization work.

The engagement that delivers it

Eight weeks, one domain, end to end. Weeks one and two agree the definitions, map the sources and pick the decisions the domain has to support. Weeks three to five build the lakehouse layers, the pipelines and the first certified semantic model, with something reviewable every week. Weeks six and seven rebuild the reports that matter on the certified model, apply security and retire the duplicates with their owners. Week eight covers capacity sizing, Purview configuration, documentation and handover.

You keep the pipelines and models as code in your repository, the written metric definitions with their owners, the security model, the capacity sizing evidence and a backlog of the domains we deliberately did not touch.

This assumes a governed Azure estate underneath. Where the platform is being built into an environment without management groups, policy or a network model, the Azure landing zone comes first, because a data platform deployed into an ungoverned subscription inherits every problem that subscription has.

Questions we get

Do we have to move everything into Fabric?

No, and starting that way is the usual mistake. We build one domain end to end, prove the numbers against the source, then extend. Existing warehouses, Azure SQL databases and line-of-business systems stay where they are and feed the governed layer. A platform nobody trusts yet does not deserve a full migration.

Is this a technology problem or an organizational one?

Both, and the organizational half is larger. Two teams reporting different revenue usually have two defensible definitions, not a broken pipeline. The engagement spends real time getting those definitions agreed and written down with a named owner, because a semantic model encodes a decision and someone has to make it.

What F-SKU capacity do we need?

We will not size it before measuring. Capacity bought ahead of a workload is either throttling under load or idle at cost, and both undermine the platform politically. We start on a small capacity, measure real refresh and query behavior during the build, and size the reservation against evidence at the end.

Our AI project stalled on data quality. Is this the fix?

Frequently, yes. Grounding an assistant on numbers that two departments dispute produces confident answers nobody accepts. One governed layer serving both Power BI and AI retrieval removes the argument, and it is why we treat reporting and AI grounding as the same platform rather than two stacks.

What happens to the forty existing Power BI reports?

They get triaged, not preserved wholesale. Most estates have a handful of reports that carry real decisions and a long tail that duplicates them. We identify the ones that matter, rebuild them on certified semantic models with row-level security, and retire the rest with their owners rather than around them.

How do we stop it drifting back?

Certified datasets, workspace and deployment pipeline discipline, lineage and classification in Microsoft Purview, and a rule that a report used in a governed decision must be built on a certified model. Governance that is not enforced at the point of consumption becomes a wiki page within a quarter.

Bring us the workload. We bring the architecture.

Thirty minutes, one architect, no deck. Come with a problem and leave with an opinion you can act on.