Skip to content

bgbailey/TMDLInheritanceDemo

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

21 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TMDL Inheritance Demo — NovaTel Communications

A reference architecture for semantic model inheritance across Microsoft Fabric workspaces using TMDL (Tabular Model Definition Language) and Git-based synchronisation.


Overview

This repository demonstrates how an enterprise-level semantic model can share a governed subset of its tables and measures with business domain–level semantic models in Microsoft Fabric.

The fictional company NovaTel Communications is used as a realistic telecoms scenario with consumer and business divisions. The enterprise workspace owns company-wide KPIs; the consumer domain workspace reuses those definitions while adding its own domain-specific analytics.

Key capabilities demonstrated:

  • Enterprise semantic model with 8 tables and 3 measure groups (10 core KPIs, network, support)
  • Consumer domain model with 3 domain tables + 5 inherited enterprise tables and 9 domain measures
  • Git-driven sync pipeline that auto-propagates TMDL changes via GitHub Actions and Pull Requests
  • One-click deployment from this template repo to live Fabric workspaces
  • Sample web application (portal) that queries live semantic models via REST APIs

What's New — Notable Enhancements

Since the original concept, this demo has been substantially built out:

Area Enhancement
Enterprise model Full 8-table data model (4 dimensions, 4 facts) with Direct Lake partitions; 3 grouped measure files for core KPIs, network, and support metrics
Consumer domain model 3 domain-specific tables (campaigns, NPS, plans) with 9 domain measures; _inherited/ subdirectories holding synced enterprise objects
Sync pipeline sync_enterprise_to_domain.py adds provenance headers to all inherited files; validate_sync.py performs byte-for-byte comparison after header stripping
GitHub Actions sync-and-validate.yml workflow triggers on enterprise model pushes, runs sync, validation, and pytest, then commits changes and opens a PR
Test coverage test_sync.py (copy logic, headers, dry-run, change detection) and test_tmdl_validity.py (structure, lineageTags, syntax)
PBIR reports Enterprise overview report (3 pages) and consumer dashboard (4 pages) in PBIR enhanced format with proper visual definitions
Portal / web app portal/index.html — a single-page Metrics Hub that queries Fabric semantic models via MSAL authentication and Power BI REST API
Deployment scripts setup/deploy_demo.ps1 (one-click Fabric workspace setup + GitHub repo injection) and setup/run_demo.ps1 (interactive live-demo walkthrough script)
Tools tools/validate_tmdl_structure.py standalone TMDL structure validator

Architecture

graph TB
    subgraph NovaTel_Enterprise["🏢 NovaTel_Enterprise Workspace"]
        EL["enterprise_lakehouse\n(8 Delta tables)"]
        EM["NovaTel_Enterprise_Model\n(Direct Lake Semantic Model)"]
        ER["NovaTel_Enterprise_Overview\n(PBIR Report — 3 pages)"]
        EL -->|Direct Lake| EM
        EM --> ER
    end

    subgraph NovaTel_Consumer_Domain["📱 NovaTel_Consumer_Domain Workspace"]
        CL["consumer_lakehouse\n(3 domain Delta tables)"]
        CM["NovaTel_Consumer_Model\n(Direct Lake Semantic Model)"]
        CR["NovaTel_Consumer_Dashboard\n(PBIR Report — 4 pages)"]
        CL -->|Direct Lake| CM
        CM --> CR
    end

    EM -->|"TMDL sync — 5 tables\n+ core measures"| CM

    GIT["🔄 GitHub Repository\n(TMDL source of truth)"]
    GH["⚙️ GitHub Actions\nsync-and-validate.yml"]
    Portal["🌐 Metrics Hub Portal\n(portal/index.html)"]

    GIT --> GH
    GH -->|"auto-sync on push to\nenterprise/"| GIT
    Portal -->|"Power BI REST API\n(MSAL auth)"| EM
    Portal -->|"Power BI REST API\n(MSAL auth)"| CM
Loading

See docs/architecture.md for detailed Mermaid diagrams including the data-flow and sync-pipeline sequences.


The Problem — Why Inheritance Matters

Traditional Power BI / Fabric semantic models are monolithic: a single .pbix file or TMSL JSON blob that represents the entire model. When multiple business units want to share enterprise KPIs (e.g. Total MRR, Churn Rate), each team maintains its own copy. This leads to:

  • Metric drift — different definitions of the same KPI across teams
  • Governance failure — enterprise changes do not propagate to domain models
  • Duplication — the same DAX is written, tested, and maintained N times

The Solution — TMDL + Git Sync

TMDL (Tabular Model Definition Language) serialises each model object into its own file:

File TMDL Object
model.tmdl Database metadata
tables/dim_customer.tmdl Table definition with columns
measures/_enterprise_core_measures.tmdl Measure group
relationships.tmdl All relationships
roles/enterprise_reader.tmdl Security role

This file-per-object granularity makes it natural to:

  1. Copy a subset — only the tables and measures a domain needs
  2. Track changes with Git — diff, blame, and PR review per-object
  3. Automate propagation — GitHub Actions copies updated files, validates them, and opens a PR for the domain team to review

Files in _inherited/ directories are managed exclusively by the sync pipeline and must not be edited manually.


Repository Structure

TMDLInheritanceDemo/
├── README.md
├── docs/
│   ├── architecture.md          # Mermaid diagrams + deployment model
│   ├── sync-guide.md            # How the sync pipeline works end-to-end
│   ├── deployment-guide.md      # Deploy from template to live Fabric workspaces
│   └── demo-walkthrough.md      # Step-by-step CI/CD propagation demo script
│
├── enterprise/
│   ├── lakehouse/
│   │   └── generate_sample_data.py    # Generates ~87K rows across 8 tables
│   └── semantic-model/
│       ├── NovaTel_Enterprise_Model.SemanticModel/
│       │   ├── definition/
│       │   │   ├── model.tmdl
│       │   │   ├── relationships.tmdl
│       │   │   ├── tables/            # 8 TMDL table files
│       │   │   ├── measures/          # 3 measure groups: core, network, support
│       │   │   ├── roles/
│       │   │   └── expressions/
│       │   └── definition.pbism
│       └── NovaTel_Enterprise_Overview.Report/   # PBIR report (3 pages)
│
├── consumer-domain/
│   ├── lakehouse/
│   │   └── generate_sample_data.py    # Generates ~2K rows across 3 tables
│   └── semantic-model/
│       ├── NovaTel_Consumer_Model.SemanticModel/
│       │   ├── definition/
│       │   │   ├── model.tmdl
│       │   │   ├── relationships.tmdl
│       │   │   ├── tables/
│       │   │   │   ├── _inherited/    # ← Synced from enterprise (do not edit)
│       │   │   │   │   ├── dim_customer.tmdl
│       │   │   │   │   ├── dim_product.tmdl
│       │   │   │   │   ├── dim_date.tmdl
│       │   │   │   │   ├── fact_subscriptions.tmdl
│       │   │   │   │   └── fact_billing.tmdl
│       │   │   │   ├── fact_consumer_campaigns.tmdl
│       │   │   │   ├── fact_consumer_nps.tmdl
│       │   │   │   └── dim_consumer_plans.tmdl
│       │   │   ├── measures/
│       │   │   │   ├── _inherited/
│       │   │   │   │   └── _enterprise_core_measures.tmdl  # ← Synced
│       │   │   │   └── consumer_measures.tmdl
│       │   │   ├── roles/
│       │   │   └── expressions/
│       │   └── definition.pbism
│       └── NovaTel_Consumer_Dashboard.Report/    # PBIR report (4 pages)
│
├── sync/
│   ├── sync_config.yaml              # Which tables/measures to inherit
│   ├── sync_enterprise_to_domain.py  # Copies TMDL subsets with provenance headers
│   └── validate_sync.py              # Validates _inherited/ matches source (byte-for-byte)
│
├── pipelines/
│   ├── fabric_helpers.py             # Shared Fabric API utilities (sempy_labs)
│   ├── deploy_enterprise.py          # Deploy enterprise model to Fabric workspace
│   └── deploy_consumer_domain.py     # Sync then deploy consumer model
│
├── portal/
│   └── index.html                    # NovaTel Metrics Hub — single-page web app
│                                     # (queries Fabric models via MSAL + REST API)
│
├── setup/
│   ├── deploy_demo.ps1               # One-click Fabric workspace setup + repo injection
│   └── run_demo.ps1                  # Interactive live-demo walkthrough script
│
├── tools/
│   └── validate_tmdl_structure.py    # Standalone TMDL structure validator
│
├── .github/workflows/
│   └── sync-and-validate.yml         # GitHub Action: auto-sync on enterprise change
│
└── tests/
    ├── test_sync.py                  # Unit tests for sync logic
    └── test_tmdl_validity.py         # TMDL structural validation

NovaTel Communications — Data Model

Enterprise Workspace

Table Rows (sample) Description
dim_customer 1 000 Customers with Segment, Region, Status
dim_product 50 Products across Mobile, Broadband, TV, Bundle categories
dim_date 1 461 Date dimension 2023–2026 with fiscal year/quarter
dim_region 5 UK regions with Tier classification
fact_subscriptions 10 000 MRR, status, customer-product-region grain
fact_network_usage 50 000 Data GB, voice minutes, SMS by customer and date
fact_support_tickets 5 000 Tickets with category, priority, resolution hours
fact_billing 20 000 Billed/paid amounts, payment method, days overdue

Enterprise Measure Groups:

File Measures
_enterprise_core_measures.tmdl Total MRR, Active Subscribers, Cancelled Subscribers, Churn Rate, ARPU, Revenue, Collection Rate, Net Subscriber Growth, New Subscribers
network_measures.tmdl Network-specific KPIs (enterprise-only)
support_measures.tmdl Support ticket KPIs (enterprise-only)

Consumer Domain Workspace (domain-specific)

Table Rows (sample) Description
fact_consumer_campaigns 20 Marketing campaigns with budget, impressions, conversions
fact_consumer_nps 2 000 NPS surveys with score, feedback category
dim_consumer_plans 15 Prepaid/postpaid plans with data/voice caps

Consumer Domain Measures (in consumer_measures.tmdl): Campaign ROI, NPS Score Avg, Promoter Pct, Detractor Pct, Net Promoter Score, Consumer Churn Rate, Total Campaign Impressions, Total Campaign Conversions, Campaign Conversion Rate

Inherited from Enterprise: 5 tables (dim_customer, dim_product, dim_date, fact_subscriptions, fact_billing) and all 9 core KPI measures from _enterprise_core_measures.tmdl.

Inheritance Whitelist

The objects synced from enterprise to consumer are declared in sync/sync_config.yaml:

inherit:
  tables:
    - dim_customer
    - dim_product
    - dim_date
    - fact_subscriptions
    - fact_billing
  measures:
    - _enterprise_core_measures

How the Sync Works — Step by Step

  1. Enterprise team modifies a table or measure in enterprise/semantic-model/ and pushes to main

  2. GitHub Actions detects the change via the paths: filter on enterprise/semantic-model/**

  3. sync_enterprise_to_domain.py reads sync_config.yaml and copies the specified TMDL files to consumer-domain/…/_inherited/, prepending a provenance header that marks them as managed

  4. validate_sync.py strips the header and confirms each inherited file matches its source byte-for-byte

  5. GitHub Actions commits the updated _inherited/ files, creates a new branch (chore/tmdl-sync-<timestamp>), and opens a Pull Request for the consumer domain team

  6. Consumer domain team reviews the PR, optionally validates in a Fabric development workspace, then merges

  7. Both Fabric workspaces pick up the change via Fabric Git integration ("Update all" in Source Control)


Metrics Hub Portal

The portal/index.html is a self-contained single-page web application that demonstrates how domain teams can build custom dashboards on top of the inherited semantic models.

image

Features:

  • MSAL browser-based authentication against Azure AD / Entra
  • Live Power BI REST API queries to both semantic models
  • Enterprise KPIs panel (Total MRR, ARPU, Active Subscribers, Churn Rate)
  • Consumer domain panel (NPS, Campaign ROI, Consumer Churn Rate)
  • Inherited measure badges showing which metrics come from enterprise
  • No server required — open portal/index.html in any browser

To run the portal, follow the setup in docs/deployment-guide.md.


Deploy from Template

This repository is a public template. To stand up a live demo:

Quick Start (One-Click)

# 1. Discover available Fabric capacities
.\setup\deploy_demo.ps1 -ListCapacities -TenantId "contoso.onmicrosoft.com"

# 2. Deploy everything to Fabric + create a private deployed repo
.\setup\deploy_demo.ps1 -TenantId "contoso.onmicrosoft.com" -CapacityId "<guid>"

The script:

  1. Authenticates to Azure / Fabric via Connect-AzAccount
  2. Creates NovaTel_Enterprise and NovaTel_Consumer_Domain Fabric workspaces
  3. Creates lakehouses and loads sample data (≈87K enterprise + 2K consumer rows)
  4. Creates OneLake shortcuts so inherited tables point to enterprise data (zero duplication)
  5. Clones this template to a new private GitHub repo with real GUIDs injected
  6. Prints Fabric portal URLs and Fabric Git integration instructions

See docs/deployment-guide.md for full setup guidance including prerequisites, step-by-step instructions, and troubleshooting.

Local Development Only

# Clone the repository
git clone https://github.com/bgbailey/TMDLInheritanceDemo.git
cd TMDLInheritanceDemo

# Install Python dependencies
pip install pyyaml pytest faker pandas pyarrow

# Run the sync manually
python sync/sync_enterprise_to_domain.py --repo-root .

# Validate
python sync/validate_sync.py --repo-root .

# Run tests
pytest tests/ -v

Live Demo Walkthrough

The included setup/run_demo.ps1 script guides you through a live audience demo that shows:

Enterprise developer adds a measure → GitHub Actions fires → sync pipeline runs → PR opens for domain team → merge → Fabric workspaces update automatically

See docs/demo-walkthrough.md for the complete step-by-step guide, or run:

# From within the deployed repo (TMDLInheritanceDemo-Deployed)
.\setup\run_demo.ps1

GitHub Actions Workflow

The workflow at .github/workflows/sync-and-validate.yml:

Trigger Condition
Push Any file under enterprise/semantic-model/** on main
Manual workflow_dispatch with optional dry_run flag

Pipeline steps:

checkout → setup Python → install pyyaml
    → sync_enterprise_to_domain.py
    → validate_sync.py
    → pytest tests/
    → detect changes in _inherited/
    → [if changed] commit + push feature branch + open PR

No secrets or environment variables are required. The workflow uses the built-in GITHUB_TOKEN with contents: write and pull-requests: write permissions.


References

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors