Microsoft Purview: Data Map, Lineage & Classification

Unified data governance with automated discovery, classification, lineage, and data catalog capabilities

Purview Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    MICROSOFT PURVIEW ARCHITECTURE                    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    PURVIEW ACCOUNT                            │   │
│  │                                                               │   │
│  │  DATA MAP                                                     │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │  Sources         Collections      Assets            │    │   │
│  │  │  ┌──────────┐   ┌──────────┐   ┌──────────┐       │    │   │
│  │  │  │ ADLS     │──>│ Sales    │──>│ Tables   │       │    │   │
│  │  │  │ Gen2     │   │ Data     │   │ Files    │       │    │   │
│  │  │  └──────────┘   └──────────┘   └──────────┘       │    │   │
│  │  │  ┌──────────┐   ┌──────────┐   ┌──────────┐       │    │   │
│  │  │  │ Synapse  │──>│ Finance  │──>│ Views    │       │    │   │
│  │  │  │ SQL      │   │ Data     │   │ Procs    │       │    │   │
│  │  │  └──────────┘   └──────────┘   └──────────┘       │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  │                                                               │   │
│  │  DATA CLASSIFICATION                                          │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ Built-in Classifiers    Custom Classifiers           │    │   │
│  │  │ • Email Address         • Employee ID                │    │   │
│  │  │ • Phone Number          • Customer Code              │    │   │
│  │  │ • Credit Card           • Internal Reference         │    │   │
│  │  │ • Social Security #     • Product SKU                │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  │                                                               │   │
│  │  LINEAGE                                                      │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ ADF Pipeline ──> Table ──> View ──> Power BI         │    │   │
│  │  │ Databricks Job ──> Table ──> Dashboard               │    │   │
│  │  │ Synapse SQL ──> View ──> Report                      │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  │                                                               │   │
│  │  DATA GOVERNANCE                                              │   │
│  │  ┌─────────────────────────────────────────────────────┐    │   │
│  │  │ Business Glossary    Access Policies    Policies     │    │   │
│  │  │ Data Domains          Sensitivity Labels  Rules      │    │   │
│  │  └─────────────────────────────────────────────────────┘    │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Data Classification

# Purview SDK for classification
from azure.purview.datamap import PurviewDataMapClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = PurviewDataMapClient(
    credential=credential,
    account_name="purview-prod"
)

# Get classification recommendations
recommendations = client.classification.get_recommendation_list()
print("Recommended classifications:")
for rec in recommendations:
    print(f"  - {rec.name}: {rec.description}")

# Apply classification to entity
client.classification.add_classification(
    entity_type="azure_datalake_gen2_path",
    entity_guid="entity-guid-here",
    classifications=[
        {"typeName": "Microsoft.DataFactory.Sensitivity"},
        {"typeName": "PII.Email"}
    ]
)

# Get classification details
classifications = client.classification.get_classification(
    entity_type="azure_datalake_gen2_path",
    entity_guid="entity-guid-here"
)

Lineage Tracking

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    DATA LINEAGE FLOW                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  SOURCE             PROCESSING            DESTINATION           │
│  ┌──────────┐      ┌──────────────┐     ┌──────────────┐     │
│  │ ADLS     │      │ ADF Pipeline │     │ Synapse      │     │
│  │ Gen2     │─────>│              │────>│ SQL Pool     │     │
│  │          │      │ • Copy       │     │              │     │
│  │ raw/     │      │ • Transform  │     │ curated/     │     │
│  │ sales/   │      │ • Validate   │     │ fact_sales   │     │
│  └──────────┘      └──────────────┘     └──────┬───────┘     │
│                                                 │              │
│                                                 ▼              │
│                                        ┌──────────────┐       │
│                                        │ Power BI     │       │
│                                        │ Dashboard    │       │
│                                        │              │       │
│                                        │ Sales Report │       │
│                                        └──────────────┘       │
│                                                                 │
│  LINEAGE METADATA:                                               │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Pipeline: pl_sales_etl                                   │   │
│  │ │                                                         │   │
│  │ ├─ Source: /raw/sales/day=2024-01-15/*.parquet           │   │
│  │ │  Type: Azure Data Lake Gen2 Path                       │   │
│  │ │  Format: Parquet                                       │   │
│  │ │                                                         │   │
│  │ ├─ Transform: spark_sales_transformation                  │   │
│  │ │  Type: Azure Databricks Notebook                       │   │
│  │ │  Output: Delta Lake format                             │   │
│  │ │                                                         │   │
│  │ └─ Sink: /curated/fact_sales/                            │   │
│  │    Type: Azure Data Lake Gen2 Path                       │   │
│  │    Format: Delta Parquet                                 │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Purview Scanning Configuration

# Create scan for ADLS Gen2
scan = client.scan.create_scan(
    scan_name="adls-sales-scan",
    collection_name="SalesData",
    properties={
        "dataSource": {
            "type": "AzureDataLakeStorageGen2",
            "properties": {
                "url": "https://stdatalake001.dfs.core.windows.net",
                "tenantId": "tenant-id"
            }
        },
        "scanRuleset": {
            "type": "System",
            "name": "AzureDataLakeStorageGen2"
        },
        "schedule": {
            "frequency": "Daily",
            "time": "02:00"
        }
    }
)

# Run scan
client.scan.run_scan(
    collection_name="SalesData",
    scan_name="adls-sales-scan"
)

ℹ️

Pro Tip: Use Purview's automatic classification to discover sensitive data. Configure custom classifiers for domain-specific data patterns (e.g., internal customer codes).

Interview Questions

Q1: How does Purview help with data governance? A: Purview provides automated data discovery, classification of sensitive data, end-to-end lineage tracking, a business glossary for standardization, and access policies for data protection—all in a unified portal.

Q2: Explain the difference between technical and business glossary in Purview. A: Technical glossary maps technical assets (tables, columns) to business terms. Business glossary defines domain-specific terminology and relationships. Together they bridge the gap between technical and business stakeholders.

Q3: How do you implement Purview scanning for a large data lake? A: Create multiple scans per data source (different schedules/filters), use collection hierarchies to organize assets, configure scan rulesets for specific data types, and use managed Private Endpoints for secure scanning.