Data Catalog: Purview, Metadata Scanning & Glossary

Enterprise data cataloging with Purview for metadata discovery, classification, and business glossary management

Data Catalog Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    DATA CATALOG ARCHITECTURE                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  DATA SOURCES           SCANNING            CATALOG                 │
│  ┌──────────┐        ┌──────────────┐     ┌──────────────┐        │
│  │ ADLS     │───────>│ Purview      │────>│ Data Map     │        │
│  │ Gen2     │        │ Scanner      │     │              │        │
│  └──────────┘        └──────────────┘     │ • Assets     │        │
│                                           │ • Lineage    │        │
│  ┌──────────┐        ┌──────────────┐     │ • Classif.   │        │
│  │ Synapse  │───────>│ Purview      │────>│              │        │
│  │ SQL      │        │ Scanner      │     │ Collections  │        │
│  └──────────┘        └──────────────┘     │              │        │
│                                           │ • Domain A   │        │
│  ┌──────────┐        ┌──────────────┐     │ • Domain B   │        │
│  │ Power BI │───────>│ Purview      │────>│ • Domain C   │        │
│  │          │        │ Integration  │     └──────────────┘        │
│  └──────────┘        └──────────────┘                               │
│                                                                     │
│  DISCOVERY & GOVERNANCE:                                             │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                                                               │   │
│  │  BUSINESS GLOSSARY      DATA CLASSIFICATION                  │   │
│  │  ┌─────────────────┐   ┌─────────────────┐                 │   │
│  │  │ Sales Revenue   │   │ PII.Email       │                 │   │
│  │  │ Customer Segment│   │ PII.Phone       │                 │   │
│  │  │ Product Category│   │ Financial.Card  │                 │   │
│  │  │ Order Status    │   │ Custom.Code     │                 │   │
│  │  └─────────────────┘   └─────────────────┘                 │   │
│  │                                                               │   │
│  │  SEARCH & DISCOVERY     ACCESS POLICIES                      │   │
│  │  ┌─────────────────┐   ┌─────────────────┐                 │   │
│  │  │ Full-text search │   │ RBAC per asset  │                 │   │
│  │  │ Tag-based        │   │ Sensitivity     │                 │   │
│  │  │ Column-level     │   │ labels          │                 │   │
│  │  │ Impact analysis  │   │ Compliance      │                 │   │
│  │  └─────────────────┘   └─────────────────┘                 │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Purview Scanning Setup

from azure.purview.datamap import PurviewDataMapClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = PurviewDataMapClient(
    credential=credential,
    account_name="purview-prod"
)

# Create collection for data domain
collection = client.collection.create_collection(
    collection={
        "name": "SalesData",
        "parentCollectionName": "Root",
        "description": "Sales domain data assets"
    }
)

# Create scan for ADLS Gen2
scan = client.scan.create_scan(
    scan_name="adls-sales-scan",
    collection_name="SalesData",
    properties={
        "dataSource": {
            "type": "AzureDataLakeStorageGen2",
            "properties": {
                "url": "https://stdatalake001.dfs.core.windows.net",
                "tenantId": "tenant-id"
            }
        },
        "scanRuleset": {
            "type": "System",
            "name": "AzureDataLakeStorageGen2"
        },
        "schedule": {
            "frequency": "Daily",
            "time": "02:00"
        }
    }
)

# Run scan
client.scan.run_scan(
    collection_name="SalesData",
    scan_name="adls-sales-scan"
)

Business Glossary Management

# Create glossary term
term = client.glossary.create_glossary_term(
    glossary_name="BusinessGlossary",
    glossary_term={
        "name": "Sales Revenue",
        "description": "Total revenue from product sales",
        "abbreviation": "Rev",
        "termStatus": "Approved",
        "steward": "data-team@company.com",
        "relatedTerms": ["Net Revenue", "Gross Revenue"],
        "synonyms": ["Sales Income", "Revenue"]
    }
)

# Link term to data asset
client.relationship.create_relationship(
    entity1_type="AtlasGlossaryTerm",
    entity1_guid="term-guid",
    entity2_type="azure_datalake_gen2_path",
    entity2_guid="asset-guid",
    relationshipType="AtlasGlossaryTermAtlasGlossaryTerm"
)

ℹ️

Pro Tip: Use Purview's automated classification to discover sensitive data. Create custom classifiers for domain-specific patterns (e.g., internal customer codes, product SKUs).

Interview Questions

Q1: How does Purview differ from traditional data catalogs? A: Purview is cloud-native, integrates with Azure services, provides automated scanning/classification, and supports hybrid environments. Traditional catalogs are often on-premises with manual metadata entry.

Q2: What is the benefit of linking glossary terms to data assets? A: Links business terminology to technical assets, enabling non-technical users to discover relevant data. Supports impact analysis when terms change and provides context for data governance.

Q3: How do you implement data catalog governance? A: 1) Define ownership per collection/domain, 2) Establish scanning schedules, 3) Configure classification rules, 4) Create business glossary, 5) Set up access policies, 6) Monitor catalog usage and quality.