Azure AD, IAM & Managed Identities

Mastering identity management and access control for secure data engineering pipelines

Identity Architecture for Data Engineering

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    IDENTITY & ACCESS MANAGEMENT                      │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  IDENTITY PROVIDERS         AUTHENTICATION         AUTHORIZATION   │
│  ┌──────────────┐          ┌──────────────┐      ┌──────────────┐ │
│  │ Azure AD     │──────────│ OAuth 2.0    │─────>│   RBAC      │ │
│  │ (Entra ID)   │          │ OpenID Conn  │      │   Roles     │ │
│  └──────────────┘          └──────────────┘      └──────────────┘ │
│         │                        │                       │         │
│         ▼                        ▼                       ▼         │
│  ┌──────────────┐          ┌──────────────┐      ┌──────────────┐ │
│  │ Managed      │          │ Token        │      │ Data         │ │
│  │ Identities   │          │ Exchange     │      │ Factory      │ │
│  │              │          │ Service      │      │ RBAC         │ │
│  │ ┌──────────┐│          └──────────────┘      └──────────────┘ │
│  │ │ System   ││                                                  │
│  │ │ Assigned ││          ┌──────────────┐      ┌──────────────┐ │
│  │ └──────────┘│          │ Conditional  │      │ Synapse      │ │
│  │ ┌──────────┐│          │ Access       │      │ RBAC         │ │
│  │ │ User     ││          │ Policies     │      │              │ │
│  │ │ Assigned ││          └──────────────┘      └──────────────┘ │
│  │ └──────────┘│                                                  │
│  └──────────────┘          ┌──────────────┐      ┌──────────────┐ │
│                            │ Key Vault    │      │ Storage      │ │
│  ┌──────────────┐          │ Access       │      │ RBAC         │ │
│  │ Service      │──────────│ Policies     │      │              │ │
│  │ Principals   │          └──────────────┘      └──────────────┘ │
│  └──────────────┘                                                  │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Managed Identities Deep Dive

System-Assigned vs User-Assigned

Feature	System-Assigned	User-Assigned
Lifecycle	Tied to resource	Independent
Sharing	Single resource	Multiple resources
Cleanup	Auto-deleted	Manual cleanup
Use Case	Single-service auth	Multi-service scenarios
Maximum	1 per resource	Unlimited

Managed Identity Configuration for Data Engineering

# Python: Using Managed Identity with Azure SDKs
from azure.identity import DefaultAzureCredential
from azure.storage.filedatalake import DataLakeServiceClient
from azure.synapse.artifacts import ArtifactsClient

# DefaultAzureCredential tries multiple auth methods automatically
credential = DefaultAzureCredential()

# ADLS Gen2 access with Managed Identity
datalake_client = DataLakeServiceClient(
    account_url="https://stdatalake001.dfs.core.windows.net",
    credential=credential
)

# List files in data lake
file_system_client = datalake_client.get_file_system_client("raw")
paths = list(file_system_client.list_paths(path="2024/01/"))
for path in paths:
    print(f"Path: {path.name}, Size: {path.size}")

# Synapse Artifacts access
artifacts_client = ArtifactsClient(
    credential=credential,
    endpoint="https://syn-workspace.dev.azuresynapse.net"
)

# List pipelines
pipelines = artifacts_client.pipeline.get_pipeline_by_name("etl_pipeline")

Service Principal Configuration

{
  "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "displayName": "sp-dataengineering-prod",
  "password": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

# Service Principal Authentication
from azure.identity import ClientSecretCredential

credential = ClientSecretCredential(
    tenant_id="your-tenant-id",
    client_id="sp-dataengineering-prod",
    client_secret="your-client-secret"
)

# Use with Azure Storage
from azure.storage.filedatalake import DataLakeServiceClient

client = DataLakeServiceClient(
    account_url="https://stdatalake001.dfs.core.windows.net",
    credential=credential
)

RBAC Roles for Data Engineering

Built-in Roles

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│                    RBAC ROLE HIERARCHY                           │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  MANAGEMENT GROUP                                               │
│  └── Subscription                                               │
│      └── Resource Group                                         │
│          └── Resource                                           │
│                                                                 │
│  SCOPE LEVELS (top to bottom):                                  │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ MG > Sub > RG > Resource                                │   │
│  │                                                          │   │
│  │ Roles assigned at higher scope inherit downward          │   │
│  │ More specific scope overrides inherited role             │   │
│  └─────────────────────────────────────────────────────────┘   │
│                                                                 │
│  DATA ENGINEERING ROLES:                                        │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ Storage Blob Data Contributor   - ADLS read/write       │   │
│  │ Storage Blob Data Reader        - ADLS read-only        │   │
│  │ Synapse Administrator           - Full Synapse access   │   │
│  │ Synapse SQL Administrator       - SQL pool management   │   │
│  │ Key Vault Secrets User          - Read secrets          │   │
│  │ Data Factory Contributor        - ADF management        │   │
│  │ Contributor                     - General management    │   │
│  │ Reader                          - View-only access      │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Custom Role for Data Engineers

{
  "Name": "Data Engineer Custom Role",
  "Description": "Custom role for data engineering operations",
  "AssignableScopes": [
    "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  ],
  "Actions": [
    "Microsoft.Storage/storageAccounts/read",
    "Microsoft.Storage/storageAccounts/write",
    "Microsoft.Storage/storageAccounts/blobServices/containers/read",
    "Microsoft.Storage/storageAccounts/blobServices/containers/write",
    "Microsoft.Synapse/workspaces/read",
    "Microsoft.Synapse/workspaces/sqlPools/read",
    "Microsoft.Synapse/workspaces/sqlPools/write",
    "Microsoft.Synapse/workspaces/notebooks/read",
    "Microsoft.Synapse/workspaces/notebooks/write",
    "Microsoft.DataFactory/pipelines/read",
    "Microsoft.DataFactory/pipelines/write",
    "Microsoft.DataFactory/factories/read",
    "Microsoft.DataFactory/factories/write",
    "Microsoft.KeyVault/vaults/secrets/read"
  ],
  "NotActions": [
    "Microsoft.Authorization/*/Delete",
    "Microsoft.Authorization/*/Write",
    "Microsoft.Authorization/elevateAccess/Action"
  ]
}

⚠️

Security Critical: Never store Service Principal secrets in code, environment variables, or configuration files. Always use Azure Key Vault with Managed Identities for secret retrieval.

Azure AD Authentication Flow for Data Services

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    AUTHENTICATION FLOW                                │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  1. REQUEST TOKEN                                                   │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐                  │
│  │ ADF/     │────>│ Azure AD │────>│ Token    │                  │
│  │ Databricks│     │ Endpoint │     │ Service  │                  │
│  └──────────┘     └──────────┘     └────┬─────┘                  │
│                                         │                          │
│  2. VALIDATE & ISSUE                     │                          │
│                                         ▼                          │
│  ┌──────────────────────────────────────────────────────────┐    │
│  │ Azure AD validates:                                       │    │
│  │ • Service Principal exists                                │    │
│  │ • Credentials are valid                                   │    │
│  │ • SP has required permissions                            │    │
│  │ • Conditional Access policies pass                       │    │
│  └──────────────────────────────────────────────────────────┘    │
│                                         │                          │
│  3. RECEIVE TOKEN                        ▼                          │
│  ┌──────────┐     ┌──────────┐     ┌──────────┐                  │
│  │ ADF/     │<────│ JWT      │<────│ Response │                  │
│  │ Databricks│     │ Token    │     │          │                  │
│  └────┬─────┘     └──────────┘     └──────────┘                  │
│       │                                                           │
│  4. ACCESS RESOURCE                                                │
│       ▼                                                           │
│  ┌──────────┐     ┌──────────┐                                   │
│  │ ADLS/    │<────│ Use JWT  │                                   │
│  │ Synapse  │     │ as Bearer│                                   │
│  └──────────┘     │ Token    │                                   │
│                   └──────────┘                                   │
└─────────────────────────────────────────────────────────────────────┘

Bicep Template for IAM Setup

// Managed Identity for ADF
resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'mi-datafactory-prod'
  location: location
  tags: {
    Environment: 'Production'
    Project: 'DataEngineering'
  }
}

// Role Assignment for ADF on ADLS
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(resourceGroup().id, 'Storage Blob Data Contributor', managedIdentity.id)
  scope: storageAccount
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'ba92f5b4-2d11-453d-a403-e96b0029c9fe')
    principalId: managedIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

// Role Assignment for Key Vault access
resource keyVaultRoleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(resourceGroup().id, 'Key Vault Secrets User', managedIdentity.id)
  scope: keyVault
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '4633458b-17de-408a-b875-068636670185')
    principalId: managedIdentity.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

// Output the Managed Identity Client ID
output managedIdentityClientId string = managedIdentity.properties.clientId
output managedIdentityPrincipalId string = managedIdentity.properties.principalId

ℹ️

Best Practice: Use User-Assigned Managed Identities when multiple services (ADF, Databricks, Synapse) need to access the same resources. This simplifies RBAC management and avoids role assignment proliferation.

Conditional Access Policies for Data Engineering

Architecture Diagram

┌─────────────────────────────────────────────────────────────────┐
│              CONDITIONAL ACCESS POLICY FLOW                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  ┌──────────┐     ┌──────────────┐     ┌──────────────┐       │
│  │ User/SP  │────>│ Sign-in      │────>│ Policy       │       │
│  │ Request  │     │ Request      │     │ Evaluation   │       │
│  └──────────┘     └──────────────┘     └──────┬───────┘       │
│                                                │                │
│                    ┌───────────────────────────┤                │
│                    │                           │                │
│                    ▼                           ▼                │
│            ┌──────────────┐           ┌──────────────┐         │
│            │ Block:       │           │ Grant:       │         │
│            │ • Outside IP │           │ • MFA        │         │
│            │ • No MFA     │           │ • Compliant  │         │
│            │ • Non-Comply │           │   Device     │         │
│            └──────────────┘           │ • App Cond.  │         │
│                                       └──────────────┘         │
│                                                                 │
│  POLICY EXAMPLES FOR DATA ENGINEERING:                          │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 1. Block sign-ins from outside corporate network        │   │
│  │ 2. Require MFA for all admin operations                 │   │
│  │ 3. Require compliant device for Synapse access          │   │
│  │ 4. Block legacy authentication protocols                │   │
│  │ 5. Session timeout for production portal access         │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

Python SDK for IAM Management

from azure.identity import DefaultAzureCredential
from azure.mgmt.authorization import AuthorizationManagementClient
from azure.mgmt.authorization.models import RoleAssignmentCreateParameters

credential = DefaultAzureCredential()
auth_client = AuthorizationManagementClient(credential, subscription_id)

# Assign 'Storage Blob Data Contributor' to ADF Managed Identity
role_assignment_params = RoleAssignmentCreateParameters(
    role_definition_id=f"/subscriptions/{subscription_id}/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe",
    principal_id=adf_managed_identity_principal_id,
    principal_type="ServicePrincipal"
)

auth_client.role_assignments.create(
    scope=f"/subscriptions/{subscription_id}/resourceGroups/rg-datalake-prod/providers/Microsoft.Storage/storageAccounts/stdatalake001",
    role_assignment_name="adf-blob-contributor",
    parameters=role_assignment_params
)

# List all role assignments for a resource
assignments = auth_client.role_assignments.list_for_scope(
    scope=f"/subscriptions/{subscription_id}/resourceGroups/rg-datalake-prod"
)
for assignment in assignments:
    print(f"Role: {assignment.role_definition_id}")
    print(f"Principal: {assignment.principal_id}")
    print(f"Type: {assignment.principal_type}")

Interview Questions

Q1: Why should you never use Storage Account Keys for data engineering pipelines? A: Storage Account Keys provide full access to the storage account and are long-lived credentials that can be compromised. Managed Identities eliminate credential management, provide automatic rotation, and enable granular RBAC. If keys must be used, store them in Key Vault and rotate regularly.

Q2: Explain the difference between RBAC at the Storage Account level vs Container level. A: Storage Account-level RBAC applies to all containers and blobs. Container-level RBAC (using resource scope) provides more granular control. For example, grant a service principal access to only the "raw" container but not "curated."

Q3: How do you troubleshoot a 403 Forbidden error when ADF tries to access ADLS? A: Check: 1) Managed Identity is enabled on ADF, 2) Correct RBAC role is assigned at the right scope, 3) No Deny assignments override the role, 4) Private Endpoints/Firewall rules allow traffic, 5) Azure AD tenant matches between resources.

Azure Active Directory, IAM & Managed Identities

Azure AD, IAM & Managed Identities

Identity Architecture for Data Engineering

Managed Identities Deep Dive

System-Assigned vs User-Assigned

Managed Identity Configuration for Data Engineering

Service Principal Configuration

RBAC Roles for Data Engineering

Built-in Roles

Custom Role for Data Engineers

Azure AD Authentication Flow for Data Services

Bicep Template for IAM Setup

Conditional Access Policies for Data Engineering

Python SDK for IAM Management

Interview Questions