Data Security: Key Vault, Encryption & Managed Identity

Enterprise data security with Key Vault, encryption, Managed Identities, and network isolation

Security Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                    DATA SECURITY ARCHITECTURE                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  LAYER 1: IDENTITY                                                  │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Azure AD (Entra ID) authentication                        │   │
│  │ • Managed Identities (System + User-assigned)               │   │
│  │ • Service Principals with certificate auth                  │   │
│  │ • Conditional Access Policies                               │   │
│  │ • MFA for privileged access                                 │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  LAYER 2: NETWORK                                                   │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Private Endpoints (no public exposure)                    │   │
│  │ • VNet Integration                                          │   │
│  │ • NSG Rules                                                 │   │
│  │ • Azure Firewall                                            │   │
│  │ • DDoS Protection                                           │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  LAYER 3: DATA PROTECTION                                           │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Encryption at rest (Microsoft-managed / Customer-managed) │   │
│  │ • Encryption in transit (TLS 1.2)                           │   │
│  │ • Key Vault for key management                              │   │
│  │ • Azure Confidential Computing                              │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  LAYER 4: MONITORING & AUDIT                                        │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │ • Azure Monitor + Log Analytics                             │   │
│  │ • Microsoft Defender for Storage                             │   │
│  │ • Key Vault logging                                         │   │
│  │ • Diagnostic settings                                       │   │
│  └─────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────┘

Key Vault Integration

from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
client = SecretClient(
    vault_url="https://kv-dataengineering.vault.azure.net/",
    credential=credential
)

# Store secret
client.set_secret("adls-storage-key", "your-storage-key")

# Retrieve secret
secret = client.get_secret("adls-storage-key")
print(f"Secret value: {secret.value}")

# Store connection string
client.set_secret(
    "synapse-connection",
    "Server=tcp:syn-prod.sql.azuresynapse.net,1433;..."
)

Customer-Managed Keys (CMK)

{
  "resources": [
    {
      "type": "Microsoft.KeyVault/vaults/keys",
      "apiVersion": "2023-02-01",
      "name": "cmk-adls",
      "properties": {
        "kty": "RSA-HSM",
        "keySize": 2048,
        "keyOps": ["encrypt", "decrypt", "wrapKey", "unwrapKey"]
      }
    }
  ]
}

Encryption Configuration

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: 'stdatalake001'
  location: location
  kind: 'StorageV2'
  sku: { name: 'Standard_LRS' }
  properties: {
    encryption: {
      services: {
        blob: { enabled: true, keyType: 'Account' }
        file: { enabled: true, keyType: 'Account' }
      }
      keySource: 'Microsoft.Keyvault'
      keyVaultProperties: {
        keyName: 'cmk-adls'
        keyVaultUri: 'https://kv-dataengineering.vault.azure.net/'
      }
    }
    minimumTlsVersion: 'TLS1_2'
    supportsHttpsTrafficOnly: true
    allowBlobPublicAccess: false
  }
}

⚠️

Security Critical: Always use Customer-Managed Keys (CMK) for production workloads. Store keys in Key Vault with automatic rotation. Never hardcode credentials in code or configuration files.

Interview Questions

Q1: Explain the difference between Microsoft-managed keys and Customer-managed keys. A: Microsoft-managed keys are fully managed by Azure (simpler). Customer-managed keys are stored in Key Vault and managed by the customer (more control, compliance). Use CMK for regulated industries.

Q2: How do you implement zero-trust security for data engineering? A: 1) Private Endpoints for all services, 2) Managed Identities (no secrets), 3) RBAC with least privilege, 4) Network segmentation, 5) Encryption at rest and in transit, 6) Continuous monitoring and alerting.

Q3: What are the best practices for Key Vault security? A: 1) Enable purge protection, 2) Use soft delete, 3) Restrict access with RBAC, 4) Enable logging, 5) Use private endpoints, 6) Rotate keys regularly, 7) Use HSM-backed keys for high security.