Data Security: Key Vault, Encryption & Managed Identity
Enterprise data security with Key Vault, encryption, Managed Identities, and network isolation
Security Architecture
Key Vault Integration
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
client = SecretClient(
vault_url="https://kv-dataengineering.vault.azure.net/",
credential=credential
)
# Store secret
client.set_secret("adls-storage-key", "your-storage-key")
# Retrieve secret
secret = client.get_secret("adls-storage-key")
print(f"Secret value: {secret.value}")
# Store connection string
client.set_secret(
"synapse-connection",
"Server=tcp:syn-prod.sql.azuresynapse.net,1433;..."
)
Customer-Managed Keys (CMK)
{
"resources": [
{
"type": "Microsoft.KeyVault/vaults/keys",
"apiVersion": "2023-02-01",
"name": "cmk-adls",
"properties": {
"kty": "RSA-HSM",
"keySize": 2048,
"keyOps": ["encrypt", "decrypt", "wrapKey", "unwrapKey"]
}
}
]
}
Encryption Configuration
resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
name: 'stdatalake001'
location: location
kind: 'StorageV2'
sku: { name: 'Standard_LRS' }
properties: {
encryption: {
services: {
blob: { enabled: true, keyType: 'Account' }
file: { enabled: true, keyType: 'Account' }
}
keySource: 'Microsoft.Keyvault'
keyVaultProperties: {
keyName: 'cmk-adls'
keyVaultUri: 'https://kv-dataengineering.vault.azure.net/'
}
}
minimumTlsVersion: 'TLS1_2'
supportsHttpsTrafficOnly: true
allowBlobPublicAccess: false
}
}
â ī¸
Security Critical: Always use Customer-Managed Keys (CMK) for production workloads. Store keys in Key Vault with automatic rotation. Never hardcode credentials in code or configuration files.
Interview Questions
Q1: Explain the difference between Microsoft-managed keys and Customer-managed keys. A: Microsoft-managed keys are fully managed by Azure (simpler). Customer-managed keys are stored in Key Vault and managed by the customer (more control, compliance). Use CMK for regulated industries.
Q2: How do you implement zero-trust security for data engineering? A: 1) Private Endpoints for all services, 2) Managed Identities (no secrets), 3) RBAC with least privilege, 4) Network segmentation, 5) Encryption at rest and in transit, 6) Continuous monitoring and alerting.
Q3: What are the best practices for Key Vault security? A: 1) Enable purge protection, 2) Use soft delete, 3) Restrict access with RBAC, 4) Enable logging, 5) Use private endpoints, 6) Rotate keys regularly, 7) Use HSM-backed keys for high security.