Azure Cloud Overview & Global Infrastructure
Understanding Microsoft Azure's global footprint, resource management, and foundational services for data engineering
Azure Global Infrastructure
Azure operates the second-largest cloud infrastructure globally with 60+ announced regions spanning 140+ countries. As a data engineer, understanding this infrastructure is critical for designing high-availability, low-latency data solutions.
Regions and Availability Zones
Key Concepts
| Concept | Description | Data Engineering Impact |
|---|---|---|
| Region | Geographic area with 1-3+ AZs | Data residency, latency optimization |
| Availability Zone | Physically separate datacenter | HA for Synapse, Databricks clusters |
| Region Pair | Paired regions for DR | Geo-redundant backup of ADLS |
| Resource Group | Logical container for resources | Organize data engineering assets |
| Subscription | Billing and access boundary | Cost allocation per project |
| Management Group | Policy hierarchy | Enterprise governance |
âšī¸
Pro Tip: When designing data pipelines, always place your compute (ADF Integration Runtime, Databricks) in the same region as your data storage (ADLS, Synapse) to avoid data transfer costs and latency.
Resource Hierarchy
ARM Template Example
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"storageAccountName": {
"type": "string",
"metadata": { "description": "ADLS Gen2 storage account name" }
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
}
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2023-01-01",
"name": "[parameters('storageAccountName')]",
"location": "[parameters('location')]",
"sku": { "name": "Standard_LRS", "tier": "Standard" },
"kind": "StorageV2",
"properties": {
"isHnsEnabled": true,
"supportsHttpsTrafficOnly": true,
"minimumTlsVersion": "TLS1_2",
"accessTier": "Hot",
"encryption": {
"services": {
"blob": { "enabled": true },
"file": { "enabled": true }
},
"keySource": "Microsoft.Storage"
},
"networkAcls": {
"defaultAction": "Deny",
"virtualNetworkRules": [],
"ipRules": []
}
},
"tags": {
"Environment": "Production",
"Project": "DataEngineering"
}
}
],
"outputs": {
"storageAccountId": {
"type": "string",
"value": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
}
}
}
bash
#!/bin/bash
# Create Resource Group
az group create \
--name "rg-dataengineering-prod" \
--location "eastus2" \
--tags Environment=Production Project=DataEngineering
# Create Storage Account with HNS (ADLS Gen2)
az storage account create \
--name "stdatalakeprodeastus2" \
--resource-group "rg-dataengineering-prod" \
--location "eastus2" \
--sku Standard_LRS \
--kind StorageV2 \
--enable-hierarchical-namespace true \
--min-tls-version TLS1_2 \
--allow-blob-public-access false \
--https-only true
# Create containers for data lake zones
az storage container create \
--name "raw" \
--account-name "stdatalakeprodeastus2"
az storage container create \
--name "curated" \
--account-name "stdatalakeprodeastus2"
az storage container create \
--name "sandbox" \
--account-name "stdatalakeprodeastus2"
# Create Synapse Workspace
az synapse workspace create \
--name "syn-prod-workspace" \
--resource-group "rg-dataengineering-prod" \
--location "eastus2" \
--storage-account "stdatalakeprodeastus2" \
--file-system "synapsefs" \
--sql-admin-login-user "sqladmin" \
--sql-admin-login-password "YourPassword123!"
# Create Synapse SQL Pool (Dedicated)
az synapse sql pool create \
--name "SQLPool01" \
--workspace-name "syn-prod-workspace" \
--resource-group "rg-dataengineering-prod" \
--performance-level DW100c
â ī¸
Important: Always enable HTTPS-only access and TLS 1.2 minimum for all storage accounts. Disable public blob access to prevent data leaks. Use Managed Identities instead of connection strings.
SLA and Performance Guarantees
| Service | SLA | RPO | RTO |
|---|---|---|---|
| ADLS Gen2 (RA-GRS) | 99.99% | <15 min | <30 min |
| Synapse Dedicated Pool | 99.9% | Point-in-time restore | Hours |
| Azure Functions | 99.95% | N/A | Seconds |
| Event Hubs | 99.95% | 0 (with capture) | Minutes |
| Cosmos DB (Multi-region) | 99.999% | 0 | 0 |
| Databricks | 99.9% | N/A | Minutes |
Pricing Tiers Overview
Best Practices Summary
- Always use Managed Identities instead of storage keys or connection strings
- Enable soft delete on storage accounts for accidental deletion protection
- Use Private Endpoints to keep traffic off the public internet
- Tag all resources consistently for cost management and governance
- Use Availability Zones for production workloads requiring high availability
- Implement Azure Policy to enforce security standards across subscriptions
- Monitor costs using Azure Cost Management and set up budget alerts
- Use ARM/Bicep templates for infrastructure as code (IaC) to ensure consistency
Interview Questions
Q1: Explain the difference between Azure Regions and Availability Zones. A: Regions are geographic areas containing multiple datacenters. Availability Zones are physically separate datacenters within a region, each with independent power, cooling, and networking. For data engineering, use Availability Zones for high availability of critical services like Synapse and Databricks clusters.
Q2: Why should you deploy compute and storage in the same Azure region? A: Deploying in the same region eliminates data transfer costs (which can be significant at scale) and minimizes network latency. For example, ADF Integration Runtime in East US reading from ADLS in East US avoids the $0.01/GB transfer fee.
Q3: What is the benefit of using Azure Resource Groups for data engineering projects? A: Resource Groups provide logical organization, simplified access control (RBAC at RG level), cost tracking per project, and easy cleanup of resources when a project is complete.