Azure VNet, Private Endpoints & Networking
Securing data engineering workloads with private networking, endpoints, and network isolation
Network Architecture for Data Engineering
Private Endpoints Configuration
Bicep Template
// Private DNS Zone for ADLS
resource privateDnsZone 'Microsoft.Network/privateDnsZones@2024-01-01' = {
name: 'privatelink.dfs.core.windows.net'
location: 'global'
properties: {
maxNumberOfRecordSets: 25000
}
}
// VNet Link for Private DNS Zone
resource vnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2024-01-01' = {
name: 'vnet-link-spoke'
parent: privateDnsZone
location: 'global'
properties: {
virtualNetwork: {
id: spokeVnet.id
}
registrationEnabled: false
}
}
// Private Endpoint for ADLS Gen2
resource privateEndpoint 'Microsoft.Network/privateEndpoints@2024-01-01' = {
name: 'pe-adls-gen2'
location: location
properties: {
subnet: {
id: spokeVnet.properties.subnets[1].id // Data subnet
}
privateLinkServiceConnections: [
{
name: 'adls-connection'
properties: {
privateLinkServiceId: storageAccount.id
groupIds: ['dfs']
}
}
]
}
}
// DNS Zone Group
resource dnsZoneGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2024-01-01' = {
name: 'default'
parent: privateEndpoint
properties: {
privateDnsZoneConfigs: [
{
name: 'dnsZoneConfig'
properties: {
privateDnsZoneId: privateDnsZone.id
}
}
]
}
}
Network Security Groups Rules
{
"securityRules": [
{
"name": "AllowDatabricksInbound",
"properties": {
"priority": 100,
"direction": "Inbound",
"access": "Allow",
"protocol": "Tcp",
"sourcePortRange": "*",
"destinationPortRange": "443",
"sourceAddressPrefix": "AzureDatabricks",
"destinationAddressPrefix": "*"
}
},
{
"name": "AllowSynapseManagement",
"properties": {
"priority": 200,
"direction": "Inbound",
"access": "Allow",
"protocol": "Tcp",
"sourcePortRange": "*",
"destinationPortRanges": ["1433", "1443"],
"sourceAddressPrefix": "SqlManagement",
"destinationAddressPrefix": "*"
}
},
{
"name": "DenyAllInbound",
"properties": {
"priority": 4096,
"direction": "Inbound",
"access": "Deny",
"protocol": "*",
"sourcePortRange": "*",
"destinationPortRange": "*",
"sourceAddressPrefix": "*",
"destinationAddressPrefix": "*"
}
}
]
}
json
{
"name": "adf-prod-vnet",
"type": "Microsoft.DataFactory/factories",
"apiVersion": "2018-06-01",
"location": "eastus2",
"identity": {
"type": "SystemAssigned"
},
"properties": {
"provisioningState": "Succeeded",
"publicNetworkAccess": "Disabled",
"networkAcls": {
"defaultAction": "Deny"
}
}
}
Self-Hosted Integration Runtime in VNet
{
"name": "ir-selfhosted-prod",
"type": "Microsoft.DataFactory/factories/integrationRuntimes",
"apiVersion": "2018-06-01",
"properties": {
"type": "SelfHosted",
"typeProperties": {
"linkedInfo": {
"type": "LinkedIntegrationRuntimeKey",
"key": "<EncryptedKey>"
}
},
"hostCaching": "Enabled"
}
}
python
# Python script to audit NSG rules
from azure.identity import DefaultAzureCredential
from azure.mgmt.network import NetworkManagementClient
credential = DefaultAzureCredential()
network_client = NetworkManagementClient(credential, subscription_id)
# List all NSGs and their rules
nsgs = network_client.network_security_groups.list_all()
for nsg in nsgs:
print(f"\nNSG: {nsg.name}")
print(f"Resource Group: {nsg.id.split('/')[4]}")
for rule in nsg.security_rules:
if rule.direction == 'Inbound' and rule.access == 'Allow':
print(f" â ī¸ Allow Rule: {rule.name}")
print(f" Priority: {rule.priority}")
print(f" Source: {rule.source_address_prefix}")
print(f" Dest Port: {rule.destination_port_range}")
Interview Questions
Q1: Explain the Hub-Spoke network topology for data engineering. A: The Hub VNet contains shared services (firewall, VPN gateway, Bastion). Spoke VNets contain data engineering workloads (Databricks, Synapse, ADF). Hub-Spoke peering allows centralized management while maintaining network isolation between workloads.
Q2: When would you use Self-Hosted Integration Runtime vs Managed VNet in ADF? A: Self-Hosted IR is needed when connecting to on-premises data sources or when you need custom network routing. Managed VNet with Managed Private Endpoints is preferred for cloud-only scenarios as it eliminates VM management.
Q3: How do Private Endpoints affect data transfer costs? A: Private Endpoints route traffic through Azure Private Link, which avoids public internet traversal. However, data transfer within the same region via Private Endpoints still incurs standard Azure data transfer costs. Cross-region transfers via Private Link incur additional charges.