Azure Event Hubs: Partitions, Consumer Groups & Capture
Real-time event streaming with partitioning, capture, and exactly-once processing guarantees
Event Hubs Architecture
Partitioning Strategy
Event Hubs Capture Configuration
{
"properties": {
"captureDescription": {
"enabled": true,
"encoding": "Avro",
"destination": {
"name": "EventHubArchiveImageFormat",
"properties": {
"storageAccountResourceId": "/subscriptions/xxx/resourceGroups/rg/providers/Microsoft.Storage/storageAccounts/stdatalake001",
"blobContainer": "event-hubs-capture",
"archiveNameFormat": "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}",
"timeWindow": "00:05:00",
"sizeLimitInBytes": 104857600,
"emptyWriterBehavior": "DropIfEmpty"
}
},
"skipEmptyArchive": true
}
}
}
Python Producer/Consumer
# Producer
from azure.eventhub import EventHubProducerClient, EventData
import json
producer = EventHubProducerClient.from_connection_string(
conn_str="Endpoint=sb://ns-prod.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;...",
eventhub_name="sales-events"
)
events = [
{"deviceId": "sensor-001", "temperature": 72.5, "timestamp": "2024-01-15T10:30:00Z"},
{"deviceId": "sensor-002", "temperature": 68.3, "timestamp": "2024-01-15T10:30:01Z"}
]
event_batch = producer.create_batch()
for event in events:
event_batch.add(EventData(json.dumps(event)))
producer.send_batch(event_batch)
producer.close()
# Consumer
from azure.eventhub import EventHubConsumerClient
def on_event(partition_context, event):
print(f"Received event from partition {partition_context.partition_id}")
print(f"Event body: {event.body_as_str()}")
print(f"Sequence number: {event.sequence_number}")
partition_context.update_checkpoint(event)
client = EventHubConsumerClient.from_connection_string(
conn_str="Endpoint=sb://ns-prod.servicebus.windows.net/...",
consumer_group="$Default",
eventhub_name="sales-events"
)
with client:
client.receive(
on_event=on_event,
starting_position="-1" # Start from beginning
)
âšī¸
Pro Tip: Use partition keys that evenly distribute events across partitions. Avoid using timestamps or sequential IDs as partition keys, as they create hot partitions.
Interview Questions
Q1: Explain the difference between Event Hubs and Event Grid. A: Event Hubs is a high-throughput event streaming service (millions of events/sec). Event Grid is a reactive event routing service (smart routing, filtering). Use Event Hubs for data ingestion pipelines; Event Grid for event-driven architectures.
Q2: How do you handle message ordering in Event Hubs? A: Event Hubs guarantees ordering within a partition. Use the same partition key for related events. For global ordering, use a single partition (limits throughput). For most use cases, partition-level ordering is sufficient.
Q3: What is the cost impact of Event Hubs Capture? A: Capture is included in the Event Hub cost (no additional charges). However, storage costs apply for the captured files in ADLS. Capture reduces the need for custom ETL jobs, saving compute costs.