📨 Amazon MQ

Master Amazon MQ message brokering with ActiveMQ and RabbitMQ for data engineering.

Module: AWS Data Engineering • Topic 15 of 65 • Premium Content

Amazon MQ Architecture

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    AMAZON MQ ARCHITECTURE                                     │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  PRODUCERS                                                           │    │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐                         │    │
│  │  │  EC2     │  │  Lambda  │  │  On-Prem │                         │    │
│  │  │  Apps    │  │          │  │  Apps    │                         │    │
│  │  └────┬─────┘  └────┬─────┘  └────┬─────┘                         │    │
│  └───────┼──────────────┼──────────────┼──────────────────────────────┘    │
│          ▼              ▼              ▼                                   │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  AMAZON MQ BROKER                                                    │    │
│  │                                                                     │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  Engine Options                                               │  │    │
│  │  │                                                               │  │    │
│  │  │  ┌─────────────────┐        ┌─────────────────┐              │  │    │
│  │  │  │   ActiveMQ      │        │   RabbitMQ      │              │  │    │
│  │  │  │                 │        │                 │              │  │    │
│  │  │  │ • JMS 1.1/2.0  │        │ • AMQP 0-9-1   │              │  │    │
│  │  │  │ • OpenWire     │        │ • MQTT 3.1.1    │              │  │    │
│  │  │  │ • STOMP        │        │ • STOMP         │              │  │    │
│  │  │  │ • WebSocket    │        │ • WebSocket     │              │  │    │
│  │  │  │ • Classic Queue│        │ • Exchanges     │              │  │    │
│  │  │  │ • Conflation   │        │ • Bindings      │              │  │    │
│  │  │  └─────────────────┘        └─────────────────┘              │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  │                                                                     │    │
│  │  ┌───────────────────────────────────────────────────────────────┐  │    │
│  │  │  Deployment Options                                           │  │    │
│  │  │                                                               │  │    │
│  │  │  Single Instance:  $0.30/hr, 1 AZ, dev/test                  │  │    │
│  │  │  Active/Standby:   $0.60/hr, Multi-AZ, production            │  │    │
│  │  │  Cluster:          $1.80/hr, 3 AZ, high availability         │  │    │
│  │  │                                                               │  │    │
│  │  │  Storage: EBS (GP2), 1 GB - 200 GB                          │  │    │
│  │  └───────────────────────────────────────────────────────────────┘  │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│                                ▼                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  CONSUMERS                                                           │    │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐                         │    │
│  │  │  EC2     │  │  Lambda  │  │  ECS     │                         │    │
│  │  │  Apps    │  │          │  │  Tasks   │                         │    │
│  │  └──────────┘  └──────────┘  └──────────┘                         │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

ActiveMQ vs RabbitMQ Comparison

Feature	ActiveMQ	RabbitMQ
Protocol	JMS, OpenWire, STOMP	AMQP, MQTT, STOMP
Message Model	Queue, Topic	Exchange, Queue
Routing	Simple	Complex (bindings)
clustering	Active/Standby	Quorum Queues
Management	Web Console	Management UI
Best For	Enterprise Java	Microservices

Amazon MQ Configuration

import boto3

mq = boto3.client('mq')

# Create ActiveMQ broker
response = mq.create_broker(
    BrokerName='data-pipeline-broker',
    EngineType='ACTIVEMQ',
    EngineVersion='5.17.6',
    HostInstanceType='mq.m5.large',
    DeploymentMode='ACTIVE_STANDBY_MULTI_AZ',
    Users=[
        {
            'Username': 'admin',
            'Password': 'SecurePassword123!',
            'Groups': ['admin']
        },
        {
            'Username': 'pipeline-user',
            'Password': 'PipelinePass456!',
            'Groups': ['users']
        }
    ],
    SecurityGroups=['sg-12345678'],
    SubnetIds=['subnet-12345678', 'subnet-87654321'],
    PubliclyAccessible=False,
    AutoMinorEngineUpgrade=True,
    StorageType='EBS',
    SecurityPolicy='arn:aws:mq:us-east-1:123456789012:securitypolicy/MySecurityPolicy',
    Logs={
        'General': True,
        'Audit': True
    },
    MaintenanceWindowStartTime={
        'DayOfWeek': 'SUNDAY',
        'TimeOfDay': '03:00',
        'TimeZone': 'UTC'
    },
    Tags={'Environment': 'production', 'Team': 'data-engineering'}
)

broker_id = response['BrokerId']
print(f"Broker ID: {broker_id}")

Message Patterns

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    MESSAGE PATTERNS FOR DATA ENGINEERING                      │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  POINT-TO-POINT (Queue)                                             │    │
│  │                                                                     │    │
│  │  Producer ──► Queue ──► Consumer (one message = one consumer)      │    │
│  │                                                                     │    │
│  │  Use Case: Task distribution, work queues                          │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  PUBLISH-SUBSCRIBE (Topic)                                          │    │
│  │                                                                     │    │
│  │  Publisher ──► Topic ──► Subscriber 1                              │    │
│  │                     ──► Subscriber 2                              │    │
│  │                     ──► Subscriber 3                              │    │
│  │                                                                     │    │
│  │  Use Case: Event broadcasting, notifications                       │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  FAN-OUT (Exchange/Binding)                                         │    │
│  │                                                                     │    │
│  │  Producer ──► Exchange ──► Queue 1 ──► Consumer A                  │    │
│  │              │            Queue 2 ──► Consumer B                  │    │
│  │              │            Queue 3 ──► Consumer C                  │    │
│  │                                                                     │    │
│  │  Use Case: Parallel processing, data distribution                  │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  REQUEST-REPLY                                                      │    │
│  │                                                                     │    │
│  │  Client ──► Request Queue ──► Server                              │    │
│  │  Client ◄── Reply Queue ◄── Server                                │    │
│  │                                                                     │    │
│  │  Use Case: Synchronous processing, RPC                            │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

Interview Questions & Answers

Q1: When should you use Amazon MQ vs. SQS/SNS?

Answer:

Amazon MQ: Migrating from on-premises, existing JMS/AMQP apps
SQS/SNS: New cloud-native applications, serverless

Amazon MQ is for existing message broker migrations. SQS/SNS are for new cloud-native designs.

Q2: What is the difference between ActiveMQ and RabbitMQ?

Answer:

ActiveMQ: JMS-based, enterprise Java, Active/Standby HA
RabbitMQ: AMQP-based, microservices, Quorum Queues HA

Choose based on your protocol requirements and existing codebase.

Q3: How does Amazon MQ handle high availability?

Answer:

Active/Standby: 2 AZs, automatic failover
Cluster: 3 AZs, quorum-based consensus
Storage: EBS replication across AZs
RTO: ~minutes for failover

Q4: What are the cost factors for Amazon MQ?

Answer:

Broker instance hours ( $0.30-$ 1.80/hr)
Storage ($0.11/GB/month)
Data transfer (standard AWS rates)
No per-message charges

Q5: How do you monitor Amazon MQ?

Answer:

CloudWatch metrics (CPU, memory, storage)
Broker logs (General, Audit)
JMX monitoring (ActiveMQ)
Management UI (both engines)
CloudTrail API logging

Summary

Amazon MQ is AWS's managed message broker service. Key takeaways:

Engines: ActiveMQ (JMS) and RabbitMQ (AMQP)
Deployment: Single, Active/Standby, Cluster modes
Use Case: Migrating from on-premises message brokers
Protocols: JMS, AMQP, MQTT, STOMP, OpenWire
HA: Active/Standby (2 AZs) or Cluster (3 AZs)
Cost: Per broker-hour + storage, no per-message fees

Amazon MQ for Data Engineers

📨 Amazon MQ

Amazon MQ Architecture

ActiveMQ vs RabbitMQ Comparison

Amazon MQ Configuration

Message Patterns

Interview Questions & Answers

Q1: When should you use Amazon MQ vs. SQS/SNS?

Q2: What is the difference between ActiveMQ and RabbitMQ?

Q3: How does Amazon MQ handle high availability?

Q4: What are the cost factors for Amazon MQ?

Q5: How do you monitor Amazon MQ?

Summary