What is Amazon Managed Grafana?
Amazon Managed Grafana is a fully managed, secure data visualization service that makes it easy to create, query, and understand operational data from multiple data sources. It is based on the open-source Grafana project and provides enterprise features like SSO, collaboration, and data source management.
Core Concepts
| Concept | Description |
|---|---|
| Workspace | A logical instance of Grafana with its own configuration, users, and data sources |
| Data Source | A connection to a backend system that provides metrics, logs, or traces |
| Dashboard | A visual representation of data using panels and variables |
| Panel | A single visualization within a dashboard (graph, table, stat, etc.) |
| Alert | A rule that triggers notifications when conditions are met |
| Organization | A logical grouping of users, dashboards, and data sources |
| Variable | A dynamic parameter that allows dashboard filtering |
Amazon Managed Grafana Architecture
How Managed Grafana Works
Amazon Managed Grafana provisions a dedicated Grafana workspace with enterprise security features. Users authenticate via IAM Identity Center (SSO), and the workspace connects to AWS data sources using IAM roles. Grafana queries data sources, renders visualizations, and delivers alerts through configured notification channels.
Real-World Project Structure
managed-grafana-production/
āāā workspace/
ā āāā workspace-config.json
ā āāā iam-roles/
ā ā āāā grafana-service-role.json
ā ā āāā data-source-role.json
ā āāā vpc-config/
ā āāā endpoint-config.json
āāā data-sources/
ā āāā cloudwatch/
ā ā āāā metrics-datasource.json
ā ā āāā logs-datasource.json
ā āāā prometheus/
ā ā āāā prometheus-datasource.json
ā āāā opensearch/
ā ā āāā opensearch-datasource.json
ā āāā rds/
ā āāā rds-datasource.json
āāā dashboards/
ā āāā infrastructure/
ā ā āāā ec2-overview.json
ā ā āāā rds-performance.json
ā ā āāā network-health.json
ā āāā application/
ā ā āāā api-metrics.json
ā ā āāā error-tracking.json
ā ā āāā user-analytics.json
ā āāā data-pipeline/
ā ā āāā glue-job-monitoring.json
ā ā āāā kinesis-throughput.json
ā ā āāā redshift-query-perf.json
ā āāā business/
ā āāā revenue-dashboard.json
ā āāā customer-metrics.json
āāā alerting/
ā āāā notification-channels/
ā ā āāā sns-topic.yaml
ā ā āāā slack-webhook.yaml
ā ā āāā pagerduty.yaml
ā āāā alert-rules/
ā āāā infrastructure-alerts.json
ā āāā application-alerts.json
ā āāā pipeline-alerts.json
āāā provisioning/
ā āāā dashboards/
ā ā āāā auto-provisioned.json
ā ā āāā folder-structure.json
ā āāā datasources/
ā āāā auto-datasources.json
āāā monitoring/
āāā cloudwatch-alarms.yaml
āāā dashboards/
āāā grafana-health.json
Production Python Code
import boto3
import json
import logging
import time
from datetime import datetime, timedelta
from typing import Dict, List, Optional
logger = logging.getLogger(__name__)
class ManagedGrafanaManager:
"""Production-grade Amazon Managed Grafana manager."""
def __init__(self, region: str = 'us-east-1'):
self.client = boto3.client('grafana', region_name=region)
self.workspace_id = None
def set_workspace(self, workspace_id: str):
"""Set the workspace ID for operations."""
self.workspace_id = workspace_id
def create_workspace(
self,
name: str,
account_access_type: str = 'CURRENT_ACCOUNT',
authentication_providers: List[str] = ['AWS_SSO'],
workspace_role_arn: Optional[str] = None,
organization_role_name: Optional[str] = None,
workspace_data_sources: Optional[List[str]] = None,
workspace_notification_destinations: Optional[List[str]] = None
) -> Dict:
"""Create a Managed Grafana workspace."""
try:
kwargs = {
'AccountAccessType': account_access_type,
'AuthenticationProviders': authentication_providers,
'Name': name,
'WorkspaceDataSources': workspace_data_sources or [
'AMAZON_OPENSEARCH_SERVICE',
'CLOUDWATCH',
'PROMETHEUS',
'XRAY'
],
'WorkspaceNotificationDestinations': workspace_notification_destinations or [
'SNS'
],
'WorkspaceRoleArn': workspace_role_arn,
'Tags': {
'Environment': 'production',
'Service': 'observability'
}
}
response = self.client.create_workspace(**kwargs)
workspace_id = response['workspace']['id']
logger.info(f"Created workspace: {workspace_id}")
# Wait for workspace to be active
self._wait_for_workspace(workspace_id, 'ACTIVE')
return response['workspace']
except Exception as e:
logger.error(f"Failed to create workspace: {e}")
raise
def create_api_key(
self,
workspace_id: str,
key_name: str,
key_role: str = 'ADMIN',
seconds_to_live: int = 86400
) -> Dict:
"""Create an API key for programmatic access."""
try:
response = self.client.create_workspace_api_key(
KeyName=key_name,
KeyRole=key_role,
SecondsToLive=seconds_to_live,
WorkspaceId=workspace_id
)
logger.info(f"Created API key: {key_name}")
return {
'key': response['key'],
'keyName': key_name
}
except Exception as e:
logger.error(f"Failed to create API key: {e}")
raise
def create_role(
self,
workspace_id: str,
role: str,
users: List[str],
groups: Optional[List[str]] = None
) -> Dict:
"""Create a workspace role with user/group assignments."""
try:
response = self.client.create_workspace_role_mapping(
WorkspaceId=workspace_id,
RoleMapping={
'role': role,
'users': users,
'groups': groups or []
}
)
logger.info(f"Created role: {role} for workspace {workspace_id}")
return response
except Exception as e:
logger.error(f"Failed to create role: {e}")
raise
def describe_workspace(self, workspace_id: str) -> Dict:
"""Get workspace details and status."""
try:
response = self.client.describe_workspace(
WorkspaceId=workspace_id
)
return response['workspace']
except Exception as e:
logger.error(f"Failed to describe workspace: {e}")
raise
def list_workspaces(self) -> List[Dict]:
"""List all Grafana workspaces."""
try:
response = self.client.list_workspaces()
return response['workspaces']
except Exception as e:
logger.error(f"Failed to list workspaces: {e}")
raise
def update_workspace(
self,
workspace_id: str,
workspace_name: Optional[str] = None,
workspace_data_sources: Optional[List[str]] = None,
workspace_description: Optional[str] = None
) -> Dict:
"""Update workspace configuration."""
try:
kwargs = {'WorkspaceId': workspace_id}
if workspace_name:
kwargs['WorkspaceName'] = workspace_name
if workspace_data_sources:
kwargs['WorkspaceDataSources'] = workspace_data_sources
if workspace_description:
kwargs['WorkspaceDescription'] = workspace_description
response = self.client.update_workspace(**kwargs)
logger.info(f"Updated workspace: {workspace_id}")
return response['workspace']
except Exception as e:
logger.error(f"Failed to update workspace: {e}")
raise
def delete_workspace(self, workspace_id: str) -> None:
"""Delete a Grafana workspace."""
try:
self.client.delete_workspace(WorkspaceId=workspace_id)
logger.info(f"Deleted workspace: {workspace_id}")
self._wait_for_workspace(workspace_id, 'DELETED')
except Exception as e:
logger.error(f"Failed to delete workspace: {e}")
raise
def _wait_for_workspace(
self,
workspace_id: str,
target_state: str,
poll_interval: int = 30,
max_wait: int = 600
):
"""Wait for workspace to reach target state."""
start_time = time.time()
while time.time() - start_time < max_wait:
workspace = self.describe_workspace(workspace_id)
state = workspace.get('status', '')
if state == target_state:
logger.info(f"Workspace {workspace_id} is {target_state}")
return
elif state == 'FAILED':
raise RuntimeError(
f"Workspace {workspace_id} failed: "
f"{workspace.get('statusReason', 'Unknown')}"
)
logger.info(
f"Workspace {workspace_id} state: {state}, waiting..."
)
time.sleep(poll_interval)
raise TimeoutError(
f"Workspace {workspace_id} did not reach {target_state} "
f"within {max_wait}s"
)
class DashboardProvisioner:
"""Provision dashboards and data sources for Grafana."""
def __init__(self, workspace_url: str, api_key: str):
self.workspace_url = workspace_url
self.api_key = api_key
self.headers = {
'Authorization': f'Bearer {api_key}',
'Content-Type': 'application/json'
}
def import_dashboard(
self,
dashboard_json: Dict,
overwrite: bool = True
) -> Dict:
"""Import a dashboard from JSON definition."""
import requests
payload = {
'dashboard': dashboard_json,
'overwrite': overwrite,
'message': 'Imported via API'
}
try:
response = requests.post(
f'{self.workspace_url}/api/dashboards/db',
headers=self.headers,
json=payload,
timeout=30
)
response.raise_for_status()
result = response.json()
logger.info(
f"Imported dashboard: {result.get('slug', 'unknown')}"
)
return result
except requests.exceptions.RequestException as e:
logger.error(f"Failed to import dashboard: {e}")
raise
def provision_datasource(
self,
datasource_name: str,
datasource_type: str,
url: str,
access: str = 'proxy',
is_default: bool = False,
jsonData: Optional[Dict] = None
) -> Dict:
"""Provision a data source."""
import requests
payload = {
'name': datasource_name,
'type': datasource_type,
'url': url,
'access': access,
'isDefault': is_default,
'jsonData': jsonData or {}
}
try:
response = requests.post(
f'{self.workspace_url}/api/datasources',
headers=self.headers,
json=payload,
timeout=30
)
response.raise_for_status()
logger.info(f"Provisioned datasource: {datasource_name}")
return response.json()
except requests.exceptions.RequestException as e:
logger.error(f"Failed to provision datasource: {e}")
raise
# Production usage
if __name__ == '__main__':
grafana = ManagedGrafanaManager()
workspace = grafana.create_workspace(
name='production-observability',
workspace_data_sources=[
'CLOUDWATCH',
'PROMETHEUS',
'AMAZON_OPENSEARCH_SERVICE'
],
workspace_role_arn='arn:aws:iam::*:role/grafana-service-role'
)
print(f"Workspace ID: {workspace['id']}")
print(f"Endpoint: {workspace.get('endpoint', 'pending')}")
Production Bash Commands
#!/bin/bash
# Amazon Managed Grafana workspace management script
set -euo pipefail
WORKSPACE_ID="${1:-}"
REGION="${AWS_REGION:-us-east-1}"
ACTION="${2:-status}"
echo "=== Amazon Managed Grafana Management ==="
echo "Workspace: ${WORKSPACE_ID:-auto-detect}"
echo "Region: ${REGION}"
echo "Action: ${ACTION}"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
# List workspaces if no ID provided
if [ -z "${WORKSPACE_ID}" ]; then
echo ""
echo "=== All Grafana Workspaces ==="
aws grafana list-workspaces \
--region "${REGION}" \
--query 'workspaces[*].[id,name,status,endpoint]' \
--output table
exit 0
fi
# Get workspace details
WORKSPACE_INFO=$(aws grafana describe-workspace \
--workspace-id "${WORKSPACE_ID}" \
--region "${REGION}" \
--query 'workspace' \
--output json)
STATUS=$(echo "$WORKSPACE_INFO" | jq -r '.status')
ENDPOINT=$(echo "$WORKSPACE_INFO" | jq -r '.endpoint // "N/A"')
NAME=$(echo "$WORKSPACE_INFO" | jq -r '.name')
AUTH_PROVIDERS=$(echo "$WORKSPACE_INFO" | jq -r '.authenticationProviders[]' | tr '\n' ', ')
DATA_SOURCES=$(echo "$WORKSPACE_INFO" | jq -r '.workspaceDataSources[]' | tr '\n' ', ')
echo ""
echo "=== Workspace Details ==="
echo "Name: ${NAME}"
echo "Status: ${STATUS}"
echo "Endpoint: ${ENDPOINT}"
echo "Auth Providers: ${AUTH_PROVIDERS}"
echo "Data Sources: ${DATA_SOURCES}"
if [ "${ACTION}" = "status" ]; then
# List API keys
echo ""
echo "=== API Keys ==="
aws grafana list-workspace-api-keys \
--workspace-id "${WORKSPACE_ID}" \
--region "${REGION}" \
--query 'apiKeys[*].[keyName,keyRole,expiresAt]' \
--output table 2>/dev/null || echo "No API keys found"
# List role assignments
echo ""
echo "=== Role Mappings ==="
aws grafana describe-workspace-configuration \
--workspace-id "${WORKSPACE_ID}" \
--region "${REGION}" \
--query 'configuration' \
--output json 2>/dev/null || echo "No configuration found"
# Check CloudWatch metrics for workspace
echo ""
echo "=== Workspace Metrics (Last Hour) ==="
aws cloudwatch get-metric-statistics \
--namespace AWS/Grafana \
--metric-name WorkspaceEndpointHit \
--dimensions Name=WorkspaceId,Value="${WORKSPACE_ID}" \
--start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
--period 300 \
--statistics Sum \
--region "${REGION}" | jq -r '.Datapoints[] | "\(.Timestamp): \(.Sum) hits"'
elif [ "${ACTION}" = "create-api-key" ]; then
KEY_NAME="automation-key-$(date +%s)"
echo ""
echo "=== Creating API Key ==="
API_KEY=$(aws grafana create-workspace-api-key \
--workspace-id "${WORKSPACE_ID}" \
--key-name "${KEY_NAME}" \
--key-role "ADMIN" \
--seconds-to-live 86400 \
--region "${REGION}" \
--query 'key' \
--output text)
echo "API Key created: ${KEY_NAME}"
echo "Key: ${API_KEY}"
elif [ "${ACTION}" = "delete" ]; then
echo ""
echo "=== Deleting Workspace ==="
read -p "Are you sure you want to delete workspace ${WORKSPACE_ID}? (yes/no): " CONFIRM
if [ "${CONFIRM}" = "yes" ]; then
aws grafana delete-workspace \
--workspace-id "${WORKSPACE_ID}" \
--region "${REGION}"
echo "Workspace deletion initiated."
else
echo "Deletion cancelled."
fi
fi
echo ""
echo "Management operation complete."
Why This Matters
Amazon Managed Grafana provides enterprise-grade observability without the operational overhead of managing Grafana servers. It handles patching, backups, and scaling automatically while providing built-in security features like IAM integration, VPC deployment, and SSO. For data engineers, it serves as the visualization layer for monitoring data pipelines, query performance, and infrastructure health across AWS services.
Mathematical Formulas
Performance Considerations
| Factor | Recommendation | Impact |
|---|---|---|
| Panel Refresh Rate | Use 30s-60s for most dashboards | High - affects load and data freshness |
| Query Optimization | Use aggregated queries instead of raw data | High - reduces data source load |
| Dashboard Variables | Use cascading variables to reduce query scope | Medium - improves query performance |
| Caching | Enable data source caching for repeated queries | Medium - reduces backend load |
| Template Variables | Limit variable cardinality for fast rendering | Medium - improves dashboard load time |
| Row Collapsing | Collapse unused dashboard rows | Low - improves initial load |
| Panel Limits | Keep dashboards under 30 panels | Medium - affects rendering speed |
| Data Source Optimization | Use downsampled metrics for dashboards | High - reduces query complexity |
Security Considerations
- IAM Identity Center: Use SSO for centralized user management
- VPC Deployment: Deploy workspaces within VPC for network isolation
- Encryption at Rest: Enable KMS encryption for workspace data
- API Key Management: Rotate API keys regularly and use least privilege
- Data Source IAM: Use IAM roles for data source authentication
- Audit Logging: Enable CloudTrail for all Grafana API calls
- Role-Based Access: Implement RBAC with workspace roles
- Shared Dashboard Security: Restrict public dashboard access
Common Pitfalls
| Pitfall | Consequence | Solution |
|---|---|---|
| Too many panels per dashboard | Slow rendering and load times | Keep under 30 panels per dashboard |
| Unoptimized queries | High latency and backend load | Use aggregated and filtered queries |
| No caching enabled | Repeated queries hit data sources | Enable data source caching |
| Missing alert configuration | Undetected issues | Set up alerts for critical metrics |
| Ignoring API key rotation | Security vulnerabilities | Rotate keys quarterly |
| No VPC deployment | Network exposure | Deploy in VPC with private endpoints |
| Overly complex dashboards | Difficult to maintain | Simplify and use template variables |
| Disabling audit logs | No visibility into access | Enable CloudTrail logging |
Interview Questions & Answers
Q1: What is Amazon Managed Grafana and when would you use it?
Answer: Amazon Managed Grafana is a fully managed, secure data visualization service based on open-source Grafana. Use it for operational dashboards, monitoring infrastructure and applications, visualizing time-series metrics from CloudWatch/Prometheus, creating business analytics dashboards, and building real-time observability platforms. It provides enterprise features like SSO, RBAC, and data source management without server management.
Q2: What data sources does Managed Grafana support?
Answer: Managed Grafana supports native AWS data sources including CloudWatch (metrics and logs), Amazon Managed Service for Prometheus, Amazon OpenSearch Service, AWS X-Ray, and Amazon Timestream. It also supports third-party data sources through plugins including Prometheus, InfluxDB, Elasticsearch, and SQL databases. Data source connections use IAM roles for secure access.
Q3: How does Managed Grafana handle authentication and authorization?
Answer: Authentication uses IAM Identity Center (SSO) for user management, supporting SAML 2.0 and OIDC identity providers. Authorization uses workspace roles (ADMIN, VIEWER, EDITOR) mapped to users and groups. API keys provide programmatic access with role-based permissions. VPC deployment adds network-level security for private access.
Q4: What are the pricing considerations for Managed Grafana?
Answer: Managed Grafana charges based on instance hours (grafana-1h or grafana-2h). Costs depend on: (1) Instance size selected. (2) Number of hours running. (3) Dashboard rendering load. Use smaller instances for development, larger for production. Consider scaling down during off-hours to reduce costs.
Q5: How do you provision dashboards in Managed Grafana?
Answer: Dashboards can be provisioned using: (1) Terraform with the aws_grafana_workspace and grafana_dashboard resources. (2) Grafana API with API keys for programmatic import. (3) S3 bucket provisioning for bulk dashboard deployment. (4) Manual creation through the Grafana UI. Version control dashboard JSON for reproducibility.
Q6: What alerting capabilities does Managed Grafana provide?
Answer: Managed Grafana provides unified alerting with: (1) Threshold-based alerts on metrics. (2) Multi-condition alert rules. (3) Notification channels including SNS, email, Slack, and PagerDuty. (4) Alert grouping and silencing. (5) Contact point management. Alerts evaluate queries at configured intervals and trigger notifications when conditions are met.
Q7: How do you monitor Managed Grafana itself?
Answer: Monitor Managed Grafana using: (1) CloudWatch metrics for workspace endpoint hits, authentication failures, and API latency. (2) CloudTrail logs for all API operations. (3) Built-in Grafana health dashboards. (4) Workspace status checks for availability. (5) API key usage monitoring for security.
Q8: Can Managed Grafana be used for business analytics dashboards?
Answer: Yes, Managed Grafana supports business analytics through: (1) SQL data sources for relational databases. (2) Table panels for tabular data display. (3) Variable-based filtering for interactive exploration. (4) Annotations for marking business events. (5) Dashboard sharing for stakeholder collaboration. Use appropriate data sources and optimize queries for analytical workloads.