šŸŽ‰ 75% of content is free forever — Unlock Premium from $10/mo →
CW
šŸ’¼ Servicesā„¹ļø Aboutāœ‰ļø ContactView Pricing Plansfrom $10

Amazon Managed Grafana for Data Engineers

AWS Data EngineeringGrafana Observability & Analytics⭐ Premium

Advertisement

Amazon Managed Grafana

Master dashboards, alerting, and real-time observability with managed Grafana on AWS.

20 min readIntermediate

What is Amazon Managed Grafana?

Amazon Managed Grafana is a fully managed, secure data visualization service that makes it easy to create, query, and understand operational data from multiple data sources. It is based on the open-source Grafana project and provides enterprise features like SSO, collaboration, and data source management.

Core Concepts

ConceptDescription
WorkspaceA logical instance of Grafana with its own configuration, users, and data sources
Data SourceA connection to a backend system that provides metrics, logs, or traces
DashboardA visual representation of data using panels and variables
PanelA single visualization within a dashboard (graph, table, stat, etc.)
AlertA rule that triggers notifications when conditions are met
OrganizationA logical grouping of users, dashboards, and data sources
VariableA dynamic parameter that allows dashboard filtering

Amazon Managed Grafana Architecture

Amazon Managed Grafana ArchitectureUsers & SSOIAM Identity CenterAPI EndpointHTTPS EndpointGrafana WorkspaceManaged Grafana EngineEnterprise Plugins & FeaturesCloudWatchMetrics & LogsData Source ConnectionsCloudWatchMetricsPrometheusManaged ServiceOpenSearchLogs & AnalyticsX-RayDistributed TracingRDS / AuroraSQL MetricsDashboard VisualizationsGraph PanelTime Series ChartsStat PanelSingle Values & KPIsTable PanelTabular Data ViewHeatmap PanelDistribution VisualsAlert RulesThreshold NotificationsAlert NotificationsSNS | Email | Slack | PagerDutyCollaborationShared Dashboards | AnnotationsSecurityIAM | VPC | KMS | Audit LogsPricinggrafana-1h | grafana-2h

How Managed Grafana Works

Amazon Managed Grafana provisions a dedicated Grafana workspace with enterprise security features. Users authenticate via IAM Identity Center (SSO), and the workspace connects to AWS data sources using IAM roles. Grafana queries data sources, renders visualizations, and delivers alerts through configured notification channels.

Real-World Project Structure

Architecture Diagram
managed-grafana-production/
ā”œā”€ā”€ workspace/
│   ā”œā”€ā”€ workspace-config.json
│   ā”œā”€ā”€ iam-roles/
│   │   ā”œā”€ā”€ grafana-service-role.json
│   │   └── data-source-role.json
│   └── vpc-config/
│       └── endpoint-config.json
ā”œā”€ā”€ data-sources/
│   ā”œā”€ā”€ cloudwatch/
│   │   ā”œā”€ā”€ metrics-datasource.json
│   │   └── logs-datasource.json
│   ā”œā”€ā”€ prometheus/
│   │   └── prometheus-datasource.json
│   ā”œā”€ā”€ opensearch/
│   │   └── opensearch-datasource.json
│   └── rds/
│       └── rds-datasource.json
ā”œā”€ā”€ dashboards/
│   ā”œā”€ā”€ infrastructure/
│   │   ā”œā”€ā”€ ec2-overview.json
│   │   ā”œā”€ā”€ rds-performance.json
│   │   └── network-health.json
│   ā”œā”€ā”€ application/
│   │   ā”œā”€ā”€ api-metrics.json
│   │   ā”œā”€ā”€ error-tracking.json
│   │   └── user-analytics.json
│   ā”œā”€ā”€ data-pipeline/
│   │   ā”œā”€ā”€ glue-job-monitoring.json
│   │   ā”œā”€ā”€ kinesis-throughput.json
│   │   └── redshift-query-perf.json
│   └── business/
│       ā”œā”€ā”€ revenue-dashboard.json
│       └── customer-metrics.json
ā”œā”€ā”€ alerting/
│   ā”œā”€ā”€ notification-channels/
│   │   ā”œā”€ā”€ sns-topic.yaml
│   │   ā”œā”€ā”€ slack-webhook.yaml
│   │   └── pagerduty.yaml
│   └── alert-rules/
│       ā”œā”€ā”€ infrastructure-alerts.json
│       ā”œā”€ā”€ application-alerts.json
│       └── pipeline-alerts.json
ā”œā”€ā”€ provisioning/
│   ā”œā”€ā”€ dashboards/
│   │   ā”œā”€ā”€ auto-provisioned.json
│   │   └── folder-structure.json
│   └── datasources/
│       └── auto-datasources.json
└── monitoring/
    ā”œā”€ā”€ cloudwatch-alarms.yaml
    └── dashboards/
        └── grafana-health.json

Production Python Code

import boto3
import json
import logging
import time
from datetime import datetime, timedelta
from typing import Dict, List, Optional

logger = logging.getLogger(__name__)

class ManagedGrafanaManager:
    """Production-grade Amazon Managed Grafana manager."""

    def __init__(self, region: str = 'us-east-1'):
        self.client = boto3.client('grafana', region_name=region)
        self.workspace_id = None

    def set_workspace(self, workspace_id: str):
        """Set the workspace ID for operations."""
        self.workspace_id = workspace_id

    def create_workspace(
        self,
        name: str,
        account_access_type: str = 'CURRENT_ACCOUNT',
        authentication_providers: List[str] = ['AWS_SSO'],
        workspace_role_arn: Optional[str] = None,
        organization_role_name: Optional[str] = None,
        workspace_data_sources: Optional[List[str]] = None,
        workspace_notification_destinations: Optional[List[str]] = None
    ) -> Dict:
        """Create a Managed Grafana workspace."""
        try:
            kwargs = {
                'AccountAccessType': account_access_type,
                'AuthenticationProviders': authentication_providers,
                'Name': name,
                'WorkspaceDataSources': workspace_data_sources or [
                    'AMAZON_OPENSEARCH_SERVICE',
                    'CLOUDWATCH',
                    'PROMETHEUS',
                    'XRAY'
                ],
                'WorkspaceNotificationDestinations': workspace_notification_destinations or [
                    'SNS'
                ],
                'WorkspaceRoleArn': workspace_role_arn,
                'Tags': {
                    'Environment': 'production',
                    'Service': 'observability'
                }
            }

            response = self.client.create_workspace(**kwargs)
            workspace_id = response['workspace']['id']
            logger.info(f"Created workspace: {workspace_id}")

            # Wait for workspace to be active
            self._wait_for_workspace(workspace_id, 'ACTIVE')
            return response['workspace']
        except Exception as e:
            logger.error(f"Failed to create workspace: {e}")
            raise

    def create_api_key(
        self,
        workspace_id: str,
        key_name: str,
        key_role: str = 'ADMIN',
        seconds_to_live: int = 86400
    ) -> Dict:
        """Create an API key for programmatic access."""
        try:
            response = self.client.create_workspace_api_key(
                KeyName=key_name,
                KeyRole=key_role,
                SecondsToLive=seconds_to_live,
                WorkspaceId=workspace_id
            )
            logger.info(f"Created API key: {key_name}")
            return {
                'key': response['key'],
                'keyName': key_name
            }
        except Exception as e:
            logger.error(f"Failed to create API key: {e}")
            raise

    def create_role(
        self,
        workspace_id: str,
        role: str,
        users: List[str],
        groups: Optional[List[str]] = None
    ) -> Dict:
        """Create a workspace role with user/group assignments."""
        try:
            response = self.client.create_workspace_role_mapping(
                WorkspaceId=workspace_id,
                RoleMapping={
                    'role': role,
                    'users': users,
                    'groups': groups or []
                }
            )
            logger.info(f"Created role: {role} for workspace {workspace_id}")
            return response
        except Exception as e:
            logger.error(f"Failed to create role: {e}")
            raise

    def describe_workspace(self, workspace_id: str) -> Dict:
        """Get workspace details and status."""
        try:
            response = self.client.describe_workspace(
                WorkspaceId=workspace_id
            )
            return response['workspace']
        except Exception as e:
            logger.error(f"Failed to describe workspace: {e}")
            raise

    def list_workspaces(self) -> List[Dict]:
        """List all Grafana workspaces."""
        try:
            response = self.client.list_workspaces()
            return response['workspaces']
        except Exception as e:
            logger.error(f"Failed to list workspaces: {e}")
            raise

    def update_workspace(
        self,
        workspace_id: str,
        workspace_name: Optional[str] = None,
        workspace_data_sources: Optional[List[str]] = None,
        workspace_description: Optional[str] = None
    ) -> Dict:
        """Update workspace configuration."""
        try:
            kwargs = {'WorkspaceId': workspace_id}
            if workspace_name:
                kwargs['WorkspaceName'] = workspace_name
            if workspace_data_sources:
                kwargs['WorkspaceDataSources'] = workspace_data_sources
            if workspace_description:
                kwargs['WorkspaceDescription'] = workspace_description

            response = self.client.update_workspace(**kwargs)
            logger.info(f"Updated workspace: {workspace_id}")
            return response['workspace']
        except Exception as e:
            logger.error(f"Failed to update workspace: {e}")
            raise

    def delete_workspace(self, workspace_id: str) -> None:
        """Delete a Grafana workspace."""
        try:
            self.client.delete_workspace(WorkspaceId=workspace_id)
            logger.info(f"Deleted workspace: {workspace_id}")
            self._wait_for_workspace(workspace_id, 'DELETED')
        except Exception as e:
            logger.error(f"Failed to delete workspace: {e}")
            raise

    def _wait_for_workspace(
        self,
        workspace_id: str,
        target_state: str,
        poll_interval: int = 30,
        max_wait: int = 600
    ):
        """Wait for workspace to reach target state."""
        start_time = time.time()
        while time.time() - start_time < max_wait:
            workspace = self.describe_workspace(workspace_id)
            state = workspace.get('status', '')
            if state == target_state:
                logger.info(f"Workspace {workspace_id} is {target_state}")
                return
            elif state == 'FAILED':
                raise RuntimeError(
                    f"Workspace {workspace_id} failed: "
                    f"{workspace.get('statusReason', 'Unknown')}"
                )
            logger.info(
                f"Workspace {workspace_id} state: {state}, waiting..."
            )
            time.sleep(poll_interval)

        raise TimeoutError(
            f"Workspace {workspace_id} did not reach {target_state} "
            f"within {max_wait}s"
        )


class DashboardProvisioner:
    """Provision dashboards and data sources for Grafana."""

    def __init__(self, workspace_url: str, api_key: str):
        self.workspace_url = workspace_url
        self.api_key = api_key
        self.headers = {
            'Authorization': f'Bearer {api_key}',
            'Content-Type': 'application/json'
        }

    def import_dashboard(
        self,
        dashboard_json: Dict,
        overwrite: bool = True
    ) -> Dict:
        """Import a dashboard from JSON definition."""
        import requests

        payload = {
            'dashboard': dashboard_json,
            'overwrite': overwrite,
            'message': 'Imported via API'
        }

        try:
            response = requests.post(
                f'{self.workspace_url}/api/dashboards/db',
                headers=self.headers,
                json=payload,
                timeout=30
            )
            response.raise_for_status()
            result = response.json()
            logger.info(
                f"Imported dashboard: {result.get('slug', 'unknown')}"
            )
            return result
        except requests.exceptions.RequestException as e:
            logger.error(f"Failed to import dashboard: {e}")
            raise

    def provision_datasource(
        self,
        datasource_name: str,
        datasource_type: str,
        url: str,
        access: str = 'proxy',
        is_default: bool = False,
        jsonData: Optional[Dict] = None
    ) -> Dict:
        """Provision a data source."""
        import requests

        payload = {
            'name': datasource_name,
            'type': datasource_type,
            'url': url,
            'access': access,
            'isDefault': is_default,
            'jsonData': jsonData or {}
        }

        try:
            response = requests.post(
                f'{self.workspace_url}/api/datasources',
                headers=self.headers,
                json=payload,
                timeout=30
            )
            response.raise_for_status()
            logger.info(f"Provisioned datasource: {datasource_name}")
            return response.json()
        except requests.exceptions.RequestException as e:
            logger.error(f"Failed to provision datasource: {e}")
            raise


# Production usage
if __name__ == '__main__':
    grafana = ManagedGrafanaManager()

    workspace = grafana.create_workspace(
        name='production-observability',
        workspace_data_sources=[
            'CLOUDWATCH',
            'PROMETHEUS',
            'AMAZON_OPENSEARCH_SERVICE'
        ],
        workspace_role_arn='arn:aws:iam::*:role/grafana-service-role'
    )

    print(f"Workspace ID: {workspace['id']}")
    print(f"Endpoint: {workspace.get('endpoint', 'pending')}")

Production Bash Commands

#!/bin/bash
# Amazon Managed Grafana workspace management script

set -euo pipefail

WORKSPACE_ID="${1:-}"
REGION="${AWS_REGION:-us-east-1}"
ACTION="${2:-status}"

echo "=== Amazon Managed Grafana Management ==="
echo "Workspace: ${WORKSPACE_ID:-auto-detect}"
echo "Region: ${REGION}"
echo "Action: ${ACTION}"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# List workspaces if no ID provided
if [ -z "${WORKSPACE_ID}" ]; then
  echo ""
  echo "=== All Grafana Workspaces ==="
  aws grafana list-workspaces \
    --region "${REGION}" \
    --query 'workspaces[*].[id,name,status,endpoint]' \
    --output table
  exit 0
fi

# Get workspace details
WORKSPACE_INFO=$(aws grafana describe-workspace \
  --workspace-id "${WORKSPACE_ID}" \
  --region "${REGION}" \
  --query 'workspace' \
  --output json)

STATUS=$(echo "$WORKSPACE_INFO" | jq -r '.status')
ENDPOINT=$(echo "$WORKSPACE_INFO" | jq -r '.endpoint // "N/A"')
NAME=$(echo "$WORKSPACE_INFO" | jq -r '.name')
AUTH_PROVIDERS=$(echo "$WORKSPACE_INFO" | jq -r '.authenticationProviders[]' | tr '\n' ', ')
DATA_SOURCES=$(echo "$WORKSPACE_INFO" | jq -r '.workspaceDataSources[]' | tr '\n' ', ')

echo ""
echo "=== Workspace Details ==="
echo "Name: ${NAME}"
echo "Status: ${STATUS}"
echo "Endpoint: ${ENDPOINT}"
echo "Auth Providers: ${AUTH_PROVIDERS}"
echo "Data Sources: ${DATA_SOURCES}"

if [ "${ACTION}" = "status" ]; then
  # List API keys
  echo ""
  echo "=== API Keys ==="
  aws grafana list-workspace-api-keys \
    --workspace-id "${WORKSPACE_ID}" \
    --region "${REGION}" \
    --query 'apiKeys[*].[keyName,keyRole,expiresAt]' \
    --output table 2>/dev/null || echo "No API keys found"

  # List role assignments
  echo ""
  echo "=== Role Mappings ==="
  aws grafana describe-workspace-configuration \
    --workspace-id "${WORKSPACE_ID}" \
    --region "${REGION}" \
    --query 'configuration' \
    --output json 2>/dev/null || echo "No configuration found"

  # Check CloudWatch metrics for workspace
  echo ""
  echo "=== Workspace Metrics (Last Hour) ==="
  aws cloudwatch get-metric-statistics \
    --namespace AWS/Grafana \
    --metric-name WorkspaceEndpointHit \
    --dimensions Name=WorkspaceId,Value="${WORKSPACE_ID}" \
    --start-time "$(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ)" \
    --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
    --period 300 \
    --statistics Sum \
    --region "${REGION}" | jq -r '.Datapoints[] | "\(.Timestamp): \(.Sum) hits"'

elif [ "${ACTION}" = "create-api-key" ]; then
  KEY_NAME="automation-key-$(date +%s)"
  echo ""
  echo "=== Creating API Key ==="
  API_KEY=$(aws grafana create-workspace-api-key \
    --workspace-id "${WORKSPACE_ID}" \
    --key-name "${KEY_NAME}" \
    --key-role "ADMIN" \
    --seconds-to-live 86400 \
    --region "${REGION}" \
    --query 'key' \
    --output text)
  echo "API Key created: ${KEY_NAME}"
  echo "Key: ${API_KEY}"

elif [ "${ACTION}" = "delete" ]; then
  echo ""
  echo "=== Deleting Workspace ==="
  read -p "Are you sure you want to delete workspace ${WORKSPACE_ID}? (yes/no): " CONFIRM
  if [ "${CONFIRM}" = "yes" ]; then
    aws grafana delete-workspace \
      --workspace-id "${WORKSPACE_ID}" \
      --region "${REGION}"
    echo "Workspace deletion initiated."
  else
    echo "Deletion cancelled."
  fi
fi

echo ""
echo "Management operation complete."

Why This Matters

Amazon Managed Grafana provides enterprise-grade observability without the operational overhead of managing Grafana servers. It handles patching, backups, and scaling automatically while providing built-in security features like IAM integration, VPC deployment, and SSO. For data engineers, it serves as the visualization layer for monitoring data pipelines, query performance, and infrastructure health across AWS services.

Mathematical Formulas

Performance Considerations

FactorRecommendationImpact
Panel Refresh RateUse 30s-60s for most dashboardsHigh - affects load and data freshness
Query OptimizationUse aggregated queries instead of raw dataHigh - reduces data source load
Dashboard VariablesUse cascading variables to reduce query scopeMedium - improves query performance
CachingEnable data source caching for repeated queriesMedium - reduces backend load
Template VariablesLimit variable cardinality for fast renderingMedium - improves dashboard load time
Row CollapsingCollapse unused dashboard rowsLow - improves initial load
Panel LimitsKeep dashboards under 30 panelsMedium - affects rendering speed
Data Source OptimizationUse downsampled metrics for dashboardsHigh - reduces query complexity

Security Considerations

  • IAM Identity Center: Use SSO for centralized user management
  • VPC Deployment: Deploy workspaces within VPC for network isolation
  • Encryption at Rest: Enable KMS encryption for workspace data
  • API Key Management: Rotate API keys regularly and use least privilege
  • Data Source IAM: Use IAM roles for data source authentication
  • Audit Logging: Enable CloudTrail for all Grafana API calls
  • Role-Based Access: Implement RBAC with workspace roles
  • Shared Dashboard Security: Restrict public dashboard access

Common Pitfalls

PitfallConsequenceSolution
Too many panels per dashboardSlow rendering and load timesKeep under 30 panels per dashboard
Unoptimized queriesHigh latency and backend loadUse aggregated and filtered queries
No caching enabledRepeated queries hit data sourcesEnable data source caching
Missing alert configurationUndetected issuesSet up alerts for critical metrics
Ignoring API key rotationSecurity vulnerabilitiesRotate keys quarterly
No VPC deploymentNetwork exposureDeploy in VPC with private endpoints
Overly complex dashboardsDifficult to maintainSimplify and use template variables
Disabling audit logsNo visibility into accessEnable CloudTrail logging

Interview Questions & Answers

Q1: What is Amazon Managed Grafana and when would you use it?

Answer: Amazon Managed Grafana is a fully managed, secure data visualization service based on open-source Grafana. Use it for operational dashboards, monitoring infrastructure and applications, visualizing time-series metrics from CloudWatch/Prometheus, creating business analytics dashboards, and building real-time observability platforms. It provides enterprise features like SSO, RBAC, and data source management without server management.

Q2: What data sources does Managed Grafana support?

Answer: Managed Grafana supports native AWS data sources including CloudWatch (metrics and logs), Amazon Managed Service for Prometheus, Amazon OpenSearch Service, AWS X-Ray, and Amazon Timestream. It also supports third-party data sources through plugins including Prometheus, InfluxDB, Elasticsearch, and SQL databases. Data source connections use IAM roles for secure access.

Q3: How does Managed Grafana handle authentication and authorization?

Answer: Authentication uses IAM Identity Center (SSO) for user management, supporting SAML 2.0 and OIDC identity providers. Authorization uses workspace roles (ADMIN, VIEWER, EDITOR) mapped to users and groups. API keys provide programmatic access with role-based permissions. VPC deployment adds network-level security for private access.

Q4: What are the pricing considerations for Managed Grafana?

Answer: Managed Grafana charges based on instance hours (grafana-1h or grafana-2h). Costs depend on: (1) Instance size selected. (2) Number of hours running. (3) Dashboard rendering load. Use smaller instances for development, larger for production. Consider scaling down during off-hours to reduce costs.

Q5: How do you provision dashboards in Managed Grafana?

Answer: Dashboards can be provisioned using: (1) Terraform with the aws_grafana_workspace and grafana_dashboard resources. (2) Grafana API with API keys for programmatic import. (3) S3 bucket provisioning for bulk dashboard deployment. (4) Manual creation through the Grafana UI. Version control dashboard JSON for reproducibility.

Q6: What alerting capabilities does Managed Grafana provide?

Answer: Managed Grafana provides unified alerting with: (1) Threshold-based alerts on metrics. (2) Multi-condition alert rules. (3) Notification channels including SNS, email, Slack, and PagerDuty. (4) Alert grouping and silencing. (5) Contact point management. Alerts evaluate queries at configured intervals and trigger notifications when conditions are met.

Q7: How do you monitor Managed Grafana itself?

Answer: Monitor Managed Grafana using: (1) CloudWatch metrics for workspace endpoint hits, authentication failures, and API latency. (2) CloudTrail logs for all API operations. (3) Built-in Grafana health dashboards. (4) Workspace status checks for availability. (5) API key usage monitoring for security.

Q8: Can Managed Grafana be used for business analytics dashboards?

Answer: Yes, Managed Grafana supports business analytics through: (1) SQL data sources for relational databases. (2) Table panels for tabular data display. (3) Variable-based filtering for interactive exploration. (4) Annotations for marking business events. (5) Dashboard sharing for stakeholder collaboration. Use appropriate data sources and optimize queries for analytical workloads.

QuizBox

See Also

šŸ”’

Premium Content

Amazon Managed Grafana for Data Engineers

You've previewed the first section. Unlock this full lesson and 900+ advanced tutorials with a Premium plan.

šŸŽÆEnd-to-end Projects
šŸ’¼Interview Prep
šŸ“œCertificates
šŸ¤Community Access

Already a member? Log in

Advertisement