🔐 AWS IAM for Data Engineering

Master IAM users, groups, roles, policies, and service accounts essential for secure data engineering on AWS.

Module: AWS Data Engineering • Topic 2 of 65 • Premium Content

IAM Architecture Overview

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                        AWS IAM ARCHITECTURE                                   │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                    AWS ACCOUNT ROOT                                   │    │
│  │  (email@company.com) - Full access, use only for account setup      │    │
│  └─────────────────────────────┬───────────────────────────────────────┘    │
│                                │                                           │
│              ┌─────────────────┼─────────────────┐                         │
│              ▼                 ▼                 ▼                         │
│  ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐        │
│  │   IAM USERS       │ │   IAM GROUPS      │ │  IAM ROLES        │        │
│  │                   │ │                   │ │                   │        │
│  │ • data-engineer   │ │ • DataTeam        │ │ • GlueServiceRole │        │
│  │ • analytics-admin │ │ • Admins          │ │ • LambdaExecRole  │        │
│  │ • dev-readonly    │ │ • ReadOnly        │ │ • RedshiftRole    │        │
│  │ • ci-cd-pipeline  │ │ • SecurityTeam    │ │ • EMR_EC2_Role    │        │
│  └─────────┬─────────┘ └─────────┬─────────┘ └─────────┬─────────┘        │
│            │                     │                     │                   │
│            ▼                     ▼                     ▼                   │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                     IAM POLICIES                                     │    │
│  │                                                                     │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌───────────┐ │    │
│  │  │  Managed    │  │  Inline     │  │  Service    │  │  SCP      │ │    │
│  │  │  Policies   │  │  Policies   │  │  Policies   │  │  (Orgs)   │ │    │
│  │  │             │  │             │  │             │  │           │ │    │
│  │  │ AWS Managed │  │ Per-object  │  │ Built-in    │  │ Guardrails│ │    │
│  │  │ Custom      │  │ policies    │  │ service     │  │ Org-wide  │ │    │
│  │  │             │  │             │  │ roles       │  │           │ │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └───────────┘ │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                │                                           │
│                                ▼                                           │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                  AUTHENTICATION & AUTHORIZATION                      │    │
│  │                                                                     │    │
│  │  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐         │    │
│  │  │  MFA         │    │  STS         │    │  Identity    │         │    │
│  │  │  (Virtual/   │    │  (Temporary  │    │  Federation  │         │    │
│  │  │   Hardware)  │    │   Creds)     │    │  (SAML/OIDC) │         │    │
│  │  └──────────────┘    └──────────────┘    └──────────────┘         │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘

IAM Policy Document Structure

Basic Policy Document

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3BucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::data-lake-raw",
        "arn:aws:s3:::data-lake-raw/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "us-east-1"
        }
      }
    }
  ]
}

Complex Policy with Multiple Conditions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowGlueAccess",
      "Effect": "Allow",
      "Action": [
        "glue:CreateDatabase",
        "glue:CreateTable",
        "glue:GetDatabase",
        "glue:GetTable",
        "glue:GetPartitions",
        "glue:BatchCreatePartition"
      ],
      "Resource": [
        "arn:aws:glue:us-east-1:123456789012:database/*",
        "arn:aws:glue:us-east-1:123456789012:table/*",
        "arn:aws:glue:us-east-1:123456789012:crawler/*"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Environment": "production",
          "aws:RequestedRegion": "us-east-1"
        }
      }
    }
  ]
}

ℹ️

Pro Tip: Always use conditions to restrict access to specific regions, tags, or VPC endpoints. This prevents accidental cross-region data access and reduces attack surface.

IAM Roles for Data Engineering Services

Glue Service Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::data-lake-*",
        "arn:aws:s3:::data-lake-*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:CreateDatabase",
        "glue:CreateTable",
        "glue:GetDatabase",
        "glue:GetTable",
        "glue:GetPartitions",
        "glue:BatchCreatePartition",
        "glue:BatchDeletePartition"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:/aws-glue/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}

Lambda Execution Role for Data Processing

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::data-pipeline-*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:UpdateItem",
        "dynamodb:Query"
      ],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/pipeline-state"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:DescribeStream",
        "kinesis:ListStreams"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:123456789012:stream/data-stream"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Redshift Cluster Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::data-warehouse-*",
        "arn:aws:s3:::data-warehouse-*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabase",
        "glue:GetTable",
        "glue:GetPartitions"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sns:Publish"
      ],
      "Resource": "arn:aws:sns:us-east-1:123456789012:redshift-alerts"
    }
  ]
}

IAM for Cross-Account Data Sharing

AssumeRole Policy (Account A → Account B)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::ACCOUNT_B:role/CrossAccountDataAccess",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "unique-external-id-12345"
        }
      }
    }
  ]
}

Cross-Account S3 Bucket Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCrossAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::ACCOUNT_B:role/DataAnalyticsRole"
      },
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::shared-data-lake",
        "arn:aws:s3:::shared-data-lake/*"
      ],
      "Condition": {
        "StringEquals": {
          "s3:ExistingObjectTag/Classification": "internal"
        }
      }
    }
  ]
}

IAM for Lake Formation

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    LAKE FORMATION PERMISSION MODEL                           │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  Lake Formation Permissions                                          │    │
│  │                                                                     │    │
│  │  Database Level:    CREATE, ALTER, DROP                             │    │
│  │  Table Level:       CREATE, ALTER, DROP, SELECT, INSERT, DELETE     │    │
│  │  Column Level:      SELECT (specific columns)                       │    │
│  │  Row Level:         SELECT (with row-level filters)                 │    │
│  │  Cell Level:        SELECT (specific cells)                         │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐        │
│  │  IAM Policies   │    │ Lake Formation  │    │  Data Catalog   │        │
│  │  (AWS Resources)│    │ (Fine-grained)  │    │  (Metadata)     │        │
│  │                 │    │                 │    │                 │        │
│  │  s3:*, glue:*   │    │ Database/Table  │    │  Tables/Columns │        │
│  │  ec2:*          │    │ Row/Column      │    │  Partitions     │        │
│  │                 │    │ Cell-level      │    │                 │        │
│  └─────────────────┘    └─────────────────┘    └─────────────────┘        │
│                                                                             │
│  PERMISSION CHECK FLOW:                                                     │
│  1. User/Role → IAM Policy Check → Deny/Allow                              │
│  2. Allow → Lake Formation Permission Check → Deny/Allow                    │
│  3. Allow → Data Catalog Permission Check → Deny/Allow                      │
│  4. Allow → S3 Object Permission Check → Deny/Allow                         │
└─────────────────────────────────────────────────────────────────────────────┘

⚠️

Security Warning: Lake Formation permissions override IAM policies. Always check both layers when debugging access issues. A user might have IAM permission but not Lake Formation permission.

IAM Best Practices for Data Engineering

Principle of Least Privilege

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│              LEAST PRIVILEGE IMPLEMENTATION                                   │
│                                                                             │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │  ROLE                  │ IAM POLICIES           │ LF PERMISSIONS    │    │
│  ├─────────────────────────────────────────────────────────────────────┤    │
│  │  Data Engineer         │ S3, Glue, Lambda       │ All tables        │    │
│  │  Data Analyst          │ Athena, QuickSight     │ SELECT on gold/   │    │
│  │  Pipeline Operator     │ Step Functions, Glue   │ bronze/silver     │    │
│  │  Security Auditor      │ CloudTrail, Config     │ Read-only catalog │    │
│  │  DevOps Engineer       │ EC2, VPC, CloudWatch   │ No data access    │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                                                             │
│  IMPLEMENTATION:                                                            │
│  1. Start with deny-all policy                                              │
│  2. Add permissions incrementally                                           │
│  3. Use condition keys to limit scope                                       │
│  4. Review and audit quarterly                                              │
└─────────────────────────────────────────────────────────────────────────────┘

IAM Policy Evaluation Logic

Architecture Diagram

┌─────────────────────────────────────────────────────────────────────────────┐
│                    IAM POLICY EVALUATION FLOW                                │
│                                                                             │
│  Request ──► Default Deny ──► Explicit Deny? ──► YES ──► DENIED            │
│                    │                                                   │    │
│                    NO                                                  │    │
│                    │                                                   │    │
│                    ▼                                                   │    │
│  Service Control Policy (SCP) ──► Deny? ──► YES ──► DENIED               │
│                    │                                                   │    │
│                    NO                                                  │    │
│                    │                                                   │    │
│                    ▼                                                   │    │
│  Permission Boundary ──► Deny? ──► YES ──► DENIED                        │
│                    │                                                   │    │
│                    NO                                                  │    │
│                    │                                                   │    │
│                    ▼                                                   │    │
│  Resource-based Policy ──► Deny? ──► YES ──► DENIED                     │
│                    │                                                   │    │
│                    NO                                                  │    │
│                    │                                                   │    │
│                    ▼                                                   │    │
│  Identity-based Policy ──► Allow? ──► NO ──► DENIED                     │
│                    │                                                   │    │
│                    YES                                                 │    │
│                    │                                                   │    │
│                    ▼                                                   │    │
│              ┌─────────┐                                               │    │
│              │ ALLOWED │                                               │    │
│              └─────────┘                                               │    │
└─────────────────────────────────────────────────────────────────────────────┘

IAM for Service Accounts (IRSA on EKS)

Creating IRSA for EKS Workloads

# Create IAM role for service account
eksctl create iamserviceaccount \
  --name glue-service-account \
  --namespace data-engineering \
  --cluster my-cluster \
  --role-name GlueEKSRole \
  --attach-policy-arn arn:aws:iam::123456789012:policy/GlueAccessPolicy \
  --approve

# Verify the role
aws iam get-role --role-name GlueEKSRole

Trust Policy for IRSA

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:sub": "system:serviceaccount:data-engineering:glue-service-account",
          "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

IAM Monitoring and Auditing

CloudTrail for IAM Events

{
  "Source": ["iam.amazonaws.com"],
  "DetailType": ["AWS API Call via CloudTrail"],
  "Detail": {
    "eventSource": ["iam.amazonaws.com"],
    "eventName": [
      "CreateUser",
      "DeleteUser",
      "CreateAccessKey",
      "DeleteAccessKey",
      "AttachUserPolicy",
      "DetachUserPolicy",
      "PutRolePolicy",
      "DeleteRolePolicy"
    ]
  }
}

IAM Access Analyzer Findings

Finding Type	Severity	Action Required
Public Access	Critical	Restrict immediately
Cross-Account	Medium	Review and validate
Unused Permissions	Low	Remove if not needed
Overly Permissive	High	Apply least privilege
MFA Not Enabled	High	Enable MFA

Cost Considerations

ℹ️

IAM is Free: AWS IAM is provided at no additional charge. You only pay for the AWS services that IAM controls access to. However, consider these cost implications:

Overly permissive roles can lead to accidental resource usage
Cross-account access may incur data transfer costs
STS calls are free but have rate limits

Summary

IAM is the foundation of AWS security for data engineering. Key takeaways:

Roles over Users: Always prefer IAM roles for services and applications
Least Privilege: Grant minimum necessary permissions
Cross-Account: Use STS AssumeRole with ExternalId
Lake Formation: Additional layer for fine-grained data access
MFA: Enable for all human users
Monitoring: Use CloudTrail and Access Analyzer
Tagging: Use resource tags for policy conditions

AWS IAM for Data Engineers

🔐 AWS IAM for Data Engineering

IAM Architecture Overview

IAM Policy Document Structure

Basic Policy Document

Complex Policy with Multiple Conditions

IAM Roles for Data Engineering Services

Glue Service Role

Lambda Execution Role for Data Processing

Redshift Cluster Role

IAM for Cross-Account Data Sharing

AssumeRole Policy (Account A → Account B)

Cross-Account S3 Bucket Policy

IAM for Lake Formation

IAM Best Practices for Data Engineering

Principle of Least Privilege

IAM Policy Evaluation Logic

IAM for Service Accounts (IRSA on EKS)

Creating IRSA for EKS Workloads

Trust Policy for IRSA

IAM Monitoring and Auditing

CloudTrail for IAM Events

IAM Access Analyzer Findings

Cost Considerations

Summary