AWS IAM for Data Engineers

Users, Groups, Roles, Policies and Permissions - the foundation of secure data engineering on AWS.

10 min readIntermediate

Why This Matters

IAM is the single most critical security service in AWS. For data engineers, IAM governs who can read from S3, which Glue jobs can assume which roles, how cross-account data sharing works, and whether your data pipeline is compliant. Misconfigured IAM is the most common cause of security breaches in cloud data pipelines. A single overly permissive policy can expose an entire data lake containing sensitive customer data, financial records, or PII.

What is AWS IAM?

Identity and Access Management (IAM) controls who can do what to which resources. Every API call to AWS - from launching an EC2 instance to querying a Redshift cluster - is authenticated and authorized through IAM.

IAM is a global service - identities and policies are not tied to a single region. Resources that IAM protects (S3 buckets, Glue crawlers, Redshift clusters) are regional, but the permission rules live globally.

Core Components

Users: Individual identities (people or programmatic actors) that authenticate with AWS
Groups: Collections of users that share the same permission set
Roles: Temporary credentials assumed by trusted entities - users, AWS services, or external accounts
Policies: JSON documents that define explicit Allow or Deny statements on AWS actions and resources

IAM Users and Groups

IAM Users

An IAM User is an identity created for a specific person or application. Each user has a unique name, security credentials (password for console, access keys for programmatic access), and optionally an MFA device.

For data engineers, programmatic access is the norm. Access keys are used by the AWS CLI, Boto3 in Python, or SDK-based tools to interact with S3, Glue, EMR, and Redshift.

import boto3

iam = boto3.client('iam')

# Create an IAM user
response = iam.create_user(UserName='data-engineer-jane')
print(f"Created user: {response['User']['UserName']}")

# Create access keys for programmatic access
keys = iam.create_access_key(UserName='data-engineer-jane')
print(f"Access Key ID: {keys['AccessKey']['AccessKeyId']}")
print(f"Secret Access Key: {keys['AccessKey']['SecretAccessKey']}")

Users are a long-lived identity. Unlike temporary credentials from roles, access keys persist until manually rotated or deleted.

IAM Groups

Groups are containers for users. Instead of attaching policies to each individual user, attach policies to a group and add users to that group.

# Create a group for data engineers
iam.create_group(GroupName='DataEngineers')

# Attach a policy to the group
iam.attach_group_policy(
    GroupName='DataEngineers',
    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess'
)

# Add a user to the group
iam.add_user_to_group(
    GroupName='DataEngineers',
    UserName='data-engineer-jane'
)

Groups cannot be nested - a group cannot contain other groups.

Policy Types

Type	Scope	Managed by	Best For
AWS Managed	Pre-built by AWS	AWS	Common use cases, quick start
Customer Managed	Created and maintained by you	You	Custom requirements, reusable
Inline	Embedded directly in user/group/role	You	One-off, specific use cases

IAM Roles

Roles are the preferred way to grant permissions to AWS services. Unlike users, roles produce temporary security credentials valid for minutes to hours, automatically rotated.

Why Roles Matter for Data Engineers

A Glue job running on EMR does not have an IAM user with static keys. It assumes a role. That role grants exactly the permissions needed - nothing more.

EC2 Instance Roles: Attach to EC2 so any process can access AWS without hardcoded keys
Lambda Execution Roles: Every Lambda function must have a role for S3, DynamoDB access
Glue Service Roles: Crawlers and jobs assume roles to read S3, write to catalog
Cross-Account Roles: Allow Account A to access resources in Account B

Trust Policy

Every role has two policy documents:

Trust Policy (AssumeRolePolicyDocument) - Defines who can assume the role
Permission Policy - Defines what actions the role can perform

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

import boto3
import json

iam = boto3.client('iam')

trust_policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {"Service": "glue.amazonaws.com"},
        "Action": "sts:AssumeRole"
    }]
}

response = iam.create_role(
    RoleName='GlueJobRole',
    AssumeRolePolicyDocument=json.dumps(trust_policy),
    Description='Role for Glue ETL jobs'
)
print(f"Role ARN: {response['Role']['Arn']}")

Assume Role Flow

IAM Policies

A policy is a JSON document specifying the effect, actions, and resources that apply.

Policy Syntax

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ReadDataLake",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::company-data-lake",
        "arn:aws:s3:::company-data-lake/*"
      ]
    },
    {
      "Sid": "DenyDeleteFromProduction",
      "Effect": "Deny",
      "Action": ["s3:DeleteObject", "s3:DeleteBucket"],
      "Resource": ["arn:aws:s3:::company-data-lake-prod/*"],
      "Condition": {
        "StringNotEquals": {"aws:RequestedRegion": "us-east-1"}
      }
    }
  ]
}

Policy Elements

Effect: Allow or Deny. Explicit Deny always overrides any Allow
Action: The API operation(s) this policy applies to. Use wildcards: s3:*, glue:Create*
Resource: The ARN of the target resource. Use * for all resources
Condition: Optional constraints based on request context (IP, time, MFA, region)
Sid: Optional human-readable identifier

Resource ARN Patterns

Resource Type	ARN Pattern
S3 Bucket	`arn:aws:s3:::bucket-name`
S3 Object	`arn:aws:s3:::bucket-name/*`
Glue Table	`arn:aws:glue:region:account:table/db/table`
Glue Database	`arn:aws:glue:region:account:database/db`
Redshift Cluster	`arn:aws:redshift:region:account:cluster:name`
KMS Key	`arn:aws:kms:region:account:key/key-id`

IAM Best Practices for Data Engineers

1. Use Roles for Everything That Isn't Human

Never hardcode access keys in Lambda functions, Glue jobs, or EC2 instances. Roles provide temporary credentials that are automatically rotated.

# BAD: Hardcoded credentials
import os
os.environ['AWS_ACCESS_KEY_ID'] = 'AKIA...'
os.environ['AWS_SECRET_ACCESS_KEY'] = '...'

# GOOD: Use default credentials from role
import boto3
s3 = boto3.client('s3')  # Automatically uses instance/execution role

2. Enforce Least Privilege

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": [
    "arn:aws:s3:::data-lake/raw/*",
    "arn:aws:s3:::data-lake/processed/*"
  ]
}

3. Use Permission Boundaries

Permission boundaries set the maximum permissions an entity can have. They do not grant permissions - they limit them.

iam.put_user_permission_boundary(
    UserName='data-engineer-jane',
    PermissionBoundary='arn:aws:iam::123456789012:policy/MaxPermissionsBoundary'
)

4. Enable CloudTrail for IAM Activity

Monitor for unauthorized changes: role trust policy modifications, policy attachment events, access key creation, MFA deactivation.

5. Rotate Access Keys Regularly

For users requiring access keys, rotate every 90 days. Use AWS Secrets Manager for programmatic rotation.

6. Use IAM Access Analyzer

Identifies resources shared with external entities and generates policies based on CloudTrail logs.

Common IAM Patterns in Data Engineering

Pattern 1: Glue Job Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::data-lake/*", "arn:aws:s3:::data-lake"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "glue:CreateTable", "glue:UpdateTable",
        "glue:GetTable", "glue:GetDatabase",
        "glue:BatchCreatePartition"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Pattern 2: Cross-Account Data Sharing

import boto3

sts = boto3.client('sts')

# Assume role in Account B from Account A
response = sts.assume_role(
    RoleArn='arn:aws:iam::ACCOUNT_B:role/CrossAccountDataAccess',
    RoleSessionName='cross-account-session'
)

credentials = response['Credentials']
print(f"Access Key: {credentials['AccessKeyId']}")
print(f"Expires: {credentials['Expiration']}")

Pattern 3: S3 Bucket Policy with HTTPS Enforcement

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::data-lake",
    "arn:aws:s3:::data-lake/*"
  ],
  "Condition": {
    "Bool": {"aws:SecureTransport": "false"}
  }
}

Real-World Project Structure

Architecture Diagram

iam-configuration/
├── policies/
│   ├── glue-job-policy.json
│   ├── lambda-etl-policy.json
│   ├── redshift-load-policy.json
│   ├── s3-data-lake-policy.json
│   └── cross-account-policy.json
├── roles/
│   ├── glue-job-role.json
│   ├── lambda-execution-role.json
│   ├── ec2-instance-role.json
│   └── cross-account-role.json
├── groups/
│   ├── data-engineers.json
│   ├── data-analysts.json
│   └── devops.json
├── scripts/
│   ├── create-iam-resources.py
│   ├── rotate-access-keys.py
│   └── audit-iam-permissions.py
└── terraform/
    ├── iam-main.tf
    ├── iam-variables.tf
    └── iam-outputs.tf

Production Python Code

import boto3
import json
from botocore.exceptions import ClientError

class IAMManager:
    """Manages IAM resources for data engineering pipelines."""

    def __init__(self):
        self.iam = boto3.client('iam')

    def create_glue_role(self, role_name, s3_bucket_arns):
        """Create IAM role for Glue jobs with least-privilege permissions."""
        trust_policy = {
            "Version": "2012-10-17",
            "Statement": [{
                "Effect": "Allow",
                "Principal": {"Service": "glue.amazonaws.com"},
                "Action": "sts:AssumeRole"
            }]
        }

        try:
            role = self.iam.create_role(
                RoleName=role_name,
                AssumeRolePolicyDocument=json.dumps(trust_policy),
                Description='Role for Glue ETL jobs',
                MaxSessionDuration=3600
            )

            self.iam.attach_role_policy(
                RoleName=role_name,
                PolicyArn='arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole'
            )

            s3_resources = s3_bucket_arns + [arn + "/*" for arn in s3_bucket_arns]
            policy_doc = {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
                        "Resource": s3_resources
                    },
                    {
                        "Effect": "Allow",
                        "Action": [
                            "logs:CreateLogGroup",
                            "logs:CreateLogStream",
                            "logs:PutLogEvents"
                        ],
                        "Resource": "arn:aws:logs:*:*:*"
                    }
                ]
            }

            self.iam.put_role_policy(
                RoleName=role_name,
                PolicyName='GlueJobS3Access',
                PolicyDocument=json.dumps(policy_doc)
            )

            return role['Role']['Arn']
        except ClientError as e:
            print(f"Error creating Glue role: {e.response['Error']['Message']}")
            raise

    def create_cross_account_role(self, role_name, trusted_account_id, bucket_arns):
        """Create cross-account role for data sharing."""
        trust_policy = {
            "Version": "2012-10-17",
            "Statement": [{
                "Effect": "Allow",
                "Principal": {"AWS": f"arn:aws:iam::{trusted_account_id}:root"},
                "Action": "sts:AssumeRole",
                "Condition": {
                    "StringEquals": {
                        "sts:ExternalId": "data-sharing-external-id"
                    }
                }
            }]
        }

        try:
            role = self.iam.create_role(
                RoleName=role_name,
                AssumeRolePolicyDocument=json.dumps(trust_policy),
                Description=f'Cross-account role for account {trusted_account_id}'
            )

            s3_resources = bucket_arns + [arn + "/*" for arn in bucket_arns]
            policy_doc = {
                "Version": "2012-10-17",
                "Statement": [{
                    "Effect": "Allow",
                    "Action": ["s3:GetObject", "s3:ListBucket"],
                    "Resource": s3_resources
                }]
            }

            self.iam.put_role_policy(
                RoleName=role_name,
                PolicyName='CrossAccountS3Access',
                PolicyDocument=json.dumps(policy_doc)
            )

            return role['Role']['Arn']
        except ClientError as e:
            print(f"Error creating cross-account role: {e.response['Error']['Message']}")
            raise

    def audit_unused_permissions(self):
        """Identify unused permissions using IAM Access Analyzer."""
        analyzer = boto3.client('accessanalyzer')
        try:
            response = analyzer.list_findings(
                analyzerArn='arn:aws:access-analyzer:us-east-1:123456789012:analyzer/ConsoleAnalyzer-xxxx',
                filter={'findAll': {'eq': [{'value': 'ACTIVE'}]}}
            )
            return response.get('findings', [])
        except ClientError as e:
            print(f"Error auditing permissions: {e.response['Error']['Message']}")
            return []

if __name__ == '__main__':
    manager = IAMManager()

    role_arn = manager.create_glue_role(
        'GlueJobRole',
        ['arn:aws:s3:::my-data-lake']
    )
    print(f"Created Glue role: {role_arn}")

    cross_role = manager.create_cross_account_role(
        'CrossAccountDataAccess',
        '111122223333',
        ['arn:aws:s3:::shared-data-lake']
    )
    print(f"Created cross-account role: {cross_role}")

Mathematical Formulas

Blast Radius Calculation

def calculate_blast_radius(permission_level, resource_count, exposure_type):
    """
    Calculate security blast radius of IAM misconfiguration.

    Formula: Blast Radius = (Permission Level * Resource Count) / Total Resources
    """
    permission_multiplier = {
        'read': 0.3,
        'write': 0.6,
        'admin': 1.0
    }.get(permission_level, 1.0)

    exposure_multiplier = {
        'internal': 0.2,
        'cross_account': 0.6,
        'public': 1.0
    }.get(exposure_type, 1.0)

    blast_radius = permission_multiplier * exposure_multiplier * (resource_count / 100)
    return min(blast_radius, 1.0)

# Example: s3:* on 50 buckets exposed publicly
radius = calculate_blast_radius('admin', 50, 'public')
print(f"Blast Radius: {radius:.0%}")  # 100% - catastrophic

Credential Rotation Interval

def calculate_optimal_rotation_interval(risk_score, compliance_requirement_days=90):
    """
    Calculate optimal key rotation interval.

    Lower risk = longer interval (max 90 days)
    Higher risk = shorter interval (min 30 days)
    """
    base_interval = compliance_requirement_days
    risk_adjustment = (risk_score / 10) * 30

    optimal_days = max(30, int(base_interval - risk_adjustment))
    return optimal_days

# High-risk pipeline: rotate every 30 days
print(f"High risk rotation: {calculate_optimal_rotation_interval(9)} days")

# Low-risk pipeline: rotate every 90 days
print(f"Low risk rotation: {calculate_optimal_rotation_interval(2)} days")

Performance Considerations

Factor	Recommendation	Impact
Role Sessions	Set appropriate MaxSessionDuration	Prevents premature credential expiration
Policy Size	Keep under 6,144 characters	Larger policies increase evaluation latency
Group Count	Max 10 groups per user	Each group adds policy evaluation time
Wildcard Actions	Avoid `*` in production policies	Increases attack surface and slows evaluation
Cross-Account	Use ExternalId for trust	Prevents confused deputy attacks
Permission Boundaries	Apply in multi-tenant accounts	Limits maximum blast radius

Security Considerations

Enable MFA on all IAM user accounts
Never use root account for data engineering work
Use IAM roles instead of access keys for all service workloads
Implement least-privilege policies for all roles
Enable CloudTrail for IAM activity monitoring
Use IAM Access Analyzer to detect overly permissive policies
Rotate access keys every 90 days
Use permission boundaries in shared accounts
Monitor for unauthorized IAM changes using CloudWatch Events
Use AWS Organizations SCPs for multi-account governance

Common Pitfalls

Pitfall	Consequence	Prevention
Using `s3:*` for Glue jobs	Full access to all S3 buckets in account	Scope to specific buckets and prefixes
Hardcoding access keys in code	Permanent credential exposure risk	Use IAM roles and instance profiles
Not using MFA on users	Account compromise risk	Enable MFA on all human users
Creating inline policies	Difficult to manage and audit	Use customer managed policies instead
Not rotating access keys	Stolen keys remain valid indefinitely	Automate rotation with Secrets Manager
Overly permissive trust policies	Unauthorized role assumption	Restrict Principal and use ExternalId
Ignoring CloudTrail logs	Cannot detect unauthorized IAM changes	Enable and monitor IAM events
Not using permission boundaries	Privilege escalation in shared accounts	Set maximum permissions for all entities

Interview Questions & Answers

Q1: What is the difference between an IAM policy and a permission boundary?

Answer: An IAM policy grants permissions. A permission boundary sets the maximum permissions an entity can have - it does not grant anything. The effective permissions are the intersection of the policy attached to the entity and the permission boundary. If a user has S3FullAccess but a permission boundary of S3ReadOnly, the user can only read from S3.

Q2: Why should data engineers use IAM roles instead of access keys?

Answer: Access keys are long-lived credentials. If leaked (committed to git, stored in logs), they provide persistent access until manually revoked. IAM roles provide temporary credentials (valid minutes to hours) automatically rotated by STS. Roles eliminate the need to manage, rotate, and store secrets. Every AWS service (Lambda, Glue, EMR) has a native mechanism to assume a role.

Q3: How does IAM evaluate policies for a user in multiple groups?

Answer: IAM evaluates all policies attached to the user and all groups. All applicable Allow statements are collected. If any policy explicitly denies the action, the request is denied. If no explicit deny exists and at least one Allow matches, the request is allowed. Conditions on policies are also evaluated - a non-matching condition causes that statement to be ignored.

Q4: What is a cross-account IAM role and when would you use it?

Answer: A cross-account role is an IAM role in Account B whose trust policy allows a principal from Account A to assume it via sts:AssumeRole. In data engineering, this is used when a staging pipeline needs to read from a production data lake, or when a centralized analytics account needs data across multiple business unit accounts. The role grants temporary credentials scoped to Account B's resources.

Q5: How do you enforce all S3 access is encrypted in transit?

Answer: Use a bucket policy with the aws:SecureTransport condition. This evaluates to true only when the request uses HTTPS (TLS). Any HTTP request is denied regardless of IAM permissions. This is defense-in-depth - IAM controls identity-based access, while bucket policies enforce transport-level security.

Q6: What is the blast radius of giving a Glue job `s3:*` on all buckets?

Answer: The Glue job would have full access to every S3 bucket in the account, including sensitive data (PII, financial records, credentials). If the job is compromised or a developer makes an error, the job could delete, overwrite, or exfiltrate any object in any bucket. Scoped policies limit the blast radius to only authorized buckets.

Q7: How do you detect when someone creates a new IAM user?

Answer: AWS CloudTrail logs all IAM API calls. Use CloudWatch Events (EventBridge) to create a rule that triggers on CreateUser events from CloudTrail. This can send alerts via SNS, log to a security dashboard, or trigger a Lambda function for automated remediation.

Q8: What is the difference between IAM Identity Center and IAM?

Answer: IAM Identity Center (formerly AWS SSO) manages access to multiple AWS accounts and business applications with a single sign-on. IAM manages identities and permissions within a single AWS account. For data engineering with multiple accounts (dev, staging, prod), use Identity Center for centralized user management and IAM within each account for fine-grained resource permissions.

AWS IAM for Data Engineers

AWS IAM for Data Engineers

Why This Matters

What is AWS IAM?

Core Components

IAM Users and Groups

IAM Users

IAM Groups

Policy Types

IAM Roles

Why Roles Matter for Data Engineers

Trust Policy

Assume Role Flow

IAM Policies

Policy Syntax

Policy Elements

Resource ARN Patterns

IAM Best Practices for Data Engineers

1. Use Roles for Everything That Isn't Human

2. Enforce Least Privilege

3. Use Permission Boundaries

4. Enable CloudTrail for IAM Activity

5. Rotate Access Keys Regularly

6. Use IAM Access Analyzer

Common IAM Patterns in Data Engineering

Pattern 1: Glue Job Role

Pattern 2: Cross-Account Data Sharing

Pattern 3: S3 Bucket Policy with HTTPS Enforcement

Real-World Project Structure

Production Python Code

Mathematical Formulas

Blast Radius Calculation

Credential Rotation Interval

Performance Considerations

Security Considerations

Common Pitfalls

Interview Questions & Answers

Q1: What is the difference between an IAM policy and a permission boundary?

Q2: Why should data engineers use IAM roles instead of access keys?

Q3: How does IAM evaluate policies for a user in multiple groups?

Q4: What is a cross-account IAM role and when would you use it?

Q5: How do you enforce all S3 access is encrypted in transit?

Q6: What is the blast radius of giving a Glue job s3:* on all buckets?

Q7: How do you detect when someone creates a new IAM user?

Q8: What is the difference between IAM Identity Center and IAM?

Quiz

See Also

Premium Content

Q6: What is the blast radius of giving a Glue job `s3:*` on all buckets?