Why This Matters
AWS Lake Formation is the foundational governance layer for data lakes on AWS, providing fine-grained access control, data sharing, and audit capabilities. It builds on top of AWS Glue Data Catalog to deliver enterprise-grade security and compliance for data lake architectures.
For data engineers, Lake Formation is critical because it enables self-service analytics while maintaining strict governance controls. Understanding Lake Formation's permission model, cross-account sharing mechanisms, and integration patterns is essential for building scalable, secure data platforms that meet enterprise compliance requirements.
Real-World Project Structure
A production Lake Formation deployment requires careful orchestration of permissions, data sharing, and governance policies.
Complete Architecture
Data Sources â Glue Catalog â Lake Formation â Data Consumers
â â â â
S3 Buckets Table Defs Permissions Athena/Redshift
Applications Crawlers Cross-Account QuickSight
Databases ETL Jobs Audit Logs EMR/Spark
Directory Structure
lake-formation-governance/
âââ infrastructure/
â âââ cdk/
â â âââ lib/
â â â âââ lake-formation-stack.ts
â â â âââ iam-roles.ts
â â â âââ s3-buckets.ts
â â âââ bin/
â â âââ app.ts
â âââ terraform/
â âââ main.tf
â âââ lake-formation.tf
â âââ permissions.tf
âââ permissions/
â âââ resource-permissions/
â â âââ database-permissions.json
â â âââ table-permissions.json
â â âââ column-permissions.json
â âââ data-cell-permissions/
â â âââ row-level-security.json
â â âââ column-level-security.json
â âââ cross-account/
â âââ sharing-policies.json
â âââ trusted-accounts.json
âââ governance/
â âââ policies/
â â âââ data-classification.json
â â âââ retention-policies.json
â â âââ compliance-rules.json
â âââ audit/
â âââ audit-config.json
â âââ alerting-rules.json
âââ scripts/
â âââ permission-management.py
â âââ data-sharing.py
â âââ audit-monitor.py
âââ monitoring/
â âââ dashboards/
â â âââ lake-formation-usage.json
â âââ alarms/
â âââ permission-alerts.json
âââ tests/
âââ unit/
â âââ test-permissions.py
âââ integration/
âââ test-cross-account.py
Lake Formation Overview
AWS Lake Formation is a fully managed service that makes it easy to set up a secure data lake in days. It builds on the AWS Glue Data Catalog to provide centralized permissions management, fine-grained access control, and audit capabilities.
Core Concepts
| Concept | Description |
|---|---|
| Data Lake Administrator | Users with full Lake Formation permissions |
| Grant/Revoke | Grant or remove permissions on resources |
| Data Lake Permissions | Fine-grained permissions on databases, tables, columns |
| Data Cells | Row and column-level filters for fine-grained access |
| Cross-Account Sharing | Share data across AWS accounts |
| Tag-Based Access Control | Use LF-Tags for permission management |
Lake Formation Architecture Diagram
Permission Model
Lake Formation provides a hierarchical permission model:
Data Lake Administrator
â
Database Permissions
â
Table Permissions
â
Column Permissions
â
Data Cell Permissions (Row-Level)
Permission Types
| Permission | Scope | Description |
|---|---|---|
| DESCRIBE | Database/Table | View metadata |
| SELECT | Table/Column | Read data |
| INSERT | Table | Write data |
| DELETE | Table | Delete data |
| ALTER | Table | Modify schema |
| DROP | Table | Remove table |
| CREATE | Database | Create tables |
Fine-Grained Access Control
Column-Level Security
import boto3
import json
from typing import Dict, Any, List
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class LakeFormationManager:
"""Manages AWS Lake Formation permissions and governance."""
def __init__(self, region: str = 'us-east-1'):
self.client = boto3.client('lakeformation', region_name=region)
self.glue_client = boto3.client('glue', region_name=region)
def grant_database_permissions(
self,
database_name: str,
principal: str,
permissions: List[str],
grant_option: bool = False
) -> Dict[str, Any]:
"""Grant permissions on a database."""
try:
response = self.client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': principal
},
Resource={
'Database': {
'Name': database_name
}
},
Permissions=permissions,
GrantOption=grant_option
)
logger.info(
f"Granted {permissions} on {database_name} to {principal}"
)
return response
except Exception as e:
logger.error(f"Failed to grant database permissions: {e}")
raise
def grant_table_permissions(
self,
database_name: str,
table_name: str,
principal: str,
permissions: List[str],
column_permissions: List[str] = None,
grant_option: bool = False
) -> Dict[str, Any]:
"""Grant permissions on a table with optional column-level access."""
try:
resource = {
'Table': {
'DatabaseName': database_name,
'Name': table_name
}
}
if column_permissions:
resource['TableWithColumns'] = {
'DatabaseName': database_name,
'Name': table_name,
'ColumnNames': column_permissions
}
response = self.client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': principal
},
Resource=resource,
Permissions=permissions,
GrantOption=grant_option
)
logger.info(
f"Granted {permissions} on {database_name}.{table_name} to {principal}"
)
return response
except Exception as e:
logger.error(f"Failed to grant table permissions: {e}")
raise
def create_data_cell_filter(
self,
database_name: str,
table_name: str,
filter_name: str,
row_filter: str,
column_names: List[str] = None,
description: str = ''
) -> Dict[str, Any]:
"""Create a data cell filter for row-level security."""
try:
filter_expression = {
'FilterExpression': row_filter,
'PartitionKey': {
'Name': 'region',
'Value': {
'ComparisonOperator': 'EQUALS',
'StringValue': 'us-east-1'
}
}
}
if column_names:
filter_expression['Columns'] = [
{'Name': col} for col in column_names
]
response = self.client.create_data_cells_filter(
TableData={
'TableCatalogId': boto3.client('sts').get_caller_identity()['Account'],
'DatabaseName': database_name,
'TableName': table_name,
'Name': filter_name,
'RowFilter': {
'FilterExpression': row_filter
},
'ColumnWildcard': {
'ExcludedColumnNames': [] if column_names else None,
'IncludedColumnNames': column_names
},
'Description': description
}
)
logger.info(f"Data cell filter created: {filter_name}")
return response
except Exception as e:
logger.error(f"Failed to create data cell filter: {e}")
raise
def create_lf_tag(
self,
tag_key: str,
tag_values: List[str]
) -> Dict[str, Any]:
"""Create a Lake Formation tag."""
try:
response = self.client.create_lf_tag(
TagKey=tag_key,
TagValues=tag_values
)
logger.info(f"LF-Tag created: {tag_key}")
return response
except Exception as e:
logger.error(f"Failed to create LF-Tag: {e}")
raise
def grant_lf_tag_permissions(
self,
principal: str,
lf_tag: Dict[str, str],
permissions: List[str],
grant_option: bool = False
) -> Dict[str, Any]:
"""Grant permissions based on LF-Tag."""
try:
response = self.client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': principal
},
Resource={
'LFTag': {
'TagKey': lf_tag['Key'],
'TagValues': lf_tag['Values']
}
},
Permissions=permissions,
GrantOption=grant_option
)
logger.info(f"Granted {permissions} on LF-Tag {lf_tag['Key']}")
return response
except Exception as e:
logger.error(f"Failed to grant LF-Tag permissions: {e}")
raise
def describe_resource_permissions(
self,
resource_arn: str
) -> List[Dict[str, Any]]:
"""Describe permissions on a resource."""
try:
response = self.client.describe_resource(
ResourceArn=resource_arn
)
permissions = response.get('ResourceInfo', {}).get(
'LastModifiedBy', []
)
logger.info(f"Permissions on {resource_arn}: {len(permissions)} grants")
return permissions
except Exception as e:
logger.error(f"Failed to describe permissions: {e}")
raise
def main():
"""Example usage of Lake Formation manager."""
manager = LakeFormationManager(region='us-east-1')
# Grant database permissions
manager.grant_database_permissions(
database_name='analytics_db',
principal='arn:aws:iam::123456789012:role/analyst-role',
permissions=['DESCRIBE'],
grant_option=False
)
# Grant table permissions with column access
manager.grant_table_permissions(
database_name='analytics_db',
table_name='customer_data',
principal='arn:aws:iam::123456789012:role/analyst-role',
permissions=['SELECT'],
column_permissions=['customer_id', 'name', 'email']
)
# Create row-level security filter
manager.create_data_cell_filter(
database_name='analytics_db',
table_name='sales_data',
filter_name='region-filter',
row_filter='region = us-east-1',
column_headers=['amount', 'product', 'date']
)
if __name__ == '__main__':
main()
Cross-Account Data Sharing
Sharing Architecture
import boto3
import json
from typing import Dict, Any, List
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class LakeFormationSharing:
"""Manages cross-account data sharing with Lake Formation."""
def __init__(self, account_id: str, region: str = 'us-east-1'):
self.client = boto3.client('lakeformation', region_name=region)
self.glue_client = boto3.client('glue', region_name=region)
self.account_id = account_id
self.region = region
def register_resource_in_data_catalog(
self,
resource_arn: str,
role_arn: str
) -> Dict[str, Any]:
"""Register an external resource in the data catalog."""
try:
response = self.client.register_resource(
ResourceArn=resource_arn,
RoleArn=role_arn,
UseServiceLinkedRole=False
)
logger.info(f"Resource registered: {resource_arn}")
return response
except Exception as e:
logger.error(f"Failed to register resource: {e}")
raise
def create_resource_link(
self,
database_name: str,
resource_arn: str,
description: str = ''
) -> Dict[str, Any]:
"""Create a resource link to share a database."""
try:
response = self.glue_client.create_database(
DatabaseInput={
'Name': database_name,
'Description': description,
'TargetDatabase': {
'CatalogId': self.account_id,
'DatabaseName': 'source-database'
}
}
)
logger.info(f"Resource link created: {database_name}")
return response
except Exception as e:
logger.error(f"Failed to create resource link: {e}")
raise
def share_table_cross_account(
self,
source_database: str,
source_table: str,
target_account_id: str,
target_database: str,
target_table: str,
permissions: List[str] = ['SELECT']
) -> Dict[str, Any]:
"""Share a table across AWS accounts."""
try:
# Create resource link in target account
self.glue_client.create_table(
DatabaseName=target_database,
TableInput={
'Name': target_table,
'TargetTable': {
'CatalogId': self.account_id,
'DatabaseName': source_database,
'Name': source_table
}
}
)
# Grant permissions to target account
response = self.client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': f'arn:aws:iam::{target_account_id}:root'
},
Resource={
'Table': {
'DatabaseName': source_database,
'Name': source_table
}
},
Permissions=permissions,
GrantOption=False
)
logger.info(
f"Table {source_table} shared with account {target_account_id}"
)
return response
except Exception as e:
logger.error(f"Failed to share table cross-account: {e}")
raise
def create_cross_account_grant(
self,
target_account_id: str,
resource_type: str,
resource_name: str,
permissions: List[str]
) -> Dict[str, Any]:
"""Create a cross-account grant for Lake Formation."""
try:
resource = {}
if resource_type == 'database':
resource['Database'] = {'Name': resource_name}
elif resource_type == 'table':
parts = resource_name.split('.')
resource['Table'] = {
'DatabaseName': parts[0],
'Name': parts[1]
}
elif resource_type == 'lf-tag':
resource['LFTag'] = {
'TagKey': resource_name.split(':')[0],
'TagValues': resource_name.split(':')[1].split(',')
}
response = self.client.grant_permissions(
Principal={
'DataLakePrincipalIdentifier': f'arn:aws:iam::{target_account_id}:root'
},
Resource=resource,
Permissions=permissions,
GrantOption=False
)
logger.info(
f"Cross-account grant created for {target_account_id}"
)
return response
except Exception as e:
logger.error(f"Failed to create cross-account grant: {e}")
raise
def main():
"""Example usage of Lake Formation sharing."""
sharing = LakeFormationSharing(
account_id='123456789012',
region='us-east-1'
)
# Share table cross-account
sharing.share_table_cross_account(
source_database='analytics_db',
source_table='customer_data',
target_account_id='987654321098',
target_database='shared_data',
target_table='customer_data_shared',
permissions=['SELECT']
)
# Create cross-account grant
sharing.create_cross_account_grant(
target_account_id='987654321098',
resource_type='database',
resource_name='analytics_db',
permissions=['DESCRIBE']
)
if __name__ == '__main__':
main()
Cross-Account Sharing Formula
Shared Resources = Source Tables à Target Accounts à Permission Types
For 10 tables shared with 5 accounts, each with 3 permission types:
Shared Resources = 10 Ã 5 Ã 3 = 150 permission grants
Audit and Compliance
Audit Configuration
import boto3
import json
from typing import Dict, Any
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class LakeFormationAudit:
"""Manages Lake Formation audit and compliance."""
def __init__(self, region: str = 'us-east-1'):
self.client = boto3.client('lakeformation', region_name=region)
self.cloudtrail = boto3.client('cloudtrail', region_name=region)
def get_lf_events(
self,
start_time: str,
end_time: str,
event_name: str = None
) -> List[Dict[str, Any]]:
"""Get Lake Formation events from CloudTrail."""
try:
events = []
response = self.cloudtrail.lookup_events(
LookupAttributes=[
{
'AttributeKey': 'EventName',
'AttributeValue': 'GrantPermissions'
}
],
StartTime=start_time,
EndTime=end_time,
MaxResults=100
)
for event in response.get('Events', []):
event_data = json.loads(event.get('CloudTrailEvent', '{}'))
if event_data.get('eventSource') == 'lakeformation.amazonaws.com':
events.append({
'event_id': event['EventId'],
'event_name': event['EventName'],
'event_time': event['EventTime'].isoformat(),
'user_identity': event_data.get('userIdentity', {}),
'request_parameters': event_data.get('requestParameters', {})
})
logger.info(f"Retrieved {len(events)} Lake Formation events")
return events
except Exception as e:
logger.error(f"Failed to get LF events: {e}")
raise
def analyze_permission_changes(
self,
events: List[Dict[str, Any]]
) -> Dict[str, Any]:
"""Analyze permission changes for compliance."""
analysis = {
'total_events': len(events),
'grant_events': 0,
'revoke_events': 0,
'unique_users': set(),
'affected_resources': set(),
'permission_types': {}
}
for event in events:
event_name = event.get('event_name', '')
user = event.get('user_identity', {}).get('arn', 'unknown')
resource = event.get('request_parameters', {}).get('resource', {})
if 'Grant' in event_name:
analysis['grant_events'] += 1
elif 'Revoke' in event_name:
analysis['revoke_events'] += 1
analysis['unique_users'].add(user)
if 'database' in resource:
analysis['affected_resources'].add(resource['database'])
if 'table' in resource:
analysis['affected_resources'].add(
f"{resource['table']['databaseName']}.{resource['table']['name']}"
)
analysis['unique_users'] = len(analysis['unique_users'])
analysis['affected_resources'] = len(analysis['affected_resources'])
return analysis
def main():
"""Example usage of Lake Formation audit."""
audit = LakeFormationAudit(region='us-east-1')
# Get events for last 24 hours
from datetime import datetime, timedelta
end_time = datetime.utcnow().isoformat()
start_time = (datetime.utcnow() - timedelta(days=1)).isoformat()
events = audit.get_lf_events(start_time, end_time)
analysis = audit.analyze_permission_changes(events)
print(f"Total events: {analysis['total_events']}")
print(f"Grant events: {analysis['grant_events']}")
print(f"Revoke events: {analysis['revoke_events']}")
print(f"Unique users: {analysis['unique_users']}")
if __name__ == '__main__':
main()
Compliance Metrics
| Metric | Target | Alert Threshold |
|---|---|---|
| Permission Grants | Audited | > 100/day |
| Cross-Account Shares | Approved | > 10/day |
| RLS Filters | Active | < 90% coverage |
| Audit Coverage | 100% | < 95% |
| Unusual Access | None | Any detection |
Security Considerations
Permission Hierarchy
| Level | Scope | Permissions |
|---|---|---|
| Account | Root | All permissions |
| Database | Catalog | CREATE, ALTER, DROP |
| Table | Database | SELECT, INSERT, DELETE |
| Column | Table | SELECT (specific columns) |
| Data Cell | Row/Column | Filtered access |
Encryption Configuration
| Layer | Configuration | Key Management |
|---|---|---|
| Data at Rest | S3 SSE-KMS | Customer-managed key |
| Data in Transit | TLS 1.2+ | AWS Certificate Manager |
| Catalog Metadata | Encrypted | AWS-managed |
| Audit Logs | CloudTrail | S3 with MFA Delete |
Access Control Best Practices
# IAM policy for Lake Formation access
policy = {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"lakeformation:GetDataAccess",
"lakeformation:GrantPermissions",
"lakeformation:RevokePermissions",
"lakeformation:BatchGrantPermissions",
"lakeformation:BatchRevokePermissions"
],
"Resource": "arn:aws:lakeformation:us-east-1:123456789012:*"
},
{
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartitions"
],
"Resource": [
"arn:aws:glue:us-east-1:123456789012:catalog",
"arn:aws:glue:us-east-1:123456789012:database/*",
"arn:aws:glue:us-east-1:123456789012:table/*/*"
]
}
]
}
Interview Questions & Answers
Q1: What is AWS Lake Formation and how does it differ from AWS Glue?
Answer: Lake Formation builds on top of AWS Glue to provide governance capabilities:
| Aspect | AWS Glue | AWS Lake Formation |
|---|---|---|
| Primary Function | ETL and data cataloging | Data lake governance |
| Permissions | IAM-based | Fine-grained (database, table, column) |
| Sharing | Limited | Cross-account with resource links |
| Audit | CloudTrail | Built-in audit with CloudTrail |
| RLS/CLS | Manual implementation | Native data cell filters |
Lake Formation provides centralized permissions management for data lakes, while Glue focuses on ETL and data discovery.
Q2: Explain Lake Formation's permission model.
Answer: Lake Formation provides a hierarchical permission model:
- Database Permissions: CREATE, ALTER, DROP on databases
- Table Permissions: SELECT, INSERT, DELETE, ALTER, DROP on tables
- Column Permissions: SELECT on specific columns
- Data Cell Permissions: Row-level filters with column restrictions
Permissions cascade down the hierarchy. Granting SELECT on a database grants SELECT on all tables. Column-level permissions override table-level permissions.
Q3: How does cross-account data sharing work in Lake Formation?
Answer: Cross-account sharing uses resource links:
- Source Account: Registers resource and grants permissions to target account
- Target Account: Creates resource link (pointer to source resource)
- Access: Target account queries data through resource link
Key Points:
- Data stays in source account (no data movement)
- Source account controls permissions
- Target account sees data as local resource
- Supports databases, tables, and LF-Tags
Q4: What are Lake Formation Tags (LF-Tags) and when should you use them?
Answer: LF-Tags are key-value pairs for attribute-based access control:
Use Cases:
- Large number of tables with similar permission patterns
- Dynamic permission management
- Compliance-driven access control
Example:
LF-Tag: DataClassification = [Public, Internal, Confidential, Restricted]
LF-Tag: Department = [Sales, Marketing, Engineering]
Benefits:
- Scalable permission management
- Easy to audit and report
- Dynamic assignment without modifying individual grants
Q5: How do you implement row-level security in Lake Formation?
Answer: Row-level security uses Data Cell Filters:
- Create Filter: Define row filter expression
- Apply to Table: Associate filter with specific table
- Assign Principal: Grant access to specific users/roles
Example:
# Create row filter for region-based access
filter = {
'FilterExpression': 'region = :current_user_region',
'Columns': ['region', 'department', 'team']
}
Benefits:
- No data duplication
- Centralized management
- Dynamic based on user attributes
Q6: What audit capabilities does Lake Formation provide?
Answer: Lake Formation provides comprehensive auditing:
- CloudTrail Integration: All API calls logged
- Permission Changes: Track grants and revokes
- Data Access: Monitor SELECT queries
- Cross-Account Activity: Track sharing events
- Compliance Reports: Built-in compliance dashboards
Key Events to Monitor:
- GrantPermissions / RevokePermissions
- GetDataAccess
- BatchGrantPermissions
- Cross-account sharing events
Q7: How do you migrate from IAM-based permissions to Lake Formation?
Answer: Migration strategy:
- Assessment: Audit existing IAM permissions
- Hybrid Mode: Enable both IAM and Lake Formation permissions
- Gradual Migration: Convert IAM policies to Lake Formation grants
- Validation: Test permissions in parallel
- Cutover: Disable IAM-based permissions
Key Considerations:
- Lake Formation takes precedence when enabled
- Use
UseLakeFormationCredentialsflag for existing IAM roles - Test thoroughly before disabling IAM permissions
Q8: What are best practices for Lake Formation governance?
Answer: Governance best practices:
- Principle of Least Privilege: Grant minimum required permissions
- LF-Tags for Scalability: Use tags for attribute-based control
- Data Cells for RLS: Implement row-level security
- Regular Audits: Monitor permission changes
- Cross-Account Governance: Use resource links for sharing
- Compliance Monitoring: Enable CloudTrail and GuardDuty
- Documentation: Maintain permission inventory
Common Pitfalls
| Pitfall | Impact | Prevention |
|---|---|---|
| Over-privileged grants | Data exposure | Use least-privilege principle |
| Missing RLS filters | Unauthorized row access | Implement data cells |
| No audit monitoring | Compliance violations | Enable CloudTrail logging |
| Ignoring LF-Tags | Permission sprawl | Use tags for scalability |
| Cross-account without governance | Shadow IT | Centralize sharing policies |
| IAM/LF conflicts | Permission confusion | Use Lake Formation exclusively |
Performance Considerations
| Metric | Target | Optimization |
|---|---|---|
| Permission Evaluation | < 10ms | Use LF-Tags, cache grants |
| Cross-Account Query | < 5s | Optimize resource links |
| Audit Log Processing | < 1 min | Use CloudWatch Logs Insights |
| Data Cell Filter | < 100ms | Optimize filter expressions |
| Bulk Permission Grants | < 30s | Use batch operations |
Security Considerations
| Layer | Threat | Mitigation |
|---|---|---|
| Network | Unauthorized access | VPC endpoints, private connectivity |
| Authentication | Credential compromise | IAM roles, MFA |
| Authorization | Over-privileged access | LF-Tags, data cells |
| Data | Unauthorized queries | Column/row-level security |
| Audit | Untracked changes | CloudTrail, GuardDuty |
| Cross-Account | Shadow IT | Centralized governance |