What is a Data Catalog?

A Data Catalog is a centralized metadata repository that organizes, describes, and indexes data assets across your organization. Think of it as a searchable inventory of all your data � what it is, where it lives, who owns it, and how it's used.

Why Data Catalogs Matter

In modern data engineering, organizations often have thousands of datasets spread across multiple accounts, regions, and storage systems. Without a catalog, data engineers waste hours searching for the right dataset, understanding its schema, or determining who to contact about data quality issues.

A Data Catalog solves these problems by providing:

• Discovery: Find datasets across S3, Redshift, RDS, and other sources
• Understanding: Know what each column means, its data type, and sample values
• Governance: Track data ownership, access policies, and compliance status
• Lineage: Trace where data comes from and how it transforms over time
• Collaboration: Enable teams to share and reuse trusted datasets

Core Components of a Data Catalog

Every data catalog, regardless of implementation, contains these essential components:

Data Catalog vs Data Lake vs Data Warehouse

Glue Data Catalog Deep Dive

AWS Glue Data Catalog is the default metadata store for AWS analytics services. It's a Hive-compatible metastore that stores table definitions, schema information, and other metadata needed to query data in S3 and other stores.

Architecture and Key Concepts

The Glue Data Catalog is organized in a hierarchical structure:

AWS Account (1 per region)
+-- Data Catalog
    +-- Database: sales_db
    �   +-- Table: orders
    �   �   +-- Columns: order_id, customer_id, amount, order_date
    �   �   +-- Partitions: year=2024/month=01
    �   �   +-- StorageDescriptor: S3 location, SerDe info
    �   +-- Table: customers
    �       +-- Columns: customer_id, name, email, created_at
    +-- Database: analytics_db
    �   +-- Table: user_events
    +-- Database: raw_data
        +-- Table: clickstream

Databases and Tables

A database in Glue is a logical grouping of tables. It's equivalent to a schema in traditional databases. Each table represents a dataset with:

• Columns: Name, type, and description of each field
• Partitions: How data is physically organized (e.g., by date)
• StorageDescriptor: Where the data lives and how it's serialized
• Table metadata: Creation time, owner, parameters

Crawlers and Classification

Crawlers are programs that scan your data stores to automatically discover schema. They:

1. Connect to data sources (S3, RDS, DynamoDB)
2. Extract schema information
3. Create or update table definitions in the Data Catalog
4. Classify data formats automatically

Built-in classifiers recognize:

• JSON, CSV, Avro, Parquet, ORC
• XML, Protobuf
• Relational database schemas

Custom classifiers handle proprietary formats using grok patterns or XML/JSON path expressions.

Table Properties and Parameters

Every Glue table has properties that control how data is accessed:

{
  "Name": "orders",
  "DatabaseName": "sales_db",
  "StorageDescriptor": {
    "Location": "s3://my-bucket/sales/orders/",
    "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
    "OutputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
    "SerdeInfo": {
      "SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
    },
    "Columns": [
      {"Name": "order_id", "Type": "bigint", "Comment": "Unique order identifier}, {"Name": "customer_id", "Type": "bigint", "Comment": "Customer foreign key}, {"Name": "amount", "Type": "decimal(10,2)", "Comment": "Order amount in USD}, {"Name": "order_date", "Type": "date", "Comment": "Order creation date"}
    ]
  },
  "PartitionKeys": [
    {"Name": "year", "Type": "int}, {"Name": "month", "Type": "int"}
  ],
  "TableType": "EXTERNAL_TABLE",
  "Parameters": {
    "classification": "parquet",
    "compressionType": "snappy",
    "parquet.compression": "SNAPPY"
  }
}

Glue Crawler Configuration

When setting up crawlers, you define:

• Data source: S3 paths, JDBC connections, or DynamoDB tables
• IAM role: Permissions to access the data and write to the catalog
• Database target: Where to store discovered schemas
• Schedule: How often to run (cron expression or on-demand)
• Classifier priority: Which classifiers to use for schema detection

import boto3

glue = boto3.client('glue')

# Create a crawler
response = glue.create_crawler(
    Name='sales-orders-crawler',
    Role='GlueCrawlerRole',
    DatabaseName='sales_db',
    Description='Crawls S3 sales data',
    Targets={
        'S3Targets': [
            {'Path': 's3://my-bucket/sales/orders/'}
        ]
    },
    SchemaChangePolicy={
        'UpdateBehavior': 'UPDATE_IN_DATABASE',
        'DeleteBehavior': 'LOG'
    },
    RecrawlPolicy={
        'RecrawlBehavior': 'CRAWL_EVERYTHING'
    }
)

# Start the crawler
glue.start_crawler(Name='sales-orders-crawler')

Schema Registry

The Glue Schema Registry allows you to centrally manage and enforce data schemas for streaming data. It ensures data producers and consumers agree on data formats, preventing schema evolution issues.

Why Schema Registry Matters

In streaming architectures, schemas can change over time. Without a registry:

• Producers might send data in unexpected formats
• Consumers break when they encounter unknown fields
• Debugging becomes difficult without schema versioning

The Schema Registry solves this by:

• Validating schemas at produce time
• Versioning schemas with compatibility rules
• Storing schemas centrally for discovery

Compatibility Modes

Avro Schema Example

{
  "type": "record",
  "name": "Order",
  "namespace": "com.company.sales",
  "fields": [
    {"name": "order_id", "type": "long", "doc": "Unique order identifier}, {"name": "customer_id", "type": "long", "doc": "Customer foreign key}, {"name": "amount", "type": {"type": "bytes", "logicalType": "decimal", "precision": 10, "scale": 2}, "doc": "Order amount}, {"name": "order_date", "type": {"type": "int", "logicalType": "date"}, "doc": "Order date}, {"name": "items", "type": {"type": "array", "items": "string"}, "doc": "List of item SKUs"}
  ]
}

Data Discovery and Lineage

Data discovery is the process of finding and understanding data assets. A good catalog makes this effortless through:

• Full-text search: Find tables by name, column name, or description
• Tagging: Add business tags like "PII", "financial", "quarterly"
• Browse by database: Navigate the hierarchy visually
• View column details: See data types, descriptions, and sample values
• Access previews: View sample data without querying the source

Data Lineage Tracking

Data lineage shows the complete journey of data from source to destination. It answers:

• Where did this data come from?
• What transformations were applied?
• Where does this data flow to?
• Who modified it and when?

Using AWS Glue for Lineage

AWS Glue provides lineage tracking through:

• Job bookmarks: Track which data has been processed
• Run history: Record of all ETL job executions
• CloudTrail integration: Log API calls for audit
• Custom lineage: Store additional metadata via API

# Get job run history for lineage
response = glue.get_job_runs(
    JobName='transform-orders',
    MaxResults=10
)

for run in response['JobRuns']:
    print(f"Run: {run['Id'
  ]}, Status: {run['JobRunState'
  ]}")
    print(f"Started: {run['StartedOn'
  ]}, Completed: {run['CompletedOn'
  ]}")
    print(f"Output: {run['OutputDataS3Path'
  ]}")

Catalog for Data Engineering

As a data engineer, the Glue Data Catalog is central to your daily workflow. Here's how it integrates with common patterns:

Key Integration Patterns

Best Practices for Data Engineers

1. Use consistent naming conventions: db_layer_source_system (e.g., raw_s3_salesforce)
2. Add descriptions: Document what each table and column contains
3. Partition strategically: Partition by date for time-series data
4. Set up crawlers: Automate schema discovery on schedule
5. Enable Lake Formation: Add fine-grained permissions
6. Use parameters: Store custom metadata like data owner, SLA, quality score

Cost Optimization

• Standard tier: 1 per million requests
• Free tier: First 1 million objects and 1 million requests per month

Athena Query Example

Once tables are registered in the catalog, querying is straightforward:

-- Query S3 data through Athena using Glue catalog
SELECT 
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spent
FROM sales_db.orders
WHERE year = 2024 
  AND month = 12
GROUP BY customer_id
HAVING SUM(amount) > 1000
ORDER BY total_spent DESC;

Redshift Spectrum Integration

-- Create external schema pointing to Glue catalog
CREATE EXTERNAL SCHEMA IF NOT EXISTS spectrum_schema
FROM DATA CATALOG
DATABASE 'sales_db'
IAM_ROLE 'arn:aws:iam::123456789012:role/SpectrumRole'
REGION 'us-east-1';

-- Query S3 data directly from Redshift
SELECT * FROM spectrum_schema.orders
WHERE order_date >= '2024-01-01';

Architecture Flow

Interview Q&A

Q1: What is the difference between Glue Data Catalog and Lake Formation?

Answer: The Glue Data Catalog is the metadata store that holds table definitions, schema, and location information. Lake Formation is a permission management layer that sits on top of the catalog and provides fine-grained access control at the database, table, column, and row level. Think of it as: Catalog = what exists, Lake Formation = who can access it.

Q2: How do you handle schema evolution in Glue Data Catalog?

Answer: Schema evolution is handled through:

• Crawler updates: Re-run crawlers to detect schema changes
• Manual updates: Use the Glue console or API to modify table definitions
• Schema change policy: Configure UpdateBehavior and DeleteBehavior in crawlers
• Backward compatibility: Add new columns as optional, avoid removing required columns
• Partition evolution: Add new partitions without affecting existing data

Q3: Explain the cost model for Glue Data Catalog.

Answer: The Data Catalog charges 1 per million requests per month. There's a generous free tier of 1 million objects and 1 million requests. A "object" is each table, partition, or database definition. For most use cases, costs remain minimal even at scale.

Q4: How would you optimize crawlers for a large data lake?

Answer:

• Partition projection: Use partition projection for known partition formats to skip crawler scans
• Scheduled crawlers: Run crawlers only when new data arrives, not continuously
• Targeted paths: Crawl specific S3 prefixes rather than entire buckets
• Classifier priority: Use custom classifiers to reduce classification time
• Incremental crawling: Use LastCrawl information to only process new data

Q5: Describe how you would set up data quality checks with Glue.

Answer: Data quality can be integrated through:

• Glue DataBrew: Profile data and create quality rules
• Custom classifiers: Validate data patterns during crawl
• Job bookmarks: Track processed data to avoid duplicates
• CloudWatch metrics: Monitor job success/failure rates
• Lambda triggers: Run validation Lambda after ETL jobs
• Athena queries: Schedule data quality checks using scheduled queries

Q6: What is the difference between Hive metastore and Glue Data Catalog?

Answer: The Glue Data Catalog is a Hive-compatible metastore but with AWS-specific enhancements:

• Fully managed (no Hive Metastore server to maintain)
• Native integration with AWS services (Athena, Redshift, EMR)
• Built-in versioning and lineage tracking
• Lake Formation integration for fine-grained permissions
• API-based access with IAM integration

Q7: How do you migrate an existing Hive metastore to Glue?

Answer: Migration steps:

1. Export metadata from existing Hive metastore using hms-dump tool
2. Import into Glue using the Glue API or AWS CLI
3. Update connection strings in applications to point to Glue
4. Validate table definitions and permissions
5. Run crawlers to sync any schema changes
6. Update ETL jobs to use Glue connectors

Q8: Explain the use of table parameters and custom metadata.

Answer: Table parameters in Glue allow storing custom key-value metadata such as:

• owner: Team or person responsible
• sla: Expected freshness (e.g., "daily", "hourly")
• pii: Contains personally identifiable information (true/false)
• quality_score: Data quality metric
• source_system: Origin system (e.g., "salesforce", "snowflake")

This metadata enables data discovery, governance, and operational monitoring.

Q9: How do you handle cross-account data access with Glue?

Answer: Cross-account access is configured through:

1. Lake Formation permissions: Grant cross-account access to databases/tables
2. IAM roles: Create roles with lakeformation:GetDataAccess
3. Resource policies: Attach S3 bucket policies for data access
4. Glue resource policies: Allow cross-account Glue API calls
5. Federated access: Use AWS SSO or identity federation

Q10: What monitoring and alerting do you set up for Glue crawlers?

Answer: Monitoring setup includes:

• CloudWatch metrics: CrawlTime, TablesAdded, TablesUpdated
• CloudWatch Logs: Enable for detailed logging
• SNS notifications: Alert on crawler failures or warnings
• EventBridge rules: Trigger Lambda on crawler state changes
• Custom dashboards: Track crawl success rates, duration, schema changes
• Cost monitoring: Track object count growth for billing alerts

Key Takeaways:

• The Glue Data Catalog is the central metadata hub for AWS analytics
• Crawlers automate schema discovery and table creation
• Schema Registry ensures data format compatibility for streaming data
• Data discovery and lineage enable self-service analytics
• Integration with Athena, Redshift, and EMR makes the catalog the backbone of data engineering
• Fine-grained access control through Lake Formation secures data assets

Summary

This topic covered the key concepts of AWS data engineering. Review the architecture diagrams, practice the interview questions, and understand the trade-offs between different service options.

Next Steps

Continue to the next topic to build on your AWS data engineering knowledge.

Knowledge Check

See Also

• Glue Data Catalog
• AWS Glue
• Lake Formation