What is Data Quality?

Data quality refers to the condition of your data based on factors such as accuracy, completeness, consistency, timeliness, and validity. High-quality data is essential for making reliable business decisions, training accurate machine learning models, and maintaining operational efficiency.

Poor data quality costs organizations an average of $12.9 million annually according to Gartner. In AWS environments, ensuring data quality is critical because cloud data pipelines often process massive volumes of data from diverse sources, making errors more likely to propagate and compound.

Data Quality Dimensions

Understanding the core dimensions of data quality helps engineers design comprehensive validation strategies:

Accuracy measures whether data correctly represents the real-world entities or events it describes. For example, a customer's age should match their actual age, not a random number.

Completeness assesses whether all required data is present. Missing values in critical fields can break downstream analytics and machine learning pipelines.

Consistency ensures data doesn't contradict itself across different systems or records. If a customer's address is "123 Main St" in one system, it shouldn't be "456 Oak Ave" in another.

Timeliness evaluates whether data is available when needed for its intended use. Stale data can lead to outdated insights and poor decisions.

Validity checks whether data conforms to defined formats, ranges, and business rules. Email addresses should contain "@", dates should be in the correct format, and amounts should be positive.

Uniqueness ensures each record is represented only once. Duplicate records can inflate metrics and cause analytical errors.

Data Quality Framework

A robust data quality framework provides a systematic approach to measuring, monitoring, and improving data quality across your organization. The framework typically follows a four-stage pipeline: Validate → Profile → Monitor → Alert.

Validation Stage

Validation is the first line of defense against bad data. It involves checking incoming data against predefined rules before it enters your data pipeline. Common validation techniques include:

• Schema validation: Ensuring data conforms to expected structure
• Type checking: Verifying data types match specifications
• Range validation: Confirming values fall within acceptable ranges
• Format validation: Checking that data follows required patterns (emails, phone numbers, dates)
• Referential integrity: Ensuring foreign keys reference valid records

Profiling Stage

Data profiling analyzes data to understand its structure, content, and quality. It provides statistics such as:

• Column-level statistics (nulls, distinct values, min/max, mean)
• Cross-column relationships
• Data patterns and distributions
• Anomaly detection

Monitoring Stage

Continuous monitoring tracks data quality metrics over time, detecting degradation or sudden changes. Key monitoring activities include:

• Trend analysis of quality scores
• Comparison against baselines
• Distribution shift detection
• Volume and freshness monitoring

Alerting Stage

When data quality drops below acceptable thresholds, alerting ensures the right teams are notified immediately. Effective alerting includes:

• Threshold-based alerts for critical metrics
• Anomaly detection for unexpected patterns
• Escalation workflows based on severity
• Integration with incident management systems

AWS Glue DataBrew

AWS Glue DataBrew is a visual data preparation tool that enables data engineers and data scientists to clean, normalize, transform, and label data in the cloud. It provides over 250 pre-built transformations for data preparation without writing code.

Key Features

Visual Data Profiling DataBrew automatically profiles your data, providing insights into:

• Column statistics (nulls, unique values, data types)
• Value distributions and patterns
• Correlations between columns
• Data quality issues and anomalies

Built-in Transformations DataBrew offers a rich library of transformations:

• Format conversions (date parsing, string manipulation)
• Statistical transformations (normalization, binning)
• Data enrichment (lookup, geocoding)
• Error handling (imputation, deduplication)

Job Scheduling and Orchestration

• Schedule recurring data preparation jobs
• Integrate with AWS Step Functions for complex workflows
• Trigger jobs based on data arrival or events
• Monitor job runs through CloudWatch

Integration with AWS Services

• Reads from and writes to S3, Redshift, RDS, and other data stores
• Integrates with AWS Glue Data Catalog for metadata management
• Uses IAM for access control and security
• Outputs to formats optimized for analytics (Parquet, ORC, Avro)

DataBrew Workflow

DataBrew Example: Data Profiling

import boto3

# Create DataBrew client
databrew = boto3.client('databrew')

# Create a dataset from S3
dataset = databrew.create_dataset(
    name='customer-data',
    input={
        's3InputDefinition': {
            'bucket': 'my-data-lake',
            'key': 'raw/customers/'
        }
    }
)

# Create a profile job
profile_job = databrew.create_profile_job(
    datasetName='customer-data',
    name='customer-profile',
    outputLocation={
        'bucket': 'my-data-lake',
        'key': 'profiled/customers/'
    },
    configuration={
        'datasetStatisticsConfiguration': {
            'includedStatistics': ['ALL']
        },
        'profileConfiguration': {
            'numericStatistics': ['MEAN', 'MIN', 'MAX', 'STANDARD_DEVIATION'],
            'nonNumericStatistics': ['UNIQUENESS', 'UNIQUENESS_RATIO']
        }
    },
    roleArn='arn:aws:iam::role/DataBrewRole'
)

# Run the profile job
databrew.start_job_run(
    name='customer-profile'
)

Amazon Deequ

Amazon Deequ is an open-source data quality library built on top of Apache Spark. Developed by Amazon, it enables data engineers to define and measure data quality at scale, making it ideal for large-scale data processing on AWS.

Core Concepts

Metrics are measures of data quality computed from your data. Deequ provides built-in metrics for:

• Size (row count)
• Completeness (non-null ratio)
• Uniqueness
• Distinctness
• Entropy
• Approximate count distinct

Constraints define business rules that data must satisfy. They use metrics to verify conditions:

• Minimum/maximum value constraints
• Range constraints
• In-set/not-in-set constraints
• Regex matching constraints
• Functional dependency constraints
• Custom SQL constraints

Checks are collections of constraints applied to your data. Checks can be configured to:

• addErrorIf(): Fail the check on violation
• addWarningIf(): Log a warning on violation

Verification Suites execute checks against your data and produce verification results.

Deequ Architecture

Deequ Example: Defining Quality Checks

import com.amazon.deequ.checks.Check
import com.amazon.deequ.checks.CheckStatus
import com.amazon.deequ.VerificationSuite
import com.amazon.deequ.constraints.Constraint

// Define data quality checks
val verificationSuite = VerificationSuite()
  .onData(data)
  .addCheck(
    Check(sparkContext, Check.Level.Error)
      .isComplete("customer_id")                    // No nulls
      .isUnique("customer_id")                      // No duplicates
      .contains("email", ".+@.+\\..+")              // Valid emails
      .isNonNegative("order_amount")                 // Positive amounts
      .isContainedIn("status", Array("ACTIVE", "INACTIVE", "PENDING"))  // Valid statuses
      .hasSize(_ > 1000)                             // Minimum row count
  )
  .addCheck(
    Check(sparkContext, Check.Level.Warning)
      .containsRatio("phone", "\\d{10}", _ > 0.95)  // 95% valid phones
      .isComplete("created_at")                       // Track missing timestamps
  )
  .run()

// Get results
val results = verificationSuite

results.checkResults.foreach { case (check, result) =>
  println(s"Check: ${check.description}")
  println(s"Status: ${result.status}")
  result.constraintResults.foreach { constraintResult =>
    println(s"  Constraint: ${constraintResult.constraint}")
    println(s"  Status: ${constraintResult.status}")
    println(s"  Message: ${constraintResult.message}")
  }
}

Deequ in AWS Glue

Amazon Deequ is available as a library in AWS Glue, making it easy to integrate data quality checks into your ETL jobs:

from awsglue.transforms import *
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame

# Initialize Glue context
glueContext = GlueContext(SparkContext.getOrCreate())
spark = glueContext.spark_session

# Read data from S3
dynamic_frame = glueContext.create_dynamic_frame.from_catalog(
    database="my_database",
    table_name="customers",
    transformation_ctx="customers_source"
)

# Convert to Spark DataFrame for Deequ
data = dynamic_frame.toDF()

# Run Deequ checks
from pydeequ.checks import Check
from pydeequ.verification import VerificationSuite

check = Check(spark, Check.Level.Error) \
    .isComplete("customer_id") \
    .isUnique("customer_id") \
    .isNonNegative("order_total")

result = VerificationSuite(spark) \
    .onData(data) \
    .addCheck(check) \
    .run()

# Convert results back to DynamicFrame for downstream processing
if result.status == "Success":
    output_frame = DynamicFrame.fromDF(data, glueContext, "clean_data")
    glueContext.write_dynamic_frame.from_options(
        frame=output_frame,
        connection_type="s3",
        connection_options={"path": "s3://clean-data/customers/"},
        format="parquet"
    )
else:
    # Route failed records to dead letter queue
    print("Data quality check failed!")
    print(result.checkResults)

Data Quality Patterns

Implementing effective data quality requires applying proven patterns that balance thoroughness with performance. Here are essential patterns for AWS data engineering.

Pattern 1: Schema-on-Read Validation

Validate data structure as it enters your pipeline rather than at the source. This is especially important when dealing with semi-structured data in data lakes.

import boto3
import json

glue = boto3.client('glue')

# Define expected schema
expected_schema = {
    "columns": {
        "customer_id": {"type": "string", "nullable": False},
        "email": {"type": "string", "pattern": r".+@.+\..+"},
        "order_amount": {"type": "double", "min": 0, "max": 100000},
        "created_at": {"type": "timestamp", "nullable": False}
    }
}

def validate_schema(record, schema):
    errors = []
    for col, rules in schema["columns"].items():
        if col not in record:
            errors.append(f"Missing column: {col}")
            continue
        value = record[col]
        
        if value is None and not rules.get("nullable", True):
            errors.append(f"Null value in non-nullable column: {col}")
        
        if "pattern" in rules and value:
            import re
            if not re.match(rules["pattern"], value):
                errors.append(f"Invalid format in {col}: {value}")
    
    return errors

Pattern 2: Anomaly Detection with Statistical Profiling

Use statistical methods to detect unexpected changes in data distributions.

Pattern 3: Data Contracts

Define explicit contracts between data producers and consumers to ensure data quality is maintained across team boundaries.

from dataclasses import dataclass
from typing import Optional, List

@dataclass
class DataContract:
    schema_version: str
    table_name: str
    owner: str
    columns: List[dict]
    quality_rules: List[dict]
    sla_hours: int

# Define a data contract
customer_contract = DataContract(
    schema_version="1.0",
    table_name="customers",
    owner="data-platform-team",
    columns=[
        {"name": "customer_id", "type": "string", "nullable": False, "unique": True}, {"name": "email", "type": "string", "nullable": False, "pattern": r".+@.+\..+}, {"name": "created_at", "type": "timestamp", "nullable": False},
    ],
    quality_rules=[
        {"metric": "completeness", "column": "customer_id", "threshold": 1.0}, {"metric": "uniqueness", "column": "customer_id", "threshold": 1.0}, {"metric": "freshness", "max_delay_hours": 24},
    ],
    sla_hours=4
)

Pattern 4: Automated Quality Scoring

Calculate a comprehensive quality score based on multiple dimensions.

def calculate_quality_score(data, schema):
    scores = {}
    
    # Completeness score
    null_counts = data.isnull().sum()
    total_rows = len(data)
    completeness = 1 - (null_counts.sum() / (total_rows * len(data.columns)))
    scores['completeness'] = completeness
    
    # Validity score
    valid_count = 0
    total_checks = 0
    for col, rules in schema.items():
        if 'pattern' in rules:
            total_checks += total_rows
            valid_count += data[col].str.match(rules['pattern']).sum()
    scores['validity'] = valid_count / total_checks if total_checks > 0 else 1.0
    
    # Uniqueness score
    uniqueness = {}
    for col in schema:
        if schema[col].get('unique'):
            unique_ratio = data[col].nunique() / len(data)
            uniqueness[col] = unique_ratio
    scores['uniqueness'] = sum(uniqueness.values()) / len(uniqueness) if uniqueness else 1.0
    
    # Overall score (weighted average)
    weights = {'completeness': 0.4, 'validity': 0.4, 'uniqueness': 0.2}
    overall = sum(scores[k] * weights[k] for k in weights)
    
    return {
        'overall': round(overall * 100, 2),
        'dimensions': {k: round(v * 100, 2) for k, v in scores.items()}
    }

Architecture Flow

Interview Q&A

Q1: What is data quality and why is it important in data engineering?

Answer: Data quality refers to the condition of data based on accuracy, completeness, consistency, timeliness, validity, and uniqueness. In data engineering, high-quality data is essential because:

• Poor data quality leads to incorrect analytics and business decisions
• It can cause machine learning models to produce unreliable predictions
• It wastes computational resources processing invalid data
• It erodes trust in data systems across the organization
• Regulatory compliance requires accurate and complete data

Data engineers must implement quality checks at every stage of the pipeline to ensure data remains reliable from ingestion to consumption.

Q2: Explain the difference between data validation and data profiling.

Answer: Data validation and data profiling serve different purposes:

Data Validation:

• Checks data against predefined rules and constraints
• Validates specific expectations (e.g., "email must contain @")
• Typically runs on each record or batch as it enters the pipeline
• Binary outcome: pass or fail
• Examples: Schema validation, type checking, range validation

Data Profiling:

• Analyzes data to understand its structure and characteristics
• Discovers patterns, distributions, and anomalies
• Usually runs periodically (not on every record)
• Produces statistics and reports rather than pass/fail results
• Examples: Column statistics, value distributions, correlation analysis

In practice, validation uses rules derived from profiling results, and profiling helps discover new validation rules needed.

Q3: What is Amazon Deequ and when would you use it?

Answer: Amazon Deequ is an open-source data quality library built on Apache Spark, developed by Amazon. It enables defining and measuring data quality at scale.

When to use Deequ:

• Processing large-scale data in AWS (millions/billions of records)
• Need Spark-based distributed quality checks
• Want to define quality checks as code (version-controlled)
• Need incremental quality checking on streaming data
• Require metrics computation for dashboards and monitoring

Key features:

• Metrics computation (size, completeness, uniqueness, etc.)
• Constraint-based checks with error/warning levels
• Incremental analysis for streaming data
• Anomaly detection based on historical metrics
• Integration with AWS Glue and EMR

Q4: How would you handle data quality failures in a production pipeline?

Answer: A comprehensive approach to handling data quality failures includes:

Immediate Response:

1. Quarantine bad data - Route failed records to a dead letter queue or reject table
2. Alert stakeholders - Notify data owners and downstream consumers
3. Log detailed error information - Capture which rules failed and sample records

Recovery Process:

1. Analyze failure patterns - Determine if it's a systemic issue or isolated incident
2. Implement fixes - Correct upstream sources or adjust transformation logic
3. Reprocess if needed - Re-run pipeline with corrected data
4. Update quality rules - Add new checks to prevent recurrence

Prevention:

• Implement data contracts with upstream producers
• Set up monitoring dashboards for quality metrics
• Create runbooks for common failure scenarios
• Use circuit breakers to stop bad data propagation

Q5: What are the key differences between AWS Glue DataBrew and Amazon Deequ?

Answer:

When to use DataBrew:

• Interactive data exploration and profiling
• One-time data cleaning tasks
• Business users who prefer visual interfaces
• Quick data preparation without coding

When to use Deequ:

• Automated quality checks in ETL pipelines
• Large-scale data requiring distributed processing
• Version-controlled quality rules
• Complex constraint logic and anomaly detection

Q6: How do you measure and track data quality over time?

Answer: Measuring and tracking data quality requires a systematic approach:

Metrics to Track:

• Completeness - Percentage of non-null values per column
• Accuracy - Percentage of values passing validation rules
• Timeliness - Data freshness (time since last update)
• Volume - Row count trends and anomalies
• Distribution - Statistical properties of numeric/categorical fields

Implementation Strategy:

1. Baseline Establishment: Profile data to establish quality baselines
2. Automated Monitoring: Run quality checks on every pipeline run
3. Dashboard Creation: Build dashboards showing quality trends (QuickSight, Grafana)
4. Alerting: Set up alerts for quality degradations (SNS, CloudWatch)
5. Historical Storage: Store quality metrics for trend analysis

Tools for Tracking:

• CloudWatch Metrics for pipeline-level monitoring
• S3/Hive tables for historical quality metrics
• QuickSight for quality dashboards
• Custom tables in Redshift/Athena for analysis

Q7: What is a data contract and how does it improve data quality?

Answer: A data contract is a formal agreement between data producers and consumers that defines:

• Schema: Column names, types, constraints
• Semantics: Business meaning and acceptable values
• Quality: Expected quality thresholds and rules
• SLA: Freshness and availability requirements
• Ownership: Who is responsible for the data

How it improves quality:

• Clear expectations - Both parties agree on what "good" data looks like
• Early detection - Violations are caught at the source
• Accountability - Clear ownership for quality issues
• Automation - Contracts can be enforced programmatically
• Documentation - Serves as living documentation of data assumptions

Implementation:

• Define contracts as code (JSON, YAML, Python classes)
• Store in a central registry (AWS Glue Catalog, custom database)
• Validate incoming data against contracts
• Alert on contract violations
• Version contracts to track changes over time

Q8: How do you implement data quality checks for streaming data?

Answer: Streaming data quality requires different approaches than batch:

Challenges:

• Cannot see entire dataset at once
• Must maintain state across micro-batches
• Latency requirements limit complex checks
• Schema evolution is more complex

Strategies:

1. Window-based Profiling:

   • Compute quality metrics over time windows (e.g., last 5 minutes)
   • Compare current window to historical baselines
   • Detect anomalies in distributions

2. Stateful Validation:

   • Maintain counters for quality metrics
   • Use Apache Flink or Kinesis Data Analytics for state management
   • Implement sliding window aggregations

3. Deequ Incremental Analysis:

   • Use Deequ's IncrementalAnalysis mode
   • Maintain metric states across batches
   • Efficient computation without full reprocessing

4. Sample-based Validation:

   • Validate statistical sample of records
   • Trade perfect accuracy for performance
   • Suitable for high-volume streams

AWS Services:

• Kinesis Data Analytics for real-time analysis
• Lambda for lightweight validation
• MSK (Managed Kafka) with custom validators
• Deequ on EMR for Spark Streaming

Q9: What are common data quality anti-patterns to avoid?

Answer:

Anti-pattern 1: No Quality Checks

• Processing data without any validation
• Discovering issues only when reports look wrong
• Fix: Implement checks at every pipeline stage

Anti-pattern 2: Checking Everything Everywhere

• Duplicating quality checks across multiple pipelines
• Wasting compute on redundant validation
• Fix: Define quality ownership per pipeline stage

Anti-pattern 3: Hardcoded Thresholds

• Quality rules with magic numbers
• No documentation for why thresholds exist
• Fix: Use data contracts and document thresholds

Anti-pattern 4: Alert Fatigue

• Too many non-actionable alerts
• Teams stop paying attention to alerts
• Fix: Implement severity levels and escalation

Anti-pattern 5: Manual Quality Checks

• Human verification of data quality
• Doesn't scale and introduces errors
• Fix: Automate all quality validation

Anti-pattern 6: Ignoring Data Profiling

• Not understanding data characteristics
• Missing opportunities to improve quality
• Fix: Profile data regularly and use results to inform rules

Q10: Design a data quality framework for a new data lake on AWS.

Answer: A comprehensive data quality framework for an AWS data lake:

Architecture Components:

1. Ingestion Layer:

   • Schema validation at ingestion (Glue crawlers)
   • Source freshness checks (CloudWatch Events)
   • Volume anomaly detection (Lambda)

2. Raw Zone:

   • Immutable storage for all incoming data
   • Partitioning by source and arrival time
   • Metadata tracking (Glue Catalog)

3. Processing Layer:

   • Deequ checks in Glue jobs
   • DataBrew for profiling and transformation
   • Custom validation Lambda functions

4. Quality Gate:

   • Centralized quality check orchestration
   • Error handling and routing
   • Metrics computation and storage

5. Clean Zone:

   • Only quality-validated data
   • Versioned and time-stamped
   • Ready for analytics consumption

6. Monitoring & Alerting:

   • CloudWatch dashboards for quality metrics
   • SNS alerts for threshold violations
   • QuickSight for quality reporting

Implementation Steps:

1. Define data contracts for all sources
2. Implement automated profiling on ingestion
3. Create quality check libraries in Glue
4. Set up monitoring and alerting
5. Build quality dashboards
6. Establish incident response runbooks
7. Train teams on quality practices

Technology Stack:

• AWS Glue for ETL and profiling
• Amazon Deequ for quality checks
• S3 for data storage
• Glue Catalog for metadata
• CloudWatch for monitoring
• QuickSight for visualization
• SNS/SQS for alerting

This comprehensive guide covers the essential concepts and practices for implementing data quality on AWS. Master these patterns and interview topics to build reliable, trustworthy data pipelines.

Summary

This topic covered the key concepts of AWS data engineering. Review the architecture diagrams, practice the interview questions, and understand the trade-offs between different service options.

Next Steps

Continue to the next topic to build on your AWS data engineering knowledge.

Knowledge Check

See Also

• AWS Glue DataBrew
• AWS Data Governance
• AWS Data Catalog