Data Cleaning: Missing Values, Outliers and Types

Master data cleaning — handling missing values, outliers, duplicates, and type conversions.

Missing Data Mechanisms

Detecting Missing Values

Handling Missing Values

Deletion

Imputation

Advanced Imputation

Outlier Detection

IQR Method

Z-Score Method

Handling Outliers

Duplicates

Type Conversions

Text Cleaning

Complete Cleaning Pipeline

Key Takeaways

Practice Exercise

1. Given a dataset with 20% MCAR missing values, compare the bias introduced by mean imputation, median imputation, KNN imputation, and listwise deletion. Compute the difference between imputed and true means.
2. Write a function that detects outliers using IQR, MAD, and Z-score methods simultaneously, then creates a consensus flag (outlier if 2+ methods agree).
3. Build a text cleaning pipeline that handles: extra whitespace, mixed case, special characters, encoding issues (mojibake), and inconsistent formatting (e.g., "NY", "New York", "NYC" → "New York").
4. Given a DataFrame with mixed types and messy data, implement the complete cleaning pipeline and verify data quality with a summary report.
5. Analyze the missing data mechanism in a real-world dataset using Little's MCAR test or pattern analysis with missingno library.