ML Foundations
Feature Engineering — Where Domain Knowledge Meets Data Science
Feature engineering transforms raw data into representations that dramatically improve model performance. It is often the single most impactful step in any machine learning pipeline.
- Numerical Scaling — StandardScaler, MinMaxScaler, and RobustScaler prepare features for distance-based models
- Categorical Encoding — one-hot, label, and target encoding convert categorical data into model-ready formats
- Feature Creation — interaction terms, date components, and aggregations unlock hidden patterns in your data
"Coming up with features is difficult, time-consuming, requires expert knowledge. Applied machine learning is basically feature engineering." — Andrew Ng
Feature Engineering — Complete Guide
Feature engineering transforms raw data into features that improve model performance. It's often the most impactful step in ML.
Mathematical Foundations
Standardization (Z-score)
where
and
Min-Max Scaling
Information Gain (Feature Selection)
where
is the entropy.
Feature Engineering Pipeline
Numerical Features
Encoding Methods Diagram
RobustScaler:
Uses median and IQR
Robust to outliers
Use for: Data with outliers
Log Transform:
x_log = log(x + 1)
Use for: Skewed distributions, power laws
Feature Creation
Date features:
Year, Month, Day, Hour
Day of week, Is weekend
Is holiday, Season
Days since event
Text features:
Word count, Character count
TF-IDF vectors
Word embeddings
Sentiment scores
Interaction features:
x₁ × x₂ (product)
x₁ / x₂ (ratio)
x₁ - x₂ (difference)
x₁², x₂² (polynomial)
Aggregation features:
Mean, Median, Std per group
Count per category
Rolling statistics
Lag features
Feature Selection
Feature Selection Methods
Method 1: Filter (statistical tests)
Correlation with target
Chi-squared test
Mutual information
ANOVA F-test
Method 2: Wrapper (model-based)
Forward selection
Backward elimination
Recursive feature elimination (RFE)
Genetic algorithms
Method 3: Embedded (built into model)
L1 regularization (Lasso)
Feature importance (Tree-based)
Permutation importance
Python Implementation
Key Takeaways
What to Learn Next
-> Dimensionality Reduction Reduce high-dimensional features using PCA, t-SNE, and UMAP while preserving key information.
-> Model Evaluation Measure how much your engineered features actually improve model performance.
-> Linear Regression See how feature scaling and encoding directly impact linear model accuracy.
-> Clustering Use unsupervised techniques to discover hidden groups and create new features.
-> Model Selection Choose the best algorithm and tune hyperparameters for your engineered features.
-> Model Deployment Package your feature engineering pipeline into production-ready APIs and services.