Feature Engineering: Selection, Creation, and Transformation

Module: Machine Learning | Difficulty: Advanced

Feature Selection Methods

Feature Creation

Polynomial Features:

Interaction Features:

Automated Feature Engineering (Featuretools)

import numpy as np
from sklearn.feature_selection import mutual_info_classif, SelectKBest

class FeatureEngineer:
    def __init__(self, n_top_k=20):
        self.n_top_k = n_top_k
    def select_features(self, X, y):
        mi_scores = mutual_info_classif(X, y)
        top_k = np.argsort(mi_scores)[::-1][:self.n_top_k]
        return top_k
    def create_interactions(self, X, feature_indices):
        interactions = []
        for i, j in zip(feature_indices[:-1], feature_indices[1:]):
            interactions.append(X[:, i] * X[:, j])
        return np.hstack([X, np.array(interactions).T])
    def create_polynomial(self, X, degree=2):
        result = [X]
        for d in range(2, degree+1):
            for i in range(X.shape[1]):
                result.append(X[:, i:i+1]**d)
        return np.hstack(result)

Research Insight: Feature engineering is often more important than model selection. Domain knowledge is crucial for creating meaningful features. Automated feature engineering can discover interactions that domain experts might miss.