Support Vector Machines

What Is Support Vector Machines?

Support Vector Machines is a key concept in Machine Learning. Understanding it is essential for building effective data science solutions.

Core Idea: Support Vector Machines provides a systematic approach to svm problems by learning patterns from data and applying them to make predictions or decisions.

Key Concepts

Support Vector Machine Fundamentals

SVM finds the maximum-margin hyperplane that separates classes.

Objective (hard margin):

$\min_{\mathbf{w},b} \frac{1}{2}\|\mathbf{w}\|^2 \quad \text{s.t.} \quad y_i(\mathbf{w}^\top\mathbf{x}_i + b) \geq 1 \; \forall i$

Soft margin (C-SVM):

$\min_{\mathbf{w},b,\xi} \frac{1}{2}\|\mathbf{w}\|^2 + C\sum_{i=1}^n \xi_i$

The kernel trick maps data to higher dimensions without explicit computation:

Python Implementation

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report
import warnings
warnings.filterwarnings("ignore")

# Load example dataset
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

# Prepare data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Class distribution: {dict(pd.Series(y).value_counts())}")
print(f"Train / Test split: {len(X_train)} / {len(X_test)}")

from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

model = SVC(kernel="rbf", C=1.0, gamma="scale", probability=True, random_state=42)
model.fit(X_train_s, y_train)

# Grid search
param_grid = {"C": [0.1, 1, 10, 100], "gamma": ["scale", 0.01, 0.1]}
gs = GridSearchCV(SVC(kernel="rbf", probability=True), param_grid, cv=5, scoring="f1")
gs.fit(X_train_s, y_train)
print(f"Best params: {gs.best_params_}")
print(f"Best CV F1 : {gs.best_score_:.4f}")

Evaluation & Results

# Evaluate model performance
y_pred = model.predict(X_test_s)

print(f"Accuracy  : {accuracy_score(y_test, y_pred):.4f}")
print(f"\nClassification Report:")
print(classification_report(y_test, y_pred,
      target_names=data.target_names))

# Cross-validation for robust estimate
cv_scores = cross_val_score(model, X_train_s, y_train, cv=5, scoring="f1")
print(f"\n5-Fold CV F1: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

# Visualise results
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

# Confusion matrix
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
cm = confusion_matrix(y_test, y_pred)
ConfusionMatrixDisplay(cm, display_labels=data.target_names).plot(ax=axes[0])
axes[0].set_title("Confusion Matrix")

# CV scores
axes[1].bar(range(1, 6), cv_scores, color="#3b82f6", edgecolor="white")
axes[1].axhline(cv_scores.mean(), color="red", linestyle="--",
                label=f"Mean={cv_scores.mean():.4f}")
axes[1].set_xlabel("Fold"); axes[1].set_ylabel("F1 Score")
axes[1].set_title("Cross-Validation Scores")
axes[1].legend(); axes[1].grid(True, alpha=0.3, axis="y")

plt.tight_layout()
plt.show()

Comparison with Related Methods

Hyperparameter Tuning

from sklearn.model_selection import GridSearchCV

param_grid = {
    "C":       [0.01, 0.1, 1.0, 10.0],
    "gamma":   ["scale", "auto"],
    "kernel":  ["rbf", "linear"],
}

grid = GridSearchCV(model, param_grid, cv=5,
                    scoring="f1", n_jobs=-1, verbose=0)
grid.fit(X_train_s, y_train)

print(f"Best params : {grid.best_params_}")
print(f"Best CV F1  : {grid.best_score_:.4f}")
print(f"Test F1     : {grid.score(X_test_s, y_test):.4f}")

Key Takeaways

1. Support Vector Machines is a powerful method for svm tasks
2. Always scale features before applying distance-based or regularised methods
3. Use cross-validation — never evaluate on the same data used for training
4. Start simple — a strong baseline prevents over-engineering
5. Visualise everything — confusion matrices, learning curves, feature importances