Support Vector Machines

Introduction

Support Vector Machines find optimal hyperplanes for classification and regression with various kernel functions.

SVM Basics

from sklearn.svm import SVC, SVR
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_samples=100, n_features=2, random_state=42)

svc = SVC(kernel='linear', C=1.0)
svc.fit(X, y)

# Support vectors
print(f"Number of support vectors: {svc.n_support_}")
print(f"Support vectors: {svc.support_vectors_[:3]}")

Kernel Functions

# Linear kernel (default for linear data)
svc_linear = SVC(kernel='linear', C=1.0)

# RBF (Gaussian) kernel - most common
svc_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')

# Polynomial kernel
svc_poly = SVC(kernel='poly', degree=3, coef0=1, C=1.0)

# Sigmoid kernel
svc_sigmoid = SVC(kernel='sigmoid', coef0=0, C=1.0)

SVM Parameters

from sklearn.svm import SVC

# C parameter - regularization strength
svc = SVC(C=0.1)   # More regularization, wider margin
svc = SVC(C=10)    # Less regularization, narrower margin

# Gamma - RBF kernel parameter
svc = SVC(kernel='rbf', gamma='scale')  # Auto (1 / n_features)
svc = SVC(kernel='rbf', gamma=0.1)       # Manual

# Epsilon - SVR parameter
svr = SVR(kernel='rbf', epsilon=0.1, C=1.0)

Multi-class Classification

from sklearn.svm import SVC

# One-vs-One (default for SVC)
svc = SVC(kernel='rbf')
svc.fit(X_multi, y_multi)

# One-vs-Rest using SVC
from sklearn.multiclass import OneVsRestClassifier
ovr = OneVsRestClassifier(SVC(kernel='rbf'))
ovr.fit(X_multi, y_multi)

Practice Problems

1. Train SVM with linear kernel
2. Compare RBF vs polynomial kernels
3. Tune C and gamma parameters
4. Visualize decision boundary
5. Use SVM for regression