Face Detection and Recognition

Module: Computer Vision | Difficulty: Advanced

Face Detection

Face detection locates all faces in an image and provides bounding boxes and landmark coordinates. Modern detectors use multi-scale feature pyramids to handle faces ranging from 20×20 pixels (distant faces) to 500×500 pixels (close-up portraits). The detection pipeline typically outputs a bounding box, confidence score, and 5 facial landmarks (eyes, nose, mouth corners) for subsequent alignment.

Multi-Task Cascaded Network (MTCNN)

MTCNN detects faces through three cascaded stages:

Where each parameter means:

• — Proposal Network: generates candidate face regions at multiple scales
• — Refine Network: filters candidates and refines bounding boxes
• — Output Network: produces final bounding boxes and 5-point landmarks
• Intuition: Each stage progressively refines detections, trading recall for precision; P-Net is fast but noisy, O-Net is accurate but slower

RetinaFace

RetinaFace uses a single-stage architecture with feature pyramid networks:

Where each parameter means:

• — focal loss for face/non-face classification
• — smooth L1 loss for bounding box regression
• — L1 loss for 5-point landmark regression
• — loss weights (0.25, 0.1 typically)
• Intuition: Multi-task learning jointly optimizes detection and landmark localization, with landmarks providing additional supervision for face quality

Face Alignment

Face alignment normalizes pose variations through affine transformation based on detected landmarks. The standard approach crops and aligns faces to a canonical 112×112 resolution:

Where each parameter means:

• — 2×2 affine transformation matrix
• — 2D translation vector
• — original landmark coordinates
• — target canonical coordinates
• Intuition: By aligning eyes to fixed positions, the network only needs to learn identity features rather than pose variations

Alignment Quality Impact

Proper alignment significantly improves recognition accuracy. Studies show alignment increases verification accuracy by 2-5% on LFW benchmark.

Face Embedding

Embedding Network

The embedding network maps aligned face images to a compact vector space where same identities cluster together:

Where each parameter means:

• — CNN backbone (ResNet-100, MobileNet)
• — aligned face image (112×112×3)
• — L2-normalized embedding vector
• — L2 norm ensuring unit length
• Intuition: L2 normalization projects embeddings onto the unit hypersphere, where cosine similarity equals dot product, simplifying distance computation

Triplet Loss

FaceNet introduced triplet loss for learning discriminative embeddings:

Where each parameter means:

• — anchor embedding (reference face)
• — positive embedding (same identity as anchor)
• — negative embedding (different identity)
• — Euclidean distance in embedding space
• — margin enforcing separation (typically 0.2)
• Intuition: The loss pushes same-identity faces closer than different-identity faces by at least margin ; hard negative mining selects the most confusing pairs

Cosine Similarity

Matching uses cosine similarity between embeddings:

Where each parameter means:

• — L2-normalized face embeddings
• — similarity score
• Intuition: For L2-normalized vectors, cosine similarity equals dot product; values > 0.6 typically indicate same identity

ArcFace

ArcFace adds an angular margin penalty to increase inter-class separability:

Where each parameter means:

• — scale factor (typically 64)
• — angular margin (typically 0.5 radians = 28.6°)
• — angle between embedding and class center for true class
• — angle between embedding and class center for other classes
• Intuition: ArcFace penalizes the angle between embedding and class center, forcing same-identity embeddings to cluster more tightly; this increases the margin between classes in angular space

Face Recognition Comparison

Complete Face Embedding Pipeline

import torch
import torch.nn as nn
import torch.nn.functional as F


class DepthwiseSeparableConv(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.depthwise = nn.Conv2d(in_ch, in_ch, 3, stride, 1, groups=in_ch, bias=False)
        self.bn1 = nn.BatchNorm2d(in_ch)
        self.pointwise = nn.Conv2d(in_ch, out_ch, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_ch)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.relu(self.bn1(self.depthwise(x)))
        return self.relu(self.bn2(self.pointwise(x)))


class MobileFaceNet(nn.Module):
    def __init__(self, embedding_dim=128):
        super().__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 64, 3, 1, 1, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True)
        )
        self.stage2 = nn.Sequential(
            DepthwiseSeparableConv(64, 64, 2),
            DepthwiseSeparableConv(64, 64),
        )
        self.stage3 = nn.Sequential(
            DepthwiseSeparableConv(64, 128, 2),
            DepthwiseSeparableConv(128, 128),
        )
        self.stage4 = nn.Sequential(
            DepthwiseSeparableConv(128, 128, 2),
            DepthwiseSeparableConv(128, 128),
            DepthwiseSeparableConv(128, 128),
        )
        self.conv2 = nn.Conv2d(128, 512, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(512)
        self.relu = nn.ReLU(inplace=True)
        self.gap = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(512, embedding_dim)

    def forward(self, x):
        x = self.conv1(x)
        x = self.stage2(x)
        x = self.stage3(x)
        x = self.stage4(x)
        x = self.relu(self.bn2(self.conv2(x)))
        x = self.gap(x)
        x = x.view(x.size(0), -1)
        return F.normalize(self.fc(x), p=2, dim=1)


class ArcFaceLoss(nn.Module):
    def __init__(self, embedding_dim, num_classes, s=64.0, m=0.5):
        super().__init__()
        self.s = s
        self.m = m
        self.weight = nn.Parameter(torch.FloatTensor(num_classes, embedding_dim))
        nn.init.xavier_uniform_(self.weight)

    def forward(self, embeddings, labels):
        cosine = F.linear(F.normalize(embeddings), F.normalize(self.weight))
        theta = torch.acos(cosine.clamp(-1 + 1e-7, 1 - 1e-7))
        one_hot = torch.zeros_like(cosine)
        one_hot.scatter_(1, labels.view(-1, 1), 1.0)
        target_logits = torch.cos(theta + self.m * one_hot)
        logits = self.s * target_logits
        return F.cross_entropy(logits, labels)


model = MobileFaceNet(embedding_dim=128)
params = sum(p.numel() for p in model.parameters())
print(f"MobileFaceNet parameters: {params:,}")

Common Challenges

1. Pose Variation: Extreme poses (profile, tilted) significantly degrade accuracy, requiring 3D face models or pose-invariant training
2. Illumination: Poor lighting affects feature extraction; histogram equalization and illumination-invariant loss help
3. Occlusion: Masks, sunglasses, and hair occlude facial regions, requiring partial face handling
4. Aging: Face appearance changes over years, requiring age-invariant embeddings or longitudinal models
5. Bias and Fairness: Training data imbalances cause performance disparities across demographics

Case Study: Large-Scale Deployment

A 2022 study on airport face recognition achieved 99.7% verification accuracy on 100M identities using ArcFace with ResNet-100 trained on MS1MV2 (5.8M images, 85K identities). The system processes 1,000 faces per second on 8 V100 GPUs using FAISS indexing for gallery search. The pipeline detects faces in 12ms (RetinaFace), aligns in 1ms, embeds in 8ms, and matches in 0.1ms per face. False acceptance rate at 99.9% true acceptance rate is 0.001%. The deployment required careful attention to demographic bias, with separate performance auditing showing <0.5% accuracy gap across age, gender, and ethnicity groups after targeted data collection.

Advanced Loss Functions

CosFace (Large Margin Cosine Loss)

CosFace adds a multiplicative margin to cosine similarity:

Where each parameter means:

• — scale factor (typically 64)
• — additive margin in cosine space (typically 0.35)
• — angle between embedding and class center for true class
• Intuition: By subtracting a margin from the correct class cosine score, CosFace forces the model to learn more discriminative features with tighter clustering

SphereFace (Angular Softmax)

SphereFace uses multiplicative angular margin:

Where each parameter means:

• — angular margin multiplier (typically 4)
• Intuition: By multiplying the angle by , SphereFace creates a tighter constraint in angular space; however, this can cause optimization difficulties for large

AdaFace (Adaptive Margin)

AdaFace adjusts the margin based on image quality:

Where each parameter means:

• — margin values for different quality levels
• — quality thresholds
• Intuition: High-quality images get larger margins (harder constraint), low-quality images get smaller margins (easier constraint); this prevents noisy labels from dominating training

Gallery Search and Indexing

Brute-Force Search

Brute-force search computes similarities against all gallery entries:

Where each parameter means:

• — query embedding
• — gallery embedding
• — gallery size
• Intuition: Simple but O(N) per query; practical for galleries up to 1M entries

Approximate Nearest Neighbor (ANN)

ANN methods trade accuracy for speed using indexing structures:

Where each parameter means:

• — number of nearest neighbors to retrieve
• ANN typically achieves 95%+ recall at 10x speedup
• Intuition: Methods like FAISS, HNSW, and LSH partition the embedding space to enable sub-linear search; essential for galleries with 100M+ entries

FAISS Indexing

FAISS provides efficient similarity search on GPUs:

import faiss
index = faiss.IndexIVFFlat(quantizer, d, nlist)
index.train(xb)
index.add(xb)
D, I = index.search(xq, k)

Where each parameter means:

• — embedding dimension (typically 512)
• — number of Voronoi cells (typically 1024)
• Intuition: FAISS partitions the embedding space into clusters, then searches only within nearby clusters, achieving 100x speedup over brute force

Face Anti-Spoofing

Face recognition systems must distinguish real faces from spoofs (photos, videos, masks):

Where each parameter means:

• — live/spoof prediction
• Features include texture analysis, depth estimation, and motion patterns
• Intuition: Spoofing detection analyzes subtle cues like moiré patterns (screens), lack of depth (photos), and unnatural motion (videos)

Privacy and Ethics

Differential Privacy

Differential privacy protects individual identities in face databases:

Where each parameter means:

• — randomized mechanism
• — datasets differing in one individual
• — privacy budget (smaller = more private)
• Intuition: Differential privacy guarantees that the output distribution doesn't change significantly when one person is added or removed from the database

Fairness Auditing

Fairness auditing ensures equal performance across demographics:

Where each parameter means:

• — false acceptance rate for demographic group
• Intuition: Fair systems should have similar error rates across all demographic groups; gaps > 1% indicate potential bias requiring mitigation

Key Takeaways

• Face detection uses multi-stage cascaded networks (MTCNN) or single-stage detectors (RetinaFace)
• Face alignment normalizes pose through affine transformation based on landmark detection
• Embedding networks map faces to L2-normalized vectors for similarity comparison
• Triplet loss learns discriminative embeddings by pulling same-identity pairs closer
• ArcFace adds angular margin penalty to increase inter-class separability
• Cosine similarity on normalized embeddings enables efficient gallery search
• FAISS and HNN enable sub-linear search in large face galleries