Instance Segmentation with Mask R-CNN

Module: Computer Vision | Difficulty: Advanced

Instance Segmentation as Multi-Task Learning

Instance segmentation combines object detection (localization + classification) with semantic segmentation (pixel-wise masking) to produce a complete per-instance understanding of an image. Each detected object receives a bounding box, class label, and pixel-level mask, enabling precise spatial reasoning about individual objects.

Mask R-CNN extends Faster R-CNN by adding a parallel mask prediction branch to the detection head. The key innovation is RoIAlign, which eliminates the quantization error introduced by RoI pooling, enabling accurate mask prediction. This simple modification improves mask quality by 10-15% while maintaining detection accuracy.

RoIAlign

RoIAlign uses bilinear interpolation to compute exact feature values at each sampling point within an RoI:

Where each parameter means:

• — input feature map
• — region of interest (bounding box coordinates)
• — sampling points within the RoI (typically 7×7 grid)
• Intuition: Unlike RoI pooling which quantizes to integer coordinates, RoIAlign computes exact feature values through bilinear interpolation, preserving spatial alignment for accurate mask prediction

Mask Prediction

The mask head predicts a binary mask for each detected object:

Where each parameter means:

• — predicted binary mask for instance
• — sigmoid activation (per-pixel probability)
• — four 3×3 convolution layers with 256 channels
• — RoI coordinates for instance
• Intuition: The mask is predicted independently for each class, avoiding competition between classes; during inference, the mask is multiplied by the classification score

Mask Loss

The mask loss is computed only for the ground-truth class:

Where each parameter means:

• — number of instances in the batch
• — predicted mask (28×28×C tensor)
• — ground-truth binary mask (28×28)
• — binary cross-entropy loss
• Intuition: Only the mask corresponding to the predicted class is used for loss computation; this decouples mask prediction from classification

Feature Pyramid Network Integration

Mask R-CNN leverages FPN to detect objects at multiple scales. Each RoI is assigned to the appropriate FPN level based on its area:

Where each parameter means:

• — FPN level assigned to the RoI
• — base level (4 for standard implementation)
• — width and height of the RoI
• 224 — reference image size (ImageNet training resolution)
• Intuition: Small objects are assigned to higher-resolution FPN levels, large objects to lower-resolution levels, ensuring appropriate feature granularity for each scale

Instance Segmentation Comparison

Complete Mask R-CNN Inference

import torch
import torch.nn as nn
import torchvision.ops as ops


class MaskRCNNHead(nn.Module):
    def __init__(self, in_channels, num_classes, mask_size=28):
        super().__init__()
        self.num_classes = num_classes
        self.mask_size = mask_size
        self.conv1 = nn.Conv2d(in_channels, 256, 3, padding=1)
        self.conv2 = nn.Conv2d(256, 256, 3, padding=1)
        self.conv3 = nn.Conv2d(256, 256, 3, padding=1)
        self.conv4 = nn.Conv2d(256, 256, 3, padding=1)
        self.deconv = nn.ConvTranspose2d(256, 256, 2, stride=2)
        self.mask_pred = nn.Conv2d(256, num_classes, 1)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, features):
        x = self.relu(self.conv1(features))
        x = self.relu(self.conv2(x))
        x = self.relu(self.conv3(x))
        x = self.relu(self.conv4(x))
        x = self.relu(self.deconv(x))
        return self.mask_pred(x)


def roi_align(features, boxes, output_size=7):
    return ops.roi_align(features, boxes, output_size,
                         spatial_scale=1.0/16, sampling_ratio=2)


def mask_rcnn_inference(backbone, fpn, rpn, mask_head, image, proposals):
    feature_maps = backbone(image)
    fpn_features = fpn(feature_maps)
    rpn_scores, rpn_boxes = rpn(fpn_features)
    keep = ops.nms(rpn_boxes, rpn_scores, iou_threshold=0.7)
    proposals = rpn_boxes[keep[:2000]]
    roi_features = roi_align(fpn_features, [proposals])
    mask_logits = mask_head(roi_features)
    masks = torch.sigmoid(mask_logits)
    scores = rpn_scores[keep[:2000]]
    return proposals, scores, masks


class SimpleMaskRCNN(nn.Module):
    def __init__(self, num_classes=81):
        super().__init__()
        self.backbone = nn.Sequential(
            nn.Conv2d(3, 64, 7, 2, 3), nn.ReLU(inplace=True),
            nn.MaxPool2d(3, 2, 1),
            nn.Conv2d(64, 128, 3, 1, 1), nn.ReLU(inplace=True),
            nn.MaxPool2d(2)
        )
        self.rpn = nn.Conv2d(128, 18, 3, padding=1)
        self.mask_head = MaskRCNNHead(128, num_classes)

    def forward(self, x):
        features = self.backbone(x)
        rpn_out = self.rpn(features)
        return features, rpn_out, self.mask_head(features)


model = SimpleMaskRCNN(num_classes=81)
params = sum(p.numel() for p in model.parameters())
print(f"Mask R-CNN parameters: {params:,}")

Real-Time Instance Segmentation

YOLACT (2019)

YOLACT generates instance masks in real-time by predicting a set of prototype masks and per-instance coefficients:

Where each parameter means:

• — prototype mask matrix (k x H x W)
• — per-instance coefficient vector (k)
• — sigmoid activation
• Intuition: Instead of predicting masks directly, YOLACT learns a dictionary of mask prototypes and combines them linearly; this is fast because mask generation is a simple matrix multiplication

SOLO (2020)

SOLO segments instances by predicting masks at each grid location:

Where each parameter means:

• — feature vector at grid position
• — learned projection matrix
• Intuition: Each grid cell predicts its own mask, with instance identity determined by grid position; this eliminates the need for proposal-based detection

Mask Scoring

Mask scoring refines mask quality by predicting the IoU between predicted and ground-truth masks:

Where each parameter means:

• — predicted mask for instance
• — RoI feature vector
• — element-wise multiplication
• Intuition: Not all predicted masks are equally accurate; mask scoring learns to predict mask quality, improving AP by re-ranking detections by mask quality

Common Challenges

1. Mask Quality vs Speed Trade-off: High-quality masks require larger mask heads and more computation, limiting real-time applications
2. Small Object Detection: Small objects produce small RoIs that may not have sufficient features for accurate mask prediction
3. Overlapping Instances: Heavily overlapping objects create ambiguous boundaries, requiring sophisticated NMS or learned mask scoring
4. Memory Constraints: Storing per-instance masks for large batches requires significant GPU memory
5. Annotation Noise: Mask annotations may have inconsistencies that propagate through training

Case Study: COCO Instance Segmentation

The COCO dataset (2017 version) contains 118K training images with 860K annotated instances across 80 classes. Mask R-CNN with ResNet-101-FPN achieves 37.0 mask AP on the test-dev set after 360K iterations of training on 8 GPUs. Training requires approximately 3 days on 8 V100 GPUs, using synchronized batch normalization and learning rate warmup. The most challenging classes are small objects (toaster: 20.1 AP) and deformable objects (person: 52.1 AP). The state-of-the-art Mask2Former achieves 50.1 mask AP using a Swin-L backbone with a transformer decoder, demonstrating the advantage of attention mechanisms for instance segmentation.

Mask R-CNN Training Details

Multi-Task Loss

Mask R-CNN jointly optimizes three losses:

Where each parameter means:

• — classification loss (cross-entropy over K+1 classes)
• — bounding box regression loss (smooth L1)
• — mask prediction loss (binary cross-entropy)
• Intuition: The three losses are equally weighted; the mask loss is only applied to the ground-truth class for each RoI, preventing competition between classes

RoIAlign Mathematical Formulation

RoIAlign uses bilinear interpolation to compute exact feature values at sampling points within an RoI:

Where each parameter means:

• — input feature map
• — region of interest (bounding box coordinates)
• — sampling points within the RoI (typically 7x7 grid)
• Intuition: Unlike RoI pooling which quantizes to integer coordinates, RoIAlign computes exact feature values through bilinear interpolation, preserving spatial alignment for accurate mask prediction

Mask Prediction Architecture

The mask head predicts a binary mask for each detected object:

Where each parameter means:

• — predicted binary mask for instance
• — sigmoid activation (per-pixel probability)
• — four 3x3 convolution layers with 256 channels
• — RoI coordinates for instance
• Intuition: The mask is predicted independently for each class, avoiding competition between classes; during inference, the mask is multiplied by the classification score

Feature Pyramid Network Integration

Mask R-CNN leverages FPN to detect objects at multiple scales. Each RoI is assigned to the appropriate FPN level based on its area:

Where each parameter means:

• — FPN level assigned to the RoI
• — base level (4 for standard implementation)
• — width and height of the RoI
• 224 — reference image size (ImageNet training resolution)
• Intuition: Small objects are assigned to higher-resolution FPN levels, large objects to lower-resolution levels, ensuring appropriate feature granularity for each scale

Cascade Mask R-CNN

Cascade Mask R-CNN extends Mask R-CNN with a cascade of detection heads, each operating at different IoU thresholds:

Where each parameter means:

• Stage 1 — detects all objects with low IoU threshold (high recall)
• Stage 2 — refines proposals with medium IoU threshold
• Stage 3 — produces final detections with high IoU threshold (high precision)
• Intuition: The cascade progressively increases detection quality, with each stage trained on harder positives and negatives; this improves AP by 2-3% over single-stage Mask R-CNN

Instance Segmentation Evaluation

Average Precision (AP)

AP measures the area under the precision-recall curve:

Where each parameter means:

• — precision (TP / (TP + FP))
• — recall (TP / (TP + FN))
• Intuition: AP summarizes the trade-off between precision and recall across all confidence thresholds; AP@50 uses IoU=0.5, AP@75 uses IoU=0.75, AP@[.5:.95] averages over multiple thresholds

Panoptic Quality

PQ unifies segmentation and detection quality:

Where each parameter means:

• — segmentation quality (mean IoU of matched pairs)
• — recognition quality (F1-score of detection)
• Intuition: PQ balances how well segments match their ground truth (SQ) against how many objects are correctly detected (RQ)

Key Takeaways

• Mask R-CNN extends Faster R-CNN with a parallel mask prediction branch
• RoIAlign eliminates quantization error through bilinear interpolation
• The mask head predicts per-class binary masks independently
• Feature Pyramid Networks enable multi-scale instance detection
• Mask loss is computed only for the ground-truth class to avoid interference
• Modern transformer-based approaches achieve superior performance but with higher compute
• Cascade architectures progressively refine detection quality through multiple stages