Panoptic Segmentation

Module: Computer Vision | Difficulty: Advanced

Panoptic Segmentation

Combines semantic (stuff) and instance (things) segmentation.

Panoptic Quality

Panoptic FPN

MaskFormer

Transformer-based unified segmentation:

import torch
import torch.nn as nn

class PanopticHead(nn.Module):
    def __init__(self, in_ch, num_classes, num_queries=100):
        super().__init__()
        self.semantic = nn.Conv2d(in_ch, num_classes, 1)
        self.instance = nn.Sequential(
            nn.Conv2d(in_ch, 256, 3, padding=1),
            nn.ReLU(True),
            nn.Conv2d(256, num_queries, 1),
        )
        self.merge = nn.Conv2d(num_classes + num_queries, num_classes, 1)
    
    def forward(self, x):
        sem = self.semantic(x)
        inst = torch.sigmoid(self.instance(x))
        return self.merge(torch.cat([sem, inst], dim=1))

Key Takeaways

• Panoptic FPN unifies semantic and instance heads
• MaskFormer treats all segmentation as mask classification
• PQ is the standard metric combining quality and recognition