Video Object Segmentation

Module: Computer Vision | Difficulty: Advanced

Semi-Supervised VOS

Given first-frame mask , segment object in all subsequent frames.

Mask Propagation

Appearance Model

Space-Time Memory Network

where keys and values come from memory frames.

Unsupervised VOS

import torch
import torch.nn as nn

class MaskPropagation(nn.Module):
    def __init__(self):
        super().__init__()
        self.fnet = nn.Sequential(
            nn.Conv2d(6, 64, 3, 1, 1), nn.ReLU(True),
            nn.Conv2d(64, 128, 3, 2, 1), nn.ReLU(True),
            nn.Conv2d(128, 256, 3, 2, 1), nn.ReLU(True),
        )
        self.correlation = nn.Conv2d(256, 1, 1)
    
    def forward(self, prev_mask, curr_frame, prev_frame):
        inp = torch.cat([prev_frame, curr_frame], dim=1)
        feat = self.fnet(inp)
        corr = self.correlation(feat)
        prop_mask = nn.functional.interpolate(
            prev_mask, size=corr.shape[2:], mode='bilinear'
        )
        return prop_mask * torch.sigmoid(corr)

Key Takeaways

• Semi-supervised VOS is well-studied with strong baselines
• Space-time memory enables long-range temporal reasoning
• Unsupervised methods rely on saliency and motion cues