Image Inpainting

Module: Computer Vision | Difficulty: Intermediate

Inpainting Objective

Fill missing region given observed pixels:

Context Encoder Loss

Partial Convolution

Only convolve over valid pixels:

Perceptual Loss

import torch
import torch.nn as nn

class InpaintingNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(4, 64, 4, 2, 1), nn.ReLU(True),
            nn.Conv2d(64, 128, 4, 2, 1), nn.ReLU(True),
            nn.Conv2d(128, 256, 4, 2, 1), nn.ReLU(True),
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(256, 128, 4, 2, 1), nn.ReLU(True),
            nn.ConvTranspose2d(128, 64, 4, 2, 1), nn.ReLU(True),
            nn.ConvTranspose2d(64, 3, 4, 2, 1), nn.Tanh(),
        )
    
    def forward(self, x, mask):
        masked = x * (1 - mask)
        inp = torch.cat([masked, mask], dim=1)
        latent = self.encoder(inp)
        return self.decoder(latent) * mask + x * (1 - mask)

Key Takeaways

• Partial convolution ensures valid convolution at mask boundaries
• Perceptual loss preserves semantic content
• Diffusion-based inpainting achieves photorealistic results