Image Inpainting and Completion

Introduction to Image Inpainting

Image inpainting is the task of reconstructing missing or corrupted regions in an image to produce a visually coherent and plausible result. Applications range from removing unwanted objects and filling in scratches on old photographs to completing partial scene constructions in augmented reality. The challenge lies in generating content that is both semantically consistent with the surrounding context and structurally coherent with the image's visual patterns.

Traditional inpainting methods used diffusion-based approaches that propagated smoothness assumptions from known regions into missing areas. These methods work well for small holes but fail for large missing regions that require semantic understanding. Modern deep learning approaches learn to hallucinate plausible content by training on large datasets of images, enabling them to fill missing regions with realistic textures and structures that match the scene context.

The evolution of inpainting methods has progressed from simple patch-based synthesis to sophisticated encoder-decoder architectures with attention mechanisms. Early methods like PatchMatch searched for similar patches in the known regions and composited them into missing areas. Modern methods use deep generative models including GANs and diffusion models to generate semantically coherent content that matches the visual style and structure of the surrounding image.

Evaluation Metrics for Inpainting

Inpainting quality is evaluated using both pixel-level metrics and perceptual quality metrics. Peak Signal-to-Noise Ratio (PSNR) measures pixel-level accuracy between the inpainted region and the ground truth, providing an objective measure of reconstruction fidelity. However, PSNR does not capture perceptual quality well, as inpainted regions may score high on PSNR while containing blurry or semantically incorrect content.

Structural Similarity Index (SSIM) measures structural similarity by comparing luminance, contrast, and structural information between inpainted and ground truth regions. SSIM provides better correlation with human perceptual judgment than PSNR for inpainting evaluation, as it captures structural consistency that is critical for visual coherence.

Learned Perceptual Image Patch Similarity (LPIPS) uses deep features from pretrained networks to measure perceptual similarity between inpainted and ground truth images. LPIPS provides the best correlation with human judgment for inpainting quality assessment, as it captures high-level semantic and perceptual information that pixel-level metrics miss.

User studies remain the gold standard for evaluating inpainting quality, as they directly measure whether the inpainted result is visually plausible to human observers. However, user studies are expensive and time-consuming, making automated metrics essential for rapid development iterations and large-scale benchmarking.

Contextual Attention Mechanism

Contextual attention, introduced by Yu et al., borrows texture features from known image regions to fill missing holes. The key insight is that natural images contain repetitive patterns—textures, structures, and patterns that repeat spatially. The attention mechanism searches for matching patches in known regions for each missing pixel location, then composites these matches to create the inpainted result.

The contextual attention operation computes soft masks that indicate how much each known pixel contributes to each missing pixel location:

Where each parameter means:

• is the output feature at missing pixel location
• is the feature at known pixel location
• is the attention weight indicating similarity between locations and
• is the set of all known pixels
• The attention weights are computed using normalized cross-correlation

The attention weights are computed by matching feature patches around missing locations to feature patches around known locations. This matching is performed at multiple scales to capture both fine textures and larger structural patterns. The propagated features are then refined through a convolutional decoder to produce the final inpainted result.

Partial Convolution

Partial convolution, introduced by Liu et al., addresses the color shift and artifact issues in standard convolution-based inpainting. The core idea is to mask the convolution operation so that only valid (known) pixels contribute to the output. After each partial convolution, the mask is updated to indicate which output pixels are valid, progressively expanding the valid region from the borders inward. This mask update mechanism ensures that the network progressively fills in missing regions while maintaining consistency with known content.

The partial convolution operation is defined as:

Where each parameter means:

• is the input feature map (with missing values)
• is the binary input mask (1 for known, 0 for unknown)
• denotes element-wise multiplication
• is the standard convolution operation
• is the rescaling mask:

The rescaling ensures that the output values are not biased by the number of valid input pixels. This approach prevents the network from relying on zero-padded missing regions, resulting in cleaner inpainting results without color shifts at mask boundaries. The mask update rule ensures that the valid region expands with each successive partial convolution layer, enabling progressive filling of large missing regions.

Diffusion-Based Inpainting

Diffusion-based inpainting methods generate high-quality completions through iterative denoising processes. These methods reverse a forward diffusion process that gradually adds noise to images, learning to denoise while conditioning on known regions. The inpainting is formulated as a conditional generation problem where the known pixels constrain the generation process. This approach produces diverse, high-quality completions that respect the known context while generating plausible content for missing areas.

Diffusion models have achieved state-of-the-art results on inpainting benchmarks, surpassing GAN-based methods in both perceptual quality and diversity. The iterative nature of diffusion sampling allows for controllable generation through guidance mechanisms, enabling users to trade off between sample quality and diversity. Recent advances like Denoising Diffusion Probabilistic Models (DDPM) and Latent Diffusion Models (LDM) have made diffusion-based inpainting more practical by reducing the number of sampling steps required while maintaining high output quality.

The denoising process at each step can be formulated as:

Where each parameter means:

• is the noisy image at timestep
• is the denoised image at the previous timestep
• is the learned mean prediction from the denoising network
• is the noise schedule at step
• is random Gaussian noise

For inpainting, the known regions are kept fixed while the missing regions are iteratively denoised. This produces diverse, high-quality completions that respect the known context while generating plausible content for missing areas.

Python Implementation: Simple Diffusion Inpainting

import torch
import torch.nn as nn
import torch.nn.functional as F


class SimpleUNet(nn.Module):
    def __init__(self, in_channels=4, out_channels=3):
        super(SimpleUNet, self).__init__()
        self.enc1 = self.conv_block(in_channels, 64)
        self.enc2 = self.conv_block(64, 128)
        self.enc3 = self.conv_block(128, 256)
        self.pool = nn.MaxPool2d(2)
        self.bottleneck = self.conv_block(256, 512)
        self.up3 = nn.ConvTranspose2d(512, 256, 2, 2)
        self.dec3 = self.conv_block(512, 256)
        self.up2 = nn.ConvTranspose2d(256, 128, 2, 2)
        self.dec2 = self.conv_block(256, 128)
        self.up1 = nn.ConvTranspose2d(128, 64, 2, 2)
        self.dec1 = self.conv_block(128, 64)
        self.out = nn.Conv2d(64, out_channels, 1)

    def conv_block(self, in_ch, out_ch):
        return nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, 1, 1),
            nn.GroupNorm(8, out_ch),
            nn.SiLU(),
            nn.Conv2d(out_ch, out_ch, 3, 1, 1),
            nn.GroupNorm(8, out_ch),
            nn.SiLU()
        )

    def forward(self, x, t_emb):
        t_emb = t_emb.view(-1, 1, 1, 1).expand_as(x[:, :1])
        x = torch.cat([x, t_emb], dim=1)
        e1 = self.enc1(x)
        e2 = self.enc2(self.pool(e1))
        e3 = self.enc3(self.pool(e2))
        b = self.bottleneck(self.pool(e3))
        d3 = self.dec3(torch.cat([self.up3(b), e3], dim=1))
        d2 = self.dec2(torch.cat([self.up2(d3), e2], dim=1))
        d1 = self.dec1(torch.cat([self.up1(d2), e1], dim=1))
        return self.out(d1)


def inpaint_with_diffusion(model, image, mask, num_steps=50):
    device = image.device
    x = torch.randn_like(image)
    for t in range(num_steps - 1, -1, -1):
        t_tensor = torch.full((x.size(0),), t / num_steps, device=device)
        pred_noise = model(x, t_tensor)
        alpha = 1 - (t + 1) / num_steps
        x = (x - (1 - alpha) * pred_noise) / torch.sqrt(alpha)
        x = mask * image + (1 - mask) * x
    return x.clamp(-1, 1)

Common Challenges

1. Large Hole Completion: Filling large missing regions requires generating plausible content from limited context, leading to potential semantic inconsistencies. When more than 50% of an image is missing, the network must hallucinate entire objects or scenes, making the task increasingly ill-posed.

2. Texture and Structure Separation: Distinguishing between texture synthesis (repeating patterns) and structure completion (edges, boundaries) requires different inpainting strategies. Dual-branch architectures that separately handle texture and structure have shown improved results for complex scenes.

3. Complex Scene Understanding: Inpainting semantic objects requires understanding the scene context to generate plausible content that matches the scene's visual semantics. For example, filling a missing region in a beach scene should produce sand, water, or sky depending on spatial location.

4. Edge Artifact Removal: Maintaining smooth transitions between inpainted and original regions without visible seams or color discontinuities requires careful post-processing or specialized network designs like feature blending layers.

5. Real-time Processing: Many high-quality inpainting methods are computationally expensive, making real-time applications challenging. Lightweight architectures and model distillation techniques help bridge this gap for mobile and web deployment.

6. Temporal Consistency for Video: Video inpainting requires maintaining temporal coherence across frames to avoid flickering artifacts. Flow-guided approaches that propagate information across frames address this challenge but add computational complexity and require accurate optical flow estimation.

7. Evaluation Subjectivity: Quantitative metrics like PSNR and SSIM do not always correlate with human perceptual quality for inpainting. User studies and learned perceptual metrics like LPIPS provide better evaluation but are more expensive to compute and require careful experimental design.

8. Memory Constraints: Processing high-resolution images with large missing regions requires significant GPU memory. Patch-based processing and memory-efficient attention mechanisms help address scalability constraints for production deployment.

Case Study: Photo Restoration Service

A photo restoration company deployed DeepFill v2 for automated inpainting of damaged photographs. The system processes scanned photos with scratches, tears, and missing sections. On a dataset of 10,000 damaged photos, the automated system achieved an average SSIM of 0.89 compared to manual restoration by experts. Processing time reduced from 15 minutes per photo (manual) to 3 seconds (automated). Customer satisfaction surveys indicated that 78% of users found the automated results comparable to manual restoration. The system handles various damage types including scratches (42%), tears (31%), water damage (18%), and faded areas (9%). The deployment enabled the company to scale their restoration services from processing 500 photos per month to over 10,000 photos per month, generating a 45% increase in revenue while maintaining consistent quality standards. The inpainting system also integrates with automatic damage detection, creating an end-to-end pipeline that identifies damaged regions before filling them with plausible content. Quality assurance metrics show that 92% of restored photos pass the company's internal quality threshold without manual review.

Key Takeaways

• Image inpainting reconstructs missing regions using contextual information from known pixels
• Contextual attention borrows texture features from known regions for pattern completion
• Partial convolution prevents color shifts by only processing valid pixels
• Diffusion-based methods produce diverse, high-quality completions through iterative denoising
• The choice of inpainting method depends on mask size, quality requirements, and speed constraints
• Modern methods like LaMa achieve state-of-art results on large irregular masks
• Diffusion-based approaches provide diverse completions but require iterative sampling
• Partial convolution prevents color shift artifacts by processing only valid pixels
• Evaluation combines quantitative metrics with human perceptual assessment
• Encoder-decoder architectures with skip connections enable high-quality inpainting
• GAN-based methods produce realistic textures while diffusion methods offer diverse outputs
• Patch-based processing enables handling of high-resolution images beyond GPU memory limits
• Partial convolution prevents color shifts by normalizing convolution outputs per valid pixels
• Contextual attention propagates texture information from known regions to missing areas
• LaMa uses Fourier convolutions for large receptive field inpainting without artifacts
• Diffusion models provide state-of-art diversity but require many sampling steps for quality