Neural Style Transfer

Module: Computer Vision | Difficulty: Intermediate

Style Transfer Objective

Content Loss

where and are feature maps at layer .

Style Loss (Gram Matrix)

Real-Time Style Transfer

Feed-forward network trained per style:

import torch
import torch.nn as nn
import torchvision.models as models

def style_loss(gen_feat, style_feat):
    B, C, H, W = gen_feat.size()
    G = gen_feat.view(B, C, H*W).bmm(gen_feat.view(B, C, H*W).transpose(1, 2))
    G = G / (C * H * W)
    S = style_feat.view(B, C, H*W).bmm(style_feat.view(B, C, H*W).transpose(1, 2))
    S = S / (C * H * W)
    return nn.functional.mse_loss(G, S)

def content_loss(gen_feat, content_feat):
    return nn.functional.mse_loss(gen_feat, content_feat)

vgg = models.vgg19(pretrained=True).features.eval()
style_layers = ['3', '8', '17', '26', '35']
content_layer = '21'

Key Takeaways

• Style is captured by Gram matrices of feature correlations
• Content is captured by spatial feature activations
• Real-time methods use feed-forward networks for 1000x speedup