Implicit Neural Representations

Module: Generative AI | Difficulty: Advanced

SIREN (Sinusoidal Representation Networks)

where

Fourier Features

where

Spectral Bias

Neural networks have difficulty learning high-frequency components:

Fourier features overcome this by mapping inputs to higher frequencies.

import torch, torch.nn as nn, math

class SIREN(nn.Module):
    def __init__(self, in_dim, hidden_dim, out_dim, n_layers):
        super().__init__()
        self.first = nn.Linear(in_dim, hidden_dim)
        self.layers = nn.ModuleList([nn.Linear(hidden_dim, hidden_dim) for _ in range(n_layers-1)])
        self.out = nn.Linear(hidden_dim, out_dim)
        nn.init.uniform_(self.first.weight, -1/math.sqrt(in_dim), 1/math.sqrt(in_dim))
    def forward(self, x):
        x = torch.sin(self.first(x))
        for layer in self.layers:
            x = torch.sin(layer(x))
        return self.out(x)

Research Insight: SIREN's periodic activations enable representing sharp edges and high-frequency details that ReLU networks cannot. This is because periodic functions are eigenfunctions of the Laplacian.