Temporal Fusion Transformers: Interpretable Multi-Horizon Forecasting

Module: Machine Learning | Difficulty: Advanced

Variable Selection Network

where

Gated Residual Network (GRN)

Multi-Horizon Forecasting

Interpretability

• Variable importance:
• Temporal attention:

import torch
import torch.nn as nn

class GatedResidualNetwork(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, dropout=0.1):
        super().__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, output_dim)
        self.gate = nn.Linear(hidden_dim, output_dim)
        self.layer_norm = nn.LayerNorm(output_dim)
        self.dropout = nn.Dropout(dropout)
    def forward(self, x, context=None):
        if context is not None:
            x = x + context
        h = torch.elu(self.fc1(x))
        h = self.dropout(h)
        output = self.fc2(h)
        gate = torch.sigmoid(self.gate(h))
        return self.layer_norm(x + gate * output)

Research Insight: TFT achieves state-of-the-art performance on electricity, traffic, and retail forecasting while providing interpretable attention patterns. The key innovation is the variable selection network, which learns which inputs are most important.