Modern Time Series Forecasting: Transformers and Foundation Models

Module: Machine Learning | Difficulty: Advanced

Informer: ProbSparse Attention

Complexity reduced from to via attention sparsification.

Autoformer: Autocorrelation

PatchTST

Patching: split time series into patches, treat as tokens.

Foundation Models

Pre-trained on large corpora, fine-tuned for specific tasks.

import torch
import torch.nn as nn

class PatchTST(nn.Module):
    def __init__(self, patch_len=16, d_model=128, nhead=8, num_layers=3):
        super().__init__()
        self.patch_len = patch_len
        self.patch_embed = nn.Linear(patch_len, d_model)
        encoder_layer = nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward=d_model*4, batch_first=True)
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers)
        self.head = nn.Linear(d_model, 1)
    def forward(self, x):
        B, L = x.shape
        x = x.unfold(1, self.patch_len, self.patch_len)
        x = self.patch_embed(x)
        x = self.transformer(x)
        x = self.head(x.mean(1))
        return x

Research Insight: PatchTST shows that channel independence (treating each variable separately) outperforms channel mixing for multivariate forecasting. The key insight is that most real-world time series have weak cross-channel dependencies.