Skeleton-Based Action Recognition

Module: Computer Vision | Difficulty: Advanced

Skeleton Graph

Vertices: , Edges:

Spatial Graph Convolution

where is the neighbors of and is a normalization factor.

Temporal Convolution

ST-GCN Block

import torch
import torch.nn as nn

class GraphConv(nn.Module):
    def __init__(self, in_ch, out_ch, A):
        super().__init__()
        self.register_buffer('A', A)
        self.conv = nn.Conv2d(in_ch, out_ch, 1)
        self.bn = nn.BatchNorm2d(out_ch)
    
    def forward(self, x):
        # x: (B, C, T, V)
        B, C, T, V = x.shape
        out = torch.einsum('bctv,vw->bctw', x, self.A)
        out = self.conv(out)
        return self.bn(out)

class STGCNBlock(nn.Module):
    def __init__(self, in_ch, out_ch, A, kernel=9):
        super().__init__()
        self.gcn = GraphConv(in_ch, out_ch, A)
        self.tcn = nn.Sequential(
            nn.Conv2d(out_ch, out_ch, (kernel, 1), padding=(kernel//2, 0)),
            nn.BatchNorm2d(out_ch),
        )
        self.relu = nn.ReLU(True)
    
    def forward(self, x):
        return self.relu(self.tcn(self.gcn(x)) + nn.Conv2d(x.size(1), x.size(1), 1)(x))

Key Takeaways

• Skeleton graphs model body structure for action recognition
• ST-GCN jointly captures spatial and temporal dynamics
• GCN-based methods are efficient and interpretable