Graph Attention Networks: Attention on Graphs

Module: Machine Learning | Difficulty: Advanced

Graph Attention (GAT)

Attention Coefficients

Multi-Head Attention

Comparison

| Method | Aggregation | Expressiveness | Parameters | |--------|-------------|----------------|------------| | GCN | Mean | 1-WL | Low | | GAT | Attention | 1-WL + features | Medium | | GraphSAGE | Sampled mean | 1-WL | Medium | | GIN | Sum | 1-WL equivalent | Low |

import torch
import torch.nn as nn
from torch_geometric.nn import GATConv

class GAT(nn.Module):
    def __init__(self, in_dim, hidden, out_dim, heads=8):
        super().__init__()
        self.conv1 = GATConv(in_dim, hidden, heads=heads, concat=False)
        self.conv2 = GATConv(hidden, out_dim, heads=1)
    def forward(self, x, edge_index):
        x = torch.relu(self.conv1(x, edge_index))
        x = self.conv2(x, edge_index)
        return x

Research Insight: GAT's attention mechanism allows the model to learn different importance weights for different neighbors, unlike GCN which uses fixed weights based on node degree. However, GAT does not increase the expressive power beyond 1-WL test.