Compositional Visual Reasoning and Scene Graphs
Module: Computer Vision | Difficulty: Premium
Scene Graph
Relationship Detection
Graph Neural Network Update
| Model | VG Recall@50 | VG Recall@100 | GQA | Approach | |-------|-------------|---------------|-----|----------| | IMP | 44.8% | 54.1% | 53.2% | Message passing | | TRR | 50.8% | 60.2% | 56.1% | Tensorized | | GPSNet | 53.2% | 63.4% | 58.3% | Graph | | ParallelNet | 55.1% | 65.7% | 60.2% | Parallel |
import torch
import torch.nn as nn
import torch.nn.functional as F
class SceneGraphGenerator(nn.Module):
def __init__(self, num_objects, num_relations, hidden_dim=256):
super().__init__()
self.obj_embed = nn.Embedding(num_objects, hidden_dim)
self.rel_embed = nn.Embedding(num_relations, hidden_dim)
self.gnn = nn.ModuleList([
GraphConvLayer(hidden_dim) for _ in range(3)])
self.obj_predictor = nn.Linear(hidden_dim, num_objects)
self.rel_predictor = nn.Sequential(
nn.Linear(hidden_dim * 3, hidden_dim),
nn.ReLU(inplace=True),
nn.Linear(hidden_dim, num_relations),
)
def forward(self, obj_features, adjacency):
h = self.obj_embed(obj_features)
for layer in self.gnn:
h = layer(h, adjacency)
obj_pred = self.obj_predictor(h)
return obj_pred
class GraphConvLayer(nn.Module):
def __init__(self, hidden_dim):
super().__init__()
self.linear = nn.Linear(hidden_dim, hidden_dim)
self.aggregate = nn.Linear(hidden_dim, hidden_dim)
def forward(self, node_features, adjacency):
messages = self.linear(node_features)
aggregated = torch.bmm(adjacency, messages)
return F.relu(self.aggregate(node_features + aggregated))
Research Insight: Visual reasoning has evolved from pipeline-based approaches (detect objects, then predict relationships) to end-to-end graph neural networks that jointly optimize object detection and relationship prediction. The key challenge is compositional generalization: models must understand novel combinations of known objects and relationships (e.g., "a horse riding a person" when trained on "a person riding a horse"). Scene graph generation enables downstream tasks like visual question answering and image captioning by providing structured representations of visual content.