Compositional Visual Reasoning and Scene Graphs

Module: Computer Vision | Difficulty: Premium

Scene Graph

Relationship Detection

Graph Neural Network Update

| Model | VG Recall@50 | VG Recall@100 | GQA | Approach | |-------|-------------|---------------|-----|----------| | IMP | 44.8% | 54.1% | 53.2% | Message passing | | TRR | 50.8% | 60.2% | 56.1% | Tensorized | | GPSNet | 53.2% | 63.4% | 58.3% | Graph | | ParallelNet | 55.1% | 65.7% | 60.2% | Parallel |

import torch
import torch.nn as nn
import torch.nn.functional as F

class SceneGraphGenerator(nn.Module):
    def __init__(self, num_objects, num_relations, hidden_dim=256):
        super().__init__()
        self.obj_embed = nn.Embedding(num_objects, hidden_dim)
        self.rel_embed = nn.Embedding(num_relations, hidden_dim)
        self.gnn = nn.ModuleList([
            GraphConvLayer(hidden_dim) for _ in range(3)])
        self.obj_predictor = nn.Linear(hidden_dim, num_objects)
        self.rel_predictor = nn.Sequential(
            nn.Linear(hidden_dim * 3, hidden_dim),
            nn.ReLU(inplace=True),
            nn.Linear(hidden_dim, num_relations),
        )

    def forward(self, obj_features, adjacency):
        h = self.obj_embed(obj_features)
        for layer in self.gnn:
            h = layer(h, adjacency)
        obj_pred = self.obj_predictor(h)
        return obj_pred

class GraphConvLayer(nn.Module):
    def __init__(self, hidden_dim):
        super().__init__()
        self.linear = nn.Linear(hidden_dim, hidden_dim)
        self.aggregate = nn.Linear(hidden_dim, hidden_dim)

    def forward(self, node_features, adjacency):
        messages = self.linear(node_features)
        aggregated = torch.bmm(adjacency, messages)
        return F.relu(self.aggregate(node_features + aggregated))

Research Insight: Visual reasoning has evolved from pipeline-based approaches (detect objects, then predict relationships) to end-to-end graph neural networks that jointly optimize object detection and relationship prediction. The key challenge is compositional generalization: models must understand novel combinations of known objects and relationships (e.g., "a horse riding a person" when trained on "a person riding a horse"). Scene graph generation enables downstream tasks like visual question answering and image captioning by providing structured representations of visual content.