Referring Expression Comprehension and Visual Localization

Module: Computer Vision | Difficulty: Premium

Referring Expression Comprehension

Cross-Modal Attention

GIoU Loss for Localization

where is the smallest enclosing box.

| Model | RefCOCO | RefCOCO+ | RefCOCOg | Approach | |-------|---------|----------|----------|----------| | MAttNet | 76.4% | 62.3% | 65.8% | Modular attention | | UNITER | 81.4% | 71.3% | 74.6% | BERT + ViT | | VL-BERT | 84.5% | 74.2% | 78.1% | Unified BERT | | OFA | 89.6% | 82.7% | 84.3% | Unified transformer |

import torch
import torch.nn as nn
import torch.nn.functional as F

class CrossModalAttention(nn.Module):
    def __init__(self, v_dim, t_dim, hidden_dim, num_heads=8):
        super().__init__()
        self.v_proj = nn.Linear(v_dim, hidden_dim)
        self.t_proj = nn.Linear(t_dim, hidden_dim)
        self.multihead_attn = nn.MultiheadAttention(
            hidden_dim, num_heads, batch_first=True)
        self.out_proj = nn.Linear(hidden_dim, hidden_dim)

    def forward(self, visual_features, text_features):
        v = self.v_proj(visual_features)
        t = self.t_proj(text_features)
        attn_output, attn_weights = self.multihead_attn(
            query=t, key=v, value=v)
        return self.out_proj(attn_output), attn_weights

class GroundingModel(nn.Module):
    def __init__(self, v_dim=2048, t_dim=768, hidden_dim=256):
        super().__init__()
        self.v_proj = nn.Linear(v_dim, hidden_dim)
        self.t_proj = nn.Linear(t_dim, hidden_dim)
        self.cross_attn = CrossModalAttention(v_dim, t_dim, hidden_dim)
        self.box_predictor = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(inplace=True),
            nn.Linear(hidden_dim, 4),
            nn.Sigmoid(),
        )

    def forward(self, regions, text_features):
        attended, attn_weights = self.cross_attn(
            regions, text_features)
        box_feats = attended.mean(dim=1)
        boxes = self.box_predictor(box_feats)
        return boxes, attn_weights

Research Insight: Visual grounding has benefited enormously from pretrained vision-language models. Grounding DINO combines DETR-style detection with open-vocabulary features, enabling language-guided detection of arbitrary objects without task-specific training. The key challenge is compositional grounding: understanding complex expressions like "the red cup next to the blue plate" requires spatial reasoning beyond simple attribute binding. Current models struggle with negation, counting, and relational reasoning, pointing to the need for structured scene representations.