Sequence-to-Sequence Models for Image Description

Module: Computer Vision | Difficulty: Premium

Show and Tell Loss

CIDEr Score

where is the TF-IDF weighted n-gram vector.

Coverage Penalty

| Model | BLEU-4 | CIDEr | METEOR | Approach | |-------|--------|-------|--------|----------| | Show and Tell | 32.1 | 108.5 | 25.7 | LSTM + CNN | | Show Attend Tell | 34.4 | 115.3 | 27.0 | Spatial attention | | Up-Down | 36.9 | 120.1 | 28.4 | Bottom-up + Top-down | | OFA | 41.0 | 146.1 | 32.5 | Unified transformer |

import torch
import torch.nn as nn
import torch.nn.functional as F

class CaptioningAttention(nn.Module):
    def __init__(self, v_dim, q_dim, hidden_dim):
        super().__init__()
        self.v_proj = nn.Linear(v_dim, hidden_dim)
        self.q_proj = nn.Linear(q_dim, hidden_dim)
        self.tanh = nn.Tanh()
        self.out = nn.Linear(hidden_dim, 1)

    def forward(self, v, q):
        v_att = self.v_proj(v)
        q_att = self.q_proj(q).unsqueeze(1)
        score = self.out(self.tanh(v_att + q_att)).squeeze(-1)
        alpha = F.softmax(score, dim=1)
        context = (alpha.unsqueeze(-1) * v).sum(dim=1)
        return context, alpha

class CaptionGenerator(nn.Module):
    def __init__(self, vocab_size, embed_dim=256,
                 v_dim=2048, hidden_dim=512):
        super().__init__()
        self.v_embed = nn.Linear(v_dim, hidden_dim)
        self.attention = CaptioningAttention(v_dim, hidden_dim, hidden_dim)
        self.word_embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTMCell(embed_dim + hidden_dim, hidden_dim)
        self.fc_out = nn.Linear(hidden_dim, vocab_size)

    def forward(self, features, captions=None, max_len=20):
        B = features.shape[0]
        v = self.v_embed(features)
        h = torch.zeros(B, v.shape[-1]).to(v.device)
        c = torch.zeros(B, v.shape[-1]).to(v.device)
        outputs = []
        for t in range(max_len):
            context, alpha = self.attention(v, h)
            if captions is not None:
                word = self.word_embed(captions[:, t])
            else:
                word = self.word_embed(
                    torch.tensor([1] * B).to(v.device))
            lstm_input = torch.cat([word, context], dim=1)
            h, c = self.lstm(lstm_input, (h, c))
            outputs.append(self.fc_out(h))
        return torch.stack(outputs, dim=1)

Research Insight: Image captioning has evolved from template-based approaches to neural encoder-decoder models, and now to vision-language foundation models. The key shift is from task-specific architectures to general-purpose multimodal models that can caption, answer questions, and reason about images. Metrics like CIDEr and METEOR capture n-gram overlap but fail to assess semantic quality; newer metrics (CLIPScore, PAC-S) leverage vision-language models for more meaningful evaluation. Controllable captioning (varying detail level, style) is an emerging direction.