Image Captioning

Image Captioning Overview

Image captioning is a fundamental vision-language task that generates coherent textual descriptions from images. The process requires understanding visual content at multiple levels from objects and attributes to relationships and scenes, then translating this understanding into fluent natural language. Modern approaches use encoder-decoder architectures where visual features from CNNs are decoded into word sequences through attention-guided generation.

Theory: Encoder-Decoder Architecture

The encoder-decoder framework is the foundation of modern image captioning systems. The encoder extracts rich visual representations from the input image using pretrained convolutional networks, typically ResNet or EfficientNet, producing spatial feature maps that encode objects, attributes, and scene context. These features serve as the visual memory that the decoder references during caption generation.

The decoder generates captions word-by-word conditioned on both the visual features and previously generated words. LSTM or Transformer decoders maintain a hidden state that accumulates context from the visual input and the linguistic history. At each timestep, the decoder attends to relevant spatial regions, integrating visual information with the evolving linguistic context to predict the next word in the sequence.

Teacher forcing is used during training where the ground truth word is fed as input at each timestep rather than the model prediction. This stabilizes training by preventing error accumulation but creates a mismatch between training and inference known as exposure bias. Scheduled sampling gradually transitions from teacher forcing to using model predictions during training to mitigate this issue.

Mathematical Foundations

The caption is generated by maximizing the conditional probability:

Where each parameter means:

• is the word generated at timestep
• is the visual feature representation of the image
• is the total caption length
• The product computes the joint probability of the entire caption sequence

Beam search decoding finds the most probable caption by maintaining candidates at each step:

Where each parameter means:

• is the length-normalized log probability score
• is the length normalization parameter (typically 0.6-0.7)
• is the current timestep for normalization
• is the conditional probability from the decoder

The attention mechanism at each decoding step computes:

Where each parameter means:

• is the unnormalized attention score for spatial location at time
• is the visual feature at spatial location
• is the decoder hidden state at the previous timestep
• is the attention scoring function

Architecture Design

Implementation

import torch
import torch.nn as nn
import torchvision.models as models


class CaptionDecoder(nn.Module):
    def __init__(self, embed_dim, hidden_dim, vocab_size, num_features=2048):
        super(CaptionDecoder, self).__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTMCell(embed_dim + num_features, hidden_dim)
        self.attention = nn.Linear(hidden_dim, num_features)
        self.feature_proj = nn.Linear(num_features, hidden_dim)
        self.fc = nn.Linear(hidden_dim, vocab_size)
        self.init_h = nn.Linear(num_features, hidden_dim)
        self.init_c = nn.Linear(num_features, hidden_dim)

    def init_hidden_state(self, features):
        mean_feat = features.mean(dim=1)
        h = torch.tanh(self.init_h(mean_feat))
        c = torch.tanh(self.init_c(mean_feat))
        return h, c

    def attention_weights(self, features, hidden):
        att = torch.tanh(self.attention(hidden))
        scores = torch.bmm(features, att.unsqueeze(2)).squeeze(2)
        weights = torch.softmax(scores, dim=1)
        return weights

    def forward(self, features, captions):
        batch_size = features.size(0)
        seq_len = captions.size(1)
        embeddings = self.embed(captions)
        h, c = self.init_hidden_state(features)
        outputs = torch.zeros(batch_size, seq_len, self.fc.out_features)
        for t in range(seq_len):
            weights = self.attention_weights(features, h)
            context = torch.bmm(weights.unsqueeze(1), features).squeeze(1)
            lstm_input = torch.cat([embeddings[:, t, :], context], dim=1)
            h, c = self.lstm(lstm_input, (h, c))
            outputs[:, t, :] = self.fc(h)
        return outputs

Comparison Table

Common Challenges

1. Exposure Bias: Teacher forcing during training creates mismatch with autoregressive inference where model predictions are used instead of ground truth
2. Repetition Problem: Decoders often generate repetitive phrases or cycles, requiring techniques like coverage penalty or diverse beam search
3. Semantic Faithfulness: Generated captions may describe objects or relationships not present in the image, known as hallucination
4. Long-Tail Vocabulary: Rare words and domain-specific terms have insufficient training examples for accurate generation
5. Evaluation Gap: Automatic metrics like BLEU and CIDEr do not perfectly correlate with human judgment of caption quality

Attention Visualization and Interpretability

Attention weights in image captioning provide interpretability by revealing which image regions the model focuses on when generating each word. Visualization shows that the model learns to attend to objects when generating their names (dog, car), to regions with specific attributes when generating adjectives (red, large), and to spatial relationships when generating prepositions (on, behind). This alignment between visual attention and linguistic semantics demonstrates that the model learns meaningful cross-modal correspondences without explicit supervision.

Grad-CAM visualization applied to the decoder reveals that the model attends to semantically relevant regions even when attention weights are not directly available. Gradient-weighted attention maps highlight the visual evidence used for each word prediction, enabling error analysis when the model generates incorrect descriptions.

Advanced Decoding Strategies

Diverse beam search extends standard beam search by introducing diversity penalties that encourage beams to explore different decoding paths. The diversity penalty adds a cost for selecting tokens that appear frequently in other beams, promoting vocabulary coverage and reducing repetition. Temperature sampling with top-k or top-p filtering provides stochastic decoding that generates diverse captions while maintaining fluency.

Length normalization addresses the bias of beam search toward shorter sequences by dividing log probabilities by the sequence length raised to a power alpha. The penalty encourages generation of more detailed captions without artificial truncation. Coverage penalty discourages the model from attending to the same image regions repeatedly, promoting more comprehensive visual descriptions.

Reinforcement learning with CIDEr as reward directly optimizes the evaluation metric rather than maximizing likelihood. The REINFORCE algorithm estimates gradients using sampled captions, allowing the model to learn decoding policies that maximize the desired metric. This approach improves CIDEr scores by 5-8 points but may reduce fluency compared to maximum likelihood training.

Case Study: MSCOCO Captioning

The MSCOCO dataset contains 120K training images with 5 captions each, covering 80 object categories and complex scenes. The Up-Down model combining bottom-up attention (Faster R-CNN) with top-down attention achieved 36.2 BLEU-4 and 120.1 CIDEr by attending to 36 detected objects per image. Recent transformer-based models using ViT-Large encoders achieve 42.5 BLEU-4 with significantly improved semantic accuracy. The BLIP framework achieves state-of-the-art results by pretraining on 129M image-text pairs before fine-tuning on COCO, demonstrating the value of large-scale pretraining for caption generation. Human evaluation studies show that the best models generate captions comparable to human descriptions in 43% of cases, with main failures being incorrect object counts and rare attribute associations.

Training and Evaluation Details

Captioning models use cross-entropy loss during maximum likelihood training with label smoothing of 0.1 to prevent overconfident predictions. Teacher forcing ratio starts at 1.0 and decays linearly to 0.5 over the first 20 epochs, gradually transitioning to scheduled sampling. The vocabulary is built from tokens appearing at least 5 times, resulting in approximately 10K tokens including special markers for start, end, and padding.

Evaluation uses multiple complementary metrics: BLEU-1 through BLEU-4 measure n-gram precision, METEOR considers synonyms and stemming, CIDEr measures TF-IDF weighted n-gram similarity, and SPICE evaluates semantic propositional content. Human evaluation on 1000 images rates captions on informativeness, fluency, and relevance, providing ground truth for metric correlation studies.

Dense Captioning and Region Description

Dense captioning generates descriptions for specific regions within an image rather than producing a single global caption. The system first proposes regions using object detection or selective search, then generates a description for each region conditioned on its visual features and surrounding context. This approach produces fine-grained descriptions like "a person wearing a red jacket" for specific bounding boxes.

Region description models take region features from Faster R-CNN and generate word sequences using attention over the regional features. The training pairs region proposals with their corresponding descriptions from dense captioning datasets like Visual Genome. The evaluation uses BLEU and METEOR at the region level, with typical performance of 18-22 BLEU-4 on Visual Genome.

Video Captioning and Storytelling

Video captioning extends image captioning to temporal sequences by incorporating motion features from video encoders. The model generates descriptions that capture temporal dynamics such as "a person walks to the door and opens it". Recurrent video captioning uses encoder-decoder architecture with temporal attention over video frames.

Dense video captioning detects and describes multiple events within a video, producing timestamped captions for each event. The event proposal network generates temporal segments, while the captioning network generates descriptions conditioned on the visual content within each segment. This approach achieves 20-25 METEOR on ActivityNet Captions dataset.

Caption Quality and Diversity

Modern captioning systems generate diverse outputs for the same image, producing multiple candidate captions that describe different aspects of the scene. Nucleus sampling with p=0.9 generates diverse yet fluent captions by sampling from the smallest set of tokens whose cumulative probability exceeds the threshold. This produces more varied and interesting descriptions than beam search.

Caption quality metrics correlate differently with human judgments. CIDEr best captures consensus with human annotators at 0.62 correlation, while BLEU-4 shows weaker correlation at 0.48 due to its sensitivity to exact word matches. SPICE achieves the highest correlation for semantic accuracy at 0.68 by evaluating scene graph similarity.

Factuality evaluation checks whether generated captions describe objects and relationships actually present in the image. Hallucination detection identifies captions that mention objects not visible in the image, which occurs in 10-15% of generated captions. Counterfactual evaluation tests whether negating caption content produces captions inconsistent with the image.

Structured Captioning Approaches

Template-based captioning generates descriptions by filling slots in predefined templates like "A [color] [object] is [action] [location]". While less fluent than neural approaches, templates ensure factual accuracy and enable controlled generation. The slot values are predicted by attribute and object detectors, producing reliable but repetitive descriptions.

Scene graph captioning generates structured representations of visual content as subject-predicate-object triplets. The model detects objects and predicts relationships between them, building a graph that captures the semantic structure of the scene. This representation enables compositional generation and supports downstream tasks like visual question answering.

Conditional captioning generates different descriptions based on specified attributes or styles. The model can be conditioned to produce captions focused on specific objects, written in different languages, or adapted for different audiences. This controllability enables personalized captioning for diverse user needs.

Data Augmentation and Training Techniques

Captioning models benefit from extensive data augmentation including image augmentation (random horizontal flip, color jittering, random crop), text augmentation (synonym replacement, word dropout, back-translation), and mixed augmentation that combines image-text pairs from different sources. These techniques improve generalization and reduce overfitting on limited caption datasets.

Curriculum learning starts with simple images containing single objects and gradually introduces complex scenes with multiple objects and relationships. This training strategy stabilizes optimization and improves final performance by 2-3 BLEU points compared to random shuffling.

Knowledge distillation transfers captioning ability from large teacher models to smaller student models suitable for deployment. The student learns to match both the hard targets (ground truth captions) and soft targets (teacher probability distributions), achieving 95% of teacher performance with 4x fewer parameters.

Model Size and Efficiency Trade-offs

Captioning model accuracy scales with model size up to a point of diminishing returns. Models with 10-50M parameters achieve the best accuracy-efficiency trade-off, while larger models provide marginal improvements at significant computational cost. The optimal architecture depends on deployment constraints and latency requirements.

Efficient captioning for mobile devices uses lightweight backbones like MobileNet or EfficientNet-B0 with attention-based decoders. These models achieve 30-32 BLEU-4 on MSCOCO with 5-10x fewer FLOPs than full-size models, enabling real-time captioning on smartphones.

Streaming captioning processes video frames incrementally without storing the full video. The model maintains a hidden state that accumulates visual information across frames, generating captions that describe ongoing events. This approach enables live video description with constant memory usage regardless of video length.

Key Takeaways

• Encoder-decoder architectures form the foundation with CNNs extracting visual features and RNNs/Transformers generating captions
• Attention mechanisms dramatically improve caption quality by enabling selective focus on relevant image regions
• Beam search with length normalization outperforms greedy decoding by exploring multiple caption hypotheses
• Teacher forcing accelerates training but scheduled sampling reduces exposure bias during inference
• Bottom-up attention using object proposals provides semantically meaningful regions over grid features
• Large-scale pretraining on image-text pairs significantly boosts performance on downstream captioning tasks
• Evaluation requires multiple metrics as BLEU, CIDEr, and METEOR capture different aspects of caption quality
• Reinforcement learning with CIDEr reward directly optimizes evaluation metrics beyond likelihood training
• Structured representations like scene graphs enable compositional caption generation and editing
• Dense video captioning extends image captioning to temporal localization and description of events
• Knowledge distillation enables efficient deployment while preserving caption quality and diversity