Multimodal NLP: Text and Vision Integration
Module: Natural Language Processing | Difficulty: Advanced
Image Captioning
Visual Question Answering
CLIP
Evaluation
| Task | Model | Score | |------|-------|-------| | COCO Captioning | BLEU-4 | 36.2 | | VQA v2 | Accuracy | 72.3 | | NLVR2 | Accuracy | 68.5 |
import torch
import torch.nn as nn
class VLModel(nn.Module):
def __init__(self, vision_encoder, text_encoder, fusion_dim=768):
super().__init__()
self.vision = vision_encoder
self.text = text_encoder
self.fusion = nn.Linear(vision_encoder.output_dim + text_encoder.output_dim, fusion_dim)
self.classifier = nn.Linear(fusion_dim, n_classes)
def forward(self, image, text_ids):
img_repr = self.vision(image)
txt_repr = self.text(text_ids)
fused = self.fusion(torch.cat([img_repr, txt_repr], dim=-1))
return self.classifier(fused)
Research Insight: Vision-language models benefit from large-scale pre-training on image-text pairs. CLIP's contrastive learning approach learns robust cross-modal representations that transfer well to downstream tasks like VQA and image captioning.