CLIP, ALIGN, and Zero-Shot Visual Recognition

Module: Computer Vision | Difficulty: Premium

CLIP Objective

Zero-Shot Classification

Image-Text Similarity

| Model | Params | Data | Zero-Shot ImageNet | |-------|--------|------|-------------------| | CLIP ViT-L/14 | 400M | 400M | 76.2% | | ALIGN | 470M | 1.8B | 76.4% | | Florence | 600M | 900M | 83.0% | | BASIC | 2.5B | 6.6B | 85.7% | | OpenCLIP | 600M | 2B | 78.0% |

import torch
import torch.nn.functional as F

class CLIPZeroShot:
    def __init__(self, image_encoder, text_encoder, class_names):
        self.image_encoder = image_encoder
        self.text_encoder = text_encoder
        self.class_names = class_names
        self.text_features = None

    def encode_text(self, templates=None):
        if templates is None:
            templates = ["a photo of a {}."]
        text_features = []
        for name in self.class_names:
            texts = [t.format(name) for t in templates]
            tokens = self.tokenizer(texts)
            features = self.text_encoder(tokens)
            features = F.normalize(features, dim=1)
            text_features.append(features.mean(dim=0))
        self.text_features = torch.stack(text_features)
        self.text_features = F.normalize(self.text_features, dim=1)

    def predict(self, images):
        image_features = self.image_encoder(images)
        image_features = F.normalize(image_features, dim=1)
        similarity = image_features @ self.text_features.t()
        return similarity.argmax(dim=1)

def compute_retrieval_metrics(image_features, text_features, labels):
    image_features = F.normalize(image_features, dim=1)
    text_features = F.normalize(text_features, dim=1)
    sim = image_features @ text_features.t()
    i2t = (sim.argmax(dim=1) == labels).float().mean()
    t2i = (sim.argmax(dim=0) == labels).float().mean()
    return {'i2t_accuracy': i2t.item(), 't2i_accuracy': t2i.item()}

Research Insight: CLIP demonstrated that contrastive vision-language pretraining enables remarkable zero-shot transfer to downstream tasks. The key insight is that natural language supervision is richer than class labels: it captures relationships, attributes, and compositional semantics. Recent work on open-vocabulary detection (Grounding DINO, OWLv2) combines CLIP features with detection architectures to detect arbitrary objects described in text. The scaling laws for vision-language models suggest that performance improves log-linearly with data and compute, indicating significant room for improvement with larger datasets.