Detecting Novel Categories with Language-Guided Detectors

Module: Computer Vision | Difficulty: Premium

Open-Vocabulary Detection

Region-Text Matching

OWL-ViT Loss

| Model | LVIS zero-shot | COCO novel | Approach | |-------|---------------|------------|----------| | ViLD | 26.0 AP | 34.2 AP | Embedding | | OWL-ViT | 34.6 AP | 41.2 AP | Contrastive | | Grounding DINO | 36.7 AP | 43.5 AP | Grounding | | Grounding DINO 1.5 | 39.1 AP | 46.8 AP | Improved | | OV-DETR | 31.2 AP | 37.8 AP | DETR |

import torch
import torch.nn as nn
import torch.nn.functional as F

class OpenVocabularyDetector(nn.Module):
    def __init__(self, detector, text_encoder, class_names):
        super().__init__()
        self.detector = detector
        self.text_encoder = text_encoder
        self.class_names = class_names
        self.text_embeddings = None

    def encode_text(self):
        embeddings = []
        for name in self.class_names:
            tokens = self.text_encoder.tokenize(
                f"a photo of a {name}")
            emb = self.text_encoder.encode_text(tokens)
            embeddings.append(emb.mean(dim=0))
        self.text_embeddings = F.normalize(
            torch.stack(embeddings), dim=1)

    def forward(self, images, class_names=None):
        if class_names is not None:
            self.class_names = class_names
            self.encode_text()
        proposals, features = self.detector(images)
        region_features = self.detector.roi_heads(
            features, proposals)
        region_features = F.normalize(region_features, dim=1)
        scores = region_features @ self.text_embeddings.t()
        return proposals, scores

def compute_ovd_metrics(predictions, ground_truth,
                        known_classes, novel_classes):
    known_pred = [
        p for p in predictions if p['class'] in known_classes]
    novel_pred = [
        p for p in predictions if p['class'] in novel_classes]
    known_ap = compute_ap(known_pred,
                          [g for g in ground_truth
                           if g['class'] in known_classes])
    novel_ap = compute_ap(novel_pred,
                          [g for g in ground_truth
                           if g['class'] in novel_classes])
    return {'known_ap': known_ap, 'novel_ap': novel_ap}

Research Insight: Open-vocabulary detection has advanced rapidly with Grounding DINO 1.5, which achieves strong performance on both known and novel categories by leveraging vision-language pretraining. The key insight is that detection and grounding (localizing objects from language queries) are the same task: given a text query, find all matching regions. This unification enables zero-shot detection of arbitrary categories described in natural language. The remaining challenge is compositional generalization: detecting "red cube on blue sphere" requires understanding spatial relationships, not just individual object concepts.