OCR, Text Detection, and Document Understanding

Module: Computer Vision | Difficulty: Premium

Document Layout as Graph

Text Detection Loss

where is the differentiable binarization.

LayoutLM v3 Objective

| Model | FUNSD | DocVQA | SROIE | Approach | |-------|-------|--------|-------|----------| | LayoutLM v2 | 83.3 | 75.5 | 95.2 | Text + Layout | | LayoutLM v3 | 87.5 | 83.3 | 96.1 | Unified pretraining | | LayoutLM v4 | 90.1 | 88.2 | 97.0 | Grounding | | DiT | 88.7 | 85.0 | 96.3 | Document transformer | | GOT-OCR | 91.2 | 90.1 | 97.5 | Vision LLM |

import torch
import torch.nn as nn
import torch.nn.functional as F

class TextDetectorDB(nn.Module):
    def __init__(self, backbone_channels=256):
        super().__init__()
        self.thresh_head = nn.Sequential(
            nn.Conv2d(backbone_channels, 64, 3, padding=1, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.ConvTranspose2d(64, 64, 2, stride=2),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.ConvTranspose2d(64, 1, 2, stride=2),
            nn.Sigmoid(),
        )
        self.box_head = nn.Sequential(
            nn.Conv2d(backbone_channels, 64, 3, padding=1, bias=False),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.ConvTranspose2d(64, 64, 2, stride=2),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.Conv2d(64, 2, 3, padding=1),
        )

    def forward(self, features):
        thresh = self.thresh_head(features)
        box = self.box_head(features)
        return thresh, box

def differentiable_binarization(probability_map, threshold,
                                 beta=1.0):
    return 1.0 / (1.0 + torch.exp(-beta * (probability_map - threshold)))

Research Insight: Document AI has been transformed by large multimodal models (GOT-OCR, LayoutLM v4) that jointly process text, layout, and images. The key shift is from pipeline-based approaches (detect then recognize) to end-to-end models that directly output structured document content. LayoutLM v4's grounding capability enables the model to highlight which document regions support its answers, improving interpretability for legal and medical document analysis. The challenge remains handling degraded documents (faxes, scans) where OCR quality degrades significantly.