OCR & Document Understanding

Learning Objectives

• Trace the classical OCR pipeline (detect -> recognise -> layout) and the modern end-to-end alternatives (Donut, Qwen-VL-OCR)
• Implement CTC (Connectionist Temporal Classification) loss for sequence-to-sequence OCR training
• Use PaddleOCR or EasyOCR for production document parsing without training
• Distinguish OCR, layout parsing, and document understanding — and pick the right tool per task

The Problem

Images full of text are everywhere: receipts, invoices, IDs, scanned books, forms, whiteboards, signs, screenshots. Extracting structured data from them — not just the characters, but "this is the total amount" — is one of the highest-value applied-vision problems.

The field splits into three skill layers:

1. OCR proper: turn pixels into text.
2. Layout parsing: group OCR output into regions (title, body, table, header).
3. Document understanding: extract structured fields ("invoice_total = $42.50") from layout.

Each layer has classical and modern approaches, and the gap between "I want text from an image" and "I need the total amount from this receipt" is bigger than most teams realise.

The Concept

The classical pipeline

• Text detection produces per-line or per-word quadrilaterals.
• Recognition crops each region to a fixed height, runs a CNN + BiLSTM + CTC to produce a character sequence.
• Layout rebuilds reading order (top-to-bottom, left-to-right for Latin; different for Arabic, Japanese).

CTC in one paragraph

OCR recognition produces a variable-length sequence from a fixed-length feature map. CTC (Graves et al., 2006) lets you train this without character-level alignment. The model outputs a distribution over (vocab + blank) at every time step; CTC loss marginalises over all alignments that reduce to the target text after merging repeats and removing blanks.

raw output: "h h h _ _ e e l l _ l l o _ _"
after merge repeats and remove blanks: "hello"

CTC is the reason CRNN worked in 2015 and still trains most production OCR models in 2026.

Modern end-to-end models

• Donut (Kim et al., 2022) — a ViT encoder + a text decoder; reads an image and emits JSON directly. No text detector, no layout module.
• TrOCR — ViT + transformer decoder for line-level OCR.
• Qwen-VL-OCR / InternVL — full vision-language models fine-tuned for OCR tasks; best accuracy in 2026 on complex documents.
• PaddleOCR — classical DB + CRNN pipeline in a mature production package; still the open-source workhorse.

End-to-end models need more data and compute but skip the error accumulation of multi-stage pipelines.

Layout parsing

For structured documents, run a layout model (LayoutLMv3, or a detector trained on the DocLayNet dataset) that labels each region: Title, Paragraph, Figure, Table, Footnote. Reading order then becomes "iterate through regions in layout order, concatenate."

For forms, use Key-Value extraction models (Donut for visually-rich documents, LayoutLMv3 for plain scans). They take image + detected text + positions and predict structured key-value pairs.

Evaluation metrics

• Character Error Rate (CER) — Levenshtein distance / length of reference. Lower is better. Production target: < 2% on clean scans.
• Word Error Rate (WER) — same at the word level.
• F1 on structured fields — for key-value tasks; measures whether {invoice_total: 42.50} appears correctly.
• Edit distance on JSON — for end-to-end document parsing; the Donut paper introduced normalised tree edit distance.

cv3-ctc-collapse

Build It

Step 1: CTC loss + greedy decoder

import torch
import torch.nn as nn
import torch.nn.functional as F


def ctc_loss(log_probs, targets, input_lengths, target_lengths, blank=0):
    """
    log_probs:      (T, N, C) log-softmax over vocab including blank at index 0
    targets:        (N, S) int targets (no blanks)
    input_lengths:  (N,) per-sample time steps used
    target_lengths: (N,) per-sample target length
    """
    return F.ctc_loss(log_probs, targets, input_lengths, target_lengths,
                      blank=blank, reduction="mean", zero_infinity=True)


def greedy_ctc_decode(log_probs, blank=0):
    """
    log_probs: (T, N, C) log-softmax
    returns: list of index sequences (blanks removed, repeats merged)
    """
    preds = log_probs.argmax(dim=-1).transpose(0, 1).cpu().tolist()
    out = []
    for seq in preds:
        decoded = []
        prev = None
        for idx in seq:
            if idx != prev and idx != blank:
                decoded.append(idx)
            prev = idx
        out.append(decoded)
    return out

F.ctc_loss uses the efficient CuDNN implementation when available. The greedy decoder is simpler than a beam search and usually within 1% CER of it.

Step 2: Tiny CRNN recogniser

Minimal CNN + BiLSTM for line OCR.

class TinyCRNN(nn.Module):
    def __init__(self, vocab_size=40, hidden=128, feat=32):
        super().__init__()
        self.cnn = nn.Sequential(
            nn.Conv2d(1, feat, 3, 1, 1), nn.BatchNorm2d(feat), nn.ReLU(inplace=True),
            nn.MaxPool2d(2),
            nn.Conv2d(feat, feat * 2, 3, 1, 1), nn.BatchNorm2d(feat * 2), nn.ReLU(inplace=True),
            nn.MaxPool2d(2),
            nn.Conv2d(feat * 2, feat * 4, 3, 1, 1), nn.BatchNorm2d(feat * 4), nn.ReLU(inplace=True),
            nn.MaxPool2d((2, 1)),
            nn.Conv2d(feat * 4, feat * 4, 3, 1, 1), nn.BatchNorm2d(feat * 4), nn.ReLU(inplace=True),
            nn.MaxPool2d((2, 1)),
        )
        self.rnn = nn.LSTM(feat * 4, hidden, bidirectional=True, batch_first=True)
        self.head = nn.Linear(hidden * 2, vocab_size)

    def forward(self, x):
        # x: (N, 1, H, W)
        f = self.cnn(x)                # (N, C, H', W')
        f = f.mean(dim=2).transpose(1, 2)  # (N, W', C)
        h, _ = self.rnn(f)
        return F.log_softmax(self.head(h).transpose(0, 1), dim=-1)  # (W', N, vocab)

Fixed-height input (the CNN max-pools height to 1). Width is the time dimension for CTC.

Step 3: Synthetic OCR

Generate black-on-white strings for an end-to-end smoke test. Each character becomes a column of six bars that spell its vocabulary index in binary. The image must carry the character's identity: if every character were the same blank box, the model could not tell a from x, and the loss would stall.

import numpy as np

GLYPHS = "0123456789abcdefghijklmnopqrstuvwxyz"

def synthetic_line(text, height=32, char_width=16):
    W = char_width * len(text)
    img = np.ones((height, W), dtype=np.float32)
    for i, c in enumerate(text):
        x = i * char_width
        code = GLYPHS.find(c) + 1
        for bit in range(6):
            if code >> bit & 1:
                y = 4 + 4 * bit
                img[y:y + 3, x + 2:x + char_width - 2] = 0.0
    return img


def build_batch(strings, vocab):
    H = 32
    W = 16 * max(len(s) for s in strings)
    imgs = np.ones((len(strings), 1, H, W), dtype=np.float32)
    target_lengths = []
    targets = []
    for i, s in enumerate(strings):
        imgs[i, 0, :, :16 * len(s)] = synthetic_line(s)
        ids = [vocab.index(c) for c in s]
        targets.extend(ids)
        target_lengths.append(len(ids))
    return torch.from_numpy(imgs), torch.tensor(targets), torch.tensor(target_lengths)


vocab = ["_"] + list("0123456789abcdefghijklmnopqrstuvwxyz")
imgs, targets, lengths = build_batch(["hello", "world"], vocab)
print(f"images: {imgs.shape}   targets: {targets.shape}   lengths: {lengths.tolist()}")

A real OCR dataset adds fonts, noise, rotation, blur, and colour. The pipeline above is identical.

Step 4: Training sketch

model = TinyCRNN(vocab_size=len(vocab))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)

for step in range(400):
    strings = ["abc" + str(step % 10)] * 4 + ["xyz" + str((step + 1) % 10)] * 4
    imgs, targets, target_lens = build_batch(strings, vocab)
    log_probs = model(imgs)  # (W', 8, vocab)
    input_lens = torch.full((8,), log_probs.size(0), dtype=torch.long)
    loss = ctc_loss(log_probs, targets, input_lens, target_lens, blank=0)
    opt.zero_grad(); loss.backward(); opt.step()

Loss should drop from double digits to about 0.3 within 400 steps on this trivial synthetic data. code/main.py runs the same loop on its own string set and then decodes three test strings.

Use It

Three production paths:

• PaddleOCR — mature, fast, multilingual. One-line usage: paddleocr.PaddleOCR(lang="en").ocr(image_path).
• EasyOCR — Python-native, multilingual, PyTorch backbone.
• Tesseract — classical; still useful for old scanned documents when models struggle.

For end-to-end document parsing, use Donut or a VLM:

from transformers import DonutProcessor, VisionEncoderDecoderModel

processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")
model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-cord-v2")

For receipts, invoices, and forms with repeatable structure, fine-tune Donut. For arbitrary documents or OCR with reasoning, a VLM like Qwen-VL-OCR is the current default.

Ship It

This lesson produces:

• outputs/prompt-ocr-stack-picker.md — a prompt that picks Tesseract / PaddleOCR / Donut / VLM-OCR given document type, language, and structure.
• outputs/skill-ctc-decoder.md — a skill that writes greedy and beam-search CTC decoders from scratch, including length normalisation.

Exercises

1. (Easy) Train the TinyCRNN on 5-digit random numeric strings for 500 steps. Report CER on a held-out set.
2. (Medium) Replace greedy decoding with beam search (beam_width=5). Report CER delta. On which inputs does beam search win?
3. (Hard) Use PaddleOCR on a set of 20 receipts, extract line items, and compute F1 against hand-labelled ground truth for {item_name, price} pairs.

Key Terms

Further Reading

• CRNN (Shi et al., 2015) — the original CNN+RNN+CTC architecture
• CTC (Graves et al., 2006) — the original CTC paper; densely packed with the algorithmic ideas
• Donut (Kim et al., 2022) — OCR-free document understanding transformer
• PaddleOCR — the open-source production OCR stack