OCR Model Guide: Classic Engines, Document VLMs, or General Vision Models
← All posts
GuideOct 15, 2026· 10 min read

OCR Model Guide: Classic Engines, Document VLMs, or General Vision Models

If you searched for OCR model, you are probably trying to decide what should actually read your pages: a classic OCR engine, one of the new open document models, or a general vision model you already pay for. Short answer: for turning document pages into Markdown, the best public benchmark scores in late 2026 belong to small specialized document models of roughly 1B parameters, not the largest general models. A leaderboard only narrows the shortlist, though. The model you should run is the one that makes the fewest costly mistakes on a sample of your own pages, at a cost and deployment model you can live with.

This guide sorts OCR models into three families, summarizes what two public benchmarks say about them, and gives you a short Python script to score candidates on your documents. If you are past choosing and thinking about fine-tuning, read Train Custom OCR Model instead.

The three kinds of OCR model

“OCR model” used to mean one thing: software that finds lines of text in an image and recognizes the characters. Today the phrase covers three families with different strengths.

1. Classic and pipeline OCR engines

Tesseract is the classic example. Modern pipeline toolkits such as PaddleOCR, MinerU’s pipeline mode, and Marker chain several smaller models together: layout detection, text detection, text recognition, table and formula modules, then rules that stitch the pieces back into reading order.

These pipelines are fast, cheap to run on modest hardware, and easy to inspect because every stage is visible. Their weak point is relationships between characters. The Nougat paper from Meta puts it plainly: engines like Tesseract recognize characters and words well but process line by line, so superscripts, subscripts, and fraction layouts get flattened into ordinary text (Nougat, arXiv 2308.13418).

Small does not mean weak. The PaddleOCR 3.0 technical report describes PP-OCRv5 and PP-StructureV3 as models with fewer than 100 million parameters that reach accuracy competitive with billion-parameter vision language models, under an Apache license.

2. Specialized document VLMs

The second family reads a whole page image and writes the output in one pass, usually Markdown or HTML for tables and LaTeX for formulas. Researchers call this end-to-end OCR. A few landmarks:

  • GOT-OCR2.0 (580M parameters) handles plain text, formulas, tables, charts, and even sheet music, and can output Markdown, TikZ, or SMILES on request (arXiv 2409.01704).
  • olmOCR from Ai2 is a 7B model fine-tuned from Qwen2-VL on 260,000 pages from more than 100,000 PDFs. The authors estimate under $176 per million pages on their own GPUs, against more than $6,240 for GPT-4o at February 2025 batch prices (arXiv 2502.18443). olmOCR 2 adds reinforcement learning against binary unit tests, with the largest gains on formulas, tables, and multi-column layouts (arXiv 2510.19817).
  • DeepSeek-OCR compresses a page into very few vision tokens. Its authors report about 97% decoding precision when there are fewer than ten text tokens per vision token, about 60% at twenty, and more than 200,000 pages a day on a single A100-40G (arXiv 2510.18234).
  • PaddleOCR-VL, MinerU2.5, dots.ocr, GLM-OCR, and TeleOCR are newer entries that now lead the OmniDocBench table, covered below.

Donut from NAVER is worth knowing as the ancestor of a related idea: skip OCR text entirely and map a document image straight to structured fields (arXiv 2111.15664).

3. General vision language models

Gemini, GPT, Qwen-VL, Claude, and similar multimodal models read text as one part of understanding an image. They are the right tool when the next step is a judgment: is this form complete, what does this chart show, which of these pages is the signed one. They are rarely the cheapest or most predictable way to transcribe thousands of pages. Claude is a good example of the confusion: it reads text through vision, but there is no separate Claude OCR model behind it. Does Claude Have OCR? covers that case in detail.

What the benchmarks say

Two public benchmarks are useful for document OCR. Neither one is your workload, and they do not always agree.

OmniDocBench

OmniDocBench (CVPR 2025) scores full-page parsing across nine document sources, including academic papers, textbooks, handwritten notes, and dense newspapers, with 19 layout categories and 15 attribute labels (arXiv 2412.07626). The headline Overall score averages three things: text accuracy as (1 minus normalized edit distance) times 100, table structure (TEDS), and formula recognition (CDM).

Selected rows from the v1.6_full table in the project README, read in October 2026:

Model Type Size Overall
TeleOCR Specialized VLM 1.2B 96.91
PaddleOCR-VL-1.6 Specialized VLM 0.9B 96.34
MinerU2.5-Pro Specialized VLM 1.2B 95.75
GLM-OCR Specialized VLM 0.9B 95.22
Gemini 3 Pro General VLM not stated 92.91
dots.ocr Specialized VLM 3B 90.77
DeepSeek-OCR 2 Specialized VLM 3B 90.25
Qwen3-VL-235B General VLM 235B 89.78
GPT-5.2 General VLM not stated 86.59
MinerU pipeline Pipeline tool not stated 86.47
olmOCR Specialized VLM 7B 85.74
Mistral OCR Specialized VLM not stated 85.66
Marker Pipeline tool not stated 78.44

The pattern is hard to miss. The top of the table is held by models around 1B parameters trained for one job, and the strongest general models sit a few points behind them.

olmOCR-Bench

Ai2’s olmOCR-Bench takes a different approach: 1,400 English PDFs with pass/fail unit tests for things like tiny fonts, old scans, tables, and math. The olmOCR 2 report lists these scores (an asterisk marks numbers reported by the model’s own authors rather than reproduced by Ai2):

System olmOCR-Bench
Chandra OCR 0.1.0 83.1*
Infinity-Parser 7B 82.5*
olmOCR 2 82.4
PaddleOCR-VL 80.0*
dots.OCR 79.1*
Marker 1.10.1 76.1
DeepSeek-OCR 75.7
Mistral OCR API (March 2025) 72.0
GPT-4o 68.9
Qwen 2.5 VL 7B 65.5
Gemini Flash 2 57.8

Notice that the order shifts between benchmarks. The March 2025 Mistral OCR scores about seven points above Marker on OmniDocBench, yet Marker beats it by four points on olmOCR-Bench. That is normal: each benchmark rewards different page types and scoring rules, which is exactly why your own sample matters more than either table.

A third benchmark, OCRBench v2, tests general multimodal models on text localization, handwriting, and reasoning about text in scenes. Its authors found that most models scored below 50 out of 100 at publication. If your images are photos of signs, receipts, and labels rather than clean pages, that result is a better warning than the document leaderboards.

How to choose an OCR model

Answer these questions before you look at a leaderboard again.

  1. What does the consumer need? Plain text for search, Markdown for RAG and review, or named fields for a database. A model that writes beautiful Markdown still leaves you parsing invoice totals out of it.
  2. Can the pages leave your network? If not, you are choosing among open weights you host yourself. If they can, a hosted API removes GPU capacity, batching, and patching from your list.
  3. What licenses can you ship? The olmOCR 2 report lists licenses next to each model: Apache 2.0 and MIT for some, OpenRAIL-M or AGPL v3 with usage restrictions for others, and API terms for closed systems. Check before you build on one.
  4. Which page types hurt? Formulas, multi-column layouts, tables, handwriting, or low-resolution phone photos each move the rankings.
  5. What is the volume? At a few hundred pages a month, any reasonable option is cheap. At millions of pages, tokens per page and pages per GPU-hour decide the budget.

A rough map from situation to family:

Situation Start with
Clean printed scans, CPU only, offline Classic or pipeline engine such as PaddleOCR
Mixed PDFs to Markdown, self-hosted GPU Small specialized document VLM
Bulk extraction without running models Hosted OCR API
Judgment about the image, low volume General vision model
Proprietary layouts that still fail after all of the above Fine-tuning (guide)

Score candidates on your own pages

Take 50 to 200 real pages, including the ugly ones, and hand-correct the text for each. For pages where certain values must be exact (invoice numbers, totals, dates, IDs), list those values too. Then run every candidate model over the same pages and save one output file per page.

The script below needs only the Python standard library. It reports mean character error rate (CER), the worst page, and how many must-be-exact values appear verbatim in each model’s output.

"""Score OCR model outputs against hand-checked ground truth.

Layout:
  truth/<page>.txt            corrected text for each page
  truth/<page>.fields.json    optional: {"invoice_number": "INV-20931", ...}
  preds/<model>/<page>.md     one output file per page, per model (.md or .txt)
"""
import json
import re
import sys
from pathlib import Path


def normalize(text: str) -> str:
    text = re.sub(r"[#*_`|>]", " ", text)          # drop Markdown markup
    text = re.sub(r"^\s*[-:]{3,}\s*$", " ", text, flags=re.M)  # table rules
    return re.sub(r"\s+", " ", text).strip().lower()


def edit_distance(a: str, b: str) -> int:
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, 1):
        cur = [i]
        for j, cb in enumerate(b, 1):
            cur.append(min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb)))
        prev = cur
    return prev[-1]


def score(root: Path) -> None:
    truths = sorted((root / "truth").glob("*.txt"))
    for model_dir in sorted(p for p in (root / "preds").iterdir() if p.is_dir()):
        cers, hits, total, missing = [], 0, 0, 0
        for t in truths:
            pred = next((model_dir / f"{t.stem}{ext}" for ext in (".md", ".txt")
                         if (model_dir / f"{t.stem}{ext}").exists()), None)
            if pred is None:
                missing += 1
                cers.append((1.0, t.stem))
                continue
            gold, out = normalize(t.read_text()), normalize(pred.read_text())
            cers.append((edit_distance(gold, out) / max(len(gold), 1), t.stem))
            fields = t.with_name(f"{t.stem}.fields.json")
            if fields.exists():
                for value in json.loads(fields.read_text()).values():
                    total += 1
                    hits += normalize(str(value)) in out
        mean = sum(c for c, _ in cers) / len(cers)
        worst = max(cers)
        print(f"{model_dir.name:<12} CER {mean:6.2%}  worst {worst[1]} {worst[0]:6.2%}"
              f"  fields {hits}/{total}  missing {missing}")


if __name__ == "__main__":
    score(Path(sys.argv[1] if len(sys.argv) > 1 else "."))

Here is the output on a toy sample of two pages, an invoice and a delivery note. Model B misread one zero in the invoice number as the letter O, a capital I in the recipient’s name as a lowercase l, and a comma as a period:

model-a      CER  0.00%  worst invoice-01  0.00%  fields 3/3  missing 0
model-b      CER  2.38%  worst delivery-01  3.03%  fields 2/3  missing 0

A 2.38% character error rate sounds fine. The field column tells the real story: the one value your accounting system matches on is wrong. This is why the script tracks exact values separately. On business documents, one wrong digit in an ID costs more than a hundred harmless whitespace differences.

Two practical notes. Edit distance on long pages is slow in pure Python, so split very long documents into pages first. And a model that times out or crashes on a page counts as 100% error for that page, which is fair: a page you cannot read is a page someone has to type.

Where OCRskill fits

OCRskill is a hosted OCR API, so it belongs in your comparison as the “do not run a model at all” option. Inference runs on Nvidia hardware behind a plain REST endpoint. You do not pick, host, or patch the model. You send a page and get either Markdown or the fields you name.

To add it to the harness, write its Markdown output into its own preds folder:

export API_KEY="sk-your-key-here"   # get one: curl https://api.ocrskill.com/get-key.json
mkdir -p preds/ocrskill

for f in pages/*.pdf; do
  curl -s https://api.ocrskill.com/ocr \
    -H "Authorization: Bearer $API_KEY" \
    -H "Content-Type: application/pdf" \
    --data-binary "@$f" > "preds/ocrskill/$(basename "${f%.pdf}").md"
done

python3 score_ocr.py .

If your real target is fields rather than text, call POST /ocr.json with a fields list instead and compare the returned values to your .fields.json files directly. Dates come back as YYYY-MM-DD, and a missing required field fails with a 400 instead of a silent null. The structured JSON API reference lists supported fields and file types. Uploads are limited to 20 MB per request.

Limitations to keep in mind

  • Leaderboards move fast. The OmniDocBench README added three models in September 2026 alone, and two of them went straight to the top of the table. Treat any table here, including ours, as a snapshot.
  • Self-reported numbers are not reproductions. The olmOCR 2 table marks which scores Ai2 reran and which came from model authors. Vendor announcements are self-reported by definition.
  • Benchmark pages are not your pages. Both benchmarks lean toward PDFs, papers, and books. Receipts, shipping labels, stamped forms, and phone photos behave differently.
  • Edit distance hides critical errors. A low CER can still contain a wrong total or a swapped ID digit. Validate important values with checksums, totals that must add up, or format rules after extraction.
  • Parameter count is not cost. A 1B model that needs many output tokens per page can cost more than a larger model that writes less. Measure pages per GPU-hour or price per thousand pages on your sample.

Conclusion

The OCR model question has changed shape. Small specialized document VLMs now beat much larger general models at turning pages into Markdown on public benchmarks, pipeline engines remain a sensible offline default, and general vision models earn their cost when you need a judgment rather than a transcript. None of that tells you which model reads your invoices, forms, or scans best. Build a sample of 50 to 200 hand-checked pages, track exact values next to character error rate, and run every candidate through the same script, including a hosted API like OCRskill if you would rather not run GPUs. The model that wins that test, at a cost and license you accept, is the right one, whatever its leaderboard rank.