Two routes Docling offers

Docling exposes two top-level ways to turn a page into structured content.

The first is the traditional pipeline: a layered stack of small specialist models running in sequence. A CNN layout detector finds text blocks, tables, pictures, formulas, and code. A separate model parses table structure into rows and columns. A small vision-language model runs on cropped code and formula regions to turn them into LaTeX or source text. A figure classifier labels pictures by type. The text inside each block comes from the PDF’s own text layer, extracted by a dedicated parser. Each stage has one job, runs on one slice of the page, and produces one piece of the final document.

The second is the VLM pipeline: one large vision-language model gets the rendered page image and emits the entire structured document for that page in one shot. No separate layout step. No separate table parsing. The model decides where the text blocks are, where the tables are, what the cells contain, which regions are pictures, all at once. The Docling-blessed model for this path is Granite-Docling-258M, available as an MLX export for Apple Silicon.

Both routes produce a structurally similar DoclingDocument, and Docling lets us swap between them by changing a single configuration object.

What each route costs on our corpus

Seven research papers, 155 pages total. Each variant runs each PDF once untimed (warmup) then three times measured; we report the median wall time per variant per doc, summed across the corpus. All on Apple Silicon. Code: benchmark/pdf_pipeline/.

Roughly one order of magnitude. The traditional pipeline does cheap text extraction on the bulk of the page (the prose, where the PDF already has a clean text layer) and saves the expensive VLM passes for the few regions that genuinely need them: the cropped code and formula boxes. The VLM pipeline pays the full VLM cost on every page to re-derive content that is already sitting in the PDF text stream. For our corpus, which is born-digital papers with clean text layers, the traditional pipeline is the right trade.

Inside the traditional pipeline

The traditional pipeline is a pipeline in the literal sense: pages stream through ordered stages, with multiple pages in different stages at once. Each stage owns one piece of the page-to-document translation, and each one is a place we can attack independently.

Picture classification piggybacks on doc_enrich. Tables are a stage of their own. Code and math sit inside doc_enrich because they share the same VLM and the same per-region inference path.

Where is the bottleneck?

Per-page wall time across the corpus, with tables on:

doc_enrich dominates the average. But the average hides the most useful fact: the bottleneck depends on the document. Per-page wall time, split per paper:

A page of mistral costs 90 ms in doc_enrich. A page of polarquant costs 8 seconds. Two orders of magnitude. The reason is what the layout step finds on each page: very few formulas in the text-heavy papers, dozens of code and formula crops per page in the quantization papers. That spread is what makes a corpus-wide “where do I optimise” question deceptive: optimising the rendering backend would barely move polarquant, and optimising the formula VLM would barely move mistral. We attack the stage that dominates the documents that matter. For us, that’s the math-heavy papers, and that means the code and formula VLM is the first place to look.

First improvement: the code & formula model

doc_enrich is a per-region VLM inference. Every code block and every math formula the layout step detects gets cropped from the rendered page and sent through a vision-language model that emits the LaTeX (for formulas) or source text with a language tag (for code). On the math-heavy papers there can be dozens of these per page. The VLM is what the stage is.

Second improvement: the layout detector

The layout stage uses docling-layout-heron, an RT-DETRv2 object detector. Docling runs it on MPS through PyTorch. We ported RT-DETRv2 to MLX in mlx-vlm PR #1195, with the bf16 weights published at:

• mlx-community/docling-layout-heron-mlx-bf16
• mlx-community/docling-layout-heron-101-mlx-bf16.

Full-corpus layout inference, 155 pages, median of 3 measured loops per PDF:

About 1.4× faster on the same model, same corpus, same hardware. End-to-end, the layout stage is a small fraction of total wall (doc_enrich still dominates), so this iteration moves the corpus number by only a few seconds. The win is real but local; we keep it because the change is mechanical and the port lives in mlx-vlm anyway.

Results

The two iterations together, CodeFormulaV2 on MLX for doc_enrich and RT-DETRv2 on MLX for layout, change the parsing budget on the same corpus, same hardware, same papers. Per-stage corpus totals (s), comparing the CPU path Docling’s catalog recommends (no MLX engine exists today for CodeFormulaV2 on Apple Silicon) to the same pipeline with our two ports plugged in:

doc_enrich is the entire story: 6 480 s drops to 260 s, a 24.9× shrink on the stage that owns 99 % of the CPU baseline. The layout port adds a 2.2× speedup on its own stage. The other stages are unchanged Docling code.

Corpus-wide:

From 1 h 49 m to 5 minutes on the same laptop, same papers, same model recommended by Docling’s catalog. What lands in Postgres at the end of it is the same DoclingDocument tree, with the same bounding boxes, ready for the next chapter: chunking and the embedder.

The candidates models

Four multilingual families, each in native precision (fp16 or bf16) and in 8-bit weight quantisation. All four run on Apple Silicon through mlx-embeddings. None of them required new MLX model code: the CLS-pooling dispatch we used to need a local patch for is merged upstream now.

Dense recall@k

Benchmark: 581 questions, exact ANN search (brute-force cosine, no HNSW), the layout-anchored ground truth from earlier in the post, k=100 retrieved per question. Code: eval/.

Snowflake-arctic-embed-l-v2.0 leads at every k from 3 to 100.Qwen3-4B sits 1-2 recall points below it through k=10 to k=50. Qwen3-0.6B and bge-m3 are tied with each other and trail the leaders by 4-5 points at k=10. The q8 (dashed) lines track the full-precision ones within 0.5-1 recall point everywhere.

We chose to keep Snowflake-arctic-embed-l-v2.0.

Hybrid retrieval

Hybrid retrieval is the next chapter, but the embedder pick should hold up under both modes, so a preview is useful here.

Recall@10 with the BM25 leg added:

Hybrid lifts every family, and lifts the weaker dense retrievers more. The rank order across families is unchanged after fusion. The embedder choice is robust to retrieval mode.

We keep Snowflake-arctic-embed-l-v2.0 with BM25 hybrid search

Quantization

A 568 M model in bf16 is 1.1 GB on disk. The q8 version is 576 MB. On a 16 GB machine where the pipeline already holds the rest of the pipeline, halving the embedder’s memory footprint is real headroom.

The recall cost at k=10 is at most 0.7 points across the four families, and closes to nothing by k=50. Quality-wise, q8 is free.

We chose Snowflake-arctic-embed-l-v2.0 quantized to 8 bits with BM25 hybrid search.

Indexing speed

The 568 M and 600 M models index in well under a minute for the 581-paragraph corpus (800-3000 tokens/second, depending on family and precision). Qwen3-4B is roughly 10× slower (~180 tokens/second; about six minutes to re-index the corpus). At the 100k-paragraph scale a real-world corpus would have, that is the difference between an embed step that takes a few minutes and one that takes an hour.

q8 is not uniformly faster than full precision on MLX. It is faster for Snowflake (1.1×), slightly slower for bge-m3 and Qwen3-4B, and noticeably slower forQwen3-0.6B (2.5× slower). The gap depends on which architecture has a fully-vectorised dequantisation path in mlx-embeddings at this version, and is the kind of thing that closes upstream as more families get tuned. For now: treat q8 as a memory and disk win, not a speed win.

The cleanest Pareto point on the chart is snowflake-arctic-l-v2 q8: best recall@10 of any sub-1B variant, fastest throughput of any sub-1B variant.

The pick

snowflake-arctic-embed-l-v2.0, q8. Tops dense recall at every k tested, tops hybrid recall too, shares the 568 M size class with bge-m3, and the q8 quantisation gives us a 576 MB on-disk footprint with no big quality cost.

The multilingual options

We considered two multilingual late-interaction models:

• jinaai/jina-colbert-v2: 559 M parameters, 94 languages, 128-dim per-token vectors.
• BAAI/bge-m3 multivector head: the same bge-m3 we already load for dense, with its multi-vector head applied to per-token outputs. 1024-dim per token, eight times the storage and the MaxSim compute of jina. It is not Matryoshka-trained, so naively truncating to a comparable 128-d would discard most of the signal in unprincipled ways.

We picked jina-colbert-v2: native low dim, training designed for the task, and the multilingual coverage is strictly broader than bge-m3’s.

We ported jina-colbert-v2 to MLX (src/rag/embedding/colbert.py) on top of the same xlm-roberta-rope backbone we vendored from mlx-vlm. Storage trick: the multi-vector field in Qdrant has HNSW disabled (hnsw_config.m=0) and lives on disk. It is only ever scored exact-MaxSim against the candidates the prefetch returns, so building a graph would be wasted work.

Hit@1 across the four multilingual families

jina-colbert-v2 does not improve our retrieval. Hybrid alone wins on three of four families. The one family where ColBERT helps (bge-m3) gains less than a point. Stacking ColBERT on top of hybrid ties “+ColBERT alone” in every family, which means the BM25 leg surfaces candidates that ColBERT then deranks back. The two scoring systems are not stacking; they are trading.

The full metric table on snowflake-l-v2 (our picked embedder):

Two failure examples make the mechanism concrete. Both are queries where snowflake-hybrid had the right paragraph at rank 1 and ColBERT then demoted it.

Q: “What two-stage process does DeepSeek-V4-Flash follow when introducing sparse attention during training, and at what sequence length and token count is sparsity first activated?”

Hybrid rank 1 (correct): the training-recipe paragraph with the sparsity schedule and the token-count cutoffs.

ColBERT rank 1 (wrong): the paper’s abstract, which mentions “DeepSeek-V4-Flash”, “sparse attention”, and “training” multiple times but does not answer the specific question.

Right answer demoted to rank 16.

Why ColBERT failed here. The abstract has many tokens that match the query’s topic words. MaxSim sums those matches, so the abstract scores high. The specific training paragraph mentions the same topic words fewer times and loses on token count, even though it actually contains the answer.

Q: “How are the authors of the DeepSeek-V4 paper listed, and what does the asterisk () notation next to certain author names indicate?“*

Hybrid rank 1 (correct): the two-sentence answer, exactly: “Authors are listed alphabetically by their first name. Names marked with * denote individuals who have departed from our team.”

ColBERT rank 1 (wrong): an unrelated paragraph about Mixture-of-Experts architecture that happens to mention “DeepSeek-V4”.

Right answer demoted to rank 8.

Why ColBERT failed here. The right answer is short, about 25 tokens. MaxSim’s score scales with the number of high-similarity doc tokens, so short answers are systematically under-scored. The long MoE paragraph wins on token count.

The pattern across the 41 cases where ColBERT demoted the right answer is consistent: it rewards token-rich generic content; BM25’s IDF weighting rewards rare-word matches that pinpoint specific answers. On a corpus of short technical paragraphs answering specific questions, hybrid retrieval is the right tool. ColBERT was built for a different corpus shape.

What we ship

The retrieval pipeline stops at dense + BM25 with RRF fusion. ColBERT MaxSim does not earn its keep on this corpus, and stacking it on top of hybrid does not help in any family. We keep the MLX jina-colbert-v2 port in the codebase as a documented building block. If the corpus later shifts to long documents or to an out-of-domain mix, this is the obvious next thing to revisit.

Code state at time of writing: 783af94. Reproduce the parsing numbers: uv run python -m benchmark.pdf_pipeline --variants codeformulav2_mlx. Reproduce the embedder + late-interaction sweeps: uv run python -m eval.sweep.