have been imagined to be learn by a machine. Previous lodge invoices, financial institution statements, payslips, mortgage purposes, medical payments, customs kinds, court docket filings, work orders.
Most firms use free instruments alongside paid APIs to attempt to convert these paperwork, and in order for you structured output, APIs like Textract Structured run you as much as round $65 per 1k pages.
In the previous couple of years, although, a whole lot of new choices have appeared: smaller open-source imaginative and prescient fashions specialised for OCR, basic vision-language fashions, and doc parsing instruments like LlamaParse — altering what’s doable and the price thereof.
So it felt like a good time to do my very own experiment to check a few of these towards paperwork of various problem.
I scouted 93 docs that would act as a proxy for what firms use OCR for — handwritten notes, tables, monetary legacy docs, scanned invoices, receipts, charts, previous newspapers, tax kinds — then ran all of them by way of 14 totally different engines.
The concept was to see how they dealt with two issues: textual content restoration and the power to protect helpful desk construction.
The principle query I wished answered: do you actually need to pay $65 for 1k structured pages, or are you able to slice that all the way down to a fraction? And does the specialised fashions win over the final ones?
When doing experiments like this you at all times discover fairly a couple of unusual issues, which I’ll cowl too. However to reply that primary query I’ll take you thru what OCR is (skip if not new), the economics, the check, among the outcomes, and what else this confirmed me.
Observe: I didn’t check full area extraction, since that’s tougher to check cleanly throughout fourteen engines.
TL;DR
There is no such thing as a single finest OCR engine. OCR is a routing downside.
For clear high-volume paperwork, Tesseract remains to be arduous to beat as a result of it’s free and quick. For blended manufacturing paperwork, Gemini Flash was one of the best all-rounder on this check. For tables, Mistral OCR appeared just like the cheaper structured choice.
The smaller specialist fashions appeared good inside their consolation zone, however failed tougher at paperwork they hadn’t seen. So, for high-stakes or messy paperwork, it is smart to escalate to a bigger mannequin.
The principle takeaway is financial: don’t pay for costly structured OCR when the doc doesn’t want it. Classify your docs, check engines by yourself information, and route based mostly on price, accuracy, construction, and failure tolerance.
Benchmarks are helpful for discovery, however they won’t inform you what works in your paperwork.
Clarify the OCR house to me
OCR (Optical Character Recognition) is how a machine turns an image into machine readable textual content. Easy in precept, and for simpler docs largely solved, however tougher when issues change into extra human.
Simply to provide you a fast overview, older OCR discovered textual content on a web page, sliced it into characters, and matched every one towards a library of recognized shapes. Tesseract has accomplished this because the Nineteen Eighties.
Trendy OCR nevertheless (together with newer variations of Tesseract) often makes use of a neural community that appears on the complete web page without delay and outputs the doc as textual content. So, in case your doc is a clear PDF or a high-quality scan in a typical font, OCR is generally a solved downside.
It stops being solved the second issues get messier: photographed receipts, handwritten notes, bizarre graphs and charts, dense monetary tables, or scanned tax kinds and mortgage purposes.
Firms want this accomplished effectively for apparent causes, because it’s one thing each downstream system acts on. The higher OCR will get, the extra paperwork turns into one thing a system can purpose over as a substitute of one thing a human has to learn by hand.
There’s additionally the truth that if we feed AI methods badly parsed docs, the whole lot after it is going to be arduous to belief.
I’m all about economics, so this house caught my eye as soon as I noticed how a lot cash is being poured into it. The Clever Doc Processing (IDP) market is projected to develop to someplace between $20 billion and $90 billion by the early 2030s, relying on which analyst you ask.
Most likely pushed by firms paying $15–25 per bill in guide dealing with prices.
And since I keep near the tech world, I’ve watched a wave of specialised small OCR fashions ship over the previous 12 months (largely Chinese language), now being utilized by builders all over the place.
Which raises the query I wished to check: can the small open-source fashions really do the work the costly APIs cost for or ought to we really look in the direction of the final imaginative and prescient fashions to deal with OCR too?
Skip the following part if you wish to perceive what this experiment confirmed. I’ve to undergo the check setup first.
The docs, the engines, and the metrics
This experiment comes down to a few questions: what engines we used, what docs we examined with, and the way we determined who received.
For the engines, I wished a lineup that coated all the alternatives I talked about, this meant: previous and new, open and closed, native and cloud, specialised and basic.
Tesseract grew to become the classical alternative. It runs regionally and may be very quick. Then I added two document-parsing pipelines: Docling and Marker. Docling is slower however runs on CPU, Marker is open-weight however needing a GPU to run quick, which reveals up later within the value.
Then for the brand new wave of specialised open OCR fashions: GLM-OCR, PaddleOCR-VL, DeepSeek-OCR, and MinerU 2.5 (a borderline case, actually a pipeline with a VLM inside). I picked them off OpenDataLab’s OmniDocBench leaderboard, the place they ranked first, second, fourth, and fifth.
I hosted them on Modal and served the relevant ones with vLLM, batching to hurry issues up. I counted the scale-up time when measuring latency later.
I additionally added one closed purpose-built mannequin, Mistral OCR, which I’d heard good issues about.
On the open aspect, I used Qwen3-VL (8B, from Alibaba), additionally hosted on Modal with the remainder of the smaller fashions. I ought to flag that I gave it a plain transcription immediate quite than the optimized serving setup it was designed for, so I’ll not have given it a good shot.
On the closed aspect, for the final fashions, I picked Gemini Flash 3.1 Lite (at the moment first on the IDP Leaderboard, the western counterpart constructed on OmniDocBench v1.5) and Claude Sonnet 4.6, at sixth.
For the cloud doc companies: LlamaParse and AWS Textract, in each its textual content and structured kinds. Structured Textract can do excess of I requested of it. I solely examined its textual content accuracy throughout the board and its desk extraction towards eight of the opposite engines.
Let’s flip to the paperwork. I picked seventeen doc sorts that have been both simple, medium, or arduous. Ninety-three recordsdata in all.
Straightforward was the stuff OCR largely solved years in the past: clear invoices and receipts. Medium got here largely from the OmniAI OCR Benchmark dataset: financial institution statements, medical notes, photographed receipts, transport paperwork, tax kinds.
Laborious was chosen when issues turned tougher: charts, kinds, handwritten notes, weirdly scanned monetary tables, authorized papers, newspapers, and previous legacy stories.
Some docs have been actually fairly troublesome, such because the legacy scanned docs you see under, and this was simply because I used to be curious if some might really do it effectively.
A few of these photos got here with gold floor fact and a few didn’t, and the bottom fact I did have wasn’t at all times constant, some recordsdata labeled accurately, some not, which is why we should always briefly cowl the metrics too.
Since each engine emits totally different markup, the same old scoring didn’t fairly match. One would possibly decide Precision and Recall for a case like this.
Precision seems to be at how most of the OCR output’s phrases really match within the GT whereas Recall measures what number of instances every GT phrase was captured.
Precision would punish engines that emit markdown construction the GT doesn’t comprise, moreover the GT typically skipped labels completely which might punish the engine unfairly. Recall would measure the phrases however punish the frequency.
So, I added on a 3rd metric known as Protection. I simply wished to measure how a lot of the bottom fact reveals up someplace within the engine’s output. It isn’t excellent, but it surely tells me whether or not an engine caught most of what mattered, with out penalizing it for gaps that have been the bottom fact’s fault quite than the engine’s.
For the paperwork with no gold floor fact in any respect, I fell again on an LLM decide, with Gemini 3 Professional as the bottom mannequin and anybody who’s used one is aware of that is fickle enterprise.
What this experiment confirmed
We mapped each doc towards the Protection metric to construct a scatter chart, and tracked latency on a separate chart. The factor a generalized chart can’t inform you although is that the engines failed in numerous methods.
The bubble graph confirmed that almost all engines fall someplace within the center high, with two outliers on each side of it.
Gemini Flash and Textract Textual content did very effectively throughout the board with some edge instances. The specialised fashions all fell under the final fashions and specialised APIs. Sonnet carried out the best but additionally with a steeper price ticket.
This will likely not have been a shock because the check set was extremely uncommon. A few of the specialised fashions could not have seen lots of them. Moreover, this check was on English paperwork and most of those smaller fashions have Chinese language origin.
After we additionally mapped latency, among the fashions turned out to be very gradual, however once more most wound up someplace within the center.
The outliers right here have been: Tesseract, Claude Sonnet 4.6, and Docling. Tesseract was extremely quick in comparison with all different engines. It ought to be your go-to for simpler paperwork.
These graphs generalise throughout all of the paperwork, however I did separate the outcomes based mostly on the sort and problem degree.
To begin with the straightforward docs. On invoices, each engine did effectively, Tesseract particularly. Receipts knocked everybody down a bit of.
The one outlier was Docling, which struggled throughout a whole lot of the classes, even the straightforward ones.
Once I appeared into the Docling failures I discovered issues like Ifjointreturn as a substitute of “joint return,” and worse, strings like Metropolis,wrostffielfouaveaoreignadresalcomletacesb. DeepSeek additionally missed key particulars right here like bill quantity and date, which is why its quantity sits low.
The identical sample holds within the medium class, although that’s the place PaddleOCR began degrading on particular sorts: financial institution statements, transport, tax kinds. Tax kinds have been arduous for everybody, however PaddleOCR and Docling wound up on the backside.
Textract was one of the best engine on a whole lot of the medium sorts, together with Claude Sonnet 4.6 and Mistral OCR.
On the tougher sorts, Gemini Flash began rising, beating Textract on kinds and handwritten notes, matching it elsewhere. It did remarkably effectively all over the place. Tesseract and Docling failed arduous on handwritten, and kinds have been powerful for them too.
Nearly all of the specialised fashions didn’t pull by way of on these tougher docs besides on monetary tables, the place they held about even.
For the docs with no floor fact (newspapers, authorized, stories, some scanned legacy paperwork) we used an LLM decide. These are genuinely arduous, so it’s no shock nearly everybody failed on the stories and newspapers.
Besides Gemini Flash that did fairly effectively all over the place. Mistral OCR additionally did effectively for newspapers. Gemini Flash received all over the place with the decide, although we used Gemini Professional because the decide so take that with a grain of salt (however I did double verify myself).
Earlier than rounding off: I additionally ran 8 engines towards Textract Structured to see how they did on monetary tables, extracting an HTML desk. I used Textract Structured’s output as the bottom fact for TEDS (Tree Edit Distance Similarity) and scored Claude Sonnet 4.6, LlamaParse, Mistral OCR, Gemini Flash, Marker, MinerU, DeepSeek-OCR, and Docling towards it.
Mistral OCR, and LlamaParse, and Sonnet did very effectively whereas being less expensive. I additionally ran it by way of an LLM decide, and the winners have been the identical three (even earlier than Textract Structured) although I’d wish to construct that check higher earlier than I totally belief it.
Now, let’s speak about what it prices to scale this up, and what would make sense the place.
When does what make sense
Let’s run by way of what it prices to scale up with these engines, after which based mostly on these docs what you’ll selected the place.
First, the price of utilizing these engines range wildly, as you noticed earlier than. Typically it helps to see the price not only for one doc, however hundreds as much as one million.
We’re self-hosting on Modal, so these prices come from precise utilization there. You’ll be able to run regionally, however my laptop wouldn’t permit it and I didn’t wish to attempt it.
When you have been to only use one engine that handles each simple and arduous paperwork, I’d suppose you wound up with an even bigger invoice than mandatory. Utilizing Textract Structured for any paperwork that aren’t wanted would hand you a invoice of $6.5k per 100k docs.
I do marvel what number of firms go the straightforward manner right here and decide the costly choices for straightforward in addition to arduous docs and go away some huge cash on the desk.
The important thing concept to take with you right here is that there’s no single finest engine for each use case, it is determined by doc kind, privateness, desk construction, failure tolerance, price, and so forth.
For the docs we now have right here, Gemini Flash 3.1-Lite is a transparent winner. This one was appropriate from wanting on the leaderboards. Mistral OCR did effectively on structured tables whereas staying low-cost. Claude Sonnet 4.6 did very effectively too, but it surely’s very gradual and costly comparatively.
Docling is so very gradual on my laptop computer. I’m positive there are methods to hurry it up, but it surely additionally failed in ways in which make it inherently unstable (nonetheless a small check although).
The specialised OCR fashions have been a little bit of a headache, particularly on English docs; I noticed output errors in Chinese language that I’ll cowl in a bit, so I’m wondering if that’s a part of it.
Textract is a steady alternative, however structured buys you nearly no further textual content accuracy so when you’re paying that steep markup for structured output, be sure to really use it. I’m guessing it’s a reasonably good enterprise mannequin for them.
So, usually for this very small check: for clear, high-volume print, simply use Tesseract. For basic heterogeneous manufacturing, go Gemini Flash. For a cost-floor with desk construction, check Mistral OCR. For top-stakes docs, path to Sonnet or a bigger mannequin.
Since everybody did effectively in numerous methods you’ll must contact me for specifics but when you’ll want to go personal it could be price it to take a look at fine-tuning a mannequin in your docs. Or, use a small specialised mannequin and escalate on failures.
Let me simply shortly speak about some issues that stood out after doing this experiment.
Different stuff I ought to point out
A handful of issues surfaced from this which might be price pulling out on their very own.
First, if you wish to perceive how a mannequin or engine will do in your docs, the one manner is to check on these docs, you’ll be able to’t depend on benchmarks to inform you. This was the primary perception this confirmed. OCR usefulness relies upon by yourself doc combine, layouts, languages, scans, tables, handwriting, and failure tolerance.
Don’t pay for construction when you don’t want it. I’m wondering what number of are utilizing sure APIs or fashions for a purpose they will’t justify. Map the price to grasp what you’re dropping by not utilizing the proper engine for the paperwork.
The specialist fashions, as talked about earlier than, have sharp boundaries. That is apparent, they are often glorious inside their coaching distribution however fail outdoors of it. That is the place the final fashions will win.
If you wish to fine-tune it could assist, however provided that the stream is steady as it can additionally fail whether it is consistently launched new doc lessons.
Lastly, the failure modes advised us greater than the averages.
PaddleOCR had repetition loops, column-merging, fallback into Chinese language textbook template textual content like 书名:___ repeated a whole bunch of instances. Whereas Docling has character errors, word-merging, and column misalignment all stacking collectively.
DeepSeek OCR has chart blindness and empty outputs on some docs. Tesseract did tremendous on clear docs (as talked about) however failed on photograph/handwriting altogether outputting rubbish.
Caveats to think about
Earlier than we spherical up, let me cowl how this check is finally imperfect by naming the problems within the GT, the metrics used, and the pattern dimension.
I coated this in one of many part above, however the floor fact differs between paperwork relying on the dataset the place they have been discovered. Generally tokenization artifacts could make appropriate OCR look worse than it’s.
Most engines have totally different codecs, some return plain textual content, some markdown, some HTML/wealthy markdown and it’s arduous to generalize throughout all.
We’re utilizing Protection, after which additionally another metrics, however these aren’t excellent. Protection received’t cost the engine if it outputs an excessive amount of textual content or the construction of it’s off. Although I did discover that for the engines that failed, they did so at first or mid-way by way of quite than on the finish.
This implies it’s helpful for rating however not an ideal strategy to rating.
LLM judges usually are not impartial fact: I’ve coated this prior to now, however they’re biased and really immediate delicate.
Then I simply have to say that this check is attention-grabbing however not that huge, the pattern dimension is manner too small to make use of this as a factual research. However, I don’t totally belief these metrics nor the decide so it was the one manner for me to have the ability to double verify the outcomes alone with out this turning right into a 12 months lengthy challenge.
So, this check is beneficial for path and getting a way of what works, however for getting a way of your use case, you’ll want to run it by way of along with your particular docs.
Lastly, latency and reproducibility is unstable. Serverless chilly begins make timing noisy, and API fashions can silently change over time, so precise copy is difficult.
Like at all times with these articles, it takes fairly a bit to do an experiment like this however I don’t simply do it for content material, I do it as a result of I’m genuinely curious.
What it seems to be like although is that OCR appears to be a routing downside, and maybe an analysis downside. Classify your docs and run them by way of a number of engines, then attempt to construct an honest router and validator in your pipeline to escalate failures after which log the prices.
If you’ll want to get the total outcomes from this experiment otherwise you need me to run it by way of your docs, get in touch.
You’ll be able to comply with my writing on Medium, my website or join with me by way of LinkedIn.
❤
All datasets used on this benchmark are publicly out there and sourced from HuggingFace. Licenses embody MIT, CC-BY-4.0, and fair-use frameworks (UCSF Business Paperwork Library) overlaying analysis, scholarship, and training. No supply paperwork are reproduced — datasets have been used solely as analysis inputs to measure OCR engine efficiency.
