What Breaks When Vision Models Read Real-World Documents

Vision models score well on clean, well-lit sample pages. The demo works because the demo document is a PDF exported straight from a modeling tool: upright, high resolution, single column of text. That is not what shows up in a client folder. Real-world documents are second and third generation fax scans, photographs taken with a phone at a slight angle, PDFs with hand-marked revisions in red pen, and pages where three people used three different unit systems over two years and nobody caught it.

On BuildAEC we run vision-model takeoff against raw construction plans, the kind of drawing set a general contractor pulls off a jobsite trailer, not a clean architectural export. The gap between reading a demo PDF and reading what actually gets uploaded is where a takeoff tool either earns trust or gets abandoned after one bad estimate. A model that misreads a wall dimension because it confused a scale bar with a dimension line does not fail loudly. It fails quietly, in a number that looks plausible, and shows up three weeks later as a change order dispute.

The fix is not a better model. It is an engineering system built on the assumption that the model will misread something on every document set, and the job is catching which parts before they reach a customer.

Where vision models actually break on real documents

Scan quality and skew

Pages photographed at an angle, faxed pages with dropout, low-contrast blue-line prints - accuracy does not degrade gracefully with these. It drops off a cliff past some rotation angle and below some effective DPI. The model will still return an answer with the same confident tone it uses on a perfect scan, which is the actual problem: nothing in the output tells you the input was bad.

Tables and multi-column layouts

General-purpose vision models handle a single column of prose well. Multi-column schedules, nested tables, and door or window schedules split across a header on sheet one and data on sheet fourteen break the model's ability to map a value to the right row and column. This is a layout-understanding failure, not a reading failure, and it is much harder to catch because the individual characters are read correctly.

Handwritten annotations and revision marks

Hand-marked revisions in red pen overlapping printed text are exactly the highest-value information on the page - they are the change - and exactly the lowest-confidence read the model will produce. The model may read the annotation as the answer, ignore it entirely, or blend the two into something that matches neither. There is no way to know which happened without a human looking at that specific region.

Units, scale bars, and inconsistent legends

The model reads what is printed, not what is meant. A legend on sheet one might say a quarter inch equals one foot, while sheet fourteen was redrawn at a different scale and the title block was never updated. The model has no way to know the legend is wrong unless the pipeline cross-checks the extracted quantity against a known reference dimension somewhere on the page.

Confidence scores are a routing signal, not a grade

The useful move is per-line confidence, not a single score for the whole document. A takeoff with one hundred line items and one bad scale read should not get thrown out wholesale, and it should not get accepted wholesale either. Confidence has to live at the level of the individual extracted value, because that is the level at which someone will decide whether to trust it.

{
  "line_id": "L-0412",
  "description": "2x6 stud wall, 8ft",
  "quantity": 340,
  "unit": "LF",
  "confidence": 0.62,
  "source_region": [1180, 640, 1420, 700]
}
// routing rule
// confidence >= 0.90  -> auto-accept, sample 5% for audit
// 0.60 <= conf < 0.90 -> human review queue
// confidence < 0.60   -> flag, do not use in totals

The thresholds above are illustrative, not universal. The right cutoffs depend on what a wrong number costs you downstream. On a takeoff feeding a bid, a bad quantity is expensive enough that the review band should be wide. On an internal search index, a lower bar is fine because a human reading the source will catch the error anyway.

Build the review queue before you need it

A confidence score is only useful if there is somewhere for a low-confidence line to go. Building that queue after the first bad extraction reaches a customer is too late. It needs to exist on day one, and it needs to be fast enough that reviewing flagged lines is faster than re-checking the whole document by hand, or nobody will use it.

• Side-by-side view of the source region and the extracted value, not a separate tab the reviewer has to switch to
• A single-click correct or confirm action, not a form
• The corrected value written back to the same record the low-confidence value came from, so nothing downstream has to be re-run manually
• Every correction logged with the original model output, because that log is your evaluation set

That log matters more than it sounds like it should. Corrections are free labeled data, and they are the only labeled data you will get without paying someone to annotate documents on purpose. We've written before about evaluating an AI feature with no labelled dataset and no time, and a review queue is the most practical way to build one while the product is running.

Test on the documents that will actually arrive

Testing a vision model against clean, source-exported PDFs and then deploying it against phone photos and third-generation faxes is testing a different product than the one you shipped. Before building anything, collect fifty real documents from the messiest part of the pipeline - the folder full of scans a customer forwarded from a flip phone, the plan set with three rounds of red-pen revisions - and run every model change against that set first.

If the extracted text is going to feed a retrieval layer or get summarized by an LLM downstream, two more problems show up. First, chunking OCR output badly destroys structure that mattered, which we cover in how to chunk invoices, plans and contracts for retrieval. Second, text extracted from an untrusted scanned document can contain instructions a customer never meant to send you, which is its own failure mode covered in prompt injection hiding inside customer documents.

The cost and latency question nobody asks until the bill arrives

Running full vision inference on every page of every document a customer uploads adds up fast, especially on multi-page plan sets or long contracts where most pages carry no extractable value. Skip pages with no text or drawing content before they hit the model, cache results keyed on a document hash so a re-upload does not trigger a re-run, and batch pages instead of calling the model once per page where the API supports it. Per-tenant AI spend also needs a hard ceiling, not just a monitoring dashboard, which is the argument we made in keeping AI spend predictable with wallets and per-tenant budgets.

When a vision model is the wrong tool entirely

If the source document is digital-native - a text layer already present, a consistent template, fields in predictable positions - a vision model is slower, more expensive, and less deterministic than a parser built against that template. Save the vision model for the scans, photos, and inconsistent layouts where a parser has nothing to hold onto. Mixing the two, parser first with vision model as fallback when the text layer is missing or garbled, usually beats running everything through the model by default.

The honest tradeoff runs the other way too: at low volume with high stakes, a human transcribing the document by hand can be cheaper and more accurate than building and maintaining an extraction pipeline at all. Building the confidence scoring, review queue, and eval set described above only pays off once volume is high enough that a person reading every document is no longer realistic. Below that line, the cheaper answer is a person and a spreadsheet, not a model.

None of this is specific to construction plans. The same pattern - skewed scans, hand annotations, inconsistent layouts, quiet failure on plausible-looking numbers - shows up in invoices, medical intake forms, and shipping manifests. If a document pipeline is on your roadmap, our services page covers how we scope that work, and the approach page covers how we price it before anything gets built.