Confidence Scoring Makes AI Output Reviewable, Not Trusted

A model that is right 92% of the time and silent about which 8% is wrong is worse than a model that is right 80% of the time and tells you which lines it is unsure about. The first one makes every output suspect, so a human has to check all of it. The second one makes 20% of the output suspect, so a human checks 20% of it. Same work, very different cost.

This shows up hard in document and plan work. On BuildAEC we run vision models over raw construction drawings to produce quantity takeoff — counts, lengths, areas, pulled off PDFs that were never designed to be machine-readable. An estimator will not accept a flat list of quantities from a model. They have been burned. What they will accept is a list where each line carries a confidence score and the low ones are already flagged, sorted and linked back to the region of the drawing they came from.

The engineering problem is that most confidence numbers are fiction. The model's token probabilities do not mean what you want them to mean. A model asked "how confident are you?" will answer 0.9 to nearly everything. If you ship a fabricated score, you have not made the output reviewable — you have made it worse, because now people trust the wrong lines.

What a confidence score is actually for

A confidence score has exactly one job: to sort output into buckets that get different treatment. It is a routing signal, not a truth claim.

That reframing kills a lot of bad design. You do not need a calibrated probability accurate to two decimal places. You need a number that reliably puts the wrong answers near the bottom. Ranking quality matters more than absolute value. If your score is systematically inflated but the ordering is right, you can fix it with a threshold. If the ordering is random, no amount of calibration saves you.

So the test for a confidence signal is simple. Take a hundred labeled outputs, sort by score, and look at the error rate in the bottom decile versus the top decile. If the bottom decile is not several times worse, your score carries no information and you should stop displaying it.

Where honest confidence comes from

Self-reported confidence from the model is the weakest source and the one everybody reaches for first. Use it, but only as one input among several, and never alone.

Structural signals

These are the cheapest and often the strongest. They come from the shape of the task rather than the model.

• Did the extraction find its evidence? For a value pulled from a document, can you locate the exact source span or bounding box? No anchor, low confidence. This alone catches a large share of fabrication.
• Does it satisfy known constraints? Line items should sum to a total. A door count should not exceed the room count by an order of magnitude. Dates should fall inside the project window. Constraint violations are a hard downgrade, not a soft one.
• Was the input legible? Scan quality, OCR character confidence, page rotation, whether the region was hatched or overlapped. Garbage in should propagate as low confidence out, not be laundered into a clean number.
• Is the retrieval thin? For RAG features, the similarity score of the top chunk and the gap between the top and third chunk both tell you something. A question answered from one weak chunk deserves a different treatment than one supported by four strong ones.

Agreement signals

Run the extraction twice with different prompts, or two models, or the same model at temperature 0 and 0.4. Where the answers agree exactly, confidence rises. Where they diverge, you have found a genuinely ambiguous case without needing a human to tell you.

This costs you a second inference call, which is real money at volume. We only apply it to the categories where errors are expensive, decided by the same reasoning as whether a feature needs an LLM at all. Doubling spend on every field is waste; doubling it on the three fields that trigger a billing event is cheap insurance.

Score the line, not the document

The most common design mistake is one confidence number for the whole response. A takeoff with 140 lines gets a single 0.87 and a reviewer learns nothing. They still have to read all 140.

Every atomic unit a human could accept or reject independently needs its own score. That means per line item, per extracted field, per matched product, per classified transaction. The unit of confidence must match the unit of review.

Here is the shape we return from extraction endpoints. Keep it boring and keep the evidence attached.

{
  "field": "door_count_level_2",
  "value": 34,
  "confidence": 0.61,
  "band": "review",
  "reasons": ["symbol_overlap", "low_scan_dpi"],
  "evidence": {
    "page": 7,
    "bbox": [412, 880, 1190, 1460],
    "model_pass_a": 34,
    "model_pass_b": 31
  }
}

The reasons array matters more than the number. A reviewer who sees "low scan dpi" knows to zoom into the drawing. A reviewer who sees 0.61 knows nothing and has to redo the work from scratch. Machine-readable reasons also let you measure which failure mode is costing you the most review time, which is how you decide what to fix next.

Pick thresholds from review cost, not from feel

Three bands are enough: auto-accept, review, reject. The interesting question is where the cut lines go, and the answer comes from arithmetic rather than taste.

Work out two numbers for the specific feature. First, the cost of a human checking one item — call it the review cost. Second, the cost of one wrong item reaching the customer — the escape cost. Then the rule is: auto-accept where the expected escape cost of the band falls below the review cost of the band.

An illustrative example. Say checking a takeoff line takes an estimator 30 seconds, and a wrong quantity that reaches a bid costs an hour of rework plus a credibility hit. The escape is roughly 120 times more expensive than the check. So you auto-accept only where the error rate is below about 1 in 120 — a very high bar, and in practice that means most lines get reviewed. Fine. The win is that the reviewer starts with the 15 flagged lines instead of scanning 140 undifferentiated ones.

Flip the numbers and the answer flips. Auto-tagging product descriptions in a catalog on Retailer OS has a low escape cost — a slightly off tag is a minor annoyance, correctable later, visible to whoever spots it. There, auto-accept goes wide and only the bottom band gets queued.

Set the threshold from what a mistake costs, not from what the score looks like.

Build the review queue before you build the model call

A confidence score with nowhere to route is decoration. The queue is the product. Order matters here, and we build it in this sequence:

1. Define the atomic review unit and its accept/reject/edit actions in the data model.
2. Build the review UI against hand-written fixture data — no model in the loop at all. If a human cannot review the output efficiently with perfect data, they will not manage it with imperfect data.
3. Log every human decision: original value, corrected value, band, reasons, time taken.
4. Add the model. Route everything to review at first, regardless of score.
5. Use two to four weeks of logged corrections as your labeled set. Now you can measure whether your score ranks errors, and where the thresholds belong.
6. Raise the auto-accept cut only after the data supports it, and keep sampling a slice of auto-accepted items for audit.

Step 6 is the one teams drop. Once a band is auto-accepted, you stop seeing its errors, so silent degradation after a model or prompt change is invisible. Sample 2–5% of auto-accepted items into the review queue permanently. It costs a fraction of the review budget and it is the only alarm you have.

The correction log is also the most valuable asset the feature produces. It is your eval set, your fine-tuning data if you ever need it, and your evidence when someone asks whether the AI is actually working. We treat it as core schema from day one on every AI feature we build, not as telemetry to be added later.

When confidence scoring is the wrong tool

It does not help for open-ended generation. If the model is drafting an email or summarizing a thread, there is no atomic unit with a right answer, so there is nothing to score. The reviewable unit is the whole draft, and the review mechanism is an edit box, not a confidence band. Don't bolt a number onto it.

It also fails when there is no reviewer. A consumer feature where the user has no expertise to judge the output cannot route to human review — the user is the human, and they cannot tell a good answer from a bad one. On EzyEvite the AI designs invitations, and the answer there is not a confidence score. It is making regeneration instant and free, so a bad output costs one click. Cheap retry beats accurate self-assessment when the user cannot evaluate.

And there is a real tradeoff in the scores themselves. Every signal you add — second pass, constraint checks, evidence lookup — adds latency and cost to a call that was already the slowest thing in the request. On interactive features we compute the cheap structural signals inline and defer the agreement pass to a background job that upgrades or downgrades the band after the fact. The user sees a provisional band immediately and a settled one within a minute.

The point of all of it is to change what you are asking a person to do. Without confidence, you are asking them to trust a black box or to redo its work. With it, you are handing them a sorted worklist where the top of the list is the part that needs a human brain. That is a feature people will actually use, and it is the difference between an AI capability that ships and one that gets switched off after a month. If you're building one and want a second opinion on the review loop, get in touch.