Does That Feature Need an LLM or Just a Query?

A founder asks for "AI-powered inventory insights". You dig into what the users actually want and it is: which SKUs have not sold in 60 days, grouped by location, ordered by tied-up capital. That is a query. It is about eleven lines of SQL, runs in 40ms, costs nothing per call, and is correct every single time. Wrap it in an LLM and you get a feature that takes four seconds, costs a fraction of a cent per view, and occasionally reports a number that does not exist in the database.

This happens constantly, in both directions. Teams reach for a model where a join would do, and they also try to hand-code rules for problems that are irreducibly fuzzy — parsing a supplier invoice that arrives in a different layout every month, or reading a construction drawing that a human draftsman laid out by feel. Both mistakes are expensive, but they fail differently. The unnecessary LLM burns money and trust. The missing LLM burns engineering months on a rules engine that never stops needing rules.

We build both kinds of feature across our products and client work, and the deciding question has turned out to be narrower than most framework diagrams suggest. It is not "is this AI-shaped". It is: is the mapping from input to output enumerable?

The decision rule

Run the feature through these four questions in order. Stop at the first "no".

1. Is the input structured, or can it be made structured cheaply? Rows in a table, a form submission, an event payload — structured. A PDF a supplier emailed, a photograph of a shelf, a sentence a user typed in their own words — not structured.
2. Can you write down every valid output? If the answer is a number, a filtered list, a boolean, a category from a fixed set of twelve, or a sorted ranking, you can enumerate it. If the answer is prose, a design, a plan, or a mapping onto categories you have not fully defined, you cannot.
3. Would two competent engineers, given the same input, produce the same output? If yes, the logic is specifiable. If they would argue about it, you are dealing with judgement.
4. Does the value depend on handling inputs you have never seen? A report over your own schema does not. An importer that must accept whatever a customer sends does.

Structured in, enumerable out, specifiable logic, closed input set — write the query. Fail any one of those and a model earns its place. Fail two or more and it is not close.

The rule is deliberately biased toward the query. That is because the query is the cheaper thing to get wrong. A bad query returns a wrong number you can reproduce, trace and fix in an afternoon. A bad model call returns a plausible wrong number that you cannot reproduce, and you find out when a customer does.

Where the line actually falls in practice

Some worked examples from features we have shipped.

Query, not a model

• "Show me my slowest-moving stock." Structured in, ranked list out. In Retailer OS this is a materialized view per location, not a prompt.
• "Which customers are about to lose loyalty points?" A date comparison and a threshold. Adding a model adds latency and a failure mode to arithmetic.
• "Flag orders where the marketplace price drifted from our price." Two numbers and a tolerance. Deterministic, auditable, testable.
• "Route this invoice to the right approver." If the routing table is finite and known, it is a lookup. It becomes a model problem only when the routing depends on reading the invoice's contents in free text.

Genuinely needs a model

• Reading a construction drawing. Sheets arrive as scans, exports, or photos of printouts. Symbol conventions vary by draftsman. In BuildAEC the vision model produces quantity takeoff lines with a confidence score per line, and low-confidence lines go to a human. No rules engine survives contact with real plan sets.
• Designing an invitation from a one-line description. In EzyEvite the user types "vintage circus, muted palette, kid's fifth birthday" and expects a design. The output space is unbounded by definition.
• Answering questions over a document corpus. We have run OCR pipelines feeding a vector store of around 21,000 embedded chunks so users can ask questions in their own words. The input is free text, the answer is prose, and the retrieval step decides which chunks are relevant. All three of those are model work.
• Normalizing a supplier catalog import. "Bl. Widget, 10mm, pk/50" against your own SKU taxonomy. This is the classic case where the input set is open and no lookup table converges.

The hybrid pattern that beats both

Most features people describe as "AI" are one model call wrapped around a lot of deterministic code. The model's only job should be the part that is irreducibly fuzzy — usually translating a human's intent into structured parameters. Everything downstream is your own code, running against your own data, with your own permission checks.

Concretely: do not let the model compute the answer. Let it fill in a form.

// The model's entire output. Nothing else.
type StockQuery = {
  intent: "slow_movers" | "stockouts" | "margin_leaders";
  location_ids: string[];      // resolved against tenant's locations
  days_since_last_sale?: number;
  limit: number;               // clamped 1..100 server-side
};

// Then: validate, clamp, scope to tenant, run parameterized SQL,
// format the result. No generated SQL. No model-authored numbers.

This gets you the natural-language interface without the failure modes. The model can be wrong about intent, which is visible and correctable — the UI shows the interpreted filters and the user fixes them. It cannot be wrong about the numbers, because it never touches them. It also cannot cross a tenant boundary, because the query is scoped in code, not in a prompt. Running 100+ per-tenant Postgres databases behind 30+ Node services taught us to keep isolation in the layer that cannot be talked out of it.

The model should choose the question, never compute the answer.

What the LLM version actually costs you

Token spend is the cost people budget for and the least of the four.

• Latency. A query returns in tens of milliseconds. A model call is hundreds of milliseconds to several seconds, and you cannot cache it as easily because the input is free text. On a dashboard that loads six panels, that is the difference between a page and a wait. On our offline-capable POS it is disqualifying: a till that pauses for a network round trip at checkout is a broken till.
• Testability. A query has a fixture and an assertion. A model feature needs an eval set, a scoring function, and a decision about what regression means. That is real work, and skipping it means you ship changes blind.
• Failure surface. Rate limits, provider outages, truncated outputs, schema drift when you change models. Each needs a fallback path. Every deterministic feature you replace with a model call is a new dependency on someone else's uptime.
• Cost variance. Not the mean, the tail. One user pasting a 40-page document into a chat box can cost more than a thousand normal calls. We build usage metering, entitlement gating and AI spend wallets with auto-recharge for exactly this reason — per-tenant ceilings are not a billing feature, they are a safety feature.

Add those up and the honest comparison is not "$0.002 per call versus free". It is "a new subsystem with an eval harness, a spend cap and a fallback path" versus "a SQL file and a test".

Where this advice does not apply

Two exceptions we will defend.

The long tail of reports. If you have 300 possible questions and each individual one is a query, building 300 report screens is worse than building one natural-language interface that emits structured parameters. The per-question logic is enumerable; the set of questions is not. That is the hybrid pattern above, and it wins on maintenance, not capability.

Discovery before specification. Sometimes you do not yet know the rules. Running a model over a few hundred real examples and reading the outputs is a fast way to learn what the categories are. Then you write the deterministic version. We have shipped model-based classifiers, watched the distribution collapse into six stable buckets, and replaced them with a lookup table plus a fallback. That is a good outcome, not a failure — the model paid for itself as a research tool.

The inverse also holds. Do not build a rules engine for an open input set because rules feel safer. If your importer's ruleset has grown for six months and still gets escalations every week, you did not need more rules. You needed a model with a confidence score and a human review queue for the bottom decile.

A pre-build checklist

Before writing the first prompt, get written answers to these. If you cannot answer them, the feature is not ready regardless of implementation.

1. Write the deterministic version in one paragraph. If you can, build that first and measure how often it is wrong. That number is now your bar.
2. Name the fuzzy step. One sentence. If you cannot isolate it to one step, the design is not decomposed yet.
3. Define wrong. What does a bad output look like, who notices, and what does it cost? "Slightly odd wording" and "wrong quantity on a bid" need very different guardrails.
4. Choose the confidence mechanism. Structured output validation, a self-reported score, retrieval-grounding checks, or a human review threshold. Pick one before you build. Confidence scoring bolted on afterwards never lines up with the failure modes you actually see.
5. Set the spend ceiling per tenant and per request. Including the tail case.
6. Write ten eval cases by hand, including three that should fail. Ten real cases beat a hundred synthetic ones.

The pattern underneath all of this: models are good at ambiguity and bad at arithmetic; databases are the reverse. Features that survive production put each one where it is strong and keep a hard boundary between them. When we scope AI work, the first pass is nearly always finding which half of the proposed feature is a query in disguise — and that half usually ships in a week rather than a quarter.