Which AI Features Should Run Overnight: The Case for Batch Inference

Most AI features start life as a request that waits for a model. A user clicks, a spinner turns, an answer appears. That pattern is right for chat and wrong for a large share of what teams actually build. If the user is not waiting, nobody needs the answer in two seconds, and you are paying a premium and carrying spiky load for a latency requirement that does not exist.

Summaries of yesterday's activity, tagging of new documents, categorizing a day's transactions, drafting follow-ups for tomorrow's appointments: these are sitting in real-time code paths in many products because that is how the first prototype was wired. Moving them to a batch job is often the simplest cost and reliability change available.

The question is how to tell which features qualify, and what you take on when you move them.

Ask one question: who is waiting?

A decision rule that works in practice is to classify each AI feature by the latest moment someone needs its output.

• A person is watching the screen. Real-time, and streaming is worth it. Chat, inline suggestions, search answers.
• A person will look within minutes. Near-real-time with a queue. "Analyze this upload" where the user can leave and come back.
• A person will look later today or tomorrow. Batch. Daily digests, nightly enrichment, bulk classification, report narratives.
• No person looks at all. Batch, with lower priority. Back-filling embeddings, re-tagging history, quality sampling.

The bottom two rows are candidates for overnight work. The top row is not, and forcing it into a batch will make the product feel broken.

What you gain

Batching changes the economics in three ways, and it is worth being specific about each.

1. Price. Some providers offer a discounted rate for asynchronous batch requests in exchange for a longer completion window. Check the current terms and the window, because the discount and the guarantee both vary by provider and change over time.
2. Smoother load. Real-time features spike when your customers are awake. Batch work can be spread across the quiet hours, which keeps rate limits free for interactive traffic.
3. Better prompts for the same money. When latency is not a constraint, you can afford a larger model, a longer prompt with more context, or a second pass that checks the first. Quality often improves more than cost does.

We discuss the adjacent levers, such as prompt caching, in caching and prompt reuse as the cheapest AI cost cuts. Batch and caching are complementary: a nightly job that shares a long instruction prefix across thousands of items is close to the best case for both.

What you take on

Overnight jobs fail differently from requests. A failed real-time call is visible immediately. A failed batch is discovered when someone opens the dashboard at 9 a.m. and finds yesterday's numbers missing. Plan for these four things before you move a feature.

Partial failure

A batch of 20,000 items will have some that error out, some that return malformed output and some that the model refuses. Treat each item as its own record with a status, not the batch as a single success or failure. Retry the failed items, and put anything that keeps failing in a visible dead-letter list.

Idempotent writes

Retrying a batch must not double-apply results. Key each output to the input it came from, and make the write an upsert. Applying the same result twice should leave the same state.

A freshness contract

Decide and document how stale a result may be. "Tags are updated nightly" is a feature. "Tags are sometimes updated" is a bug report. Show a last-updated time in the interface where it matters.

A deadline and an alarm

If the job has not finished by a set hour, someone should be told. A scheduled check that compares expected output count against actual count is a few lines of code and catches most silent failures.

A minimal job shape

nightly_run(date):
  items = select unprocessed items for date
  for chunk in split(items, 1000):
    submit chunk as one batch request, tag each item with item_id
  poll until batch complete or deadline reached
  for each result:
    validate against schema
    if valid:   upsert(item_id, result), mark done
    elif retries < 2: requeue(item_id)
    else:       mark failed, add to review list
  compare done_count to expected_count
  if gap > threshold: alert owner

Nothing here is exotic. It is a queue, a validation step and a reconciliation count. The reconciliation count is the part teams skip, and it is the part that tells you the job did what you think it did. It follows the same principle we use when proving a data migration worked: count what went in, count what came out, and treat a mismatch as an event.

Where batch is the wrong call

Do not batch anything a user is waiting on, even if the cost saving looks attractive on a spreadsheet. A feature that takes hours to respond is a different feature. Also be cautious with items whose value decays quickly: a "next best action" suggestion generated at 2 a.m. may be stale by the time a rep opens it at 10 a.m. after a customer called in.

There is also a middle path worth considering: a hybrid. Precompute overnight for the common case and fall back to a real-time call when a user needs something fresh or the precomputed result is missing. It costs extra code, and it is only worth it when a meaningful share of views hit the precomputed path.

Picking your first candidate

Look for a feature with three traits: output is consumed later, volume is high enough that cost matters, and a wrong or late answer is easy to tolerate. Document tagging and daily summaries usually qualify. Move one, measure the cost and the failure rate for two weeks, and then decide on the next. If you build AI into a multi-tenant platform, per-tenant spend controls matter here too, which we cover in keeping AI spend predictable with wallets and budgets. For help classifying your own features, see our AI and agentic systems work or get in touch.