Most analytical AI workloads can be run as batch inference jobs. Like historical data processing analogs, batch processing has a number of appealing properties - faster job completion times (via higher throughput on large sets of inputs), lower costs, and lower need for constant resource allocation.
So why don't all teams rely on batch inference for non user-facing workloads?
We generally find that it comes down to a few reasons:
- Real-time inference APIs have been the norm since model companies began offering foundation models as a service. Many teams simply architected their systems around these APIs, and switching costs haven't seemed worthwhile.
- Many systems are high-volume - meaning many requests arrive over time, but not necessarily batch-oriented (all at once). It's possible to process these requests somewhat asynchronously or queue them to run as a batch, but they're realistically best handled as event-driven.
- Many teams simply aren't at a scale where batch inference, even if saving >50% on inference costs, would help them materially.
When batch is better
| Workload characteristic | Batch inference is usually better when... | Real-time inference is usually better when... |
|---|---|---|
| Latency tolerance | Results can arrive minutes or hours later without hurting the product or workflow. | The user, system, or transaction is waiting for the result immediately. |
| Input shape | Inputs arrive in large groups, scheduled exports, backfills, or periodic refreshes. | Inputs arrive one at a time or continuously as user actions and events happen. |
| Volume vs. Cost | The workload is large enough that provider batch discounts, higher utilization, or self-hosted batching materially reduce cost. | Volume is low enough that batching savings do not justify added system complexity. |
| Failure handling | Failed items can be retried, inspected, or replayed asynchronously. | Failures need immediate fallback behavior because they block a live experience. |
| Task orientation | The job is a data-processing pipeline: enrich records, classify documents, extract fields, score accounts, or reprocess historical data. | The model is part of an interactive product loop, complex agentic process, routing decision, or user-facing assistant. |
| Capacity planning | You can schedule work around available compute, rate limits, or cheaper off-peak capacity. | Capacity must be available whenever requests arrive. |
| Versioning and replay | You want to re-run the same task over a fixed dataset with a specific model, prompt, and schema version. | The freshest context matters more than reproducible replay over a bounded dataset. |
A simple rule of thumb: use batch inference when the work looks like a data pipeline; use real-time when the model sits inside an interactive user-facing application.
Batch limitations
There are a handful of system tradeoffs that should be made when considering the use of batch inference.
- Often, batch inference is best for single-turn LLM calls, where all of the context is known upfront. If there is some unknown number of tool calls, sandboxed code executions, or other steps that interrupt inference, some of the benefits of high-throughput inference can be defeated or made more challenging to architect around.
- Many providers offering batch inference services have 24-hour, or worse, 72-hour SLAs and flaky success guarantees. Not only can this be problematic to design around, it makes prototyping harder, and nearly defeats the possibility of using batch inference services for experimental work.
- Most batch inference services are also often second-class products: little observability, rigid handling of data, and inconsistent response times.
Flex/Async Processing
Although not extremely popular, a middle-ground that some inference providers have started to offer is flexible or asynchronous processing modes. These are one-off calls (rather than batch) that are designed to have longer processing times and run in the background, or with some longer SLA in exchange for lower costs. If your application is event-driven but can tolerate higher latency, this may be a good solution as an alternative to batch.