Baseten, a company that sells managed inference infrastructure for large language models, published an engineering breakdown separating two categories of work that get lumped together under performance tuning. One category shifts a deployment along a fixed tradeoff line between latency and throughput. The other expands what the deployment can do at every point on that line simultaneously.
The distinction matters because the two categories call for different engineering responses. Moving along a tradeoff means picking a target, low latency for a chat product, high throughput for a batch pipeline, and reconfiguring existing infrastructure to hit it. Expanding the tradeoff means shipping a faster kernel, a better speculative decoding method, or a smarter way to split GPUs across a model, so the same hardware serves more of both latency and throughput than before.
Baseten frames the first category around three levers engineering teams already control. Batch size is the most direct: a small batch keeps per-user response times fast but leaves GPU capacity underused, while a larger batch raises total tokens served per GPU at the cost of individual response speed. Parallelism strategy, meaning how a model’s weights get split across multiple GPUs, is the second lever. Baseten states that Tensor Parallelism favors low latency because its GPU-to-GPU communication runs over fast interconnects, while wider Expert Parallelism favors throughput, and Attention Data Parallelism trades per-request speed for system-wide capacity. Quantization, running a model at lower numerical precision, is the third: Baseten says it improves both latency and throughput at once, though it opens a separate tradeoff against output quality that the company describes as forgiving when using newer low-precision formats like MXFP4 and NVFP4.
The second category, per Baseten, is where kernel optimization, speculative decoding, and prefill-decode disaggregation live. Faster CUDA kernels cut the compute needed per generated token across the board. Speculative decoding techniques such as EAGLE-3 and DFlash let a smaller draft process guess upcoming tokens for the main model to verify, which Baseten says now works well enough on predictable outputs like code to lift throughput without much added latency. Disaggregation, splitting the prefill and decode stages of inference onto separate worker pools, lets each stage be tuned for its own bottleneck rather than sharing hardware built for neither.
Here is the operational split the source blurs by presenting both lists as equally available choices: a team hitting a latency or cost target it cannot reach today should look first at batch size, parallelism, or quantization settings, because those are configuration changes against existing infrastructure. A team whose targets are unreachable even after that tuning needs a different serving stack, engine, or hardware generation, because no combination of batching and parallelism settings manufactures capacity that was never built into the deployment. Baseten’s own business, selling the inference layer that implements both categories, gives it a direct interest in customers reaching for the second option sooner rather than settling for endless tuning of the first.
Teams running LLM inference in production should audit which category their current bottleneck falls into before their next optimization sprint. A team still adjusting batch sizes and parallelism configs six months after launch is very likely tuning a ceiling it cannot raise, and the fix is a kernel, decoding, or hardware upgrade rather than another sweep.
Baseten engineer Philip Kiely detailed the distinction in a company blog post, “The efficient frontier of LLM inference.”
