A widely read post on r/LocalLLaMA lays out the current pecking order of inference software for Apple Silicon, and the conclusion is blunt: no single runtime yet combines the optimizations CUDA-based stacks have had for years. The author, a community member who says they spent two weeks testing frameworks full time, argues that vllm-metal is the most complete option available today. Even it forces a tradeoff between two techniques Mac users need running at once, and that tradeoff, not a benchmark score, is the real decision point for anyone provisioning a Mac for local inference.
Apple Silicon has become a default target for local large-language-model inference because a single machine’s unified memory can hold weights that would otherwise require multiple discrete GPUs to fit. That capacity is part of why models like Qwen3.6 and Qwen3.8 show up on Mac hardware within days of release. The software underneath has not caught up: the post catalogs techniques such as continuous batching, flash attention, dynamic scheduling, paged KV caching, prefix caching and speculative decoding as mature on Nvidia stacks but scattered across mlx-lm, vllm-metal and a growing list of one-off forks on Apple Silicon.
The sharpest edge is architectural. Newer Qwen releases use a hybrid cache pairing a conventional KV cache with a recurrent state (a Gated DeltaNet design), and that recurrent state overwrites itself as it processes tokens rather than preserving full history the way a flat KV cache does. That difference makes prefix caching, which lets a server skip reprocessing a conversation’s earlier turns, much harder to implement for these models, and it complicates speculative decoding, which predicts several tokens ahead and needs a clean rollback whenever a prediction misses.
Among the runtimes, the post is specific. llama.cpp did not deliver competitive throughput in the author’s testing and ran noticeably hotter. mlx-lm, the base layer Apple provides and that most Mac projects build on, strips the multi-token-prediction heads that ship with newer Qwen checkpoints during conversion from SafeTensors. Models trained with built-in speculative decoding lose that capability once converted for Mac, and a pull request adding that support back has sat open for months without a maintainer merge.
vllm-metal, built out of UC Berkeley’s vLLM project, already has the widest set of these optimizations implemented, including continuous batching and paged KV cache. Its recent addition of prefix caching for hybrid-state models forces a choice between prefix caching and speculative decoding rather than both together. The post’s author recommends the community stop starting new forks and instead consolidate behind one patched branch, built on unmerged work from a contributor identified as AirRunner, until multi-token-prediction support lands in mlx-lm’s main branch.
Reader comments filled in real-world numbers the post itself did not test. One commenter running an 8-bit quantization of Qwen3.8-27B on a 128GB M5 Max Mac, paired with a draft-model speculative decoding method called DFlash, reported usable throughput. A 4-to-5-bit quantization of a separate model, Ling-3.0 flash, ran substantially faster on the same class of hardware, illustrating the precision-for-speed tradeoff at a given memory tier. A different M5 Max owner said MLX performance degraded badly as context grew and moved primary workloads to Nvidia hardware instead, keeping the Mac for lighter jobs. A developer who says they maintain a competing runtime called MTPLX disputed the post’s timeline in the replies, claiming their tool combined prefix caching and speculative decoding on hybrid GDN models weeks before vllm-metal did. That claim appears only in the comment thread and has not been independently verified.
This is a community guide and discussion posted to r/LocalLLaMA, not a vendor benchmark or a peer-reviewed comparison, and the throughput figures above come from individual setups rather than controlled test runs. Anyone provisioning a Mac for local agentic workloads this quarter should treat vllm-metal as the safer default for long-running sessions, budget for the prefix-caching-or-speculative-decoding tradeoff on hybrid models, and revisit the choice once multi-token-prediction support reaches mlx-lm’s main branch.
Source: a community guide and discussion thread posted to r/LocalLLaMA on 15 August 2026.
