Perplexity has built a local inference engine, called Lily, that runs only on Apple silicon and only for its own Qwen3.6-35B-A3B model. The company says Lily beats Apple’s general-purpose MLX-LM stack by roughly 23 percent on prefill throughput and 35 percent on decode throughput, averaged across a ten-point sweep from 256 to 128,000 tokens. Perplexity built the engine to support Hybrid Compute, its scheme for splitting a task between a cloud model and a local model that works with private files on the machine, and has promised to release the engine publicly at some point, though nobody can download it today.
The architecture explains why a purpose-built engine helps. The model carries 35 billion parameters in total, yet any single token wakes only about 3 billion of them: a router picks eight subnetworks out of 256, and one shared expert sees everything. Ten of its layers use standard attention, the other 30 run Gated DeltaNet, squeezing everything that came before into a recurrent state of fixed size rather than a cache that keeps growing. MLX-LM, built to support many architectures at once, cannot exploit that specific combination of sparse routing, mixed attention types, and recurrent state the way a single-purpose runtime can.
Lily’s gains came from a short list of targeted changes, each measured by Perplexity as an ablation of Lily against a version of itself, not as a comparison with the MLX-LM baseline. Folding the dequantization of 4-bit weights into the matrix multiplication, so an expanded array never touches unified memory, raised prefill by 77.4 percent on a prompt of 512 tokens. At that same length, a further 89 percent came from never handing the routing step back to the CPU for inspection and leaving it on the GPU throughout. On decode, grouping four query heads together so a cached key-value row is fetched a single time was worth 23.8 percent at 32,000 tokens of context, and a fixed-block attention layout added as much as 40.2 percent at 128,000 tokens.
Perplexity also reports what did not work, which is the more interesting part of the post. Speculative decoding, a technique that has produced large speedups on server GPUs, made Lily’s single-request decode 18 percent slower. Verification arrived in awkward batches of two to five rows, and those rows frequently routed to different experts, which increased the weight reads the engine was trying to minimize. Shrinking the drafter model’s vocabulary made the drafter itself faster without moving the end-to-end number. A separate round of experiments in reducing GPU launches, overlapping full phases of the pipeline, enlarging prefill tiles and broadening kernel fusion produced no measurable gain at all. Perplexity’s own hardware-limit checks show why: the core matrix operations already run at 97.9 percent and 90.3 percent of the fastest sustained memory-read rates the hardware supports, so weight reads, not compute, are the ceiling.
The result is an engine that is fast because it forfeits generality. MLX-LM has to work for whatever architecture a developer hands it. Lily was tuned kernel by kernel to one router, one attention mix and one chip generation. That is the opposite of where inference tooling has been heading, toward frameworks like MLX and vLLM that abstract hardware differences so a new model just works. Perplexity has not said what happens to Lily’s tuning the next time it changes Qwen’s architecture or Apple ships a new GPU generation, and hand-fitted kernels do not update themselves.
For teams building on-device AI products, Lily’s numbers say an M5 Max Mac can now run a 35-billion-parameter mixture-of-experts model at usable local speed. The harder decision is whether to chase that ceiling with a custom runtime, and accept the maintenance cost every time the model or the chip changes, or stay on a general framework and give up 20 to 35 percent of throughput for portability.
Perplexity Engineering detailed Lily’s design and benchmarks in a post on the company’s engineering blog on September 1, 2026.
