The ladder from chip to lane

This machine is an M1 Pro with 14 GPU cores. A Metal compute dispatch is a grid of threadgroups; a threadgroup runs on one GPU core, which keeps several resident and switches between them to hide memory latency. Within a threadgroup, threads execute in simdgroups that step through instructions in lockstep. Apple’s own documentation maps the vocabulary to other dialects: a simdgroup is what CUDA calls a warp, a lane is a thread within it.

The numbers in the figure are read from this machine (threadExecutionWidth, maxTotalThreadsPerThreadgroup, maxThreadgroupMemoryLength) and match the constants hard-coded in the kernels (NUM_SIMD_LANES=32, the 32 KB shared-memory budget documented in paged_ops.cpp).

One threadgroup per unit of query work

vllm-metal ships two paged attention kernels in kernels_v2/, and the host picks per step:

Pure-decode batches run paged_attention in pagedattention.metal. The grid is (num_heads, total_q_tokens): one threadgroup per query token per head, which for a real batch is thousands of threadgroups that the 14 cores work through a few at a time. Its 256 threads form 8 simdgroups that stride across the sequence’s KV blocks, each maintaining online-softmax running state, merged across simdgroups at the end.

Batches with more query tokens than sequences, which means any batch containing prefill, run paged_attention_tiled in pagedattention_tiled.metal when the dtype and head size qualify; the per-token kernel is the fallback for everything else. The tiled kernel is FlashAttention-2 style: one threadgroup computes a 32-token block of query rows for one head (BQ=32), with four simdgroups each owning 8 rows, and Q·Kᵀ and P·V done as 8×8 simdgroup_multiply_accumulate tiles.

Both grids are flat over the packed token axis from the previous post. The padded alternative would be a grid over (batch, max_seq_len) with most threadgroups assigned to padding. Here every threadgroup corresponds to real work, with one exception: the tiled grid is sized total_q_tokens/BQ + num_seqs, a deliberate over-estimate of how many 32-token blocks the ragged sequences actually fill. Surplus threadgroups discover they are surplus and exit. That choice is inherited.

Which sequence owns token 4,097?

A threadgroup wakes up knowing only its grid coordinates: head 3, query token 4,097. The KV cache is paged, so before it can do anything it needs seq_idx: which request’s block table to walk, which context length bounds the causal mask, where its sequence starts in the packed query tensor. The batch is ragged, so token 4,097 could belong to any request.

The mapping from token to request lives in one array: cu_seqlens_q, the exclusive prefix sum of query lengths, [0, len_0, len_0+len_1, ...], length num_seqs + 1. Finding the owner of token t means finding the largest i with cu_seqlens_q[i] <= t. The array is sorted. That is a binary search:

// pagedattention.metal
inline int find_seq_idx(const device int32_t *cu_seqlens_q,
                        int q_token_idx, int num_seqs) {
  int lo = 0, hi = num_seqs;
  while (lo < hi) {
    int mid = (lo + hi + 1) / 2;
    if (cu_seqlens_q[mid] <= q_token_idx) {
      lo = mid;
    } else {
      hi = mid - 1;
    }
  }
  return lo;
}

Step through it below. Click a token; the steps are the ones the kernel executes.

The search runs before any attention math, in every threadgroup, and nothing about it is parallel. All threads of the threadgroup, 256 in the decode kernel and 128 in the tiled one, execute it redundantly on identical inputs and take identical branches. Redundant execution is the right call: the alternative is one lane searching while the rest wait at a barrier, then a broadcast through threadgroup memory. Uniform redundant execution costs the same wall time as one lane and skips the synchronization.

The cost is at most $\lceil \log_2(N{+}1) \rceil$ iterations, each one dependent int32 load. At 16 concurrent requests that is 5 loads of a 68-byte array that every threadgroup touches, against a KV loop that streams megabytes. For a pure decode batch the search is computing the identity function: cu_seqlens_q is [0, 1, 2, ...], so token t belongs to request t. The kernel pays those loads to learn that.

Borrowed from Triton

The kernel’s own comment names its ancestor: the approach is “the same approach used by the upstream vLLM unified Triton kernel,” triton_unified_attention.py:find_seq_idx. vllm-metal adapted the upstream kernel’s tests first, then translated the kernel to Metal against them. Upstream’s find_seq_idx is also a binary search over the same prefix array, executed once per Triton program as scalar control flow (it has since moved to triton_attention_helpers.py).

The tiled kernel inherits a subtler trick. Its grid is in units of 32-token Q-blocks, but blocks are aligned per sequence, so block boundaries depend on where sequences start. Rather than materialize a second prefix array in block units, the tiled kernel and its Triton ancestor search the token-unit array through a transform: compare against cu_seqlens_q[mid] / BQ + mid instead of cu_seqlens_q[mid]. Each sequence contributes len/BQ full blocks plus one potential ragged tail, and the + mid term counts the tails. One array serves both coordinate systems.

The over-provisioned grid has the same upstream origin, and upstream documents the reason: computing the exact number of Q-blocks would require reading the query lengths on the CPU, and the kernel’s authors judged a few empty threadgroups cheaper than that synchronization. The pattern repeats: spend trivial GPU work to keep the CPU out of the launch path.

What continuous batching asks of the kernel

Trace one step end to end. The vLLM scheduler hands vllm-metal a step that mixes, say, 32 decoding requests and one mid-prefill request. The model runner packs them into one flat token tensor, decodes first, and builds cu_seqlens_q as plain Python lists in prepare_unified(): something like [0, 1, 2, ..., 32, 544]. Every attention layer makes one paged-attention dispatch; the C++ side routes this batch to the tiled kernel, sizes the grid from total_q_tokens, and each threadgroup binary-searches its way to its request. No layer of this stack ever sees a [batch, seq_len] rectangle.

Each forward pass is one attention dispatch per layer regardless of batch composition (a small pure-decode batch adds a reduce pass via the split-KV path), which is what continuous batching asks for: requests join and leave the batch every step, query lengths change every step, and none of that changes the kernel, only the contents of a 136-byte prefix array. The speculative-decoding case from the previous post lands in the same array, though not as one piece: prepare_unified() expands a k+1-token verification segment into k+1 unit deltas, each with its own context length carrying the causal staircase.

Measuring the obvious optimization

The search is $O(\log N)$ work per threadgroup to compute something the host already knows. The host builds cu_seqlens_q; it could just as easily build the inverse, a token → seq_idx array of length total_q_tokens, and the kernel would do one load instead of a loop. We implemented it behind an env flag and measured.

Method. A function constant adds a second path to both kernels: seq_idx = seq_map[q_token_idx] for the decode kernel, a Q-block map with sentinel entries for the tiled kernel’s surplus threadgroups. The host builds the token map in prepare_unified() next to cu_seqlens_q and derives the Q-block map from it at the step’s first tiled dispatch; each converts to an MLX array once per step. Correctness is checked by asserting bit-identical attention output against the binary-search path across decode, mixed, ragged-tail, and spec-verify shaped batches, and a dispatch-side counter asserts that each timing arm actually ran the path it claims. Timing is ABAB-interleaved across processes, 2 warmup batches then pooled timed repeats, medians and IQR reported. Shapes: 16 query heads, 4 KV heads, head size 128, fp16 (Qwen3-0.6B’s query-head count and head size; the model itself has 8 KV heads).

The search is not free everywhere. Per-dispatch medians on the M1 Pro:

The pattern matches the arithmetic. At context 128 the KV loop visits 8 blocks, so the search’s dependent loads are a visible fraction of each threadgroup’s runtime, and the fraction grows with the batch: 64 requests take up to 7 search iterations, 1,024 up to 11. At context 2,048 the KV loop visits 128 blocks and the search disappears into it. No configuration regressed beyond noise. The host pays for the map: at 1,024 decoding requests, 86 µs to build the Python list and 10 µs to convert it, per step. That is under 0.1% of the attention time of the step it serves.

End to end, at ordinary scale, nothing moves. Serving Qwen3-0.6B at concurrency 16 with 1,024-token inputs, 3 ABAB rounds per arm: output throughput +0.9%, median TTFT −1.2%, median TPOT +0.5%, p99 TPOT +2.6%, all inside the round-to-round IQR. Three rounds per arm can only detect complete separation, and the TPOT point estimates lean slightly against the map, so the supported read is flat. At 1,024 tokens of context, decode sits in the regime where the table above says the search costs nothing, so this is the microbench’s prediction, confirmed.

The favorable regime, end to end. Serving the same model at concurrency 64 with 128-token inputs and outputs keeps every decode step at context 128 to 256 with a 64-deep batch: the microbench’s best case. The first three ABAB rounds per arm showed output throughput +3.1% and median TPOT −3.0%. Nine rounds per arm dissolved it: +0.9% mean throughput, −0.7% mean TPOT, 6 of 9 pairwise wins, exact permutation p = 0.26 and 0.45. The early +3.1% traced to three fast server processes that landed on both arms (Fisher p = 1.0) and never recurred. An 8-11% kernel-side win dilutes to under 1% of serving time, and separating a 1% effect from 1.3% per-round noise takes about thirty rounds per arm; we stopped at nine.

Where this sits in the design space. Kernels that need token-to-request resolution have settled into four patterns. Per-program search over the prefix array, as here and in upstream’s unified Triton kernel. Grid axes that encode the request directly, as in the classic padded PagedAttention launch, paying threadgroups for padding instead. Host-precomputed work descriptors, as in FlashInfer’s plan() phase and FlashMLA’s tile scheduler metadata, which amortize scheduling onto the CPU once per step. And explicit precomputed index maps, which is the variant we tested; vLLM already ships them in FlexAttention’s doc_ids and ROCm AITER’s token_to_batch (non-default attention backends) and in Mamba2’s per-chunk seq_idx, but not in its default attention path.

Verdict. The binary search is not literally free: replace it with a host-built map and the decode kernel returns up to 11% of its time at short context and large batch, bit-identically. It is free where it matters: those are the configurations where the kernel is cheapest, so serving throughput moves by less than the run-to-run noise. The upstream trade, a few empty threadgroups and a $\log_2 N$ loop in exchange for keeping the CPU out of the launch path, survives measurement, and the seq-map branch stays parked in its experiment worktree. The patch worth upstreaming from this exercise is the bug the bit-identical assertion flushed out of the unmodified baseline instead: an exception thrown from inside mx.eval leaves MLX’s Metal eval state wedged, and a later, valid eval can return an unwritten output buffer.

Links

• vllm-metal: github.com/vllm-project/vllm-metal, kernels in vllm_metal/metal/kernels_v2/
• Upstream ancestor: triton_unified_attention.py, introduced in vllm#16828; find_seq_idx now lives in triton_attention_helpers.py
• Apple GPU execution model: Scale compute workloads across Apple GPUs, WWDC22 session 10159, Metal feature set tables
• Previous post: Contiguous vs Paged Varlen KV Cache

PR #45203 is the workaround I proposed upstream: pyav_keyframes, an opt-in lossy video loader that decodes only keyframes, so decode work is at most num_frames single-frame decodes per clip regardless of clip length. What it buys and what it costs are both measurable. All numbers below are from public datasets, with full settings and tuning logs in offline_video_vllm.

Where decode time goes

Video codecs store a complete image only at periodic keyframes (I-frames). The frames between them are motion-compensated deltas: P-frames reference earlier frames, B-frames reference earlier and later ones. A keyframe plus the frames that depend on it forms a GOP (group of pictures), typically 2–10 s in web encodes.

Decoding an arbitrary frame means starting at its GOP’s keyframe and decoding forward, because P and B frames are meaningless alone. A lossless sparse sampler pays that GOP-prefix decode for every target it touches, however few targets it keeps. Keyframes have neither problem: decoding one costs one frame decode, and finding them costs no decode at all, since the demuxer reads packet headers that already carry a keyframe flag and a timestamp.

A loader that never decodes a delta frame

pyav_keyframes makes two passes over the container. The first demuxes the stream and records every keyframe timestamp; no pixels are decoded. The second spreads num_frames picks evenly over that keyframe list, then seeks to each pick and decodes exactly one frame. A 30-second clip and a 10-minute clip cost the same: one header sweep plus at most num_frames keyframe decodes.

When the budget exceeds the keyframe count, picks repeat instead of falling back to delta frames, and the repeats stay balanced: a 2-keyframe clip asked for 16 frames returns 8 copies of each. Repeated frames are decoded once and reported at their true source positions in the frame metadata, so a model like Qwen2.5-VL, which embeds each frame’s time position, sees the same moment twice rather than motion that never happened.

The sliders below replay the pick logic: set how many keyframes the clip has and how many frames the caller asks for, then compare the decode work of lossless uniform sampling against keyframe-only sampling for the same request.

The trade, measured

Setup: Qwen2.5-VL-7B-Instruct on one A100 (TP=1), offline LLM.chat, max_tokens=1, 16 frames per clip, 1,990 multiple-choice questions from NExTQA and MVBench. The two runs are identical except the video loader: lossless OpenCV sampling vs pyav_keyframes. End-to-end wall time drops from 674 s to 380 s and throughput rises from 2.95 to 5.23 req/s, a 1.77× speedup from the loader swap alone. At the decode stage, lossless sampling costs 193 ms on a 30 s clip and 3,124 ms on a 600 s clip; pyav_keyframes stays between 43 and 77 ms across all four test clips.

Left: decode time to extract 16 frames (log scale) across clip lengths and GOP settings. Right: end-to-end wall time on the N=1990 benchmark; the only change between bars is the video loader.

Accuracy is the other side of the trade. NExTQA, scene-content QA, is unchanged at 79.6 vs 79.5. MVBench drops 11.3 points overall, and the drop concentrates in motion-sensitive subtasks: action_antonym loses 52.7 points, moving_attribute and object_existence lose 36.4 each, while 10 of 18 subtasks stay within ±2 points. Keyframes land on scene boundaries, so prompts about what a scene contains keep their signal; prompts about what changes between keyframes lose it.

Left: overall accuracy, num_frames=16. Right: MVBench per-subtask deltas, the worst and best of 18 subtasks; the remaining 10 move less than ±2 pt.

The decode savings apply to every clip by construction; the accuracy cost lands only where the prompt needs inter-keyframe information. A classification job in the eBay mold, one-token answers about what the clip shows, matches the NExTQA column. Motion reasoning matches the action_antonym column and should stay on a lossless loader.

A case study in quoting out of context

Chinese has an idiom for this failure mode: 断章取义, judging a passage by a fragment taken out of context. Keyframe sampling does it to video. The two CLEVRER-based MVBench subtasks that drop 36.4 points each, moving_attribute and object_existence, are the extreme case: all 26 CLEVRER clips we probed carry exactly one keyframe, at frame 0, so pyav_keyframes returns the opening frame duplicated to fill the budget. The model is asked about a five-second video and shown its first instant, num_frames times.

The item below is a real object_existence row. The question asks about a purple sphere; the sphere rolls in at t≈2.0 s, so the answer is yes, and frame 0 cannot know that. Drag the budget down: uniform sampling loses its glimpses one by one and collapses to frame 0 at num_frames = 1; the keyframe row sits at frame 0 from the start. On encodes like this, num_frames stops being the knob that matters. The keyframe count is, and it is a property of the file, not of the request.

Frames from the CLEVRER validation set, via MVBench. The verdict lines describe the visible evidence in each sampled frame set; no model inference was run for this figure.

object_existence is 200 rows like this one; this mechanism is what its −36.4 pt aggregates.

Using it

The loader is opt-in; nothing changes unless a run selects the pyav_keyframes backend. Until the PR lands, the same loader ships as a single-file drop-in (pyav_keyframes_v2) in the experiment repo: importing the module registers it with vLLM’s loader registry, and the README has the exact LLM(...) configuration.

Links

• vLLM PR: vllm-project/vllm#45203, “[Multimodal] Add lossy keyframe-only video loader (pyav_keyframes)”
• Experiments and benchmarks (public datasets): WindChimeRan/offline_video_vllm

Same mlx_lm code, two Macs, no other changes. Three prompts of 30K + 5K + 10 input tokens (Qwen3-0.6B, max_tokens=256), run first sequentially then concurrently. On the M1 Max 64GB the concurrent run finishes in 45.0s, against 40.3s sequential. On the M1 Pro 32GB the concurrent run takes 189.6s, against 74.8s sequential. No OOM, no crash, no warning.

Left: M1 Pro 32GB. Right: M1 Max 64GB. Sequential vs concurrent wall time, with resident memory (solid) and macOS-compressed memory (dashed) over time.

In concurrent mode mlx_lm pads its KV cache to [3, H, 30000, D] because the longest prompt is 30K. The actual tokens total 35,010 (30,000 + 5,000 + 10), but the padded rectangle is 3 × 30,000 = 90,000. 61% of every attention step is wasted on padding. On the 64GB box that padding still fits; resident memory peaks at 32GB and the workload finishes. On the 32GB box the same padded rectangle pushes the system into memory pressure, macOS starts compressing pages (orange line rises from ~2GB to ~6GB), and every decode step pays decompression before attention and recompression after.

This is a deployment cliff: identical code, no OOM, no warning, just silently slower on a smaller machine.

The one layer we replaced

The cliff lives in attention. vllm-metal is a vLLM plugin for Apple Silicon: at the model level it reuses mlx_lm’s weight loader, RMSNorm, Linear, MoE, and MLP layers. All of these are token-wise. They operate on each token independently and don’t care whether tokens are batched as a [B, T] padded rectangle or flattened to a [total_tokens] varlen strip. Attention is the only layer that needs to know about sequence boundaries; that is the one we replaced. Above the model, the scheduler is vLLM’s.

mlx_lm uses flash-style attention over a contiguous, left-padded KV cache shaped [B, H, T, D] (B = batch size, H = number of KV heads, T = sequence length uniform across the batch, D = head dim), consumed by MLX’s stock scaled_dot_product_attention. vllm-metal uses flash-style attention over a paged KV cache laid out as [total_tokens, H, D]: tokens from every sequence are packed onto a single flat token dimension, with cu_seqlens marking sequence boundaries. mlx_lm’s cache is 4D; vllm-metal’s view is 3D, with the [B, T] axes collapsed into a single [total_tokens] axis. The vLLM scheduler drives the varlen layout. Same MLP, different attention.

Why the cliff exists, and what vllm-metal does instead

The cliff is a property of the cache shape.

mlx_lm. The KV cache is a 4D tensor [B, H, T, D] with uniform T. MLX’s stock scaled_dot_product_attention requires this shape and has no cu_seqlens-style parameter; every sequence in the batch occupies T_max tokens whether it needs them or not. Prefill and decode run as mutually exclusive phases. A forward pass is either all prefill or all decode, never mixed. Per-step compute scales as $B \cdot L_{\max}$, so a batch with one long prompt and two short ones pays the long prompt’s price three times.

vllm-metal. The KV cache is paged: KV memory is sliced into fixed-size blocks indexed by a per-sequence block table, the way upstream vLLM does it. Inside the attention step, tokens from all sequences are laid out on a single flat token dimension, with cu_seqlens marking sequence boundaries. The scheduler is free to pack prefill chunks and decode tokens into the same forward pass. This is real continuous batching, not phase-separated pseudo-batching. Per-step compute scales as the total of real tokens, $\sum_b L_b$, so the 90,000 vs 35,010 imbalance from the case study collapses to 35,010. MLX’s stock SDPA accepts neither cu_seqlens nor a block table, so the attention path is a hand-written Metal kernel.

The widget below shows the case-study batch as each layout stores it.

The case-study batch as each cache layout stores it. Drag a length: the padded rectangle re-pads everything to the new longest sequence; the packed strip grows only by the tokens actually added.

Speculative decoding falls out for free. Verification scores each sequence’s draft tokens in one forward pass: k + 1 query tokens per sequence, with k varying across the batch, so the batch is ragged on the query axis, not just in KV lengths. To a cu_seqlens kernel this is just another batch. Upstream vLLM’s FlashAttention backend runs prefill, decode, and verification through the same varlen attention call; drafts merely lengthen each request’s query span, and the spec-specific code only picks which logits to verify. After rejection, the diverging accepted prefixes are just new sequence lengths over the same paged store. A 4D padded cache has to re-pad both axes every step, and making that work is involved enough to be its own paper (Batch Speculative Decoding Done Right). vllm-metal has not shipped speculative decoding yet; the layout means it will arrive with no new kernel, just a different cu_seqlens.

On the benchmark

Output throughput (Qwen3-0.6B BF16) on SiliconBench's chat and agent splits at concurrency 1, 8, 16. Hatched mlx_lm bars on the agent split mark partial-success runs (X/100 prompts returned a non-empty completion).

chat split (~1K input, max output 256)

agent split (~4K input, max output 256)

SiliconBench is our benchmark harness for local LLM inference engines on Apple Silicon. It sends 100 prompts to each engine’s OpenAI-compatible API at three concurrency levels: c=1, 8, and 16, where c is the number of in-flight requests. The chat split is single-turn; the agent split is multi-turn material.

At c=1 mlx_lm is faster on both splits. By c=8 the data structure pays off: vllm-metal scales while mlx_lm flattens on chat and collapses on agent.

The agent split also exposes a reliability cliff. mlx_lm returns zero tokens for a large fraction of the long-input prompts at every concurrency level, the same padded-cache problem the M1 Pro experiment isolated, scaled to a benchmark. vllm-metal serves all 100 prompts at all three concurrency levels.

Where this lands in the ecosystem

vLLM’s scheduler is the easy half: it emits a varlen schedule (cu_seqlens and block tables), but the schedule pays off only if that structure survives all the way down to the attention kernel. One repack anywhere in between and you are back to padded compute.

All three MLX-based stacks (mlx_lm, omlx, vllm-mlx) converge at the same MLX call: mx.fast.scaled_dot_product_attention, which requires uniform T and has no cu_seqlens argument. vllm-mlx is worth pointing out: vLLM’s varlen scheduler runs upstream, but a _left_pad_prompts() step at the kernel boundary repacks into 4D padded form. The scheduler is doing varlen bookkeeping the kernel can’t use. llama.cpp takes a third path: its Metal flash-attention kernel supports varlen via an explicit attention mask over per-stream KV ring buffers, with seq_id deciding which tokens attend to which cells.

Of the five stacks audited, vllm-metal is the only one that pairs cu_seqlens-based varlen with a flat 3D KV layout. On NVIDIA this pairing is the de facto serving pattern. Both vLLM and SGLang use it in production. Apple Silicon hasn’t shipped the same pattern until recently: vllm-metal 0.2.0 (April 2026) is the first end-to-end serving framework on Apple Silicon to ship paged varlen attention. The data-structure choice each stack makes is what shows up at concurrency in the benchmark above.

Citation

This blog post is part of the paper SiliconBench: Speed, Memory, and Fidelity for LLM Inference on Apple Silicon, coming soon to arXiv.

What it does

I turned my Timeular Tracker (the 8-sided time-tracking dice) into a physical macropad for macOS. Flip the dice, switch macOS desktops. Side 1 is my main workspace, side 2 is a side project, side 3 is another. No keyboard shortcut to remember, just a satisfying physical flip. Every side can also be bound to launching an app, running a shell command, or opening a URL, all configured from a tiny menu bar popover.

How it got built

I had the Timeular charged but unused on my desk (I stopped paying the monthly subscription). I also had Claude Opus 4.6, and even though I don’t know Swift, Opus does.

1. Scan. Asked Claude to find the Timeular over Bluetooth. It picked Python to explore the device.
2. Prototype. Read the orientation signal on flip and wired it to shell commands. A working macropad in one script.
3. Go native. Rewrote the whole thing in Swift/SwiftUI as a proper macOS menu bar app with CoreBluetooth.
4. Ship it. Works.

Source on GitHub.

LLM inference engines like vLLM were designed for discrete NVIDIA GPUs, where GPU memory is a dedicated, isolated resource. The memory allocator can safely assume it owns nearly all of VRAM — upstream vLLM defaults to claiming 90% of total GPU memory (gpu_memory_utilization=0.9), and if another process has already taken enough that this target can’t be met, vLLM simply refuses to start.

This assumption breaks on Apple Silicon. The M-series chips use a Unified Memory Architecture (UMA): CPU, GPU, and Neural Engine all share a single physical memory pool. There is no dedicated “VRAM” to claim. A Mac running vllm-metal is likely also running a browser, an IDE, and other applications — all competing for the same memory. Reserving 90% of total memory is not realistic in this scenario.

But the problem isn’t just about lowering a threshold. We also want to support the case where a Mac is used as a dedicated inference server with no other significant memory consumers.

The discussion tracks vllm-metal#97.

Why vllm-metal

An NVIDIA RTX 5090 ships with 32 GB of GDDR7 at a $1,999 MSRP — roughly $62.47 per GB. A Mac Studio with the M3 Ultra and 512 GB of unified memory starts at $9,499 — roughly $18.55 per GB, more than 3× cheaper. This is not an apple-to-Apple comparison (😉): CUDA VRAM is dedicated and faster, while unified memory is shared with the rest of the system and comes with lower bandwidth — which is precisely why the memory allocator matters so much on this platform.

Related Work

To compare the three engines below, we use a common vocabulary:

• memory hint: the value reported by Metal’s recommendedMaxWorkingSetSize — the OS’s suggestion for how much GPU memory a process should use. This is a static, per-process hint; it does not reflect memory consumed by other apps.
• RAM cap: a software-imposed ceiling, typically a fraction of total system RAM (e.g., 2/3 or 3/4), intended to leave room for the OS and other apps.
• utilization target: the fraction of the memory budget an engine attempts to claim (e.g., 0.9 means “use up to 90%”).

vLLM

vLLM follows a profile-and-claim strategy. On CUDA, it queries the GPU for total and free VRAM at startup, then claims total_vram × utilization_target (default 0.9). If the requested amount exceeds what is actually free (because another process is already using the GPU), vLLM fails immediately rather than proceeding with a smaller budget. The claimed memory is managed as a paged block pool (PagedAttention), where KV cache blocks are allocated and freed at page granularity as requests arrive and leave.

GPU memory is filled in two stages. First, model weights are loaded onto the GPU. vLLM then runs a profiling forward pass with dummy inputs to measure peak non-KV memory usage (activations, temporaries). The remaining memory — total_vram × utilization_target - weight_memory - profile_peak — becomes the KV block pool:

mistral.rs

mistral.rs adapts the same two-phase approach for Apple Silicon but introduces a RAM cap to leave room for the OS and other applications sharing the same memory pool:

In Phase 1 (model weights), mistral.rs uses the memory hint directly, computing memory_hint - current_process_allocation and reserving max(available × 0.02, 512 MB) as headroom, then greedily places layers on GPU until the budget is exhausted.

In Phase 2 (KV cache), the effective ceiling becomes min(memory_hint, ram_cap). Since Apple typically reports a memory hint in the 66–75% range of system RAM — close to what the RAM cap already computes — the min() often selects the same value. The KV budget is then:

The two safety margins stack: 25–33% is reserved by the RAM cap, and a further 10% by the utilization target (default 0.9), leaving roughly 60–68% of system RAM as the effective ceiling before model weights are subtracted.

llama.cpp

llama.cpp uses the memory hint as its ceiling, memory_hint - current_process_allocation, with no RAM cap and no utilization target. Memory consumed by other apps is invisible; the engine has no system-wide pressure signal. On macOS 15+, a background thread requests buffer residency every 500 ms to prevent OS eviction, an acknowledgment that the OS may reclaim memory under pressure even after allocation succeeds.

GPU memory is filled in three stages: model weights (last N layers offloaded to GPU, back-to-front), a KV cache (preallocated at a user-specified token capacity n_ctx), and a compute scratch buffer (worst-case activation buffer reused every forward pass). Unlike the paged designs above, llama.cpp’s KV cache is contiguous — sequences share a ring buffer with 1-token granularity and an explicit defrag pass. Total KV memory is committed at startup:

vLLM assumes exclusive ownership of a discrete memory pool. mistral.rs introduces Apple Silicon-specific caps for desktop coexistence. llama.cpp pre-commits a user-specified amount with no system-wide awareness. vllm-metal inherits from all three but needs to handle both the mixed-use desktop case and the dedicated server case, ideally without requiring the user to manually tune a single magic number.

Proposal

vllm-metal has two KV cache paths, selected by VLLM_METAL_USE_PAGED_ATTENTION. Both are tracked in vllm-metal#97.

Path 1: Contiguous Allocation (MLX)

Today, the scheduler reasons in 16-token blocks and reports phantom block counts, but MLX allocates contiguous caches in 256-token steps. None of those blocks exist at runtime. The fix is to strip the paged bookkeeping and use mlx_lm’s auto behavior: each request allocates only the KV cache memory it needs via make_prompt_cache() and releases it when done. No upfront budget, no utilization target.

Path 2: Paged Allocation (vLLM)

Path 2 is under active development on the paged-attention-v3 branch (vllm-metal#70). It maintains a global block pool backed by Metal buffers, aligned with upstream vLLM’s PagedAttention. VLLM_METAL_MEMORY_FRACTION controls how much system RAM the pool claims. At startup, the engine measures weight and activation memory, then fills the remaining budget with KV blocks:

On a dedicated inference server, set the fraction high. On a desktop Mac sharing memory with a browser, IDE, and other applications, the fraction that is actually free will be lower. If the requested allocation exceeds free memory, the engine refuses to start and reports the available memory and the fraction the user would need to set. An environment variable (not a CLI flag) matches vllm-metal’s existing configuration pattern (VLLM_METAL_PREFIX_CACHE, etc.).

The Wired Collector

macOS distinguishes between pageable memory (swappable to disk) and wired memory (pinned in physical RAM). Metal GPU buffers are wired. Under memory pressure, macOS invokes the wired collector, a kernel mechanism that reclaims GPU wired memory by evicting Metal buffers, causing silent performance degradation or crashes.

The mistral.rs community documented this behavior in mistral.rs#1348. The workaround: sudo sysctl iogpu.disable_wired_collector=1. The setting does not persist across reboots unless added to a startup script. For dedicated servers running Path 2 with an aggressive memory fraction, disabling the wired collector is recommended.