SiliconBench

Speed, memory, and fidelity for LLM serving on unified-memory desktops. All requests hit an OpenAI-compatible endpoint at concurrency 1 / 8 / 16, BF16 weights, n=100 per level.

benchmark repo weekly journals paper (soon)

SiliconBench audits the LLM serving engines that run on unified-memory desktop hardware. Nine stacks are benchmarked on Apple Silicon against the same weights and prompts, with a CUDA-native reference track on an NVIDIA DGX Spark for the three engines the two ecosystems share. A maintainer agent re-runs the benchmark, commits the raw results, and this page rebuilds from them automatically.

Speed alone is a misleading ranking on shared machines: the engine pool is the same memory your browser and IDE use, and a stack can be fastest while claiming most of it or while returning wrong output. The tables therefore keep three lenses side by side. Speed is measured on two workloads (a short-prompt chat split and an agent split whose multi-turn prompts reach several thousand input tokens), memory as the peak footprint during serving, and fidelity as weighted F1 on a classification task against an NVIDIA reference on identical weights. Try sorting by tok/s at c=1 and then at c=16: the point of the concurrency sweep is that single-stream rankings do not survive load.

Single-node serving only. Stacks are listed alphabetically by default; click a column header to sort (failed runs always sink to the bottom); no default ranking is implied; the paper's central finding is that speed-only orderings mislead. ✕ = crashed (<5/100 requests), n/100 = partial run, – = not measured. Trend sparklines show each stack's own shape across c=1/8/16 (per-row normalized; TTFT on a log scale); magnitudes are in the numbers. Hover a stack name for its run provenance; the full record is under per-framework provenance at the bottom.

Live results rebuilt automatically from every merged benchmark run

machine model

Findings from the paper frozen analysis; not refreshed by the weekly pipeline

Frozen section. Everything below is analysis from the paper, based on fixed benchmark runs (Apple chat 2026-07-03, Apple agent 2026-05-19/20, DGX Spark 2026-07-03/04, all on Qwen3-0.6B BF16). It is written once and updated only with the paper. The live tables above rebuild automatically and may diverge from these numbers as engines evolve.

TL;DR.

  • Nine stacks enter; four remain viable. Only llama.cpp, vllm-metal, mlx_lm, and omlx pass all three lenses. The chat-split throughput leader holds a near-constant 47 GB claim and fails 5-shot fidelity, so a speed-only leaderboard recommends a stack the multi-lens reading rules out.
  • Declared memory budgets are not enforced ones. Two stacks that configure explicit budgets still grow to within a few GB of physical RAM on the agent split and degrade or crash.
  • The platform gap is serving software, not silicon. Upstream engines on DGX Spark scale 4 to 8x from c=1 to c=16 at matching single-stream speed, while the best Apple stack reaches 3.3x.
  • The benchmark maintains itself. A weekly agent updates every engine, re-runs both splits, repairs what breaks inside a write allowlist, and publishes a journal; this page rebuilds from its commits.

Flat paths are enforced budgets; diving paths end in failure

Speed-memory trajectories per stack on chat and agent splits
Figure from the paper. Each stack traces its (throughput, peak memory) position across concurrency 1, 8, 16; linestyle is the stack's declared budget policy.

Stacks with enforced budgets trace flat paths: throughput grows while memory stays put (vllm-metal holds 33 to 39 GB while scaling 3.3x). The sharpest finding cuts against the audit table: mistral.rs and sglang declare explicit budgets, yet both dive to about 60 GB on a 64 GB machine and degrade to partial completion or crash on the agent split. A declared budget is a configuration knob; discipline has to be enforced end to end through the allocator. ollama sits at 98 percent of Metal's 48 GB advisory working-set hint from the first request, sized to the hint rather than to demand.

The scaling gap is software, not silicon

Bridge pairs: same engine family on Apple Silicon and DGX Spark
Figure from the paper. The three engine families the two platforms share, same workload; line-end labels give c=1 to c=16 scaling.

Upstream vllm on GB10 scales 7.7x on chat and 4.3x on agent with median TTFT of 52 and 134 ms at c=16; sglang behaves alike. The attribution is engine-local: sglang scales 6.9x on CUDA but declines below single-stream on its MLX backend, and llama.cpp plateaus on both platforms, so its ceiling is the engine design, not the hardware. The silicon itself is competitive: single-stream speed matches across platforms, and llama.cpp on the M-series finishes ahead of its own CUDA build at chat c=16 (252 vs 193 tok/s).

The benchmark maintains itself

A maintainer agent re-runs the benchmark weekly: it pulls each engine from upstream, runs both splits, diagnoses failures, applies bounded fixes inside a write allowlist, and commits a structured journal. In one week a single MLX library bump broke three stacks through three distinct failure modes; two were repaired within the same cycle. The journals are the provenance record behind the live tables above.