Comparing Five Local LLM Inference Engines on M1 Max: llama.cpp, Magnitude, Ollama, uzu, and Locals

Claimed speeds for modern Mac inference engines like Magnitude, uzu, and Locals come from disparate hardware configurations, including M4 Pro, M5 Max, and M2 Max systems. This makes direct comparisons difficult without a common testbed.
To establish a clear baseline, we ran a suite of workloads—short Q&A, code generation, long-context summarization, multi-turn conversation, and concurrent requests—on an M1 Max 64GB machine, using Ollama and llama.cpp as reference points.

Because each engine prefers distinct model formats (such as GGUF or MLX), quantization schemes, and speculative decoding setups, this benchmark separates apples-to-apples comparisons using identical weights from practical throughput measurements under each engine’s recommended settings.
Specifically, llama.cpp and Magnitude were evaluated on identical GGUF weights, while Locals and standard mlx-lm were tested against identical MLX 4-bit weights. Ollama and uzu were benchmarked in their respective standard operational configurations.

Hardware and Test Models

Spec	Value
Machine	MacBook Pro, M1 Max, 10-core CPU, 32-core GPU, 64GB unified memory
OS	macOS 27.0 (26A428)
Architecture	arm64, Metal 4
Toolchain	Command Line Tools, Apple clang 21.0.0
Ollama	0.40.11 (Homebrew)
llama.cpp	0.7.0 (build 11669, Homebrew)

On October 28, 2026, we updated the packages using brew upgrade ollama llama.cpp.
To prevent conflicts with an existing Ollama.app installation at /usr/local/bin/ollama, the Homebrew build ran as an isolated daemon on port 127.0.0.1:11436 with a dedicated models directory.

All tests used Qwen/Qwen3.5-9B.
The larger Qwen/Qwen3.8-27B requires approximately 20.5GB of memory, which leaves limited headroom for concurrent runs and long-context processing on a 64GB machine, so this benchmark focused exclusively on the 9B parameter model.

Local runtimes rely on different model formats: llama.cpp and Magnitude consume GGUF files (single-file archives packing weights and metadata), whereas Locals and Ollama load MLX format weights optimized for Apple Silicon.
Running a 9B model in native 16-bit precision consumes over 18GB of memory. To maintain efficient memory usage, all engines were tested with 4-bit quantized checkpoints that compress weights down to 4-bit representations.

The Mirai Medium checkpoint used by uzu is a distilled derivative of Qwen3.5-9B produced after 4-bit quantization.
Because its weights differ from standard GGUF and MLX 4-bit checkpoints, it is treated here as a distinct checkpoint within the Qwen family.
Magnitude’s model repository lists the download size for 9B Q4 at 8.4GB (including draft models) and 27B Q4 at 20.5GB.

Speculative decoding speeds up inference by having a lightweight draft model or additional prediction head propose several candidate tokens in advance, which the primary model verifies in a single forward pass.
Examples include llama.cpp’s multi-token prediction (MTP) and the DFlash heads used by Magnitude and uzu.

Engine	Model Format	Speculative Decoding Setup	Hardware for Published Claim
Ollama	MLX Safetensors, NVFP4	Enabled automatically via model draft head	Not claimed against vendor baseline
llama.cpp	GGUF, UD-Q4_K_XL	MTP evaluated explicitly	Not claimed against vendor baseline
Magnitude	GGUF	DFlash on 9B	M4 Pro 48GB
uzu	uzu custom Safetensors	Enabled only on M5 series in vendor setup	M5 Max 128GB
Locals	MLX 4-bit affine	Disabled	M2 Max 96GB

uzu’s advertised 273 tok/s figure on 9B was obtained on an M5 Max 128GB with speculative decoding enabled.
In uzu’s benchmark harness, DFlash is gated to run only on M5 processors, meaning speculative acceleration does not run by default on an M1 Max.
Similarly, the Locals repository does not publish reference metrics for M1 Max.

Benchmark Setup and Workloads

The evaluation comprised six distinct workloads.
All input prompts were in English, with identical prompts sent to each engine across runs.

Case	Description	Input / Output Limit
A Short Q&A	Explaining initial latency in an HTTP service	~140 / 256 tokens
B Code Generation	Writing a Python interval merger function	~121 / 1024 tokens
C ~8K Long Context	Summarizing 125 incident records in 5 sentences	8,512 / 512 tokens
D ~27K Long Context	Summarizing 400 incident records in 5 sentences	27,077 / 512 tokens
E Multi-turn Chat	9 sequential follow-up questions extending task A	Same initial A / 128 tokens per turn
F Concurrency	Batch generation of task B across 2, 4, and 8 parallel requests	Identical to B / 512 tokens per request

Tasks A through D and concurrency test F were executed 5 times each following an initial warmup run. Multi-turn chat E ran as a single 10-turn sequence.
All runs used temperature 0, thinking OFF, and uniform output length caps.

Logged metrics include prompt processing speed (tok/s), token generation speed (tok/s), time to first token (TTFT, in seconds, measured from request dispatch until the first non-whitespace token arrives), and end-to-end duration (E2E, in seconds).
Tokens per second (tok/s) measures processing or generation throughput per unit time.
When an engine finished generation naturally before hitting the token ceiling, total output length varied. Consequently, E2E is not used for fixed-length throughput comparisons; generation speed in tok/s serves as the primary metric.

Prompts and Sample Outputs

Here are excerpts from the prompts and generated outputs used across the workloads.

Task A asked the model to explain why initial requests to an HTTP service exhibit high latency and to propose two distinct diagnostic tests:

Hello! Let’s break down why your service feels slow on the very first request but speeds up afterward. This is a classic pattern in AI applications, and understanding the three distinct phases—Model Loading, Prompt Processing, and Token Generation—will help us pinpoint the exact bottleneck.

Task B required a complete Python function to merge intervals, specifying type annotations, a docstring, and four test assertions:

Write a complete Python 3 function merge_intervals(items) that accepts an iterable of (start, end) integer pairs. Reject a pair when start > end with ValueError. Return a new list of non-overlapping intervals sorted by start, merging intervals that overlap or touch. Do not mutate the input. Include type hints, a short docstring, and four assert examples covering empty input, unsorted input, touching intervals, and an invalid pair. Return only Python code.

All compliant engines produced interval consolidation loops structured similarly to this snippet:

for current_start, current_end in sorted_intervals[1:]:
    last_start, last_end = merged[-1]
    
    # Check if intervals overlap or touch
    if current_start <= last_end:
        # Merge intervals
        merged[-1] = (last_start, max(last_end, current_end))
    else:
        merged.append((current_start, current_end))

Tasks C and D fed incident registers containing service names, HTTP error codes, severity levels, durations, and ticket identifiers, asking for a 5-sentence executive summary.
Ticket IDs were randomized between test iterations to prevent prompt caching across long-context measurements:

The incident register documents 125 independent offline software system failures involving a rotating set of services including worker, gateway, scheduler, storage, indexer, search, and notifier across four geographic regions. The highest severity level recorded across these entries is “high,” which appears frequently alongside medium and low severity incidents.

Multi-turn test E asked for give one additional practical detail across 9 consecutive turns while preserving conversational history.
Concurrency test F dispatched simultaneous, state-free requests executing the code prompt from task B.

Measuring llama.cpp with Cache Disabled

During initial short prompt runs, llama-server reused 136 out of 140 prompt tokens from its internal prompt cache.
Because caching distorts raw prompt processing measurements, the server was restarted with --no-cache-prompt.
Subsequent runs processed all 140 tokens from scratch on every iteration, leaving 0 cached tokens.

The server loaded the exact Qwen3.5-9B GGUF file (UD-Q4_K_XL) downloaded from the Magnitude catalog. This matched the weights used by Magnitude.

Single-Request Performance

Settings: GGUF UD-Q4_K_XL, prompt cache disabled, speculative decoding disabled; values represent medians across 5 runs.

Case	Prompt Processing	Generation Speed	Median TTFT	Median E2E	Median Output Tokens
A Short Q&A (~140 in)	348.55 tok/s	36.59 tok/s	0.445 s	7.415 s	256
B Code (~121 in)	355.55 tok/s	37.07 tok/s	0.387 s	13.768 s	497
C ~8K Context (8,512 in)	452.33 tok/s	34.03 tok/s	18.915 s	23.717 s	168
D ~27K Context (27,077 in)	403.18 tok/s	31.38 tok/s	67.332 s	72.532 s	166

As prompt length grew, TTFT expanded from 0.445 s on task A to 67.332 s on task D.
Token generation speed stayed within a narrow band between 31 and 37 tok/s regardless of prompt length.

Multi-Turn Chat and Concurrent Scaling

Multi-turn test E executed 10 conversational turns in a single session.
By turn 10, accumulated context reached 1,433 tokens, but only 153 tokens required fresh processing while 1,280 tokens hit the KV cache.
Across the 10 turns, median TTFT was 0.565 s and median generation throughput was 37.28 tok/s.

Concurrency test F ran with --parallel 8 --ctx-size 32768 (allocating 4,096 tokens per slot), evaluating 2, 4, and 8 parallel requests.

Concurrency	Aggregate Throughput	Per-Stream Generation	Median TTFT	Median E2E
2	42.36 tok/s	21.84 tok/s	0.69 s	22.85 s
4	78.00 tok/s	20.73 tok/s	1.34 s	25.19 s
8	122.00 tok/s	16.89 tok/s	2.41 s	31.63 s

Scaling from 2 to 8 parallel requests reduced individual stream generation from 21.84 tok/s to 16.89 tok/s, while aggregate throughput climbed from 42.36 tok/s to 122.00 tok/s.

Speculative Decoding with MTP

We then enabled multi-token prediction on the same GGUF model using --spec-type draft-mtp --spec-draft-n-max 3.
The table compares performance against the non-speculative baseline across 5 iterations per workload.

Case	Non-Spec Generation	MTP Generation	Speedup	MTP TTFT	MTP E2E
A Short Q&A	36.59 tok/s	48.46 tok/s	1.32x	0.445 s	5.742 s
B Code	37.07 tok/s	60.22 tok/s	1.62x	0.387 s	8.542 s
C ~8K Context	34.03 tok/s	40.90 tok/s	1.20x	20.151 s	24.150 s
D ~27K Context	31.38 tok/s	32.53 tok/s	1.04x	72.480 s	76.966 s

In code generation task B, draft acceptance approached 80%, raising throughput 1.62x (from 37.07 to 60.22 tok/s).
Acceptance rates were approximately 58% on short Q&A task A, 52% on summary task C, and 48% on summary task D.
On long-context tasks C and D, verification overhead slightly increased TTFT, which lengthened overall E2E time.
All generated Python code passed syntax parsing and included the four required assertions.

Measuring Magnitude and DFlash Speculative Decoding

We tested Magnitude 0.2.12 using the catalog build of Qwen3.5-9B, which includes a DFlash speculative drafter.
The underlying GGUF model file is identical to the one tested with llama.cpp.

Disabling Speculation on Identical Tensors

Magnitude’s catalog models do not provide a configuration switch to detach draft heads directly.
To isolate the base model, we created an APFS clone of the GGUF file and modified a single character in the general.name metadata string (replacing a hyphen with a space).
None of the tensor weights or quantization tables were altered.
Magnitude recognized this cloned file as a distinct custom model, which let us disable speculative decoding via magnitude models drafter ... none.

Single-Request Performance

Settings: Magnitude 0.2.12, speculative decoding disabled; values represent medians across 5 runs.

Case	Prompt Processing	Generation Speed	Median TTFT	Median E2E	Median Output Tokens
A Short Q&A	333.10 tok/s	41.61 tok/s	0.451 s	6.652 s	256
B Code	340.20 tok/s	41.56 tok/s	0.328 s	12.189 s	490
C ~8K Context	541.27 tok/s	40.33 tok/s	15.786 s	19.444 s	146
D ~27K Context	382.40 tok/s	37.54 tok/s	60.786 s	64.910 s	155

Without speculative decoding, token generation stayed stable at roughly 41.6 tok/s on short text and code.
This is approximately 1.12x to 1.14x faster than llama.cpp’s non-speculative baseline of ~37 tok/s.

DFlash Acceleration

Comparing the default DFlash-enabled catalog setup against the non-speculative configuration highlights where draft prediction succeeds:

Case	Non-Spec Generation	DFlash Generation	Speedup	Non-Spec TTFT	DFlash TTFT
A Short Q&A	41.61 tok/s	41.19 tok/s	0.99x	0.451 s	0.476 s
B Code	41.56 tok/s	82.58 tok/s	1.99x	0.328 s	0.343 s
C ~8K Context	40.33 tok/s	37.64 tok/s	0.93x	15.786 s	16.461 s
D ~27K Context	37.54 tok/s	33.93 tok/s	0.90x	60.786 s	62.799 s

DFlash produced substantial gains on code generation task B: generation speed doubled from 41.56 tok/s to 82.58 tok/s (a 1.99x speedup), while E2E duration dropped from 12.189 s to 6.559 s.
Conversely, short Q&A task A saw no measurable change (0.99x), while long-context tasks C and D experienced a 7% to 10% throughput dip alongside longer TTFT due to speculative verification overhead.

Multi-Turn Chat and Concurrent Scaling

In multi-turn chat E (10 turns), Magnitude generated 128 tokens on every turn, with a median TTFT of 0.575 s and a median generation throughput of 41.40 tok/s.
On turn 10, 1,318 out of 1,473 input tokens were served directly from the prompt cache.

Concurrent batch test F (5 iterations per concurrency tier) yielded the following results:

Concurrency	Aggregate Throughput	Per-Stream Generation	Median TTFT
2	56.82 tok/s	29.91 tok/s	0.693 s
4	72.21 tok/s	19.14 tok/s	1.258 s
8	132.97 tok/s	19.51 tok/s	3.658 s

At 8 concurrent streams, aggregate throughput reached 132.97 tok/s, with all 40 test requests finishing in approximately 29 seconds.

Measuring Ollama on an Isolated Port

Homebrew-installed Ollama 0.40.11 was started on isolated port 11436 to pull and run the qwen3.5:9b tag.
The pulled model used the MLX Safetensors format with NVFP4 quantization, totaling approximately 7.64GB on disk.

Although we attempted to disable speculative decoding using draft_num_predict=0, Ollama’s MLX runner automatically initializes speculative mechanics whenever draft heads are detected in the model archive.
Server logs confirmed ongoing draft generation and token acceptance, so all reported Ollama metrics reflect active speculative decoding.

Single-Request Performance

Settings: Ollama 0.40.11, MLX NVFP4, speculative decoding active; values represent medians across 5 runs.

Case	Prompt Processing	Generation Speed	Median TTFT	Median E2E	Median Output Tokens
A Short Q&A (128 in)	298.83 tok/s	47.27 tok/s	0.565 s	5.912 s	256
B Code (~121 in)	329.76 tok/s	53.37 tok/s	0.449 s	9.716 s	498
C ~8K Context (8,500 in)	352.08 tok/s	46.40 tok/s	24.272 s	28.012 s	176
D ~27K Context (27,065 in)	328.23 tok/s	39.67 tok/s	82.603 s	87.106 s	170

Generation throughput held at 53.37 tok/s on code task B and 39.67 tok/s on long summary task D.
All 5 code generations in task B compiled cleanly in Python and satisfied the four test assertions.

Multi-Turn Chat and Concurrent Scaling

In multi-turn chat E, Ollama achieved a median TTFT of 0.459 s and a median generation throughput of 53.58 tok/s over 10 turns.
Following the initial model load, subsequent turns responded promptly with TTFT values between 0.45 s and 0.60 s.

Concurrency test F ran with OLLAMA_NUM_PARALLEL=8, sending 2, 4, and 8 simultaneous requests.

Concurrency	Aggregate Throughput	Per-Stream Generation	Median TTFT
2	53.20 tok/s	55.77 tok/s	4.85 s
4	54.54 tok/s	56.11 tok/s	13.21 s
8	54.76 tok/s	56.40 tok/s	30.05 s

Under Ollama, aggregate throughput remained flat at around 54 tok/s across all concurrency levels.
Individual stream generation stayed at 56 tok/s without degradation, but TTFT grew linearly by 8 to 9 seconds per added stream. This indicates that requests were processed largely in sequential order.

Running uzu with Mirai Medium on M1 Max

We ran uzu 0.6.13 in a dedicated virtual environment with the distilled checkpoint trymirai/Qwen3.5-9B-M (~5.38GB).

Single-Request Performance

Settings: uzu 0.6.13, Mirai Medium, speculative decoding disabled (default on M1 Max); values represent medians across 5 runs.

Case	Prompt Processing	Generation Speed	Median TTFT	Median E2E	Median Output Tokens
A Short Q&A	225.29 tok/s	55.19 tok/s	0.699 s	5.320 s	256
B Code	231.10 tok/s	55.20 tok/s	0.523 s	14.620 s	782
C ~8K Context	376.06 tok/s	51.75 tok/s	22.749 s	26.153 s	177
D ~27K Context	332.49 tok/s	47.91 tok/s	81.620 s	84.675 s	150

Single-request generation throughput was strong, ranging from 47 to 55 tok/s.
However, all 5 runs in code generation task B failed Python syntax parsing.
The model appended internal reasoning fragments to the output, such as assert ... == [...] -> [...] is incorrect logic in thought. This caused syntax errors.
This self-correction text appeared despite specifying thinking OFF, reflecting a characteristic of the distilled checkpoint.

Multi-Turn Chat and Concurrent Scaling

In multi-turn chat E, matching uzu’s session-based history specification, the engine generated 128 tokens per turn across 10 iterations.
Median TTFT was 0.228 s and median generation throughput was 57.12 tok/s, demonstrating effective cache reuse.

Concurrent batch test F produced the following metrics:

Concurrency	Aggregate Throughput	Per-Stream Generation	Median TTFT
2	54.79 tok/s	28.55 tok/s	0.748 s
4	54.96 tok/s	14.21 tok/s	1.207 s
8	54.95 tok/s	7.08 tok/s	2.149 s

Aggregate throughput in uzu stalled at approximately 55 tok/s across all concurrency tiers.
Individual stream generation throughput dropped in direct inverse proportion to concurrency (28.55 -> 14.21 -> 7.08 tok/s). This indicates time-sliced sharing of GPU execution slots.

Comparing Locals with Reference mlx-lm

To evaluate Locals 0.2.1, we set up Apple’s reference MLX runtime, mlx-lm 0.32.2, as a baseline.
Both runtimes loaded the identical local checkpoint mlx-community/Qwen3.5-9B-MLX-4bit (~5.98GB).

Running Locals’ preliminary quick tuning reduced the quantization matrix multiplication microbenchmark latency from 93.05 ms (standard kernel) to 30.90 ms (optimized kernel).

Single-Request Performance

Settings: Locals 0.2.1, MLX 4-bit, 1 slot; values represent medians across 5 runs.

Case	Generation Speed	Median TTFT	Median E2E	Median Output Tokens
A Short Q&A	59.99 tok/s	0.428 s	4.663 s	256
B Code	62.82 tok/s	0.310 s	10.418 s	636
C ~8K Context	51.17 tok/s	14.913 s	18.281 s	169
D ~27K Context	43.73 tok/s	51.840 s	55.549 s	178

Without speculative decoding, Locals delivered the highest base throughput in this benchmark, reaching roughly 60 tok/s on short text and 63 tok/s on code.
All 5 code outputs from task B were valid Python code containing the four specified assertions.

Multi-Turn Chat and Concurrent Scaling

In multi-turn chat E, Locals recorded a median TTFT of 0.165 s and a median generation throughput of 118.56 tok/s over 10 turns.
From turn 2 onward, most conversation context hit cache, resulting in fast responses.

For concurrency test F, Locals and standard mlx-lm were each configured with 8 slots to compare aggregate throughput directly on the same prompt.

Concurrency	Locals Aggregate	Standard mlx-lm Aggregate	Speedup Ratio	Locals TTFT	mlx-lm TTFT
2	100.36 tok/s	71.43 tok/s	1.40x	0.563 s	0.738 s
4	124.65 tok/s	79.35 tok/s	1.57x	1.097 s	1.655 s
8	203.14 tok/s	69.65 tok/s	2.92x	1.992 s	3.123 s

Standard mlx-lm flattened at 70 to 79 tok/s under concurrency. In contrast, Locals scaled to 203.14 tok/s at 8 parallel requests—a 2.92x throughput advantage.
When running single-stream task A under the same 8-slot allocation, Locals achieved 58.21 tok/s versus 54.63 tok/s for standard mlx-lm (1.07x).
The optimized kernels and batching scheduler in Locals show their primary advantage when handling concurrent traffic.

Engine Comparison and Environmental Context

These benchmarks highlight clear trade-offs across runtimes, distinguishing direct apples-to-apples comparisons from divergent baseline conditions.

When comparing llama.cpp and Magnitude 0.2.12 on the identical GGUF checkpoint (UD-Q4_K_XL) without speculative decoding, Magnitude maintained a 12% to 14% lead: 41.61 vs. 36.59 tok/s on task A, and 41.56 vs. 37.07 tok/s on task B.
On long contexts C and D, where Magnitude ran with a 256K limit and llama.cpp with 32K, generation speeds held at roughly 40 tok/s for Magnitude and 31 to 34 tok/s for llama.cpp.

Speculative decoding benefits varied dramatically by task.
On structured tasks with predictable token patterns, like code generation in task B, both implementations delivered major speedups: llama.cpp’s MTP reached 1.62x (37.07 -> 60.22 tok/s), while Magnitude’s DFlash reached 1.99x (41.56 -> 82.58 tok/s).
However, on long-context tasks C and D, draft acceptance dropped to roughly 50%. The associated verification overhead extended TTFT and increased overall E2E duration.
Speculative decoding is most effective when targeted at predictable workloads like code generation.

The comparison between Locals and standard mlx-lm on identical MLX 4-bit weights showed its largest divergence under concurrent load.
While single-stream performance was comparable (1.07x difference), Locals reached 203.14 tok/s at 8 concurrent streams compared to 69.65 tok/s for standard mlx-lm—a 2.92x improvement in aggregate throughput.
Aggregate throughput for Ollama and uzu leveled off at approximately 55 tok/s under concurrency, though these reflect different model formats and scheduler implementations.

Output Quality and Excluded Metrics

In terms of output quality, uzu’s code generation in task B failed syntax checks across all 5 iterations due to embedded self-correction thoughts.
The other engines (llama.cpp, Magnitude, Ollama, and Locals) consistently produced syntactically valid code that satisfied all four test assertions.

All five inference engines ran successfully on the M1 Max 64GB testbed.
We excluded 27B parameter models because 9B evaluations provided sufficient architectural clarity without the memory constraints and heavy download footprints of 27B weights.
GPU utilization metrics are omitted because macOS does not offer a consistent, non-privileged interface for recording accurate GPU saturation across isolated daemon processes.

Data Availability

Raw prompts, daemon launch parameters, and run-by-run logs with minimum and maximum values are archived in the repository under llm-engine-benchmark/results/summary.csv and llm-engine-benchmark/results/summary.md.