Measured on M1 Max 64GB: weight-only INT8 runs 41% slower than fp16, a hand-written Metal int8 GEMM 15.6x slower, MPSMatrix int8 3x, and MLX 8-bit 9% slower.
Tested on M1 Max 64GB ComfyUI: v0.24.1 fails to load int8_tensorwise, v0.30.1 hits the missing aten::_int_mm MPS kernel, and a dequantize patch runs slower than bf16.
Hands-on with Tencent Hy-MT2 1.8B Q4_K_M (1.08GB) on M1 Max 64GB via llama-server. JSON, SRT, HTML, glossary, and minority-language prompts with full input-output pairs. The 1.25bit 440MB build does not load on stock llama.cpp 8990, and 30B-A3B (hy_v3) is not in the Mac route yet.
Hypura breaks away from llama.cpp’s mmap design and streams even dense models with a three-tier NVMe placement, while TurboQuant eliminates quantization-constant overhead via a polar-coordinate transform. Includes a design comparison with Flash‑MoE and a review of scenarios where KV‑cache compression actually helps.
How should memory be allocated in reasoning models? This paper explains the trade-offs among quantization, KV cache, and test-time compute, based on 1,700 experiments.
