#Local LLM

33 articles

TechApr 21, 2026updated

Qwen3.6-35B-A3B on M1 Max via Ollama 0.20.6: 27 tok/s same as 3.5, but 13× thinking tokens

Hands-on Qwen3.6-35B-A3B (23GB 4bit GGUF) on M1 Max 64GB via Ollama 0.20.6. Generation speed stays at 27 tok/s — same as Qwen3.5-35B-A3B — but the same prompt produces 13× more thinking tokens. Multi-turn behavior, persona handling, and a three-tier NSFW probe included.

LLM Local LLM Qwen Ollama Apple Silicon MoE Experiment

TechApr 17, 2026updated

Qwen3.6-35B-A3B pairs Gated DeltaNet with MoE and raises the bar on agentic coding

Alibaba's Qwen team released Qwen3.6-35B-A3B as open weights. A 40-layer hybrid of Gated DeltaNet, Gated Attention, and MoE hits 73.4 on SWE-bench Verified, 37.0 on MCPMark, and 1397 on QwenWebBench.

LLM Local LLM Qwen MoE Agent Coding

TechApr 14, 2026

Can Local Vision LLMs Extract RPG Stats from Character Art?

I tested local Vision LLMs (Gemma 3, Qwen2.5-VL, Llama 3.2 Vision, Gemma 4) to see if they could look at character illustrations and pixel art and generate RPG-style stats in JSON format.

AI Local LLM VLM Image Recognition Ollama Gemma Qwen Apple Silicon Experiment

TechApr 13, 2026

Microsoft Ships Foundry Local GA: A ~20MB Native Library That Bundles Local AI Into Any App

Foundry Local is a local AI runtime that embeds into apps via package managers as a ~20MB native library. Built on ONNX Runtime with automatic GPU/NPU selection, it runs Phi, Qwen, Mistral and more offline through an OpenAI-compatible API.

Microsoft Local LLM Azure AI Foundry ONNX Runtime Windows ML

TechApr 6, 2026

LLM-jp-4-32B-A3B on ROCm + Strix Halo: 41% Faster Than Qwen3.5

Benchmarking NII's LLM-jp-4-32B-A3B-thinking on EVO-X2 (Ryzen AI Max+ 395) with ROCm. 62.9 t/s vs Qwen3.5-35B-A3B's 44.7 t/s. Covers thinking control issues, KV cache trade-offs, knowledge cutoff, Japanese quality comparisons, code generation tests, and training data composition.

AI LLM Local LLM llama.cpp AMD ROCm MoE Qwen Experiment

TechApr 3, 2026

Running Lemonade on Strix Halo (EVO-X2): Vulkan Shared Memory Leaks and ROCm Stability

Real-world testing of AMD Lemonade v10.0.1 on Ryzen AI Max+ 395. LLM, image generation, speech recognition, and TTS running simultaneously, NPU Hybrid execution, Vulkan vs ROCm benchmarks, and discovering shared memory leaks.

AMD Local LLM Vulkan ROCm NPU llama.cpp GPU Inference Optimization Benchmark Experiment

TechApr 3, 2026

AMD's Lemonade Local AI Server Bundles GPU, NPU, and Multi-Modal Inference Under One Roof

Lemonade is AMD's open-source local AI server that manages multiple backends like llama.cpp and FastFlowLM across GPU/NPU/CPU, serving text, image, and audio generation through an OpenAI-compatible API.

AMD Local LLM NPU GPU llama.cpp Inference Optimization ROCm Vulkan

TechApr 3, 2026updated

Google's Gemma 4 launches in four sizes (E2B–A4B), publishing Gemini 3–derived reasoning under Apache 2.0

Google DeepMind has released Gemma 4: four models—31B dense, 26B MoE (A4B), E4B, and E2B—with a 256K context, multimodal input, tool calling, and support for 140 languages.

AI LLM Google Open Model MoE Multimodal Local LLM

TechApr 2, 2026updated

SwiftLM is a Swift-based LLM inference server that integrates TurboQuant and SSD streaming into Metal shaders

SwiftLM, an Apple Silicon–only MLX inference server, provides a native Metal implementation of TurboQuant V2+V3 hybrid KV‑cache compression and NVMe SSD expert streaming.

Apple Silicon LLM MLX Local LLM Inference Optimization KV Cache MoE Swift

TechMar 31, 2026

Ollama Moves to MLX Backend, Dramatically Speeds Up Local Inference on Apple Silicon

Ollama 0.19 switches the Apple Silicon backend to MLX, achieving 1,810 tokens/s prefill and 112 tokens/s decode. NVFP4 quantization support and cache improvements landed at the same time.

Ollama MLX Apple Silicon LLM Local LLM Inference Optimization

TechMar 31, 2026updated

Qwen3.5-35B-A3B long context on Strix Halo: 65K ctx adds only 800MB VRAM on llama-server Vulkan, zero speed loss (measured)

Only 10 of 40 layers in Qwen3.5-35B-A3B consume KV cache (SSM+Attention hybrid), so raising llama-server ctx-size from 4096 to 65536 cost just 800MB VRAM with no throughput drop. Measured on Ryzen AI Max+ 395 (Strix Halo), with q8_0 KV quant numbers.

LLM Local LLM llama.cpp AMD Vulkan KV Cache Qwen Benchmark

TechMar 25, 2026

Hypura’s NVMe Streaming and TurboQuant’s KV Cache Quantization

Hypura breaks away from llama.cpp’s mmap design and streams even dense models with a three-tier NVMe placement, while TurboQuant eliminates quantization-constant overhead via a polar-coordinate transform. Includes a design comparison with Flash‑MoE and a review of scenarios where KV‑cache compression actually helps.

LLM Local LLM Quantization Apple Silicon Inference Optimization KV Cache Rust