Tested on EVO-X2 (gfx1151) under Windows + ROCm: the official b10666 binary runs it at pp512 159.25 / tg128 23.26, and the earlier crash was my own workaround flag.
Tested on M1 Max 64GB: AtomicChat's M64 GGUF keeps the 51B N-gram table in its own shard, so a llama.cpp PR #27742 build leaves it on SSD and runs the 125B MoE at 17.6 tok/s.
Tested on GMKtec EVO-X2 (Ryzen AI Max+ 395): Q8_0 + MTP beats a 4-bit M1 Max at 22 tok/s, thinking burns 32,712 chars before any HTML, and the NSFW refusal line moves.
Hands-on log of building the DEV article's PDF RAG on M1 Max 64GB, extending it with images via CLIP, and pushing through Japanese with bge-m3 + Qwen3.6 35B. Documents the modality gap, the dual inference server crash, and LLM-jp 4-8B's empty chat template silently dropping the system role.
Notes on a DEV Community article that wires up FastAPI as an OpenAI-compatible RAG API layer with llama.cpp, Chroma, and Open WebUI, plus where the architecture fits and what to watch for.
After Xiaomi MiMo-V2.5's weights went public, I checked whether it runs on Mac/ROCm or on cloud GPU (RunPod/GCE). It's still rough on local hardware, but RunPod's 4x H200 runs it for ~$14/hr and GCE Spot H100 brings it down to ~$1.6/hr.
Benchmarking NII's LLM-jp-4-32B-A3B-thinking on EVO-X2 (Ryzen AI Max+ 395) with ROCm. 62.9 t/s vs Qwen3.5-35B-A3B's 44.7 t/s. Covers thinking control issues, KV cache trade-offs, knowledge cutoff, Japanese quality comparisons, code generation tests, and training data composition.
Lemonade is AMD's open-source local AI server that manages multiple backends like llama.cpp and FastFlowLM across GPU/NPU/CPU, serving text, image, and audio generation through an OpenAI-compatible API.
Only 10 of 40 layers use KV cache, so raising llama-server ctx-size from 4096 to 65536 cost 800MB VRAM and no throughput. Measured on Ryzen AI Max+ 395.
After updating to AMD Software 26.3.1 on a GMKtec EVO-X2 (Ryzen AI Max+ 395), Vulkan backend fails to allocate device memory properly and falls back to CPU. Investigation and workaround by changing BIOS VRAM allocation from 48GB/16GB to 32GB/32GB.
The three-stage pipeline of BERT perplexity scan → LLM judgment → escalation packaged as a cross-platform Python tool. The installer automatically downloads llama-server and GGUF models.
