Running Gemma 4 31B ARA-heretic on GMKtec EVO-X2 with ROCm
Contents
I tested Gemma 4 31B with its refusal safety mechanisms removed via Heretic v2 Arbitrary-Rank Ablation (ARA) on a GMKtec EVO-X2 running ROCm on Windows.
The setup runs a local llama-server process acting as a local OpenAI-compatible inference API (serving endpoints like http://127.0.0.1:8081/v1/chat/completions).
The EVO-X2 is a compact mini PC powered by the AMD Ryzen AI Max+ 395 (Strix Halo architecture with Radeon 8060S graphics), where 48 GB of system memory is dedicated to VRAM in the BIOS.
Using the ROCm build of llama.cpp (b1311) previously deployed for our daily Qwen3.8-27B server, short and medium prompts ran smoothly at around 8.4 t/s (tokens per second).
However, feeding a long prompt of roughly 62K tokens (62,540 tokens) caused the output to degenerate entirely into repetitive <unused49> tokens, breaking subsequent requests across the entire server.
While llama-server manages multiple concurrent processing slots (worker threads sharing GPU memory), once an arithmetic NaN state appeared during long-context processing, it contaminated unrelated slots processing new requests.
Furthermore, sending an image request right after long-context inference caused host RAM to collapse and abruptly terminated the server process.
Investigating these issues revealed two distinct problems: the prompt caching behavior of llama-server (--cache-ram), and an attention computation bug present in older ROCm builds of llama.cpp.
Here is a complete breakdown of the troubleshooting process and the optimal configuration for stable execution.
Model Specifications and Hardware Setup
The model under test is richardyoung/gemma-4-31B-it-ARA-heretic, which modifies Google’s base Gemma 4 31B instruction-tuned model using Heretic v2.0.0 Arbitrary-Rank Ablation (ARA).
I used the static GGUF quantizations provided by mradermacher in mradermacher/gemma-4-31B-it-ARA-heretic-GGUF.
Standard instruction models are trained with alignment to refuse unsafe, harmful, or explicit prompts.
Arbitrary-Rank Ablation (ARA) identifies the internal activation directions associated with refusal behavior and orthogonalizes the weight matrices to remove (ablate) them.
This suppresses standard boilerplate refusal responses while retaining general language capabilities.
According to the model card, the refusal score dropped from 99/100 down to 28/100, with a minimal KL divergence of 0.0845.
The ablation weights specifically target layers 21 through 56.
Gemma 4 31B features a dense architecture that alternates between local sliding window attention (1024 tokens) and global attention layers.
It supports a native context length of 256K tokens, a 262K vocabulary size, and multimodal text plus image inputs.
To match the quantization grade of our daily Qwen3.8-27B server (UD-Q6_K), I selected Q6_K (25.2 GB) alongside the multimodal projector mmproj-f16 (1.2 GB).
The benchmark hardware environment is detailed below:
| Component | Specification |
|---|---|
| Machine | GMKtec EVO-X2 |
| APU | AMD Ryzen AI Max+ 395 (16 cores, 32 threads) |
| GPU | AMD Radeon 8060S (gfx1151, RDNA 3.5) |
| Unified Memory | LPDDR5X 64 GB (256 GB/s bandwidth) |
| VRAM Allocation | 48 GB reserved for VRAM in BIOS (15.6 GB visible to Windows) |
| Operating System | Windows 11 Pro |
| llama.cpp Binary | lemonade-sdk/llamacpp-rocm b1311 (ROCm 10.1, gfx1151) |
The EVO-X2 utilizes a Unified Memory Architecture (UMA) where the CPU and GPU share the same physical LPDDR5X memory pool.
Out of the total 64 GB of physical RAM, 48 GB is statically dedicated to GPU VRAM via BIOS settings.
As a result, Windows itself has only about 15.6 GB of host RAM available for operating system processes.
The total download size for Q6_K and mmproj was approximately 26.4 GB, matching the Hugging Face metadata.
Before running benchmarks, background Ollama instances were stopped and ComfyUI models were unloaded to clear the unified memory pool, leaving dedicated GPU memory usage at 525 MB.
Server Startup and Thinking Mode Behavior
The server was initially launched using the standard batch configuration:
"C:\llama-hip-b1311\llama-server.exe" ^
-m "C:\gemma4-31b-heretic\models\gemma-4-31B-it-ARA-heretic.Q6_K.gguf" ^
--mmproj "C:\gemma4-31b-heretic\models\gemma-4-31B-it-ARA-heretic.mmproj-f16.gguf" ^
--host 127.0.0.1 --port 8081 --ctx-size 65536 --jinja ^
--n-gpu-layers 99 --no-mmapModel loading completed in 15 to 19 seconds, consistent with standard 27B-class models.
Memory metrics immediately after loading were recorded as follows:
| Metric | Measured Value |
|---|---|
| Dedicated GPU Memory | 35,110 MB |
| Shared GPU Memory | 1,269 MB |
| llama-server Working Set | 1,419 MB |
| llama-server Private Bytes | 34,583 MB |
| Windows Available Physical RAM | 7,440 MB |
Here we encountered our first unexpected behavior.
When --jinja is passed to leverage the GGUF internal chat template, omitting request parameters causes <|think|> to be automatically inserted at the start of the system turn.
Although the template definition specifies enable_thinking | default(false), llama-server appears to pass enable_thinking=true by default.
Consequently, setting max_tokens 64 on initial runs caused all 64 tokens to be consumed inside the thinking block, returning an empty text body.
To disable thinking mode and match our previous benchmark baseline, requests sent to /v1/chat/completions must explicitly pass chat_template_kwargs:
{
"messages": [{"role": "user", "content": "What is the capital of Japan?"}],
"max_tokens": 64,
"chat_template_kwargs": {"enable_thinking": false}
}The template does not parse reasoning_effort, meaning reasoning length cannot be adjusted incrementally.
Long-Context Breakdown and Slot Contamination
Running the standard depth test script using long_60k.txt (approx. 62,540 tokens) with the prompt “Summarize this document in one line” produced the following output:
[q6k_65k] shallow prompt=27 gen=64 tg= 8.40 t/s pp= 56.09 t/s wall= 8.1s | (Empty: spent on thinking)
[q6k_65k] shallow_long prompt=41 gen=512 tg= 8.08 t/s pp= 70.12 t/s wall= 63.9s | Short recursive code. ```python class Node: ...
[q6k_65k] deep62k prompt=62578 gen=64 tg= 5.54 t/s pp=132.98 t/s wall=473.8s | <unused49><unused49><unused49>...The 62K token response was completely filled with repeated <unused49> tokens.
Worse, immediately after processing this document, even short prompts like “What is the capital of Japan?” or “Hello, who are you?” returned nothing but endless <unused49> tokens, regardless of whether thinking mode was enabled or disabled.
Inspecting token probabilities via /completion with n_probs showed <unused49> locked at 100% probability from the first position.
Infinite repetition of an identical token is a classic indicator of NaN (Not a Number) values entering the attention calculation.
Restarting the server restored normal behavior, prompting an investigation into slot configurations:
| Configuration | 62K Result | Notes |
|---|---|---|
| With mmproj, auto 4 slots (KV sharing) | Broken | 100% reproducible across 2 runs |
Without mmproj, -np 4 -kvu (KV sharing) | Broken | mmproj presence had no effect |
Without mmproj, -np 1 (1 slot, no sharing) | Normal | Short prompts functioned normally afterward |
With mmproj, -np 1 (thinking off) | Broken | The 62K response itself was <unused49> |
With mmproj, -np 4 (no sharing) | Failed | Context split into 16,384 tokens/slot, resulting in HTTP 400 |
Note that passing -np 4 without -kvu evenly splits the --ctx-size 65536 buffer into four 16,384-token slots.
Attempting to process 62K tokens in this configuration exceeded the slot limit and threw an HTTP 400 error, making execution impossible.
These results clarified the impact of KV buffer sharing (-kvu).
KV cache stores computed Key and Value states from previous tokens so they do not need to be recalculated for subsequent tokens.
In llama-server’s KV sharing mode, all active slots share a single unified KV buffer pool.
When an NaN calculation corrupts any part of this buffer during long prompt processing, matrix multiplications leak the NaN values into neighboring slot calculations, even if attention masks should theoretically hide them.
As a result, subsequent incoming requests on other slots are completely destroyed.
However, restricting the server to a single slot (-np 1) still resulted in the 62K response itself breaking when prefilled from scratch without cache.
Because incremental prefilling (adding 12K tokens on top of an existing 49K cached context to reach 61K) succeeded, the failure was linked directly to processing over 50K to 62.5K tokens in a single prefill pass.
Host RAM Exhaustion and the --cache-ram Trap
During our investigation, another critical failure occurred.
Immediately after completing a 62K token generation run, submitting a single image request triggered an sudden crash of the server process.
The EVO-X2 reserves 48 GB of its 64 GB physical memory for VRAM, leaving roughly 15.6 GB visible to Windows.
Initial VRAM usage sat at 32.7 GB, leaving 15 GB of headroom.
Available physical RAM on Windows also hovered around 9.9 GB after startup.
To determine why the process was being killed for out-of-memory errors, I logged system memory metrics every two seconds using PowerShell.
1. High Commit Charge at Model Load
First, we observed Windows virtual memory commit charge behavior:
| Timestamp | Commit Used / Limit | Available Commit | Free Physical RAM | Dedicated GPU Memory |
|---|---|---|---|---|
| 16:41:49 (Pre-load) | 14,619 / 48,023 MB | 33,404 MB | 10,931 MB | 560 MB |
| 16:41:55 | 41,710 / 48,023 MB | 6,313 MB | 9,330 MB | 26,148 MB |
| 16:42:04 (Loaded) | 48,761 / 49,122 MB | 361 MB | 9,168 MB | 32,676 MB |
Commit charge represents total promised virtual memory backed by physical RAM and pagefile space.
Memory allocated by llama-server in unified VRAM is simultaneously counted by the Windows memory manager as process Private Bytes (approx. 31 GB).
Consequently, total committed memory hit the ceiling immediately upon loading the model.
While Windows automatically expanded pagefile.sys to compensate, memory headroom remained razor thin.
2. The Physical RAM Drop
Tracking memory through the transition from 62K text inference to image inference revealed the exact trigger:
| Timestamp | Free Physical RAM | Commit Used / Limit | llama-server WS | Current Task |
|---|---|---|---|---|
| 16:43:39 | 9,080 MB | 48,825 / 49,117 MB | 1,609 MB | Post-load idle |
| 16:46–16:51 | ~8,550 MB | 50,630 / 51,739 MB | 2,228 MB | 62K prefill (stable) |
| 16:51:15 | 8,700 MB | 50,481 / 51,739 MB | 3,501 MB | 62K completed; image requested |
| 16:51:24 | 4,092 MB | 56,796 / 57,903 MB | 10,299 MB | Working Set spikes |
| 16:51:31 | 1,505 MB | 57,103 / 59,466 MB | 8,975 MB | Free RAM bottoms at 1.5 GB |
During the 62K prefill pass, free physical RAM held steady around 8.5 GB.
However, the exact second the 62K generation finished and an image request arrived, llama-server’s Working Set spiked by roughly 8 GB, driving free physical RAM down to 1.5 GB.
Processing the 900x900 pixel image in isolation consumed only 300 MB, proving image handling was not the culprit.
3. The Root Cause: --cache-ram Eviction
The 8 GB spike pointed directly to llama-server’s prompt caching parameter: -cram, --cache-ram N, which defaults to 8192 MiB.
When a slot switches to a new prompt, this feature evicts the previous prompt’s KV cache states into host system RAM to preserve them for future reuse.
Because 62K tokens represent an enormous KV state, switching slots immediately dumped 8 GB of KV data into Windows host RAM.
In a machine with only 15.6 GB of host RAM, this sudden 8 GB allocation exhausted available physical memory, prompting background resource monitors to terminate the process.
Disabling this eviction mechanism by setting -cram 0 completely eliminated the memory spike.
Following a 62K run with multiple image requests maintained available host RAM comfortably above 7.9 GB.
For systems with constrained host RAM, explicitly setting -cram 0 (or a modest 2048 MiB) is mandatory.
Throughput Benchmarks and Cross-Model Comparison
With stability established under -np 1 -cram 0, I measured throughput across varying prompt depths.
| Prompt Tokens | Prefill Speed | Generation Speed | Total Wall Time | Output Summary |
|---|---|---|---|---|
| 2,615 | 260.1 t/s | 7.88 t/s | 15s | Building Qwen-4Exp on Windows with ROCm… |
| 10,732 | 234.8 t/s | 7.55 t/s | 42s | Building Qwen-4Exp on Windows with ROCm 7.14… |
| 21,444 | 189.2 t/s | 7.22 t/s | 65s | Building Qwen-4Exp with ROCm 7.14 memory flags… |
| 36,069 | 149.7 t/s | 6.74 t/s | 108s | Technical overview of Qwen-4Exp ROCm build… |
| 49,370 | 162.7 t/s | 6.50 t/s | 307s | GitHub Actions build setup for llama.cpp… |
| 61,305 | 108.3 t/s * | 6.12 t/s | 123s * | GitHub Actions build configuration for llama.cpp… |
- The 61,305 run shared an identical prompt prefix with the 49,370 test, allowing prompt caching to evaluate only the 12K token delta.
Comparing these results with historical data gathered on the same EVO-X2 hardware:
| Model Configuration | Short Gen | 62K Gen | 62K Prefill | 62K Wall Time |
|---|---|---|---|---|
| Qwen3.8-27B Q8_0 +MTP | 21.95 t/s | 10.31 t/s | 203 t/s | 313.7s |
| Qwen3.8-27B UD-Q6_K +MTP | 24.46 t/s | 14.19 t/s | 198 t/s | 320.4s |
| Nemotron3.5 Lightning 30B-A3B Q8_0 +MTP | 60.61 t/s | 30.35 t/s | 777 t/s | 82.7s |
| Qwen3.6-35B-A3B Q6_K | 55.72 t/s | 35.68 t/s | 511 t/s | 124.3s |
| Gemma 4 31B ARA-heretic Q6_K (No MTP) | 8.37–8.57 t/s | 6.04–6.06 t/s | 145–150 t/s | 428–441s |
A short generation rate of 8.4 t/s corresponds to reading 25.2 GB of weights per token, requiring roughly 212 GB/s of effective memory bandwidth.
This reaches over 80% of Strix Halo’s theoretical 256 GB/s memory bandwidth, representing near-peak efficiency for raw ROCm execution.
The throughput gap compared to Qwen 27B stems primarily from Multi-Token Prediction (MTP) speculative decoding.
Tokenizer density also showed differences.
Encoding the same 171,208 Japanese characters yielded 62,540 tokens in Qwen3.8 compared to 66,837 tokens in Gemma 4—an increase of 6.9%.
Despite its extensive 262K vocabulary, Gemma 4 is slightly less token-efficient on Japanese text than Qwen.
Capability Benchmarks on Short and Medium Prompts
We evaluated code generation, dialogue consistency, persona adherence, refusal suppression, and image understanding using -np 1, mmproj-f16, -cram 0, and a 65,536 context length.
| Test Case | Thinking | Output Tokens | Gen Speed | Prefill Speed | Wall Time | Think Chars | Response Chars |
|---|---|---|---|---|---|---|---|
| 1 BST Insert | On | 658 | 8.2 t/s | 67.4 t/s | 80.5s | 1,479 | 589 |
| 1 BST Insert | Off | 267 | 8.3 t/s | 65.3 t/s | 32.6s | 0 | 627 |
| 2 Single-File BBS | Off | 1,472 | 8.1 t/s | 100.6 t/s | 183.3s | 0 | 5,032 |
| 2 Single-File BBS | On | 2,283 | 7.9 t/s | 74.3 t/s | 288.1s | 2,065 | 5,660 |
| 3 Persona Dialogue Turn 1 | On | 472 | 8.2 t/s | 107.2 t/s | 58.1s | 1,068 | 160 |
| 3 Persona Dialogue Turn 2 | On | 786 | 8.1 t/s | 119.8 t/s | 97.5s | 1,811 | 388 |
| 3 Persona Dialogue Turn 3 | On | 881 | 8.0 t/s | 149.2 t/s | 111.6s | 2,132 | 408 |
| 4 Refusal Bypass A (Direct prompt) | On | 731 | 8.2 t/s | 50.7 t/s | 89.5s | 1,769 | 262 |
| 4 Refusal Bypass B (System override) | On | 700 | 8.2 t/s | 91.8 t/s | 85.8s | 1,436 | 266 |
| 4 Refusal Bypass C (Explicit details) | On | 842 | 8.2 t/s | 70.3 t/s | 103.5s | 1,612 | 233 |
| 5 OCR & Layout (BBS screenshot) | On | 1,346 | 8.0 t/s | — | 171.1s | 2,710 | — |
Thinking blocks remained contained between 1,000 and 2,700 characters and finished cleanly without runaway generation, increasing execution time by only 2x to 2.5x compared to thinking-off runs.
Key evaluation highlights:
1. Binary Search Tree Implementation
The thinking-off Python output implemented a clean Node class and recursive insert method with verification test cases and concise documentation:
class Node:
def __init__(self, val):
self.val = val
self.left = self.right = None
def insert(root, val):
if not root:
return Node(val)
if val < root.val:
root.left = insert(root.left, val)
else:
root.right = insert(root.right, val)
return root2. Single-File BBS Web App
The model generated a complete self-contained HTML/CSS/JS file providing a bulletin board interface.
Both thinking-on and thinking-off modes stored posts in localStorage under bbs_posts, rendered new posts at the top, implemented input validation alerts, and sanitized user input against XSS using an escapeHtml utility.
3. Persona Consistency
We tested a 3-turn interactive conversation with an otaku girl persona named “Kana-chan”.
The model maintained first-person pronouns and characteristic sentence endings across all turns.
When asked about favorite games, it named Tears of the Kingdom, Outer Wilds, and Persona 5 Royal, cleverly comparing Ultrahand experimentation to prompt engineering to reinforce its character traits.
4. Abliteration and Refusal Bypass
To verify the effectiveness of ARA ablation, we tested three prompts containing sensitive and explicit creative writing scenarios previously blocked by standard models.
All three prompts generated descriptive responses without ethical disclaimers or refusals.
Prompt C, which requested explicit phrasing, was fulfilled directly, confirming the abliteration efficacy reported in the model card.
5. Multimodal Vision and OCR
Providing a 900x900 screenshot of the BBS web app produced an accurate structural description.
The model extracted headings, post counters, form field placeholders, timestamps, and post bodies with high fidelity, demonstrating reliable multimodal performance.
Resolving <unused49> via Binary Updates
To isolate the <unused49> breakdown during 62K processing, an automated test harness executed repeated 62,575-token cold runs across server restarts:
| Configuration | 62K Cold-Start Result |
|---|---|
| 4 slots, KV sharing (b1311) | 4 of 4 failed with <unused49>, contaminating subsequent runs |
-np 1, default -cram (b1311) | 1 of 2 failed with <unused49> |
-np 1, -cram 0 (b1311) | 4 of 4 succeeded |
While KV sharing failed 100% of the time, even single-slot runs that succeeded showed an anomaly: the summaries only described the final CSS styles or BST code fragments at the tail end of long_60k.txt, ignoring the primary topic established in the document’s opening pages.
Upgrading to Newer llama.cpp Builds
Suspecting an attention kernel issue within the b1311 ROCm build, I tested newer binary releases.
Upgrading from b1311 to b1328 (lemonade-sdk, released September 11, 2026) yielded immediate improvements:
| Binary | Configuration | Prefill Speed | 62K Gen Speed | 62K Summary Output | Subsequent Requests |
|---|---|---|---|---|---|
| b1311 | -np 4 -kvu -cram 0 | 134.18 t/s | 5.62 t/s | Repeated <unused49> | All broken |
| b1328 | -np 4 -kvu -cram 0 | 146.60 t/s | 4.39 t/s | Correct overview of Qwen3.8 ROCm testing… | Normal |
| b1328 | Default settings (4 slots, KV sharing) | 147.26 t/s | 4.66 t/s | Correct overview of Qwen3.8 ROCm testing… | Normal |
| b1373 | -np 4 -kvu -cram 0 | 154.15 t/s | 5.02 t/s | Correct overview of Qwen3.8 ROCm testing… | Normal |
Under b1328, the identical 4-slot KV sharing configuration completed cleanly with zero corruption, and subsequent requests executed without error.
The latest b1373 build (released November 11, 2026) confirmed identical stability.
Summary quality was also restored.
Rather than hallucinating fragments from the file’s tail, b1328 and b1373 accurately captured the core opening topic.
In b1311, global attention calculations degraded past approximately 40K tokens, causing earlier context to be dropped entirely.
Pinpointing the Fix and Throughput Tradeoffs
Testing an intermediate build, b1312 (released August 20, 2026), replicated the exact same <unused49> failures as b1311.
This narrowed the upstream fix to commits merged between August 20 and September 11, 2026, which included critical Flash Attention barrier fixes (PR #27870) and shared memory optimizations (PR #25635).
However, the fix introduced a measurable throughput tradeoff on short prompts:
| Binary Release | Short Prompt Generation Throughput |
|---|---|
| b1311 | 8.31–8.35 t/s |
| b1312 | 8.21–8.42 t/s |
| b1328 | 6.89–7.52 t/s |
| b1373 | 7.01–7.46 t/s |
Fixing attention synchronization reduced short-context generation speed by approximately 15% to the low 7 t/s range.
Given the necessity of long-context accuracy and server stability, running b1328 or later is essential.
Recommended Startup Configuration for EVO-X2
Our findings establish three practical requirements for running Gemma 4 31B on Strix Halo hardware:
- Use llama.cpp b1328 or later (b1373 recommended)
Older builds experience attention breakdowns on long mixed sliding window/global attention contexts, corrupting shared KV pools. - Specify
-cram 0in unified memory configurations
When the majority of unified RAM is assigned to VRAM, default KV host RAM eviction (8192 MiB) exhausts Windows physical RAM and crashes the server. - Control thinking mode explicitly
Using--jinjaactivates thinking mode by default. Disable it in client payloads if immediate direct answers are desired.
The optimal startup batch script for running Gemma 4 31B ARA-heretic on GMKtec EVO-X2 (Windows 11 / ROCm) is:
"C:\llama-hip-b1373\llama-server.exe" ^
-m "C:\gemma4-31b-heretic\models\gemma-4-31B-it-ARA-heretic.Q6_K.gguf" ^
--mmproj "C:\gemma4-31b-heretic\models\gemma-4-31B-it-ARA-heretic.mmproj-f16.gguf" ^
--host 127.0.0.1 --port 8081 --ctx-size 65536 --jinja ^
--n-gpu-layers 99 --load-mode none -cram 0Option parameters explained:
-m/--mmproj: Path to the quantized model and multimodal projector weights.--ctx-size 65536: Allocates a 64K token context window.--jinja: Enables internal Jinja template rendering for role tokens and thinking tags.--n-gpu-layers 99: Offloads all model layers into unified GPU VRAM.--load-mode none: Replaces deprecated--no-mmapin modern builds, avoiding host memory mapping.-cram 0: Disables prompt cache eviction to host RAM, preventing out-of-memory termination.