Qwen3-Next's Gated Attention: why one sigmoid gate kills the attention sink

An attention head in a Transformer can decide where to look, but it has no way to decide that nothing it found is worth using. Even when there’s nothing it wants to reference, the attention has to go somewhere, so it flows to the first token, which usually has nothing to do with the content. This is the phenomenon called the attention sink. The Qwen team gave attention the missing “don’t use it” option. The change is just multiplying the output of softmax attention by a per-head sigmoid gate, and they call it Gated Attention. With it, first-token attention drops from 46.7% to 4.8%. Qwen3-Next (the 80B-A3B released in September 2025) is a model built around this mechanism — less a new model than a substantial rework of the architecture. This came up recently at a Qwen Developer Ambassador gathering, where someone shared a write-up of this paper. On this blog I’ve been running a series of experiments that poke at where things actually happen inside a model — SFT on a single targeted layer, pulling internal features out with an SAE — so I read the paper as reference along that same line.

What Qwen3-Next reworked

Qwen3-Next has four main changes. A hybrid structure mixing linear attention and softmax attention; an ultra-sparse MoE where only about 3B of the 80B total parameters are active at inference; training-stability measures against the attention sink and massive activation; and Multi-Token Prediction (MTP) to speed up speculative decoding.

The 48-layer structure is this pattern, repeated:

$$12 \times \bigl(\,3 \times (\text{Gated DeltaNet} \to \text{MoE}) \;\to\; 1 \times (\text{Gated Attention} \to \text{MoE})\,\bigr)$$

A four-layer block, 12 times over. Three layers are the linear-attention Gated DeltaNet and one layer is the softmax-attention-based Gated Attention — a 3:1 ratio.

Item	Value
Total / active params	80B / ~3B
Layers	48 (Gated DeltaNet 36 + Gated Attention 12)
MoE experts	512 (10 routed + 1 shared activated)
Gated Attention heads	Q: 16, KV: 2 (GQA, head dim 256)
Gated DeltaNet heads	Q/K: 16, V: 32 (head dim 128)
Native context	262,144 tokens (extensible to ~1.01M with YaRN)
Training tokens	15T

This design carries over to later models. Qwen3-Coder-Next shipped as a coding-focused version on the same 80B-A3B layout, and April 2026’s Qwen3.6-35B-A3B also uses the 3:1 Gated DeltaNet / Gated Attention hybrid (the fine-grained parameters differ per model). I covered the math of the linear-attention Gated DeltaNet in the Qwen3.6 article, so from here on this is about the other half, Gated Attention.

Why the attention sink happens

The target of that concentration is, in most cases, the BOS token at the very front (a special token marking the start of the document). In the paper’s measurements, a gate-free baseline model allocated 46.7% of attention on average to the first token. Nearly half of the attention flows into a meaningless token.

The cause is in the structure of softmax. Softmax is a function that normalizes its output into a probability distribution, so the attention weights that pass through it always sum to 1. That means a head cannot choose to “look nowhere.” Even when the information it wants to reference isn’t in the context, it has to distribute a total of 1 somewhere, and the low-value first token gets used as the receptacle.

This shows up not only as first-token attention concentration but paired with massive activation (also called the residual sink), where a specific dimension of the hidden state holds an extremely large value. In the baseline model, the max hidden-state activation reached 1053. BF16 gets coarser the larger a value is, so this outlier becomes a source of numerical error and quantization degradation.

I covered a paper that explains why attention sinks and residual sinks arise — as an “implicit adjustment mechanism for stabilizing training” — in an earlier post. That paper reads the sink as a mechanism the model needs, while this Gated Attention paper is on the side that measured the sink dropping sharply once you provide a different exit. The way the first token in Qwen3-8B’s residual stream converges to a fixed pattern unrelated to content, I actually confirmed in the post where I ran Qwen-Scope’s SAE locally. That post is worth a read in its own right.

What Gated Attention is

The change proposed by Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (May 2025, Qwen team; later a NeurIPS 2025 Best Paper Award winner) is simple: multiply the output of SDPA (Scaled Dot-Product Attention, the compute core of softmax attention) elementwise by a sigmoid gate computed from the input.

$$\mathrm{Y} = \bigl(\,\sigma(X W_\theta) \odot \mathrm{SDPA}(Q, K, V)\,\bigr)\, W_o$$

$X$ is the input to the attention layer, $W_\theta$ is a linear layer added for the gate, $\sigma$ is the sigmoid, and $\odot$ is the elementwise product. If the notation is shaky for you, my math-for-reading-AI-articles series covers sigmoids and matrix products, so take a look there first. It’s fine to just get the gist — “oh, so that’s the shape of the equation” — and skip it if it’s a hassle. The gate value ranges from 0 to 1 and decides, per head and per token, whether to pass or clamp the SDPA output.

flowchart TD
    X[input hidden state] --> QKV[Q / K / V projection]
    QKV --> SDPA[SDPA<br/>softmax attention compute]
    SDPA -.->|standard attention goes<br/>straight to output projection| O[output projection]
    X --> G[gate projection]
    G --> SIG[sigmoid<br/>0-1 gate value]
    SDPA --> MUL[elementwise product]
    SIG --> MUL
    MUL --> O

The attention computation itself is untouched; the gate is applied to the output after that computation finishes.

The authors didn’t pick this position on a hunch — on a 15B-parameter MoE and a 1.7B dense model, they compared more than 30 variants that changed the gate’s position, granularity, and activation function.

Gate position	Result
Right after SDPA output	Largest effect (adopted)
Right after value projection	Second-best
Right after query / key projection	Small improvement
After the output projection	Almost no effect

On granularity and form, a head-specific gate beat a head-shared one, and a multiplicative sigmoid beat SiLU and additive variants. The largest effect comes from an elementwise gate on the output (adding 201M parameters at 15B scale), but a coarse gate with a single scalar per head performs almost on par. The latter adds around 1.6M parameters — about 1/125 of the elementwise gate.

Why just tacking a gate on the back improves things

The paper decomposes the effect into two mechanisms.

Non-linearity enters between two collapsed linear layers

The SDPA output is a linear combination of $V$, and $V$ itself is a linear projection of the input. And right after SDPA comes the output projection $W_o$, with no non-linearity between these two linear layers. Looking only inside the attention layer, the value projection $W_v$ and output projection $W_o$ can be mathematically folded into a single low-rank linear map $W_v W_o$. This is not a claim about the whole model’s expressiveness, residual connections and later layers included.

The sigmoid gate slots right between these two layers. This makes the value-side path a sandwich: linear ($W_v$) → non-linear (gate) → linear ($W_o$). Same idea as an MLP built as Linear → ReLU → Linear. The paper lists this as one of the improvement factors: the input-dependent non-linearity enters here and raises the attention layer’s expressiveness.

Heads can choose to “stay quiet this time”

The other is query-dependent sparsity. Measuring gate values in the trained model, the mean score was 0.116. Most heads clamp their output hard for most inputs.

This connects to the drop in the attention sink. The softmax sum-to-1 constraint doesn’t change when you add a gate. But because the gate can suppress, at the output stage, the output of a head that has nothing worth referencing, the need to divert attention to the first token fades — that’s the authors’ interpretation. In short, it replaces “there’s nothing, so park it on the first token” with “if there’s nothing, just throw it away,” and cuts the former as far as possible. What the paper demonstrates by measurement goes as far as: introducing the gate made first-token attention concentration and the giant activations plummet at the same time.

The effect shows up clearly in numbers.

Metric	Baseline	With gate
First-token attention share	46.7%	4.8%
Max hidden-state activation	1053	94
PPL (1.7B dense, 3.5T tokens)	6.180	6.130
MMLU (15B MoE)	58.79	60.82
RULER 128k (long-context benchmark)	31.65	58.82

The biggest gap is RULER (a benchmark for retrieval and tracking in long context) at 128k tokens: 31.65 to 58.82. With the sink and giant activations sharply reduced, the model becomes less prone to breaking when you change RoPE’s frequency setting to stretch the context length, the paper analyzes.

Training stability changes too. In a setting where a 1.7B model is trained on 1T tokens, the gated version converges even at a learning rate of 8e-3 where the baseline diverges, and loss spikes (sudden jumps in the loss during training) drop substantially. Part of the reason cited is that with the max activation down from 1053 to 94, BF16’s effective numerical precision goes up.

How it’s used across Qwen3-Next

Qwen3-Next’s Gated Attention layers take this mechanism — experimented on at 1.7B and 15B in the paper — and adopt it in a real model trained at 80B on 15T tokens. The official blog states it as “Adopt the output gating mechanism from our prior work [2],” where reference [2] is this paper, explicitly noting the adoption of output gating. The stated reason is to reduce the low-rank problem in attention. The blog also says the mechanism helps suppress attention sink and massive activation, which secures numerical stability across the model. There are 16 Q heads and 2 KV heads (GQA) with head dim 256, and RoPE is partial — applied to only the first 64 of the 256 dimensions. The official blog explains this as a way to keep performance from dropping on inputs longer than what was seen during training. Mechanically: dimensions with RoPE applied rotate by an angle that depends on position, so once you exceed the length seen in training, the rotation angle enters an unseen range and matching breaks down. Dimensions without RoPE match on content alone, independent of distance, so they don’t break as the sequence grows. Leaving 75% of the dimensions position-free means the dimensions that can break when you extend the length are limited to 25%.

Stabilizing training involves measures beyond the gate. As background: a normalization layer (RMSNorm) evens out the spread of values passing through the layer, then reapplies a learnable scale. Qwen3 used QK-Norm, applying this to Q and K, but part of that scale grew abnormally large during training. An outlier shows up here too. Qwen3-Next’s Zero-Centered RMSNorm changes how that scale is held. Instead of learning the scale directly, it splits it into “1 + delta” and learns the delta, applying weight decay to it. The delta is always pulled back toward 0, so the scale stays near 1 and abnormal growth is far less likely to occur in the first place. The MoE router is also normalized at initialization, preventing routing from skewing toward particular experts early in training.

MTP is a module that predicts not just the next token but the one after it, boosting the base model’s performance as an extra training signal while also serving at inference as a draft generator for speculative decoding. Because it drafts sharing the same context as the main model, the acceptance rate is high, and it speeds up inference by confirming multiple tokens at once. I covered the mechanics and measurements of speculative decoding in the post where I ran Gemma 4’s MTP drafter on an M1 Max.

---

Gated Attention’s gate values can be read out for all heads in a single forward pass. Which head passes its output and which clamps it is observable directly from outside the model. On this blog I’ve kept running experiments on whether you can change a trained model’s behavior by poking at it from outside. With a corrector that full-rank-SFTs a single layer, the eval_loss localized to a middle layer, but the rewriting behavior didn’t follow that number. When I read it with J-lens, I found the trained layer was writing a “nothing to fix” direction and halting the rewrite, and that post ended on the idea that turning this direction into a single gate vector added to raw Qwen might build a corrector without any SFT. In the Knowing–Using Gap paper, knowledge that was supposedly memorized never reaches the middle layers and gets stranded, and simply transplanting the hidden state into the layer where the reasoning circuit runs brought the correct answers back — without changing a single bit of the weights. All of these try to make some needed judgment or representation take effect at a targeted spot, after the fact, while leaving the body untouched. I’ve tried them in turn, but none of them has become a breakthrough of a solution for me. Gated Attention works because which head clamps its output is trained in across all heads from the pretraining stage. It sidesteps the whole after-the-fact struggle to begin with. What I want is to get close to that from outside a trained model, and this time I read the paper hoping it would help toward that. An experiment visualizing per-head attention distributions and gate values on a local Qwen3 model is something I’ll do in a separate post.