Kimi K3 activates 16 of 896 experts per MoE layer. From the July 27 tech report: experts are sliced FFNs, not specialists, and the router is one linear projection.
Qwen's Gated Attention (NeurIPS 2025 Best Paper) puts a per-head sigmoid gate on SDPA output. First-token attention drops 46.7%→4.8%, max activation 1053→94. Why it works and how Qwen3-Next uses it.
Luma AI's Uni-1 integrates image understanding and generation in one decoder-only autoregressive model. It does not use diffusion; instead, it tokenizes text and image patches in a shared vocabulary and generates them sequentially.
AttnRes to replace Transformer's fixed residual combination with softmax attention in the depth direction. Demonstration with Kimi Linear 48B improved GPQA-Diamond +7.5pt and HumanEval +3.1pt. Training overhead was kept below 4% and inference below 2%.
A paper explains that two seemingly mysterious Transformer behaviors, heavy attention on specific tokens and unusually large activations in specific dimensions, are actually manifestations of the same mechanism.
