Kimi K3's router picks 16 of 896 experts: an allocator, not a classifier
Contents
When I first read that Kimi K3 “activates 16 out of 896 experts,” I pictured a model that calls in the 16 specialists who fit the question out of a roster of 896. A math person, a code person, a Japanese person, that kind of division of labor.
Once you look at the fact that the 16 are chosen again per token at every layer, the metaphor gets in the way. An expert is not a small standalone LLM. In K3 it is a slice of the FFN (feed-forward network) part of a Transformer block.
When I wrote about the K3 announcement, the tech report was not out yet, so there was almost nothing to say about Stable LatentMoE. The weights and tech report landed on July 27, and the router equation, the dimensions an expert works on, and the load-balancing method are all in there now. That is enough to ask what an expert really is, and what the router is actually classifying.
An expert in K3 is a slice of one layer’s FFN
A normal Transformer block has attention and an FFN. Attention mixes information across tokens, the FFN transforms the features of each token.
In a dense model every token in a layer goes through the same FFN.
MoE provides several of these FFNs and runs only some of them per input.
is an expert and is the coefficient applied to its output. As a diagram:
flowchart TD
A[Hidden state of one token entering a layer] --> B[Router computes 896 scores]
B --> C[Select the top 16]
C --> D[Run the 16 expert FFNs]
D --> E[Weighted combination of outputs]
E --> F[Next layer]The actual K3 numbers:
| Item | Kimi K3 |
|---|---|
| Layers | 93 |
| Model hidden dimension | 7168 |
| Routed experts | 896 per MoE layer |
| Experts chosen per token | 16 |
| Always-on shared experts | 2 |
| Latent dimension fed to experts | 3584 |
| Intermediate dimension per expert | 3072 |
| Total parameters | 2.78T |
| Active parameters | 104.2B |
The part that matters is that each MoE layer holds its own set of 896. Expert 137 in layer 20 and expert 137 in layer 70 are different weights that happen to share a number.
In K3’s Stable LatentMoE the input handed to the routed experts is compressed from 7168 to 3584 dimensions, and the combined output of the 16 is projected back to 7168. The shared transform is handled by 2 shared experts that stay on at the original 7168 dimensions.
Simplified:
“Consult 896 people” is further from the implementation than “run two shared passes while picking 16 out of 896 small transforms to assemble an FFN on the spot.”
No expert is assigned a field like “Python”
The name suggests that fields of responsibility exist before training. In practice neither the experts nor the router receive domain labels as supervision.
At the start of training nothing about expert 137 is decided. When the router sends a hidden state to an expert, the chosen expert’s output is scaled by a mixing weight, and the gradient from the next-token prediction loss reaches the router through that mixing weight. The Top-16 selection itself is discrete, and no gradient passes through it. The load-balancing bias update runs as a separate process. Particular inputs collect at the same expert, that expert learns a transform that suits them, and it gets selected more often. Roles form out of that loop.
DeepSeekMoE split experts into finer pieces for the same reason: more freedom in combining small experts. Separating shared experts so that routed experts hold less duplicated common knowledge also carried over into K3.
Correlations with language or domain do show up after training. A 2026 analysis of multilingual MoE found routing in early and middle layers to be language-agnostic and diffuse, with language-specific specialization appearing mostly in later layers. Another multilingual analysis reported language-specific processing in early and late layers with middle layers acting as language-agnostic capacity. The two look contradictory at first, but the models and the analysis conditions differ.
The Illusion of Specialization at ACL 2026 examined MMLU routing in three models at the level of expert groups rather than individual experts. Across many domains, layers, and routing counts there is a small “Standing Committee” that always takes a large share, handling syntax and the scaffolding of reasoning, while peripheral experts handle the domain-specific knowledge.
Real expert sets mix pieces reused across many domains with pieces skewed toward one. What lives in an expert is not a field name like “Python” or “math” but a transform that worked well on some region of hidden state. Sometimes that region lines up with a category humans have a word for, sometimes it is a blend of syntax, token shape, numeric representation, and combinations of features.
The router is a compute allocator
What does a router that picks 16 out of 896 actually do? The base equation in the K3 tech report is one line.
It linearly projects the current hidden state into 896 scores . During training a per-expert load-adjustment bias is added before taking the top 16. The mixing ratios after selection use the normalized original scores.
In shape alone that resembles an 896-class classifier. But the router returns 16 indices out of 896 candidates and the corresponding FFNs run immediately after, and training backpropagates the next-token prediction loss into the router. Both what the output is used for and how it is trained differ from ordinary multi-class classification.
The entering the router is the hidden state that attention and FFNs up to that layer have already shaped. A single linear projection is enough for the final selector because the earlier layers put meaning, syntax, and position into a linearly separable form. It resembles bolting a small linear classification head onto BERT hidden states, except a MoE router has no human-assigned labels.
The router trains inside the main run
I assumed that routing a 2.8T model would require training the router itself as a small LLM. Instead K3’s router is updated together with the experts inside the main training run, with no separate model of its own and no independent training as a comprehension model or a policy model.
Load imbalance across experts is the problem around the router. When tokens skew toward particular experts, processing piles up on the GPU holding them. Experts that get selected rarely also see less training data. Push too hard for even distribution and the destination for the same input changes from step to step, which makes experts resemble each other.
Memory-Aware Routing at ACL 2026 Findings calls this state “pseudo-balance”: usage counts are even, but knowledge is duplicated and specialization never settles.
Because K3 handles the extreme count of 896, it added a load-balancing scheme called Quantile Balancing. It derives the threshold at which each expert receives its target number of tokens from a quantile of the router scores (the cutoff value for a given top percentage of the score distribution) and updates the selection bias. At inference the bias from the end of training is fixed. Quantiles are computed during training only.
Here too the router stays a simple gate, with the engineering aimed at preventing concentration on particular experts and at preventing experts from sitting unselected and untrained.
K3 is still a fixed 93-layer Transformer
If the FFN path is this dynamic, is there a reason to keep attention and the layer count fixed?
In that sense K3 is still a Transformer. Every token walks through 93 layers in order. 69 layers of Kimi Delta Attention and 24 layers of Gated MLA are used together, but which layer uses which is fixed at design time. What the router selects from the input is only the FFN path. Attention Residuals made the choice of which past-layer representation to pull dynamic, but blocks are never skipped or repeated.
flowchart LR
A[All tokens] --> B[Fixed attention]
B --> C[MoE router]
C --> D[Select 16 FFNs]
D --> E[Next fixed layer]There are several candidates for what could stop being fixed.
- Ending at an intermediate layer per token
- Varying how many times a token passes through a block
- Choosing attention or a state space model per input
- Using attention only when long-range retrieval is needed
- Varying the FFN compute assigned to a token
All of them have research examples. ADEPT (2026) brought per-token early exit into generation. Depth-Recurrent Transformer iterates the same Transformer block in latent space, decoupling parameter count from compute depth.
Retrieval-Aware Distillation keeps only the attention heads that matter for retrieval from a pretrained Transformer and replaces the rest with state-space-style recurrent heads. On a 1B model, keeping 2% of all attention heads recovered over 95% of original performance on retrieval-heavy tasks. SALSA tries a router that calls attention only for the tokens that recurrent processing fails to cover.
These are separate lines of work, all independent of K3. But research that extends conditional compute beyond the FFN into attention, depth, and iteration count has already started.
A router that selects beyond the FFN becomes control flow
Suppose the next model made these choices per token.
flowchart TD
A[Hidden state] --> B{Local gate}
B -->|Sequence processing in compressed state| C[State space model]
B -->|Exact retrieval from the past| D[Attention]
B -->|Feature transform| E[MoE FFN]
B -->|Not enough compute| F[Rerun the same block]
B -->|Enough| G[Early exit]
C --> B
D --> B
E --> B
F --> BA router like that is assembling the neural circuit that will run. It decides which module to use, how many times, and when to stop, all from the hidden state, which puts it close to control flow in a program.
Which raises the question of whether you then need a router to judge the router. As in current MoE, though, each layer can hold its own local gate, its own gate for calling attention, and its own early-exit decision, so one giant router in the middle is unnecessary. Each gate takes the shared hidden state and outputs only the decision it owns.
Stabilizing discrete path selection is the training problem. How to push gradients into the selection, how to keep training modules that go unused, whether paths skew toward certain inputs or languages, how to hide inter-GPU communication latency. K3 needed Quantile Balancing and dedicated communication infrastructure for FFN routing alone. Make the whole model dynamically wired and the paths and communication patterns to manage grow accordingly.
Better read as conditional compute on accelerators than as an artificial neuron
Early artificial neural nets started by simplifying a biological neuron into a weighted sum of inputs and a firing rule. K3 still uses linear transforms, nonlinear functions, and learnable connections. The 896 experts are designed as a set of FFNs that run on GPUs and other accelerators.
There is research on how much was lost in that simplification. Beniaguev et al. (2021) had an artificial network reproduce, at millisecond resolution, the input/output of a biophysical model of a layer 5 cortical pyramidal cell, and reported that five to eight temporally convolutional layers were required. With NMDA receptors (a type of glutamate receptor that amplifies nonlinearly when inputs overlap) removed, a fully connected network with one hidden layer was enough. Most of that depth came from NMDA-driven nonlinearity in the dendrites.
Treating “how hard this is for an artificial network to reproduce” as a complexity measure was used to compare human and rat pyramidal cells in a PNAS paper from July 2026. That one does not vary the depth: it fixes a 3-layer TCN, measures spike-prediction AUC per cell, and inverts it into a Functional Complexity Index (FCI). Lower AUC, meaning the artificial network cannot imitate the cell, means higher complexity. Sweeping depth over 1, 2, 3, 5, 7, and 9 layers was a sensitivity analysis to confirm 3 layers was a reasonable choice.
FCI came out significantly higher for humans, driven by dendritic membrane area and branching pattern plus the density and nonlinearity of NMDA receptors. Activating more and more synapses along a single oblique dendrite of an L2/3 pyramidal cell model with human synapse properties, the summation of inputs flipped from sublinear to supralinear at around 35 synapses.
What the K3 tech report spends on Stable LatentMoE is matrix-product stability after dropping to the latent dimension and load balancing across 896 experts; inter-GPU communication, GEMM (general matrix multiply) fusion, and weight transfer belong to the systems implementation chapters. The recommendation of 64 or more accelerators appears in the official blog instead. Expert count, dimensions, and quantization scheme are all set from both training quality and hardware efficiency.
K3 is not the result of reproducing a biological neural circuit. Differentiable matrix computation was scaled up on accelerators, and experts and a router were placed there to run only the parts that are needed.