One trained transformer layer matches full-parameter RL across 7 Qwen models

A paper claiming that RL-training a single transformer layer can match full-parameter training, “Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training” (arXiv:2607.01232), showed up on arXiv.
The claim is tested across 7 models and 3 RL algorithms.

The authors are mostly from the University of Minnesota, with collaborators from Peking University and Amazon.
The paper notes the work is independent of Amazon.

What the paper claims

RL training for LLMs (GRPO-style reasoning training) normally updates every parameter.
This paper breaks the resulting gain down layer by layer to see where it actually lives.
The result: training only the single best layer recovered almost the entire gain of full-parameter RL training, and often exceeded it.
The contribution concentrates in the middle layers at 40-60% depth, while layers near the input and output contribute almost nothing.
The same shape held across 7 models, 3 algorithms, and multiple task domains.

GRPO is a PPO-family RL method that generates several responses per prompt and computes advantages from rewards relative to the group; it spread through DeepSeek-R1’s training.
Since it needs no value model, it uses less memory than PPO.
I covered the infrastructure and library side of RL training before.

The layer contribution metric

The method is simple: for an L-layer model, repeat “enable gradients for layer k only, freeze everything else, and run RL training” while varying k.
Implementation-wise it is just requires_grad=False on everything except the target layer.

flowchart TD
    A[Pretrained model with L layers] --> B[Enable gradients for layer k only<br/>freeze all other layers]
    B --> C[RL training with GRPO etc.]
    C --> D[Evaluate on benchmarks to get score S_k]
    D --> E[Compute layer contribution C_k]
    E -->|repeat for every layer k| B

Layer contribution is computed from the evaluation scores.

$C(k) = \frac{S_k - S_{\mathrm{base}}}{S_{\mathrm{full}} - S_{\mathrm{base}}}$

$S_{\mathrm{base}}$ is the untrained base model, $S_{\mathrm{full}}$ is the model after full-parameter RL training, and $S_k$ is the model with only layer k trained.
$C(k) = 1.0$ means single-layer training fully reproduced the gain of full-parameter training.

The 7 models tested are below. The DeepSeek distill is also Qwen-based, so all of them are from the Qwen family.

Model	Size	Layers
Qwen3-1.7B-Base	1.7B	28
Qwen3-4B-Base	4B	36
Qwen3-8B-Base	8B	36
Qwen2.5-Math-1.5B	1.5B	28
Qwen2.5-1.5B-Instruct	1.5B	28
Qwen2.5-3B-Instruct	3B	36
DeepSeek-R1-Distill-Qwen-7B	7B	28

The RL algorithms are GRPO as described above, plus Dr. GRPO, a variant that corrects its response-length bias, and GiGPO, which nests groups within groups for agent tasks (used in the ALFWorld experiments).

Training data includes NuminaMath-CoT and DeepScaleR (math), DeepCoder (code), and ALFWorld (agents).
Evaluation ranges from math benchmarks (MATH500, GSM8K, AIME 2024/2025, OlympiadBench) to out-of-domain ones like HumanEval+, GPQA-Diamond, and IFEval.

Middle layers dominated on all 7 models

First, the math benchmark averages for the Qwen3 models.

Model	Base	Full-param RL	Best single layer	Recovery
Qwen3-1.7B-Base	44.1	50.8	51.8 (layer 10)	1.14
Qwen3-4B-Base	52.2	63.7	64.3 (layer 16)	1.06
Qwen3-8B-Base	58.0	66.5	67.1 (layer 16)	1.07

On all three models, training only the best single layer scores higher than full-parameter training.
Recovery above 1.0 means the full gain was recovered with extra on top.

The same holds across algorithms: recovery of 1.01 for Qwen2.5-Math-1.5B with Dr. GRPO, and on ALFWorld with GiGPO, single-layer training of Qwen2.5-1.5B-Instruct reached an 89.1% success rate (87.8% for full-parameter).
The distilled DeepSeek-R1-Distill-Qwen-7B came in at 1.05.

The edge layers, on the other hand, contribute little.
On Qwen3-8B, training only layer 0 (input side) ended up worse than the base model (recovery -0.51).

Normalizing layer position to 0-1 depth, the highest-contributing layers sat in the 40-60% band on every model.
Swapping the training data from NuminaMath to DeepScaleR keeps the per-layer contribution ranking at a Spearman correlation of 0.76, and crossing from math to code still keeps it at 0.59.
The authors read layer contribution as a property tied to the model’s structure rather than to the data.

Weight changes are nearly uniform across layers

The paper also measures per-layer weight change during full-parameter RL training.
The change magnitude is nearly uniform across layers (0.5-0.8) and does not line up with the heavily skewed layer contribution.

What decides contribution is not how much a layer moved but how effective that layer’s parameter subspace is for improving the task.
Why the middle layers’ subspaces are the effective ones is left as an open question.

Using this without a full layer scan

A full layer-contribution scan takes as many RL training runs as there are layers.
The paper tries three ways of putting the contribution information to use.

The first is layer-wise adaptive learning rates: raise the learning rate to 1e-5 only for high-contribution layers and leave the rest at 5e-6.
On Qwen3-1.7B this went from 50.82 to 53.70, beating full-parameter training by 2.88 points.

The second is selective training of the top-10 contribution layers, which took Qwen3-8B from 66.43 to 69.11.

The third is a heuristic that skips the scan and just trains the middle layers from the start, which moved Qwen3-1.7B from 50.82 to 51.35.

All three beat full-parameter training while updating fewer parameters.

The only one that needs no scan is the third, and the authors recommend it as the default setting.

Majority voting across single-layer models beats full-parameter RL

The paper has one more analysis: majority voting.
They trained a separate single-layer model for each of the top-7 contribution layers of Qwen3-1.7B and had them vote on OlympiadBench.

Setup	OlympiadBench
Best single-layer model	28.3%
Full-parameter RL	26.9%
Majority vote of top-7 layer models	33.6%
Self-consistency voting with one model (reference)	31.3%

The Jaccard similarity (overlap of two sets, measured as intersection over union) between the problem sets each layer model solves is only 34.1%.
Which layer you train changes which problems get solved.
The authors interpret this as diverse task-specific solutions sitting densely around the pretrained weights.

Relation to LoRA, and limitations

There is no direct comparison against LoRA or other parameter-efficient methods in the paper.
LISA (a method that samples which layers to train) and similar work only appear as related research.

Where LoRA adds low-rank deltas to every layer, this trains one layer at full rank, so the two cut the problem in completely different ways.
Whether they combine, or whether single-layer LoRA shows the same localization, is outside the paper’s scope.

The authors are explicit about the limitations.
The three approaches above were validated mostly on math reasoning, and checking them on code and agent tasks is left as future work.
Layer contribution is defined against a specific training setup, so changing the setup changes the values.
And as noted above, there is no theory yet for why the middle layers.

Also, at first glance this looks like a paper about training more efficiently, and it says no such thing.
Most of the cost of RL training sits on the rollout side, generating piles of responses, and single-layer training does not change that.
Backpropagation still has to flow down to the target layer, so what reliably shrinks is the optimizer state (about 2x the parameter count with AdamW), and the paper gives no reduction numbers either.
What it does say stops here: the RL gain is localized in the middle layers, and training only there matches or beats full-parameter training.