Single-layer full-rank SFT on Qwen3-4B: localization is real, LoRA still wins
Contents
Update (2026-07-08): Follow-up — I fitted Anthropic’s Jacobian lens for Qwen3-4B and read the internals of the “do nothing” layers from this post. The pass-through turned out to be an active, learned write, not an absence of change → Anthropic’s Jacobian lens on Qwen3-4B: what a do-nothing single-layer SFT wrote
Last time I LoRA-trained Qwen3-4B on a RunPod 4090 to build a style corrector for this blog.
The result: it learned not to break clean sentences, but that restraint was strong enough that it passed most of the actual slop through untouched.
Meanwhile, a paper I covered recently (arXiv 2607.01232) reports that RL training gains are localized in the middle layers at 40-60% depth, and that training the single best layer can even beat full-parameter training.
The catch: the paper only tested RL. It doesn’t examine layer localization under SFT.
This is a follow-up on my own task.
Same style-corrector SFT as last time, but instead of LoRA I unfreeze exactly one layer and train it full-rank, at three depths, against a LoRA baseline retrained on identical data.
Same model as before (Qwen3-4B-Instruct-2507), trained on a RunPod 4090.
The design: three depths at 25/50/75%
The paper’s layer-contribution scan means one training run per layer — 36 runs for the 36-layer Qwen3-4B.
All I want to know is whether the middle layer matters in SFT too, so I cut it down to three depths: 25%, 50%, 75%.
| Run | Target layer | Trainable params |
|---|---|---|
| depth-25 | layer 9 of 36 | 100.9M (2.51% of total) |
| depth-50 | layer 18 of 36 | 100.9M |
| depth-75 | layer 26 of 36 | 100.9M |
| LoRA baseline | r16, all-linear, all layers | same config as last time |
If only the 50% point wins, the paper’s mid-layer localization carries over to SFT.
If all three land in the same place, layer position doesn’t matter here.
If all three lose to the baseline, one full-rank layer isn’t enough for this task’s SFT.
The freeze is implemented exactly the way the paper describes: set requires_grad=False on every parameter outside the target layer.
The learning rate drops from LoRA’s 1e-4 to 1e-5 for full-rank; everything else (3 epochs, batch 4 × gradient accumulation 4, max_length 1024, bf16) matches the previous run.
The LoRA baseline is retrained on the same data instead of reusing the previous artifact.
The training data grew and the validation set changed with it, so the old eval_loss can’t be compared directly.
More training data
The teacher data comes from the same edit-history mining as before, now with the fixes from recent article reviews added, growing the style-edit pairs from 799 to 874.
The new pairs include the review feedback on the previous corrector article and on the paper writeup itself.
Feedback on the corrector experiment becomes teacher data for the next corrector.
The chain resolution — collapsing two-stage edits where Claude’s fix got re-flagged into a single “original → final” pair — is unchanged from last time; this round it connected 24 chains and dropped 25 intermediate forms.
The final dataset is train 1,574 / val 174, with rewrite and identity examples at roughly 1:1.
Only the trained layer gets collected
Last time the LoRA adapter was 132MB. Full-rank means a full-model checkpoint of 8GB per save.
Since only one layer changes, the script saves just that layer’s state_dict with torch.save — about 200MB at bf16 for 100.9M parameters.
A local merge script loads the base model and injects the layer with load_state_dict(strict=False) before quantization or evaluation.
Per-epoch checkpointing is off.
A run takes about 15 minutes, so if the pod dies, rerunning is faster than restoring.
The lesson from last time — write success markers only on success — stays in: a per-run DONE plus a global ALL_DONE.
Before uploading anything I ran a 4-step smoke test on my M4 mini with Qwen3-0.6B to confirm the freeze works, the layer state_dict saves, and the merge round-trips.
Environment setup through all four runs is a single shell script: scp it, launch with nohup, and it finishes unattended.
flowchart TD
A[Local: scp data + scripts] --> B[Env setup<br/>torch 2.6.0+cu124 pinned]
B --> C[Model download<br/>directly on the pod]
C --> D[Train depth-25]
D --> E[Train depth-50]
E --> F[Train depth-75]
F --> G[Train LoRA baseline]
G --> H{All runs OK?}
H -->|yes| I[ALL_DONE]
H -->|no| J[HAD_FAILURES<br/>check runs without DONE]From the local side, rsync pulls the output directory every two minutes.
Whenever the pod dies, the local copy is at most two minutes behind. Also unchanged from last time.
Two errors before the pod was even up
The Community Cloud 4090 ($0.35/h) showed available stock but refused to deploy with “This machine does not have the resources to deploy your pod.”
That’s three times in a row now, so I no longer trust the Community stock indicator.
On Secure Cloud ($0.70/h) the container wouldn’t start:
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.8,
please update your driver to a newer versionThe RunPod PyTorch 2.8.0 template is a CUDA 12.8 build, and the assigned machine’s driver (550.127.05) only goes up to CUDA 12.4.
Downgrading the template to PyTorch 2.4.0 got the container running.
That pins torch to 2.6.0, the newest cu124 build.
One thing to watch: a later pip install of transformers and friends can resolve a newer torch from PyPI’s default cu128 wheels and overwrite the cu124 install, so the second pip command pins torch==2.6.0 as well.
The combination (torch 2.6.0, transformers 5.13.0, trl 1.7.1) was checked with pip —dry-run before training.
The fixes for last time’s failures (uninstalling torchvision/torchaudio, the correct repo ID for the 1.7B) are baked into the bootstrap script too.
The upload is about 1.6MB of data plus two scripts; the model downloads on the pod side — 13 files in 6 seconds.
eval_loss reproduces the paper’s localization
All four runs finished at 12-15 minutes each; pod start to artifact recovery took a bit over an hour and stayed under $1.
| Run | Target layer | eval_loss |
|---|---|---|
| depth-25 | layer 9 of 36 | 1.775 |
| depth-50 | layer 18 of 36 | 1.132 |
| depth-75 | layer 26 of 36 | 1.723 |
| LoRA baseline | all layers, r16 | 0.760 |
Only the 50% depth drops clearly; the two ends land on nearly the same number.
The shape the paper reports for RL — contribution concentrated in the 40-60% depth band — shows up in SFT eval_loss as well.
At the same time, even the best single layer (1.132) doesn’t reach the all-layer LoRA (0.760).
The LoRA comparison the paper never ran gets an answer here, at least for this task: one full-rank layer loses to all-layer LoRA.
The actual rewrites disagree with the numbers
Same check as last time.
Eight slop sentences that had been flagged in real reviews, plus three natural sentences that must be left alone, through five configurations under identical conditions: the base model, the three single-layer runs, and the LoRA baseline.
The base model loads once on the M4 mini, and each run’s layer state_dict is injected and restored in memory.
Counted by how much each configuration touched, the five split cleanly.
| Configuration | Slop rewritten (of 8) | Natural sentences broken (of 3) |
|---|---|---|
| Base Qwen | 8 | 3 |
| depth-25 | 1 | 0 |
| depth-50 | 1 | 0 |
| depth-75 | 8 | 2 |
| LoRA baseline | 2 | 2 |
depth-25 and depth-50 produced identical output on all 11 inputs.
Both leave the three natural sentences untouched and pass 7 of the 8 slop sentences straight through.
The “learned restraint, not removal” behavior of the previous LoRA corrector comes out even stronger in a single layer.
depth-75 went the other way and behaves almost like the base model.
It touches all eight slop sentences, but it also translates “loss” into the Japanese word 損失 inside a natural sentence and rewrites factual phrasing it should have left alone.
Its eval_loss is nearly identical to depth-25’s, yet one of them learned “do nothing” and the other learned almost nothing.
The LoRA baseline reproduces the previous overcorrection exactly: it rewrites “eval_loss” as 評価損失 and “loss” as 損失 in the natural sentences.
This blog keeps code-level terms in ASCII, so that habit breaks the style — and 75 extra teacher pairs didn’t fix it.
One input separated every configuration: a sentence ending in 試金石となるはずだ (“this experience should become a touchstone for future development”).
| Configuration | Output |
|---|---|
| depth-25 / depth-50 | dropped the touchstone metaphor, kept a hedged ending |
| depth-75 | dropped the metaphor and the hedge |
| LoRA baseline | kept the metaphor, trimmed only the hedge |
The three single-layer models remove the metaphor itself; the LoRA baseline leaves the metaphor and shaves the sentence ending.
On the one sentence they did touch, the single-layer models cut deeper.
The rewrite quality didn’t move with any choice of training method or layer.
The previous conclusion — what matters is the teacher data built from raw diffs and the sentence-level input granularity — held through this SFT variant unchanged.