Anthropic's Jacobian lens on Qwen3-4B: what a do-nothing single-layer SFT wrote
Contents
Last time, full-rank SFT on a single layer of Qwen3-4B produced a strange style corrector.
eval_loss showed the same layer localization as the paper, but when I actually ran rewrites, the depth-25% and depth-50% runs had learned to do almost nothing, and depth-75% behaved like plain Qwen.
The numbers moved, the function never showed up. What happened inside those layers?
I had just written up Anthropic’s J-space paper, and the J-lens readout it introduces has a public implementation.
This time I fit the lens for Qwen3-4B-Instruct-2507 and read the internals of the layer-swapped models against the base model.
The lens fitting ran on a rented RTX 4090 on RunPod; all readouts ran on an M4 Mac mini (MPS).
You have to fit the lens yourself
J-lens estimates the average first-order effect of a mid-layer residual stream (the vector sequence each layer reads, modifies, and passes on) on the final-layer representation as a Jacobian matrix J_ℓ, transports the residual through that matrix, and decodes it with the model’s own unembedding.
The details are in my explainer, so I’ll skip them here.
Pre-fitted lenses exist (neuronpedia/jacobian-lens), and there is already one for Qwen3-4B.
But that one is for the original Qwen3-4B, not for Instruct-2507, which my corrector is based on.
J_ℓ is estimated from the model weights themselves, so a lens does not transfer across checkpoints.
Fitting costs one forward pass plus ceil(d_model / dim_batch) backward passes per prompt.
With d_model=2560 on Qwen3-4B and dim_batch=8, that is 320 backward passes per prompt.
Measured on the M4 mini: 636 seconds per prompt. The repo README says ~100 wikitext-style prompts are enough for a usable lens, which works out to about 18 hours. That would block my main machine for a full day, so I sent it to RunPod.
Fitting on a 4090 on RunPod
Pod operation was the same as the previous SFT run: upload one bootstrap script, let it run unattended from environment setup through fitting, and rsync artifacts back to local every two minutes.
Same library pins as last time (torch 2.6.0+cu124, transformers 5.13.0).
The first attempt with dim_batch=32 died with an OOM.
Fitting keeps the computation graph for all 35 source layers alive while running repeated backwards, so on top of the 8GB model you get roughly 14GB of graph, and 24GB is not enough.
Dropping to dim_batch=16 stabilized at 17.3GB. Total backward FLOPs are unchanged by dim_batch in this design.
The other snag was on the retrieval side: this template (RunPod PyTorch 2.4.0) ships without rsync. It is a one-line apt-get, but the pull script keeps failing silently with no error surfaced.
The fit itself ran at 30–33 seconds per prompt, 51 minutes for 100 prompts.
Since I might have to kill the pod early, I made the script save an intermediate lens from the resumable checkpoint every 25 prompts. Whenever you stop, you keep a usable lens up to that point.
Each lens is 438MB in fp16. Total pod cost, including the OOM retry and setup, was about 1 USD.
First, check that the known phenomena reproduce
To verify the fitted lens, I ran the paper’s flagship example.
Feed “Fact: The currency used in the country shaped like a boot is” and Italy — absent from both the prompt and the output — rises through the middle layers.
| Layer | Rank of Italy | Rank of Euro |
|---|---|---|
| L12 | 399 | 8657 |
| L24 | 28 | 792 |
| L30 | 29 | 14 |
| L33 | 7 | 4 |
The actual output continues with “the Euro”. Before the model says the answer, the unspoken concept behind it (boot-shaped country = Italy) is sitting in the middle layers.
One Qwen-specific caveat: the raw top-5 readouts are dominated by punctuation tokens like ”？” and ”…”.
The repo treats this as known behavior and masks to word-like tokens for visualization, so all my observations track ranks of specific tokens instead of raw top-5 lists.
Japanese works too. For 「事実: 富士山がある国の通貨は」 (“Fact: the currency of the country with Mt. Fuji is”), Japan reaches rank 23 at L18, and at L30 日本 is rank 4 and yen is rank 1.
Everything downstream in this experiment is Japanese slop sentences, so confirming that mid-layer concept readout works on Japanese input mattered before going further.
How to read a layer-swapped model
A single-layer SFT artifact is the state_dict of one trained layer, so as in the previous evaluation, I keep the base model loaded and swap just that layer in memory.
J_ℓ is determined by the weights from layer ℓ up to the final layer, so in a model where only layer k is swapped, J_ℓ is unchanged for every layer at or above k.
That means the base-fitted lens remains exactly valid for readouts above the trained layer in the swapped models. This does not hold for LoRA, which touches every layer (which is why the LoRA baseline from last time is excluded here).
flowchart TD
A[Base Qwen3-4B-Instruct-2507] --> B[Fit J-lens on 100 wikitext prompts<br/>51 min on a RunPod 4090]
A --> C[Swap in trained layer state_dict<br/>depth-25 / depth-50 / depth-75]
C --> D[Run the same rewrite prompts<br/>record residuals at every layer]
A --> E[Record the same residuals on base]
D --> F[Compare cos similarity and vocab-projected Δh<br/>lens is exact above the trained layer]
E --> FInputs are the same 11 sentences as last time (8 slop + 3 natural), through the same corrector prompt.
I measured per-layer cos similarity between base and swapped residuals, and projected the difference vector Δh (swapped residual minus base residual) through the lens to read which words the perturbation pushes toward.
I also verified behavior by generating: all 4 configurations matched the outputs reported in the previous post.
depth-75 barely changed inside either
The simplest result first. depth-75, which trained layer 26, was nearly unchanged inside.
The relative norm of the perturbation (norm of Δh divided by the norm of the base residual) is only 0.06–0.08, and cos similarity stays above 0.994 at every layer for all 11 sentences.
Projecting Δh into the vocabulary gives strings of punctuation. It points at nothing meaningful.
depth-75, which last time “behaved almost like plain Qwen”, is also almost plain Qwen inside.
Its eval_loss had dropped to the same level as depth-25, yet the representations barely changed.
depth-25’s pass-through sits on top of a large internal shift
This is the main event. depth-25, which trained layer 9, passes 7 of the 8 slop sentences through untouched — but its internals changed a lot.
Here is the cos similarity trace for the “試金石” (touchstone) sentence, the one input where all configurations diverged last time.
| Layer | depth-25 | depth-50 | depth-75 |
|---|---|---|---|
| L9 | 0.992 | 1.000 | 1.000 |
| L15 | 0.905 | 1.000 | 1.000 |
| L21 | 0.884 | 0.949 | 1.000 |
| L27 | 0.933 | 0.949 | 0.997 |
| L30 | 0.965 | 0.966 | 0.998 |
| L34 | 0.970 | 0.972 | 0.998 |
The depth-25 perturbation is small right after the trained layer 9 (cos 0.992), grows through the middle layers to a minimum at L21 (0.884), then recovers to 0.97 toward the final layer. Relative norm peaks at 0.47 mid-stack. The same shape appears on all 11 sentences.
Same output, but the intermediate representations are nearly 20% different.
Projecting Δh into the vocabulary above the trained layer shows where it points: the upper layers surface the input’s own content words.
| Input (Japanese slop/natural sentence) | Top Δh projections at L30/L33 |
|---|---|
| ここで壁に当たった。 (hit a wall here) | wall, 壁, Wall |
| つまり、データが物語っているのだ。 (in other words, the data tells the story) | therefore, thus, hence |
| この検証を通じて、多くの学びを〜 (through this verification, many learnings…) | verification, validation |
| 結局のところ、重要なのは〜 (in the end, what matters is…) | finally, conclusion, 結 |
| この経験は、〜試金石となるはずだ。 (this experience should be a touchstone…) | experience, experiences |
| モデルの挙動を注意深く〜 (carefully observing the model’s behavior…) | model, models |
| 条件を整えれば、誰でも〜 (if the conditions are right, anyone…) | requisite, prerequisites |
| 4Bの方がlossも〜 (the 4B has better loss…) | loss, bigger |
| 量子化はf16で8.05GB〜 (quantization: 8.05GB at f16…) | quant, 量化, float, FP |
| eval_lossは0.729で〜 (eval_loss was 0.729…) | validation, evaluation, loss |
10 of the 11 sentences show this pattern (the one exception, the 浮かび上がる sentence, produced no clear content word).
The wall sentence points at wall, the “in other words” sentence at therefore, the quantization sentence at quant — the perturbation’s direction changes with each input, and it comes out almost entirely on the English side.
Japanese input routing through shared English-leaning representations is the same picture the J-space paper showed with its Chinese experiments on the language-shared workspace.
If the perturbation points toward writing the input’s own content concepts back onto the workspace, then depth-25’s “do nothing” is not an absence of change.
It is implemented as a learned, active write in the direction of re-committing the input as-is.
The position distribution is skewed too.
At the token positions of the slop-marker words themselves (試金石, 物語, 観察), Δh has less than half the relative norm of the sentence-final position, and its projections decode to nothing interpretable.
The write concentrates at the final position — the one that decides the first token of the response.
depth-50 reaches the same pass-through from the negation side
depth-50, which trained layer 18, was the second “do nothing” run — its outputs matched depth-25 exactly on all 11 sentences last time.
Inside, it is a different animal.
At L21–L27, its Δh projections point at negation and hedging vocabulary: nothing, not, no, only, maybe, perhaps. The same cluster appeared on 10 of 11 sentences.
The input-content echo is weaker than depth-25’s and shifted higher, around L33.
Same pass-through, two implementations: depth-25 does it by re-committing the input’s concepts, depth-50 by a write that reads like “there is nothing to fix”.
The eval_loss localization from last time was flattening these two distinct internal changes — and depth-75’s non-change — into similar-looking numbers.
Pass-through removes half the loss on its own
The SFT loss here measures how exactly the model reproduces the teacher’s output sentence for each input.
A model that judges a natural sentence and returns it unchanged, and a model that misses the slop and returns the sentence unchanged, produce the same output and therefore the same loss. The difference between the two never reaches the training signal.
And half of the training data consists of “natural sentence in, same sentence out” pairs, added to suppress over-correction.
Which means half of the data can be answered correctly without ever judging whether anything is slop.
depth-25 learned to hold and re-write the input, depth-50 to judge that there is nothing to fix — both implementations of “return the input”.
Slop rewriting was not learned by the LoRA run, nor by any of the three single-layer runs. The reason is not model capacity; it is a data mix where returning the input gets half the answers right.
On the data side, the fix is more pairs that are close in form but opposite in answer — 「ここで壁に当たった。」 (“hit a wall here”, fix it) versus 「ここでビルドが失敗した。」 (“the build failed here”, leave it). Returning the input stops reducing the loss on those, and the decision has to depend on the slop word.
On the loss side, weight the edited tokens instead of scoring the whole sentence evenly. If a 30-token sentence needs a 2-token fix, verbatim copying already gets 28 tokens right, and missing the 2 tokens that matter barely moves the loss. Counting the edited span more heavily removes that imbalance.
There is also a way to assemble this without redoing the SFT.
Plain Qwen rewrote all 8 slop sentences — at the cost of also rewriting natural ones. The ability to rewrite was in the base model all along; the judgment of what to fix and what to leave is the only part that never worked. The training runs so far tried to teach that judgment to the same weights as the rewriter, and every time it shut down the rewriting instead.
Splitting them means giving the judgment to the detector and to the “nothing to fix” write observed in this run, sending only the sentences where the two disagree to a larger model or a human, and letting plain Qwen do the rewriting.
Whether that write actually plays the judgment role can be tested directly: ablate the direction on depth-50 and see whether the pass-through turns into rewriting, then inject it into plain Qwen and see whether the rewriting stops. If both happen, this corrector needs no SFT at all — plain Qwen plus one steering vector as the gate.