Why LLMs memorize fine-tuned facts at 0.998 but use them at 0.078 in reasoning
Contents
A paper on arXiv argues that when an LLM can’t use fine-tuned knowledge in reasoning, it’s because the knowledge never carries to the mid layers where the reasoning computation runs (arXiv:2607.08393).
It names this “knows it but can’t use it” state the Knowing–Using Gap and verifies it across six models: Qwen-2.5 1.5B/3B/7B and Llama-3.2/3.1 1B/3B/8B.
The authors are a group at HKUST and HKUST(GZ).
I’d been on a run of layer-level posts with single-layer SFT and J-lens, so I read this one next.
Memorization 0.998, reasoning with the same facts 0.078
The experiment takes 1,000 facts the model doesn’t know from STaRK, a semi-structured knowledge-base benchmark (biomedical Prime and academic MAG).
Each fact is an entity→relation→entity triple, expanded into sentences for fine-tuning. Both full-parameter and LoRA (rank 16, α=32) runs are tested.
Evaluation covers direct recall of the injected facts (memorization) and two reasoning tasks that use them.
Chaining links two memorized facts in sequence. The paper’s example:
- Fact 1: “Which protein is expressed in embryo? → IGFBP3”
- Fact 2: “Which drug targets IGFBP3? → Mecasermin”
- 2-hop question: “Which drug targets the protein expressed in embryo? → Mecasermin”
The other task, Intersection, narrows down an answer shared across facts, like “Which exposure is linked to both glioma and hypothyroidism?”
Both are solvable with nothing beyond the individually memorized facts.
Accuracy still split this far (three of the six models):
| Model | Memorization | Chaining | Intersection |
|---|---|---|---|
| Qwen-2.5-1.5B | 0.998 | 0.078 | 0.440 |
| Qwen-2.5-7B | 0.996 | 0.124 | 0.504 |
| Llama-3.1-8B | 0.986 | 0.182 | 0.458 |
The gap has a time dimension too.
With LoRA on Chaining, memorization saturated at 10.4 epochs on average while reasoning accuracy stabilized at 15.0 — a 4.6-epoch lag.
Intersection lagged less, 0.6 epochs.
Early stopping on memorization accuracy halts training before the knowledge becomes usable.
Self-patching transplants the model’s own hidden states
To find the cause, the paper builds an intervention it calls self-patching.
It’s a variant of activation patching (the standard mechanistic-interpretability technique of splicing internal states from another run into a forward pass and measuring the output change); the “self” is that both the source and the target are the same fine-tuned model.
(1) Run the memorization prompt (direct recall) and cache the hidden state at a source layer, at the token span of the head entity (the subject-side entity of the fact triple).
(2) Run the reasoning prompt and overwrite the same entity’s token span at a target layer with the cached state.
(3) Continue the forward pass and record how much the correct token’s probability changed.
(4) Repeat over all layer pairs and heat-map which transplants repair the failure.
flowchart TD
A[Run memorization prompt] --> B[Cache hidden state at<br/>entity position, source layer]
B --> C[Run reasoning prompt]
C --> D[Overwrite same entity position<br/>at target layer]
D --> E[Continue forward pass]
E --> F[Record change in correct-token probability]
F -->|sweep layer pairs| AIt runs in the opposite direction from causal tracing (which corrupts a working case with noise to find which spot breaks the output): self-patching sweeps a failing case for where a transplant repairs it.
That means it applies directly to the very examples that fail at reasoning.
The injected knowledge sat in early and late layers
The sweep found the effective transplants concentrated in two clusters:
late layers around 80% depth into mid layers around 50% depth, and early layers around 10% depth into the same mid layers.
The injected knowledge is present in early and late layers, then, but never carries to the mid layers where the reasoning computation runs.
The paper calls this the knowledge-circuit misalignment hypothesis.
The memorization loss bottoms out within a few epochs, and gradient updates stop.
The knowledge representations stop updating wherever they landed, unreadable from the mid-layer reasoning circuit — the paper’s word for it is stranded.
The oracle condition (sweep all layer pairs per test case and pick the best target) backs this reading; one patch recovered this much:
| Model | Chaining | +oracle patch | Intersection | +oracle patch |
|---|---|---|---|---|
| Qwen-2.5-1.5B | 0.078 | 0.793 | 0.440 | 0.987 |
| Qwen-2.5-7B | 0.124 | 0.774 | 0.504 | 0.956 |
| Llama-3.1-8B | 0.182 | 0.795 | 0.458 | 0.921 |
Not a single bit of the weights changed — relocating a representation to the layer that holds the reasoning circuit brought the answer back.
This is not rote memorization in the human sense. A rote learner doesn’t understand the material, so a reminder doesn’t help with word problems.
Here the knowledge already sits inside the model in an answer-ready form; it just never gets called where it’s needed.
It’s the student who recites the formula on demand but never thinks to use it in a word problem.
Incidentally, the RL layer-localization paper I read the other day had the training gains concentrated at 40–60% depth. The transplant targets centering around 50% land in the middle of that.
Two fixed layer pairs recover 58–75% of it
The oracle is a per-example brute-force sweep, so it’s not usable as-is.
The paper tries locking the two clusters into two fixed layer pairs (late 0.8L → mid 0.5L and early 0.1L → mid 0.5L, for L total layers).
That alone recovered 58–75% of what the oracle could fix. On Chaining, the average went from 0.121 unpatched to 0.357 with fixed pairs (oracle: 0.444).
Still, this is an inference-time intervention, not a training fix.
It presumes you know where the fact’s subject entity sits in every incoming prompt, which rules out serving arbitrary questions.
The paper’s own limitations section leaves the training side open: an early signal that predicts, during fine-tuning, which facts will end up unusable.
How self-patching relates to J-lens
As a tool, self-patching is the counterpart of J-lens.
J-lens projects each layer’s residual stream (the vector sequence every layer reads and writes on its way forward) into vocabulary space to check what is readable at that layer — it never touches the model.
Self-patching actually moves hidden states and confirms whether the output changes once they land elsewhere.
Restated in lens terms, the paper predicts that under direct recall the answer token surfaces in late layers, while under 2-hop reasoning the bridge entity never surfaces mid-layer.
That is the failure-side mirror of what the J-lens post confirmed: an unspoken supporting concept (boot-shaped country = Italy) riding the mid layers.
Pick the layers from a self-patching sweep and retrain only the mid-layer side, single-layer SFT style — does the misalignment go away? The paper stops short of that.