GPT-OSS keeps its answers after its hidden states are swapped for a TPR formula
Contents
This blog has spent a fair number of posts on what happens inside a model and whether poking at it changes the output, through papers and hands-on runs. The most recent one was reading concepts out with J-lens and rewriting them. A rather different story came out, so I looked into it.
The paper is The Emergent Symbolic Structure of Artificial Neural Networks, by R. Thomas McCoy (Yale), Paul Soulos (Johns Hopkins), Tal Linzen (NYU) and Paul Smolensky (Microsoft Research), dated August 30, 2026.
It starts by rebuilding the hidden states of small models from a hand-written formula, and goes all the way to swapping every layer of GPT-OSS’s hidden states for that formula and letting the model itself answer.
Addition and word order
The paper first takes on the Linear Representation Hypothesis, the view that a hidden state is a sum of concept vectors. Linear probes (a linear classifier applied to a hidden state to read out a concept), sparse autoencoders (decomposing a hidden state into a sum of a few features) and word2vec’s king − man + woman ≈ queen all rest on this view.
Addition has no order, though. If you write “cats chase dogs” as cats + chase + dogs, then “dogs chase cats” is the same three words added up and gives the same vector.
and come out identical too if all you do is add 3, − and 5. “If P then Q” versus “if Q then P”, and concatenate(x, y) versus concatenate(y, x), are the same story: the answers differ but the vectors cannot tell them apart. Cognitive science calls this the binding problem, and how to tie elements to positions has been argued about for a long time.
The paper uses Smolensky’s Tensor Product Representations (TPRs) to treat a structure as a collection of pairs, each pairing a filler (the content) with a role (the position). Smolensky introduced them in 1987, with a longer journal version in 1990.
”cats chase dogs” becomes the three pairs {subject: cats, verb: chase, object: dogs}.
Each filler and each role gets a vector, each pair is combined with the tensor product (an operation that builds a matrix from two vectors), and all the pairs are summed.
The final step is still a sum, same as the Linear Representation Hypothesis, but the filler and the role are multiplied first, so the matrix for cats on the subject side and the matrix for cats on the object side are different objects.
The paper then applies one affine transformation (a linear map plus a translation) to flatten this into a vector, so that a coordinate system that has been stretched, rotated or shifted still counts as the same TPR. The numbers become unreadable, but the straight-line relations between points survive and a neural network can undo the transformation, so the TPR content can be taken as unchanged.
DISCOVER replaces the encoder with a formula
The analysis method is called DISCOVER (DISsecting COmpositionality in VEctor Representations), a way of fitting a TPR to actual vectors. McCoy and colleagues introduced it in 2019, and this paper extends it to LLMs. The mascot is a tapir, because interleaving the letters of TPR and AI spells tapir.
You train the model you want to study and keep the vector its encoder produces. Then you train a DISCOVER model, explicitly shaped as a TPR, so that its output gets as close to as possible under mean squared error. The only trainable parts are the filler vectors, the role vectors, and the matrix and bias of the affine transformation.
Finally you plug into the original model’s decoder and check whether the same correct output comes out.
flowchart TD
A[Target model encoder] --> B[Vector e]
C[DISCOVER model<br/>filler × role TPR] --> D[Approximation ê]
B -. minimize MSE .-> D
D --> E[Target model decoder]
E --> F[Same output as before?]The metric is approximation accuracy, the fraction of test items where the decoder, fed the TPR approximation, produces the entire output sequence correctly. If replacing the encoder with the formula leaves the answers unchanged, the formula is judged to do the same job as the encoder.
Which roles to assume is a human choice. For the small letter-sequence models, five role schemes were tried.
| Role scheme | Role of each letter | Roles |
|---|---|---|
| Left-to-right | Position counted from the left | 6 |
| Right-to-left | Position counted from the right | 6 |
| Bidirectional | Pair of left and right positions (21 combinations for lengths 1 to 6) | 21 |
| Wickelroles | Which letters come before and after (26 letters plus boundary, 27 × 27) | 729 |
| Bag-of-words | Same role for every letter | 1 |
Bag-of-words gives every letter the same role, which throws order away, and is there as the no-structure baseline.
Starting with a small model that reverses strings
The first example is a GRU (a gated recurrent network) encoder and decoder that reverses Q M Z into Z M Q. Ten copies were trained, differing only in random initialization, and all ten hit 100% on the test set.
With bidirectional roles, the lowest DISCOVER approximation accuracy across the ten runs was 99.98%. Right-to-left roles came close to bidirectional, left-to-right roles did poorly. The paper’s explanation is that positions counted from the right matter more for reversal. The order-free scheme scored low.
The paper states DISCOVER’s limits up front here. A TPR is a bilinear formula (linear in the filler and linear in the role), so it can express only a narrow class of functions, and there is no guarantee the approximation works.
Even when it works, it shows the DISCOVER model captured everything in the target, not that the target holds everything in the DISCOVER model. Nor is it a claim that the network actually computes tensor products; it only describes the shape of the representation that emerged.
Varying architecture and task, 600 runs
Three tasks, copying, reversing and interleaving (alternating letters from the front and back: Q M Z V R → Q R M V Z), were trained on four architectures: an MLP (multilayer perceptron), a GRU, a Transformer and a bottleneck Transformer. Inputs were sequences of 1 to 6 uppercase letters, 57,000 pairs split into 50,000 training, 2,000 validation and 5,000 test.
The bottleneck Transformer is new in this paper. Its decoder can only read the vector at the first input position. Since the encoder is bidirectional and builds each position’s vector from the whole sequence, the first position is forced to carry the entire sequence. In an ordinary Transformer every position has its own vector, and on tasks this simple each vector might only need to hold its own letter. The bottleneck exists to create a situation where several pieces of information must be packed into one vector.
Four architectures times three tasks times ten reruns times five schemes gives 600 DISCOVER models. Bidirectional roles gave a strong approximation in all 12 architecture-task combinations; the lowest, the reversing bottleneck Transformer, averaged 0.973, and everything else was above 0.99.
Wickelroles, with 729 roles against bidirectional’s 21, were worse than bidirectional in every condition. More roles did not translate into better approximation.
Appendix C adds alphabetical sorting, a task where input order does not matter. There the order-free scheme approximates reasonably well, and even the best scheme for the GRU and the Transformer fell below the 95% seen in the other experiments.
| Architecture | Best-scheme approximation accuracy |
|---|---|
| MLP | 0.979 |
| GRU | 0.935 |
| Transformer | 0.871 |
| Bottleneck Transformer | 0.979 |
On a task that does not need order, the representations did not hold much order either.
Looking at the period position in LLMs
An LLM produces a vector for every token at every layer, so a single sentence yields a pile of vectors. The paper picks the vector for the token at the end of the sentence, the period.
The LLMs here only attend to earlier tokens, so the period is the first position where information about the whole sentence is available. Earlier work had also found that punctuation tokens carry information about the preceding sentence. Hence the hypothesis: the hidden state at the period reflects the whole preceding sentence.
The models are Gemma-3-27b, GPT-2-XL, GPT-OSS-20b, Pythia-12b, Qwen3-14b, OLMo-2-13B and Llama-3.1-8b, in each case the largest in its series that fits on a single A100. Five layers were examined: the first non-embedding layer, 25%, 50%, 75%, and the last layer. Hidden sizes run from 1600 to 5376.
A hidden state is a list of numbers and cannot be read directly. So to test the hypothesis itself, the authors first trained a separate period-unpacking model (a 6-layer Transformer, hidden size 1024, 16 attention heads) that reconstructs the preceding sentence from the period’s hidden state.
Subject-verb-object sentences were recovered perfectly at every layer. Word lists came out above roughly 85% and complex sentences above roughly 50%. Scoring counts an output as wrong if a single word differs.
DISCOVER approximations were then fed into this period-unpacking model.
| Condition | Input | Scheme that worked | Scheme that did not |
|---|---|---|---|
| Lists | 3 to 5 nouns from a 300-noun vocabulary | Bidirectional (12 roles) | Predecessor word as role (300 words plus a start marker, 301 roles) |
| Subject-verb-object | 80 occupation nouns, 16 verbs | subject / verb / object, 3 roles (accuracy 1.0) | ― |
| Complex sentences | Coordination, subordinate clauses, passives | Bidirectional | Syntactic roles, the path of dependency labels from each word to the root |
The authors write that they were surprised bidirectional roles, plain linear order, beat syntactic roles on complex sentences. In Appendix G they retrained the period-unpacking model to output a parse tree instead, and bidirectional still won, so they conclude the period encoding favors linear order over syntax. Other tokens’ encodings may still hold syntax.
There is another odd result. On complex sentences, the period-unpacking model did better when fed DISCOVER’s approximation than when fed the LLM’s real hidden state. For GPT-OSS’s middle layer, real hidden states gave 0.71 and the bidirectional approximation gave 0.96.
The unpacking model was trained on real hidden states, yet the approximation built from them reconstructs better. Since the roles used for the approximation are linear-order roles, the order structure is present on the LLM side; the paper’s reading is that it is simply realized imperfectly.
Making GPT-OSS do arithmetic, logic, code and syntax
In the period experiments, the answers came from a separately trained period-unpacking model, not the LLM itself. So the paper narrows to one LLM, replaces the vectors for every input token at every layer with approximations, and checks whether the LLM itself still answers correctly.
That LLM is GPT-OSS. Analyzing this one model took a little over 3,000 GPU hours (summed per-GPU time, with no normalization to H100-equivalents) on a cluster of H100 and H200 GPUs, which is why it stayed at one model.
| Task | Example | Answer |
|---|---|---|
| Arithmetic | 7 + -6 * 3 | -11 |
| Syllogisms | Pick which of 4 conclusions follows from 2 premises | No gamblers are polite agents. |
| Code execution | Define 2 list functions and 2 variables, call one | [“Q”, “S”, “M”, “V”] |
| Passivization | The swimmer who … avoided the accountant … | The accountant … was avoided by the swimmer … |
| Tense reinflection | Past-tense sentence to present tense | Verb agrees with its subject’s number |
| Question formation | Declarative to yes/no question | Auxiliary moves to the front |
Arithmetic problems are a * b + c or a + b * c with a, b, c integers from -9 to 9. GPT-OSS’s own accuracy is 0.76 on syllogisms and 0.94 on passivization; the rest were above 0.96. Prompts use GPT-OSS’s standard chat format (harmony), and every word in the stimuli is a single token; words that split into several tokens were avoided.
Five role schemes, split by whether each token holds only itself or all preceding tokens, and whether task-specific structure is used.
| Role scheme | Pairs held by each token | Role content |
|---|---|---|
| Order-free | Itself and all preceding tokens | All the same |
| Bidirectional (self) | Itself only | Bidirectional position in the sequence so far |
| Bidirectional (all) | Itself and all preceding tokens | Bidirectional position in the sequence so far |
| Task-specific (self) | Itself only | Position assigned by a task-specific parser |
| Task-specific (all) | Itself and all preceding tokens | Concatenation of the other token’s position and its own |
A task-specific position is something like “second letter of the list that is the value of variable x” for code, or “adjective modifying the direct object” for language. Task-specific (all) goes further and uses roles that join two positions, such as subject_noun-object_adj. In a sentence like The tall poet helped the spy, the adjective tall should affect poet, which it describes, differently from spy, which it has nothing to do with, and a joined role can keep that difference. The authors suspect these joined roles behave somewhat like attention’s queries and keys.
With one DISCOVER model per layer across all 25 layers, task-specific (all) was best on all six tasks and came out almost identical to GPT-OSS’s own accuracy. The largest gap, on arithmetic, was 2.36 points.
The paper reads this as each token holding several pairs, with those pairs corresponding to task-specific structure.
Editing only part of a representation
Because a TPR is literally the sum of its parts, subtracting one pair’s representation and adding another’s swaps out one piece of the structure. In practice, you take the original model’s hidden state, subtract the vector DISCOVER computes for the old pair, add the vector for the new pair, and hand it back to the original model. The edit shifts every component of the vector by some amount, so it makes no assumption that information sits in particular components.
Interventions on the string models were close to 1.0 in every condition. Interventions on period encodings came close to the period-unpacking model’s own accuracy in every condition, including the ones where that accuracy was low to begin with.
On GPT-OSS, 31 kinds of intervention across the six tasks averaged 0.903.
A filler intervention swaps content. With the input -2 + 3 * -4, subtract the vector for the pair “3 in the number slot” from the hidden state and add the vector for “8 in the same slot”. The input text is untouched, yet GPT-OSS’s output moves from -14, which is -2 + 3 × -4, to -34, which is -2 + 8 × -4.
| Filler intervention | Accuracy |
|---|---|
| Rewrite a number in an arithmetic expression | 0.978 |
| Change a list element in code execution | 0.99 |
| Change the repetition count in code execution | 1.00 |
| Change which function is called | 0.95 |
| Change which variable is passed as the argument | 0.96 |
A role intervention rewrites position. Moving polite in The economists stopped the polite tourists from object adjective to subject adjective made GPT-OSS behave as if the input had been The polite economists stopped the tourists.
| Role intervention | Accuracy |
|---|---|
| Swap the two numbers in an arithmetic expression | 0.978 |
| Move a letter from one list to the other in Python code | 0.950 |
| Move an adjective from object to subject in tense reinflection | 0.980 |
| Same, for a prepositional phrase | 0.892 |
| Same, for a relative clause | 0.958 |
Moving a prepositional phrase or a relative clause requires rewriting more pairs than a single adjective. Moving a relative clause also changes the subject’s number, so the output verb has to switch from singular to plural, and that went through too.
GPT-OSS handles position with RoPE (rotary position encoding, which expresses relative position through rotation) and does not add position vectors into the residual stream, so, as a footnote points out, the position information being edited here must have emerged from training.
Local interventions, editing only the target token, were tried as well. For filler changes, local edits did about as well as full edits; for role changes, local edits dropped sharply. The paper’s explanation is that a token’s content sits mostly at its own position while structure is spread across tokens.
Appendix J builds pairs of interventions that are the same move in linear order but different moves in syntax, and swaps the edit vectors between them. Only the syntactically matching combination scored high; the one that matched linear order alone scored low.
Combinations never seen
The results so far cannot distinguish whether DISCOVER represents roles and fillers compositionally or has simply memorized “cats-as-subject” as one atomic concept. The paper separates the two by withholding specific pairs when training DISCOVER.
For instance, examples with C in the third position are left out of training, and at evaluation time the count is how many examples containing that pair are handled correctly. Nothing is withheld from the target model’s training, only from DISCOVER’s, since the point is to measure the representation rather than the target model’s ability to generalize.
The comparison is what the paper calls the strong chance baseline: with withheld pairs in an example, randomly placing the leftover fillers into the leftover slots succeeds with probability .
Take a copying model evaluated on A M C D Q W where the pairs A in first, C in third and D in fourth were withheld. The trained pairs M, Q and W go to the right places, leaving A, C and D and three empty slots, so random placement gets 1 of arrangements right.
For the string models, every case except the interleaving Transformer beat the baseline by a wide margin. Period encodings beat it in all seven LLMs, and GPT-OSS beat it by a wide margin on five of the six tasks, arithmetic excepted, even with many withheld pairs.
Appendix M repeats the check on models whose internals were designed by hand. A model built to bind roles and fillers systematically generalized; a model built to memorize each pair separately did not.
Where the paper places itself
In the discussion, the authors place their result between two old positions in cognitive science: eliminativism, which holds that because neural networks succeed, symbols can be dropped from the theory, and implementationalism, which holds that anything behaving symbolically must implement a symbol system inside.
The result goes against eliminativism, since a system with no explicit symbolic structure turned out to have implicit symbolic structure. But the authors do not adopt implementationalism either. They side with limitivism, Smolensky’s 1988 position that networks approach symbol systems in the limit but realize them only approximately in practice.
The two pieces of evidence are the 0.903 average intervention accuracy on GPT-OSS, which is not 1.0, and the period-unpacking reversal above. In the authors’ interpretation, the period encodings hold a symbolic structure that is only approximately realized, DISCOVER produces a noise-free version of it, and a model that relies on that structure therefore fits the approximation better.
What the deviation from the approximation is, the paper leaves open. It might be mere noise, or it might be doing the kind of fuzzy statistical work a symbol system cannot, and both options are kept. Which limit limitivism refers to, more training, larger models, higher numerical precision, discretization, or noise going to zero, is also left undecided.
There is one more tension the paper raises itself. Representations are built from combinable parts, yet a long line of prior work finds neural networks generalize poorly to new combinations. In Kim and Linzen (2020), a model that had only seen hedgehog as a subject got it wrong as an object.
The paper’s suggestion is that a model may build compositional representations only for the combinations it has seen. It compares this to word2vec’s king − man + woman ≈ queen, which would not hold if queen were missing from the training data.
Scope
The paper covers only tasks that a simple symbolic program could solve perfectly. How representations look in domains that are only partly systematic, natural language being the obvious one, is left unanswered, and the conclusion says so.
The method it names on the Linear Representation Hypothesis side is the sparse autoencoder, the kind of feature-ID extraction covered in the Qwen-Scope SAE post.
It also cites a separate paper (Enyan and McCoy, 2026) arguing that the successes of existing interpretability methods can be explained by TPR structure, and argues that future methods should take the filler-role product as a unit of analysis.
Part of the code is at https://github.com/tommccoy1/discover/; the full version is to be released once the authors’ employers approve.
This blog has been peeking inside models for a while, poking at roughly where a given thing is represented. This paper could be read as a continuation of that, or as something else entirely, along the lines of “so is it just a huge probability calculator?”, and that ambiguity is what makes it interesting to me.