LEWM predicts the on-screen person's emotion, not the model's own

Someone on X wrote that “recent research shows LLMs perform better when given emotions and self-awareness,” and pointed at the Large Emotional World Model on arXiv.
It is a paper from Changhao Song and colleagues at Tianjin University, with v1 out on December 30, 2025 and v2 on July 29, 2026.

Reading it, the model predicts the emotion of the person shown in a video as one of seven discrete labels, then uses that label as a condition for the next world state, so it is not about giving the model emotions of its own.
Nothing corresponding to “self” or “self-awareness” appears anywhere in the body text either, so I went through where the numbers in the abstract come from as well.

What LEWM is set up to do

The paper lists two lines of world models: ones like Sora, Genie, and Cosmos that generate video frames, scenes, or action-conditioned rollouts directly, and ones like Dreamer and JEPA that predict the next step in an internal representation. It says both lines rarely carry emotion as one of the state variables.

Given the observed state $\mathcal{S}_t$ at time $t$ , the emotional state $\mathcal{E}_t$ , and the behavior trajectory $\mathcal{H}_t$ in between (a text description of what was done in that interval), the model predicts the state and emotion at $t+1$ together.

p(\mathcal{S}_{t+1}, \mathcal{E}_{t+1} \mid \mathcal{S}_t, \mathcal{E}_t, \mathcal{H}_t)

LEWM splits this into two stages.

p(\mathcal{E}_{t+1} \mid \mathcal{S}_t, \mathcal{E}_t, \mathcal{H}_t) \times p(\mathcal{S}_{t+1} \mid \mathcal{S}_t, \mathcal{E}_t, \mathcal{H}_t, \mathcal{E}_{t+1})

The first factor is the prediction of the future emotion, the second is the prediction of the future state conditioned on that emotion.
$\mathcal{S}_t$ is a synchronized set of video, audio, and image, and $\mathcal{E}_t$ is the emotion label of the person on screen.

How the EWH dataset was built

EWH (Emotion-Why-How) is the dataset this paper introduces: 2,258 first-person videos with emotional or behavioral turning points collected from TikTok and Douyin, plus 1,583 episodes and over 630 hours from eight TV series.
The series are Desperate Housewives, Frasier, Friends, How I Met Your Mother, Modern Family, The Big Bang Theory, The Office, and This Is Us, chosen because they have plenty of interpersonal interaction and emotional behavior. The scenes cover home, workplace, school, meals, travel, medical settings, and public places.

Shot boundaries are detected and the footage is cut into clips of 5 to 15 seconds, then Qwen3.5-VL identifies the protagonist and writes a timeline of their behavior.
GPT-5 reads that timeline and assigns each phase one of seven labels: neutral, happiness, sadness, anger, fear, surprise, and disgust.
The paper does not say what neutral means, but it reads as the label for when none of the other six apply.
What goes into the dataset are only the cases where a non-neutral emotion changes into a different one, and after that the person’s movement, task focus, way of interacting, or who they interact with also changes and stays changed for a while. Momentary expressions, minor postural changes, routine behavior, and actions driven by the task or imposed from outside are excluded.
The same clip goes through this process five times, and only those where the before-and-after emotion pair comes out the same in at least three runs are kept.
Out of 240,221 candidates, 10,850 remained. Three annotators checked 10% of them, and Fleiss’ $\kappa$ (agreement between raters) was 0.82, 0.76, and 0.65 for the validity of the emotion change, the plausibility of the emotion label, and the correctness of the clip boundaries, in that order.

That makes “emotion” here one of these seven labels, assigned to the single protagonist in a clip.

How the emotion label is fed in, and the loss

On the input side, features for each modality (video, audio, and image) go through one linear layer into the LLM’s embedding space.
The emotional state is looked up in a learnable emotion embedding table to get a vector in the same space, wrapped in the boundary tokens <eBOS> and <eEOS>, and placed in the context.

For the future emotion, the hidden representation $z_t$ that the LLM produces from the context is multiplied by an emotion head $W_e$ , $\mathrm{Softmax}(W_e z_t)$ gives a distribution over the seven classes, and the most probable one is picked.
The chosen label is turned back into a vector with the same table and appended to the end of the context.
From there the model generates the future-state tokens autoregressively, and per-modality decoders reconstruct video, audio, and image.

flowchart TD
    A[Current video, audio, and image] --> B[Per-modality encoder and linear layer]
    C[Current emotion label] --> D[Emotion embedding table]
    F[Behavior trajectory as text] --> E[LLM backbone]
    B --> E
    D --> E
    E --> G[Emotion head outputs a 7-class distribution]
    G --> H[Pick the most probable label]
    H --> I[Embed it with the same table and append to the context]
    I --> J[Reconstruct future video, audio, and image from the generated tokens]

Training adds the language modeling loss, the distance in embedding space between the predicted and ground-truth future state, and the cross-entropy for the future emotion (the gap between the predicted distribution and the answer).

\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \alpha \mathcal{L}_{\mathrm{state}} + \lambda \mathcal{L}_{\mathrm{emotion}}

The paper itself says that the hidden representation $z_t$ is kept in the context as is, so that the state is not decided by the predicted emotion alone.
The explanation is that this is there for when the emotion prediction is imperfect.

Words for self and self-awareness in the paper

Counting words in the full v2 HTML, awareness, consciousness, self-aware, subjective, and theory of mind all come up zero times.
The only word with “self” in it is Self-supervised, once, in the references.

What the paper says it introduces is human emotion, and the abstract puts it as “we introduce human emotion as a key state variable in world models.”
That is the human emotion the model is trying to predict, not an internal state of the model.

For research that does deal with the model’s internal state, there is the emotion vectors paper Anthropic put out in April 2026, which found 171 representation patterns corresponding to emotions inside Claude Sonnet 4.5.
That one measures the model’s internal representations, while LEWM predicts the emotion of the human on screen.

As far as I can tell from reading it, this is in no way a story about AI acquiring emotions. It reads more like research on getting the model to read the room. The model follows the flow of a person’s emotions and anticipates what comes next.

Where the numbers in the abstract come from

The 45.72%, 3.94%, 17.47%, and 6.10% in the abstract each come from a different benchmark and a different comparison.

Number	Benchmark	What is being compared
45.72%	EWH	Gap vs WorldGPT on predicting video from audio and image
3.94%	WorldNet (WorldGPT’s physical-world dataset)	Gap vs WorldGPT on predicting audio from all three modalities
17.47%	MELD (an emotion recognition dataset built from Friends dialogue)	Weighted-F1 gap in the 3-class sentiment setting (positive, neutral, negative)
6.10%	MMLU	Gap in the Political Science category alone

The metric in Table 3 of the paper (the comparison table for WorldNet and EWH) is cosine similarity, which does not match the phrase “accuracy improvement” used in the abstract.
The 45.72 figure, taken from the corresponding row of Table 3, is 0.6727 − 0.2155 = 0.4572, so it is the difference in similarity multiplied by 100, not a percentage.
The body also has a separate 45.73% figure, which is 0.6723 − 0.2150 from the row with all three modalities as input.

The 6.10% on MMLU appears in the section checking that general reasoning did not degrade after adding emotion.
The paper’s wording is “performance fluctuates across individual categories, LEWM remains overall comparable to WorldGPT,” so it states overall parity first, then cites Political Science at 6.10% as the category that gained the most.
On VQAv2 the largest gain was 5.95% in the Animal category, HellaSwag topped out at 1.24%, and NExT-QA at 2.36%.

The 17.47% on MELD also changes with the granularity.
In the 3-class sentiment setting, accuracy rises by 15.78 points and Weighted-F1 (the per-class F1 averaged with weights by sample count) by 17.47 points, but in the fine-grained emotion setting it drops to 3.37 points on MELD and 2.92 points on CMMA, a Chinese multimodal emotion dataset.

Separating the effect of the dataset from the effect of the emotion mechanism

The paper has a comparison model called LEWM*, which is WorldGPT fine-tuned on EWH with the architecture left unchanged, put there to separate the effect of the dataset from the effect of the emotion mechanism.

Comparing cosine similarity on EWH for the predictions where input and output are the same modality, image to image, video to video, the breakdown comes out like this.

Modality	WorldGPT	LEWM* (trained on EWH)	LEWM (with emotion mechanism)
Image	0.4157	0.5950	0.5967
Video	0.2232	0.5572	0.6732
Audio	0.2544	0.2974	0.4949

For image, training on EWH alone took it from 0.4157 to 0.5950. Adding the emotion mechanism on top adds 0.0017, so almost all of the gain is on the dataset side.
For video, the dataset added 33.40 points and the mechanism 11.60.
Audio is the only one where the mechanism’s share is larger, 4.30 points from the dataset and 19.75 from the mechanism.

The lineup when emotion is removed from the conditioning

The ablation, which measures effects by removing components, lines up conditions that change how emotion is fed in.
There is No Emotion, which drops all emotion input; Current Only, which uses only the current emotion; Hard Future, which conditions on the single predicted future label; Soft Future, which conditions on the full distribution; and Oracle Future, which is given the ground-truth future emotion directly.

Condition	Image	Video	Audio
WorldGPT	0.4226	0.2175	0.2493
No Emotion	0.5950	0.5572	0.2974
Current Only	0.5378	0.6649	0.2912
Hard Future	0.5407	0.6606	0.2884
Soft Future	0.5594	0.6593	0.3069
Oracle Future	0.4690	0.7515	0.4100
LEWM	0.5967	0.6732	0.4949

In the image column, No Emotion at 0.5950, with all emotion dropped, is higher than the three middle conditions that include emotion.
The three No Emotion values are the same numbers as LEWM* in Table 3.

All the paper says about Oracle Future is the comparison with Soft Future, a gain of 9.22 points on video and 10.31 on audio.
In the audio column of that table, Oracle Future, given the ground-truth future emotion, scores 0.4100, below the full LEWM at 0.4949. The body text does not mention this reversal.

Incidentally, the labels are defined as seven in the dataset construction section, yet the intervention section that swaps the emotion for another value says “each of the eight emotions,” and the analysis of emotion changes brings in “joy,” a word that is not among the seven.