Anima 4-char conditioning distillation vs fixed-template P3 on M1 Max

In the previous experiment on Anima’s Qwen3-0.6B and LLMAdapter, I crossed a short P0 and a successful P3 between the Qwen side and the T5 side. At seed 42, switching only Qwen to P3 while keeping T5 at P0 did not bring back kanachan’s visual features, and switching T5 to P3 got kanachan and keichan drawn apart correctly.

Building a long prompt from a fixed dictionary is one way to keep the actual input short. This time I did not feed Anima a long prompt at all, and tested whether the conditioning of a successful long prompt can be reproduced from short triggers. I froze Qwen3-0.6B, built a small resampler that learns to produce the conditioning right before the DiT, and used Qwen3.8 Max to grow the set of orders and scenes used for its training.

Test environment

Item	Details
Machine	MacBook Pro M1 Max, 64GB unified memory, MPS. Extraction, training and generation all ran on this one machine, no RunPod
Generation	local ComfyUI, about 33 s per image (8 steps)
Base model	anima-base-v1.0.safetensors
LoRA	anima-turbo-lora-v0.1 (speed, strength 1.0), anima-4char-v1_epoch100 (char, strength 1.0 / clip 0.8)
text encoder	qwen_3_06b_base.safetensors (CLIPTextEncode, type=qwen_image)
Teacher pair generation	Qwen3.8 Max (ambassador program, OpenAI-compatible API via ModelScope)
Resampler training	1500 steps in under a minute, tensor extraction in tens of seconds

Qwen3.8 Max returns no internal tensors, so it only picks orders and scenes

With early access to the Qwen3.8 Max preview I could pull generated text and reasoning info from the dedicated workspace’s OpenAI-compatible API. The Chat Completions API spec on Alibaba Cloud Model Studio describes responses made of generated text, reasoning, logprobs, usage and so on, with no field returning intermediate hidden states for each input token. As far as the documented spec goes, it cannot be used for feeding Qwen3.8 Max’s internal tensors directly into Anima.

Model Studio also has a dedicated embedding API. The spec uses per-input embedding vectors for search and similarity, and 1024 dimensions is one of the options. But it says nothing about per-token hidden states or compatibility with Anima’s LLMAdapter, so it cannot be used for the distillation either. Qwen3.8 Max was used only to generate training data.

Max never writes the P3 text itself; it returned 12 JSON items of character order plus scene. Local checks screened for missing characters and broken JSON, and a fixed renderer assembled P3 from the same English templates. Max decides only the order and the scene, and the text that reaches Anima always comes out of the renderer’s fixed templates.

Splitting the work per stage

Task	Where it runs	Why
Turn the 4-character dictionary into order + scene JSON	Qwen3.8 Max API	grows the pool of order and scene combinations
Assemble P3 from the JSON	local fixed renderer	fixes attribute order and sentence patterns, keeps teacher prompts reproducible
Extract successful conditioning from P3	current Anima / Qwen3-0.6B / LLMAdapter	the teacher is the exact tensor the DiT already succeeds with
Extract the hidden state from P0	current Anima / Qwen3-0.6B (frozen)	same 0.6B on the short-input side, so the comparison isolates the transform, not capacity
Train the resampler	local M1 Max (MPS)	a network around 50M params; with the Qwen3-0.6B side frozen, one machine is enough
Compare images at the same seed	local generation setup	the final question is whether the DiT draws the four characters apart

Bridging the 43-vs-179 token gap with a resampler

In the previous four-character setup, the LLMAdapter’s pre-padding output length differed a lot between P0 and P3.

Input	actual T5 tokens	LLMAdapter output
P0, triggers only	43	(1, 43, 1024)
P3, with identification, position and clothing sentences	179	(1, 179, 1024)

After padding to 512 tokens the shapes do match. But P0 is mostly zeros past token 43, while P3 carries meaningful conditioning up to token 179. Apply MSE directly and the positions describing the same character or attribute no longer line up, and matching the zero padding dominates the loss.

P0  : 43 token  ──────────────── zero padding ────────────────
P3  : 179 token ───────────────────────────── zero padding ──

Projecting the hidden size to 1024 does not bridge this token-count gap. On top of the projection, I train a resampler that turns the short P0-side hidden state into the same 179 tokens as the teacher.

flowchart TD
    S[P0 + character dict] --> M[Qwen3.8 Max API]
    M --> J[order + scene JSON]
    J --> R[fixed renderer]
    R --> L[P3 teacher prompt]
    L --> A[current Anima encoder]
    A --> G[C_good<br/>179-189 x 1024]
    S --> O[frozen Qwen3-0.6B<br/>P0 hidden state]
    O --> P[resampler<br/>200 learned queries]
    P --> C[C_pred<br/>200 x 1024]
    G --> D[distillation loss]
    C --> D

The first round is restricted to the fixed 4-character P3 prompts, with the number of learned queries (NQ) fixed at 200. Extending to pairs or trios will need either predicting the teacher mask too, or giving each combination its own effective length - I decided to add that later and started the experiment.

Step 1, fixing the teacher prompts

Input data is saved as the four fields from prompt_short to prompt_teacher.

Field	Contents
prompt_short	the short prompt you actually want to type
character_dict	per-trigger hair style, hair color, eyes, height, clothing
max_json	the JSON Qwen3.8 Max organized per character
prompt_teacher	the P3-style English prompt built by the fixed renderer

Max never gets an unknown trigger alone to guess a character from, and any output that adds attributes missing from the dictionary, drops a character, or changes the order is excluded from teacher candidates. API outputs are saved together with temperature, seed-equivalent settings, model ID and reasoning settings.

The four handwritten pairs (order1-4)

At first the Qwen3.8 Max API was not called at all: I kept the four identification sentences in the fixed template established in anima-4char-qwen-conditioning (trigger + hair/eyes/accessory identification + position sentence + height comparison + clothing sentence) and handwrote only the permutations of the four characters.

pair ID	short (P0)	tokens after renderer (actual T5)	teacher image verdict
order1	kurara, keichan, kanachan, koharu	179	Success. Visual features, order and height differences all as instructed (confirmed in the existing article)
order2	koharu, kanachan, keichan, kurara	184	Success. Visual features, order and height differences all as instructed (Step 4 images below)
order3	keichan, kurara, koharu, kanachan	189	Visual features and order as instructed, used for training. Height differences are off though: kanachan comes out shorter than koharu (see the note under the Step 4 image)
order4	kanachan, koharu, kurara, keichan	183	Success. Visual features, order and height differences all as instructed (Step 4 images below)

Max API run (Run 2)

Next I actually called Qwen3.8 Max. Usage was prompt_tokens=384, completion_tokens=2085, total_tokens=2469.

Max received only the four characters’ identification dictionary (names plus short hair/eyes/accessory descriptions) and generated 12 JSON pairs of a four-character order (permutation) and a short scene phrase. It did not write any of the sentences (identification, position, height comparison, clothing); renderer.py (the fixed renderer, described below) assembles those locally.

The self-check (local, separate from the Max call) covered only whether all four names appear with no omissions or duplicates and whether scene_short/scene_full are non-empty. 12 out of 12 passed (0 rejects).

Since the self-check can only catch name issues, I generated teacher P3 images for all 12 in Step 2 and checked them by eye. Exactly one (maxgen06) failed with generated positions that do not match the instructed order (the teacher prompt text is correct, but the generation swapped the visual features’ positions). That one was excluded from the training data, adopting 11 of 12.

pair ID	assignment	teacher P3 image verdict
the 9 of maxgen00-09 (excluding maxgen06)	training	Success. Visual features and order as instructed
maxgen06	excluded	Failure. Features are all there but the order does not match the instruction (image below)
maxgen10, maxgen11	held-out	Success. All four as instructed (Step 4 images below)

The fixed renderer (renderer.py) takes the four-character order and assembles the position sentences (is on the far left / is second from the left / … / is on the far right) and the height comparison sentences. The height comparison reuses the established identification logic as is: the tied pair kurara and keichan gets the same height as X when the second of them appears, kanachan gets half a head shorter if a tied character already appeared and half a head shorter than the tallest otherwise, and koharu is always the shortest.

A self-test confirmed this renderer can emit the exact same text as the existing order1 (4char_P3) and order2 (4char_P3_order_reversed) prompts.

Step 2, saving only successful teacher tensors

prompt_teacher goes through the current Qwen3-0.6B and LLMAdapter, and every stage of the encode path is saved: Qwen3-0.6B’s final hidden state, the T5 tokenizer IDs and weights, the LLMAdapter’s pre-padding output, the conditioning after padding to 512 tokens with its valid-token mask, and the image generated at the same seed.

Images are checked for the four characters’ identity, attributes and position, and the conditioning of a failed prompt never enters C_good. The teacher tensor’s dtype, per-token norms, and overall mean and std also go into the manifest.

Run log

step_resampler_extract.py (a 4-pair batch version of step2_extract.py) saved the P0-side and P3-side tensors for each of order1-4. CPU-pinned, and the DiT itself is never loaded.

• Input side: qwen_final from feeding P0 (triggers only) through Qwen3-0.6B. All four orders are permutations of the same four triggers, so the shape is (1, 38, 1024) float32 across the board
• Teacher side: adapter_out_weighted from feeding P3 (with identification sentences) through qwen_final → LLMAdapter (pre-padding, real length). Shapes run (1, 179, 1024) to (1, 189, 1024) per order (token counts shift with the wording of the identification sentences)

Checking the value ranges first, per-token L2 norms average 4.06-4.17 on the teacher side across all four orders, while the input-side Qwen final hidden state sits around 110, about 27x the teacher’s scale. This 27x gap is what stalled the first resampler implementation in Step 3.

Run log addendum (Run 2)

For the Step 1 Max generations, the 11 items excluding maxgen06 (whose teacher P3 image broke the order) went through the same extraction of P0-side and P3-side tensors (the PAIRS dict in step_resampler_extract.py was extended). The Max generations carry scene_short on P0, so their P0-side token counts run 43-45 instead of the constant 38 of order1-4. The resampler side handles this with a key_padding_mask on cross-attention.

That brings the training data to 16 pairs in total (4 handwritten + 12 Max-generated), of which the 15 excluding maxgen06 are confirmed by teacher images.

Step 3, trying the resampler with 0.6B as-is

From the hidden state of the current Qwen3-0.6B fed with P0, can the 179-token C_good be predicted? The outcome shows which piece is missing, Qwen capacity or a learned transform that expands a short prompt into long conditioning.

Qwen3-0.6B and the DiT stay frozen. Qwen hidden states are saved in advance, and only the projection and the resampler go on the GPU during training.

The initial loss is recorded as three terms, masked MSE, cosine and distribution.

L = λ_mse * masked_MSE(C_pred, C_good)
  + λ_cos * (1 - cosine(C_pred, C_good))
  + λ_stat * distribution_loss(C_pred, C_good)

distribution_loss matches token norms, mean and std. Verdicts never rely on the training loss alone: images generated from the held-out pairs, the ones kept out of training, decide whether the four characters are drawn apart.

Run log (Run 1)

Training used order1, order3 and order4, with order2 (reversed) as the held-out validation pair. P0 (the resampler input) has the same token count for all four orders, since they are permutations of the same four triggers, so this setup checks whether the order information alone can be read out of the Qwen hidden state to reproduce the teacher’s arrangement.

The resampler is 200 learned queries (NQ=200, headroom over the teacher max length of 189) cross-attending into the P0-side Qwen final hidden state (resampler.py).

Item	Value
Architecture	200 learned queries (200×1024) + [self-attn→cross-attn→FFN]×3 layers, 8 heads
dim	1024 (Qwen final hidden state and LLMAdapter output are both 1024, so no projection)
trainable parameters	50,598,912
optimizer	AdamW, lr=2e-4, weight_decay=1e-4
steps	1500 (3 training pairs, full batch)
Loss weights	λ_mse=1.0, λ_cos=1.0, λ_stat=0.1
Environment	local M1 Max, MPS

The first implementation did not converge even at 1500 steps. The context (Qwen final hidden state) went into the cross-attention keys and values raw, so the resampler’s internal values got dragged toward the input-side scale (per-token norm ~110). The output-layer LayerNorm alone could not close the gap to the teacher (per-token norm ~4), and train cos distance plateaued around 0.94 (the distribution_loss breakdown showed most of the initial value was this 27x norm gap itself). Inserting a pre-LayerNorm on the context and lowering the output LayerNorm’s initial weight from 1.0 to 0.2 improved it drastically at the same 1500 steps.

	train_mse	train_cos_dist	val_mse	val_cos_dist
before the fix, step 1500	0.213	0.943 (≈ nearly uncorrelated)	0.217	0.976
after the fix, step 1 (initial)	0.059	0.999	0.049	0.794
after the fix, step 1500	0.00046	0.0047 (≈ cos similarity 0.995)	0.0234	0.595 (≈ cos similarity 0.405)

The three training pairs (order1, order3, order4) were reproduced almost perfectly at cosine distance 0.005. The held-out order2 plateaued around cosine distance 0.59, with an MSE about 125x the training side. With only three training pairs, that is about what you would expect.

The images turned out more tangled (images in Step 4). The resampler’s held-out (order2) output did draw all four characters’ visual features apart correctly (kanachan’s ahoge and side ponytail, koharu’s black hair and red eyes, kurara’s rose-brown hair and earrings, keichan’s blonde hair and blue ribbon). That is clearly better than raw P0, where features bleed into other characters or drop out. But the order matched not order2’s instruction (koharu, kanachan, keichan, kurara) but order4 from the training set (kanachan, koharu, kurara, keichan) almost exactly. The P0 token sequences are mere permutations of the same four triggers, and the resampler looks like it picks whichever trained order pattern is closest. The visual features did partially carry over to an unseen order, but whether the order itself is being read out of the Qwen hidden state cannot be judged from this 3-pair training.

Run log (Run 2, 12 training / 3 held-out)

Run 1 showed that 3 training pairs cannot tell whether the order is being read out, so using the Step 1 Max generations I retrained with the same settings (NQ=200, 3-layer resampler, AdamW lr=2e-4, 1500 steps): training = order1, order3, order4 plus the 9 of maxgen00-09 excluding maxgen06, 12 pairs in total, and held-out = order2, maxgen10, maxgen11, 3 pairs.

	train_mse	train_cos_dist	val_mse (combined)	val_cos_dist (combined)
Run 1 (3 train / 1 held-out), step 1500	0.00046	0.0047	0.0234	0.595
Run 2 (12 train / 3 held-out), step 1500	0.00045	0.0049	0.0203	0.605

Training-side fit (cosine distance around 0.005) barely moved from Run 1, and the value over the three held-out pairs combined was flat too, 0.595 to 0.605. Quadrupling the training pairs did not improve the held-out numbers.

The per-pair numbers were not far apart either.

Held-out	mse	cos_dist
order2	0.0213	0.649
maxgen10	0.0197	0.571
maxgen11	0.0199	0.596

The combined numbers were flat, but the generated images broke differently for each of the three (images in Step 4). This is no longer the simple memorize-one-trained-pattern result.

Held-out	Instructed order	How it breaks
order2	koharu, kanachan, keichan, kurara	1st (koharu) and 4th (kurara) as instructed. kanachan and keichan swap at 2nd and 3rd
maxgen10	kanachan, keichan, koharu, kurara	3rd (koharu) and 4th (kurara) as instructed. kanachan and keichan swap at 1st and 2nd
maxgen11	koharu, kanachan, kurara, keichan	Only the 1st (koharu) as instructed. The 2nd and 3rd both come out looking like keichan (blonde with ribbon), kurara never appears, and kanachan gets pushed out to 4th

What the three share is that koharu’s standing position (which slot from the left) is as instructed, and it is kanachan and keichan that swap as neighbors or duplicate. The order information does not come entirely from rote recall (three different failure modes rule out collapse onto a single memorized pattern). A character whose looks sit far from everyone else’s, like koharu (black hair, red eyes, shortest), tends to land in the instructed standing position, while kanachan and keichan, the pair whose positions varied most across the training data, still do not get drawn apart correctly. The combined loss alone read as “no change”; only lining up the per-pair numbers and the images made the difference in failure modes visible.

Step 4, comparing prompt expansion and tensor distillation in images

Same prompt, same seed, four conditions from raw P0 input to the resampler path.

1. Feed P0 to the current Anima as is
2. Build P3 from the fixed dictionary and feed it to the current Anima
3. Build P3 from Qwen3.8 Max’s JSON and feed it to the current Anima
4. Build C_pred from P0 with Qwen3-0.6B plus the resampler

Condition	Four characters correct	Feature bleed	Position	Notes
P0	breaks (image below)	yes	does not match the instructed order	sometimes only three characters appear
fixed-renderer P3 (4 handwritten, no distillation)	4/4 success	none	as instructed	order1-4, see Step 1
fixed-renderer P3 (12 Max-generated, no distillation)	11/12 success	order swapped only in maxgen06	as instructed except maxgen06	the Max→P3 results, see Step 1
0.6B resampler (distilled)	partial (below)	features drawn apart with less breakage than P0	Run 1: nearly reproduces a different trained order. Run 2: koharu/kurara land right, kanachan/keichan swap	cosine distance Run 1: 0.595 / Run 2: 0.605 (3 held-out combined)

The distillation-free fixed-renderer P3 drew 15 of 16 pairs apart as instructed, while the distilled 0.6B resampler broke the held-out order. Letting Max decide order and scene in front of a short input prompt, then rendering P3 and feeding it to Anima, stays as the control condition for the distillation.

Except for the multi-seed section, most comparisons ran once at seed 42 only. A one-seed success is not generalized into stability. Character count, identity, attributes and order are judged separately.

Run log (seed 42 only)

The order3 and order4 teacher P3s were also generated standalone and checked before entering the training data. First, order3’s teacher P3 came out like this.

I only noticed later that this order3 does not get the height differences right. The teacher prompt text follows the renderer’s height logic (koharu is always the shortest), yet in the generated image kanachan is shorter than koharu. Teacher images were judged mostly on visual features and order, so this slipped through and the pair went into the training data as is. The damage to the training result seems limited though, and height differences come out fine in the generations after this. Possibly because the prompt side, whose height text is correct, is also present.

Here is order4’s teacher P3. This one gets the height differences right too.

order1 is a training pair: raw P0, teacher P3 and the resampler’s C_pred are compared. C_pred saw this combination in training, so the check is whether it reproduces the teacher almost exactly. The first image is raw P0.

The second image is the teacher P3.

The third image is the resampler’s C_pred. It comes out almost identical to the teacher P3.

order2 is the held-out validation pair, an order never used in the resampler’s training. Raw P0, teacher P3 and the resampler’s C_pred line up with order4’s teacher P3 re-shown for comparison, four images in total. The first is raw P0.

The second is the teacher P3. The instructed order looks like this image.

The third is the resampler’s C_pred.

The fourth is a re-post for comparison: order4’s teacher P3, which was used in training.

Comparing the four images, the visual-feature breakage clearly shrinks from raw P0 (1st image), while the order matches order4 from the training set (4th image) rather than the instruction (teacher P3, 2nd image). This reads less like the order information being read from the Qwen hidden state and reproduced toward the teacher, and more like the closest order seen during training getting picked, a result close to memorization.

Run log (Run 2, seed 42 only)

After retraining the resampler with 12 training and 3 held-out pairs, the same three conditions (raw P0, teacher P3, C_pred) were regenerated for the three held-out pairs (order2, maxgen10, maxgen11). order2’s raw-P0 and teacher-P3 images are identical to Run 1 (the prompts did not change), so the two above are reused and only the C_pred is replaced with the retrained one.

maxgen10 is held-out; the instructed order is kanachan, keichan, koharu, kurara, and the scene is waiting on a station platform. The first image is raw P0 (trigger plus scene phrase), the second the teacher P3, the third the C_pred from the Run 2 retrained resampler.

maxgen11 is held-out; the instructed order is koharu, kanachan, kurara, keichan, and the scene is playing in a rainy courtyard. Same lineup as maxgen10: raw P0, teacher P3, Run 2 C_pred.

Lining up the three held-out pairs, the combined loss barely moved from Run 1, but the failure mode changed from Run 1’s near-reproduction of one trained pattern. koharu’s standing position is right in all three, while kanachan and keichan either swap (order2, maxgen10) or one duplicates and the other disappears (maxgen11), so the instructed order is still not reproduced. Raw P0 (first image of each) still loses the fourth character or falls back to school uniforms, breaking at the level of character count and attributes before order even matters.

Multi-seed check (seeds 1234 and 9999 added)

The held-out results so far were single runs at seed 42, so with C_pred fixed (same resampler weights, same input P0), only the generation seed was changed to 1234 and 9999 for the same three held-out pairs. C_pred is the same tensor regardless of seed, so this checks whether the way the four characters are drawn apart shifts when the DiT’s sampling changes. Below, order2, maxgen10 and maxgen11 in that order, two images each for seeds 1234 and 9999.

Across the three seeds, order2 and maxgen10 reproduced exactly the same wrong arrangement every time. That swap is hard to explain by the sampling seed alone, and the influence looks to sit on the C_pred side. Only maxgen11 broke differently per seed (seeds 42 and 1234 duplicate keichan and drop kurara, seed 9999 shows all four but in yet another wrong order). Among the three, maxgen11’s conditioning may be the one most open to different readings by the DiT’s sampling.

Distillation breaks the order, the fixed renderer went 15 of 16

On the training side, both Run 1 (3 pairs) and Run 2 (12 pairs) reproduced the teacher conditioning down to cosine distance around 0.005, while the cosine distance over the three held-out pairs combined went 0.595 to 0.605 - quadrupling the training pairs did not move the number. What did change is how the generated images break, from Run 1’s near-reproduction of one trained pattern to Run 2’s pattern where koharu’s standing position lands right but kanachan and keichan swap or duplicate. This change never shows up in the combined loss. order2 and maxgen10 reproduced the identical wrong arrangement across all three seeds, and that swap is hard to explain by the sampling seed alone.

Meanwhile, the distillation-free route - Max’s JSON rendered into P3 by the fixed renderer and fed straight to Anima - drew 15 of 16 pairs apart as instructed. Since MSE and cosine distance dropped far enough without fixing the held-out order, the feature-level loss used here (masked MSE + cosine + distribution loss) appears not to capture which differences in the conditioning the DiT actually uses to draw the characters apart. Instead of pulling C_pred toward C_good, there is room to move to a loss that matches the DiT’s own noise prediction at the same latent and timestep (L_dit = MSE(DiT(x_t, t, C_pred), DiT(x_t, t, C_good))), but that needs backprop through the DiT and a lot more compute, so it was not tried within this article.