Inside a 4-Character Anima LoRA, From Prompt Mistakes to a Biased DiT

Compared with a one-character LoRA, the merged 4-character LoRA of Kei, Kana, Koharu, and Kurara is much harder to control: posing and keeping the four apart break easily.
Still, all four are definitely inside the LoRA — call them up one at a time and they come out fine.
Then what does my instruction actually look like by the time it has passed through the encoder and entered the model and the LoRA?
If I knew that, I could probably get the image I want without fiddling with prompts all year round.
I ran eight experiments between August 7 and August 14 poking at exactly that, but the individual posts came out dense, and even I find them hard to reread.
This rolls the eight posts into one and explains them more plainly.

The terms that come up repeatedly are collected here first.

Term	Meaning
conditioning	The prompt text converted by an encoder into a sequence of numbers. The image-drawing part works from this, not from the text
encoder	The part that converts the prompt into conditioning. Anima has two: Qwen3-0.6B and T5
DiT	The main body that receives the conditioning and actually draws the image
LLMAdapter	A component that converts the Qwen output into a form the DiT can accept. Its output is the conditioning right before the DiT
trigger word	A unique word like kanachan, written in the prompt to call up a character
training captions	The captions attached to each image when the LoRA was trained. The closer a prompt is to how these were written, the closer the output follows it
teacher conditioning	Conditioning built from a long, successful prompt, used as the target (teacher) when training a model to reproduce it from a short prompt

flowchart TD
    A[Missing prompt phrases<br/>Aug 7] --> B[Reproducing conditioning<br/>from short prompts<br/>Aug 9]
    B --> C[Conditioning right before the DiT<br/>replaces a character<br/>Aug 9]
    C --> D[Outfit lands on the wrong person<br/>in a classroom scene<br/>Aug 9]
    D --> E[Swapping positions<br/>does not move the outfit<br/>Aug 11]
    E --> F[Outfit mix-up comes from<br/>the DiT LoRA's bias<br/>Aug 12]
    F --> G[Pose and relations follow<br/>description order and slots<br/>Aug 13]
    G --> H[Band lineup exposes<br/>training-data bias<br/>Aug 14]

Most of it was missing prompt phrases

Right after the 4-character LoRA went out, prompting all four from my own generation server kept producing the same characters over and over.
Starting from trigger words only and restoring prompt information in four stages (appearance descriptions, position lines, outfit lines) across 11 combinations of 2–4 characters × 3 seeds, most of the problem turned out to be that my prompts were missing phrases that exist in the training captions.
The LoRA itself wasn’t defective; my prompts at generation time had drifted from how the training captions were written.

Training a model to reproduce conditioning from short prompts failed

If missing phrases were all it took, couldn’t the conditioning of a successful long prompt be generated automatically from a short trigger?
In an experiment that added a small converter module called a resampler to a frozen Qwen3-0.6B to reproduce teacher conditioning, I measured how close the produced conditioning came to the teacher with cosine distance, a metric that approaches 0 as two sequences of numbers point in the same direction, with 0 meaning a match.
On the pairs used for training, cosine distance reached about 0.005, an almost exact match. On validation prompts, though, it only came down to about 0.6, and the character ordering didn’t improve either.
Meanwhile, a method with no such training — having Qwen3.8 Max fill a fixed prompt template — drew the characters as instructed in 15 out of 16 cases.
The model could reproduce the pairs it was trained on almost exactly, but it had not become able to build conditioning for an ordering it had never seen.

Zeroing the conditioning right before the DiT replaced a character

In a post that separated the Qwen path from the T5 path, I directly manipulated the internals that the training in the previous section failed to reproduce.
Zeroing kanachan’s description span at the final Qwen layer barely changed the image, but zeroing the same span at the LLMAdapter output (right before the DiT) erased kanachan’s appearance traits (side ponytail, ahoge, brown hair) and replaced her with someone nearly identical to keichan, blonde with a blue ribbon.

Intervention	Where	Change in the image
A	Final Qwen layer, zero kanachan	Nearly identical to baseline, kanachan’s traits remain
B	Final Qwen layer, zero koharu	Nearly identical to baseline, koharu’s traits remain
C	Final Qwen layer, swap the kanachan and koharu description spans	No swap; order and traits unchanged
D	LLMAdapter output (right before the DiT), zero kanachan	kanachan disappears, replaced by a keichan-like character

Intervention D reproduced the same breakage across all 3 seeds, and the replacement was keichan every time.
However I altered the final Qwen layer, the image barely changed; the character changed only when I touched the conditioning right before the DiT.

In a classroom scene, the specified outfit was worn by the wrong person

In an experiment putting Kurara and Kei in conversation, Koharu with crossed arms, and Kana waving, all in one classroom, some of the 8 setups × 3 seeds got the roles clearly right, yet 0 out of 3 images were usable as they were.
The grey skirt and black tights specified for Kana vanished in 3 out of 3, and Koharu was wearing them instead.
Faces and rough roles were drawn apart correctly; only the mid-conversation hands and the lower-body outfits went to the wrong person.

Can the prompt say who wears an outfit

In a post that swapped position and outfit assignments independently, whether the black tights were assigned to Kana or to Koharu, and whether or not their positions were swapped, the person at front right wore them in 12 out of 12 images.
Restating the outfit at the end of the prompt, shortening the prompt, or writing the outfit next to the name didn’t fix it.
With the 3-character LoRA, stripping appearance descriptions out of the captions and absorbing them into the trigger words kept the characters distinct; but removing all four characters’ appearance descriptions the same way dropped the distinct-character images to 0/3, and the outfit mix-up stayed.
The appearance descriptions can’t simply be removed, yet once appearance, roles, and outfits pile up together, the person at front right kept wearing the black tights regardless of the assignment.

The outfit mix-up is the DiT LoRA’s bias

In a post that applied the LoRA to the LLMAdapter side and the DiT side separately, I checked which side the mix-up comes from.
With the inputs to Qwen and T5 kept identical across all conditions, plain Anima with no LoRA put the black tights on Koharu at the back, exactly as specified.
The moment the DiT LoRA was applied to the same input, Kana at front right was wearing them instead.
With the LoRA on the LLMAdapter side alone the tights stayed on the right person; applying it to the DiT side alone made the same mistake as applying both.
Rather than the 0.6B encoder lacking information, the DiT side of the 4-character LoRA had learned a strong bias toward putting black tights on the front-right person, and it overrode a mapping that plain Anima got right.

Poses and relations stabilized with a fixed order and named slots

In a post probing how far poses and character-to-character relations can be specified, I found cases where the output ignored the instructions: an expression cue alone turned a character into someone else’s face, swapping Koharu and Kana’s order in the prompt changed who was drawn, and the high five happened between the two center characters no matter whose names I wrote.
Adding more text didn’t stabilize it; changing the structure did.
Each character’s appearance description (hair, eyes, accessories) became a short fixed phrase that I never expanded or trimmed along the way. The order in which characters were described was fixed once and never moved. The two who should touch went into adjacent center slots, and the front and back rows named exactly who was in them, not just how many.
In that form, with three in front and one in back, the two center characters high-fived in all 3 seeds, Kana in the back kept her arms down, and Koharu on the right crossed her arms.

The training-data bias that a band lineup exposed

In a post checking whether the four could be drawn as a band, I applied everything above to an actual band-lineup prompt.
Assigning bass on the left, guitar and vocals at center, guitar on the right, and drums in the back row, I compared a handwritten prompt, prompts written by Qwen3-4B through 14B, a linear bridge that converts the 4B output into the same 1024 dimensions as the 0.6B in order to swap in the larger encoder, and compression down to 507 tokens; the instruments and roles never fully lined up.
Checking the 896 training captions, only 10 distinct ones contain all four trigger words at once, and every one of them is a symmetric composition: standing in a row, white background, everyone in the same motion.
Training examples where the four hold different positions, different props, and different actions: zero.
Adding roles tended to pull characters toward the center, or snap them back to the same-motion poses that existed in training. That bias looked like the cause.
Next, I’ll add 24–40 captions that permute the four characters’ roles, positions, and props while keeping the solo, duo, and trio captions intact, and rerun the same band-lineup prompts.