LoRA: Setting Up a Training Environment on an RTX 3060 Laptop (6 GB VRAM)

Introduction

In the previous article, I was planning to train on a Mac mini M4 Pro (24 GB), but due to circumstances I ended up with the base M4.

The base M4 just doesn’t have enough VRAM to run training. So I decided to let an idle laptop (RTX 3060 Laptop / 6 GB VRAM) grind overnight.

This article summarizes how to set up a LoRA training environment on Windows 11 + RTX 3060 Laptop (6 GB VRAM). With only 6 GB of VRAM, SDXL is tough, so this guide targets a SD1.5-based model.

Environment

Item	Spec
OS	Windows 11 Home
GPU	RTX 3060 Laptop (6GB VRAM)
Storage	1TB SSD

1. Install Prerequisites

1-1. Git

URL: https://git-scm.com/download/win
Install with default settings

1-2. Python 3.10.6

Use this exact version (other versions often cause issues).

URL: https://www.python.org/downloads/release/python-3106/
Choose Windows installer (64-bit)
Make sure to check “Add Python to PATH.”

1-3. CUDA Toolkit 11.8

URL: https://developer.nvidia.com/cuda-11-8-0-download-archive
Windows > x86_64 > 11 > exe (local)

1-4. cuDNN 8.x (for CUDA 11.x)

URL: https://developer.nvidia.com/cudnn
Requires an NVIDIA account
After download, copy into the CUDA install directory (overwrite)

2. Install kohya_ss

# Create a working folder
mkdir D:\AI
cd D:\AI

# Clone
git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss

# Run setup
.\setup.bat

During setup, answer:

CUDA version → 11.8
Install accelerate → Yes

3. Place Base Model(s)

Destination: D:\AI\models\

Recommended models (manga/anime style)

Model	URL
Anything V5	https://civitai.com/models/9409
Counterfeit-V3.0	https://civitai.com/models/4468

4. Prepare Training Data

Folder layout

D:\AI\datasets\my_character\
  └── 10_chara_name\         ← "<repetitions>_<trigger word>"
        ├── image001.png
        ├── image001.txt     ← caption
        ├── image002.png
        ├── image002.txt
        └── ...

Repetition count guideline

100 images × 10 repeats = 1,000 steps/epoch
If you have 50 images, use 20_chara_name to compensate

5. How to Write Captions

Basic structure

<trigger word>, <core attributes>, <variable details>

Examples

For the same character (a short black-haired, red‑eyed girl):

image001.txt (school uniform, smiling)

chara_name, 1girl, short black hair, red eyes, smile, school uniform, white shirt, blue skirt, upper body, simple background

image002.txt (casual outfit, neutral face)

chara_name, 1girl, short black hair, red eyes, casual clothes, hoodie, jeans, upper body, outdoor

image003.txt (swimsuit, side view)

chara_name, 1girl, short black hair, red eyes, swimsuit, bikini, from side, beach, summer

Tag categories

Fixed tags (shared across all images)

chara_name ← trigger word
1girl
short black hair ← hairstyle
red eyes ← eye color

Variable tags (change per image)

Outfit: uniform, casual, swimsuit…
Expression: smile, serious, angry…
Composition: upper body, portrait, full body…
Background: simple background, outdoor, indoor…
Pose: standing, sitting, from side…

Tips for the trigger word

Coin a unique token that won’t collide with existing words
Examples: mycharaname, xyz_oc, aaa_character
If it’s too short, it can mix with other concepts

Notes

NG	OK
`black_hair`	`black hair` (space-separated)
`trigger,1girl`	`trigger, 1girl` (space after comma)
Japanese tags	English tags only

6. Auto-generate Captions → Manual Cleanup

Method 1: WD Tagger (built into kohya_ss)

In kohya_ss, Utilities tab → “WD14 Captioning”
Select the image folder
Run → .txt files are generated per image
Manually edit:
- Add the trigger word to the beginning
- Normalize character-specific feature tags
- Remove unnecessary tags

Method 2: BooruDatasetTagManager

URL: https://github.com/starik222/BooruDatasetTagManager
Handy GUI to batch-edit tags

Workflow

Auto-generate with WD Tagger (~5 minutes)
Open in BooruDatasetTagManager
Add the trigger word to all images at once
Normalize character-specific tags (hair color, eye color)
Fix obviously wrong tags
Save

7. On Image Sizes

If the originals are large (e.g., 2800x2800), no need to resize.

If you enable enable_bucket in kohya_ss, you can train with variable sizes as-is.

Settings:
- enable_bucket: ON
- resolution: 512
- bucket_no_upscale: ON

8. Launch

cd D:\AI\kohya_ss
.\gui.bat

Open http://127.0.0.1:7860 in your browser.

9. Training Settings for 6 GB VRAM

Item	Value
Network Rank (Dim)	32–64
Network Alpha	16–32
Batch Size	1
Resolution	512
Epoch	around 5 (adjust after checking)
Learning Rate	1e-4
Optimizer	AdaFactor or Lion
Mixed Precision	fp16
Gradient Checkpointing	✓ ON
Cache Latents	✓ ON
enable_bucket	✓ ON
bucket_no_upscale	✓ ON

About epochs: Don’t start with 10. First run 5 and inspect the outputs. If it’s not enough, run more. With more images × repeats, overfitting happens more easily, so start conservative.

10. Training Data Guidelines

Images	Result
10–20	Bare minimum; tends to overfit to specific poses
30–50	Practical
50–100	Very good
100+	More than enough

Qualities of a good dataset

Variety of expressions
Variety of outfits
Variety of compositions (front, side, diagonal)
Variety of backgrounds
Stable line quality
Include a few full-body shots
Include a few close-up face shots

11. Handling 3‑view sheets

If you include a 3‑view sheet as-is, it will also learn the “3‑view layout”.

Fix: split and use individually

Cut out front/side/back and include them as separate images.

Learns the character traits
Doesn’t learn the 3‑view layout
Effectively adds three images’ worth of data

12. Training Time Estimate

RTX 3060 Laptop / 100 images / SD1.5 base:

About 1–2 hours to finish

Troubleshooting

CUDA out of memory

Lower Batch Size to 1
Turn ON Gradient Checkpointing
Lower Network Dim to 32

Loss becomes NaN

Lower the Learning Rate (e.g., 5e-5)
Change Mixed Precision to bf16

Outputs look blurry

Undertrained → increase epochs
Raise Network Dim (64–128)

Character doesn’t show up

Forgot to include the trigger word in the prompt
Mismatch between the training trigger word and the prompt

Closing

I expect the M4 Pro to arrive sometime next year—when it does, I’ll ask for a higher spec. Hopefully I can write the second part then.