Setting Up LLM/RAG for Work, So I Tried LoRA Training on the Side (Part 1)

Why I Decided to Make a LoRA

I’ve been tasked with building an automated help desk reply system at work. Using an existing SaaS solution was an option, but since it involves internal confidential information, we can’t send it to an external service. The decision was to build a RAG system within the LAN.

A Mac mini M4 Pro was approved for purchase for this project. But there’s still time before the specs are finalized.

Item	Spec
Chip	Apple M4 Pro
CPU	12-core
GPU	16-core
Neural Engine	16-core
Memory	24GB (Unified Memory)
Storage	512GB SSD

Minimum M4 Pro configuration. With unified memory, all 24GB is available as GPU memory (in theory).

So while waiting, why not use this Apple Silicon machine to run some LoRA training? That was the plan.

(I’m also doing actual work, don’t worry.)

Goal for This Series

I want to be able to generate this character.

The goal is to make a LoRA with my art style (well, kind of) and freely change poses and compositions using ComfyUI. Part 2 will report on the results, so stay tuned.

If you’re wondering “who is this?”, check the About page.

What’s in Part 1

This article covers “here’s what I plan to do.” The actual verification after the Mac mini arrives will be covered in Part 2.

Phase 1: Environment Setup (Common Foundation)

The setup to do first when getting a new Mac. This becomes the common foundation for both RAG building and LoRA training.

Install Homebrew

One command in the terminal. The foundation of development environment setup on macOS.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Python Environment Setup

This is the biggest pitfall. With the latest Python 3.12 or 3.13, libraries like PyTorch often don’t work.

The image generation AI world runs stably on Python 3.10 or 3.11. Managing versions with pyenv and creating virtual environments is strongly recommended.

brew install pyenv
pyenv install 3.10.13
pyenv global 3.10.13

Phase 2: Image Generation AI (LoRA Training) Setup

I’ll use Kohya_ss GUI for training. It’s currently the de facto standard tool for LoRA training in the image generation community.

1. Training Tool: Kohya_ss

A GUI tool for creating LoRAs, operated from the browser.

The official README has macOS installation instructions: bmaltais/kohya_ss

# Clone the repository
git clone https://github.com/bmaltais/kohya_ss.git
cd kohya_ss

# Setup
./setup.sh

# Launch
./gui.sh

During setup, Accelerate config questions appear. The key here is selecting mps (Metal Performance Shaders) as the platform.

Launching it opens a training interface in the browser.

2. M4 Pro Configuration Tips

“Lightweight settings” to complete training within 24GB memory. These matter.

Setting	Recommended Value	Reason
Optimizer	AdamW8bit	Memory saving
Mixed Precision	bf16	M4 supports bf16, recommended
Cache Latents	On	VRAM saving
Gradient Checkpointing	On	Essential. Dramatically reduces memory usage

3. Workflow

1. Prepare training images (about 15–30)
2. Start training in Kohya_ss
3. Leave the Mac alone and wait for completion

Regarding training time — the MPS (Metal Performance Shaders) backend tends to be slower compared to NVIDIA GPUs. SDXL LoRA training could take several hours, so running it before bed seems reasonable.

Phase 3: Generation Environment (ComfyUI) Setup

To generate images using the trained LoRA, I’ll use ComfyUI. Its node-based UI allows for flexible workflow construction.

1. Install ComfyUI

The official README has an “Installing on macOS” section: comfyanonymous/ComfyUI

# Clone the repository
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

# Install dependencies
pip install -r requirements.txt

# Launch
python main.py

On Apple Silicon, PyTorch’s Metal support enables GPU acceleration without CUDA.

2. Install ComfyUI-Manager

After installing ComfyUI, the first thing to add is ComfyUI-Manager: ltdrdata/ComfyUI-Manager

cd ComfyUI/custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Manager.git

With this installed, missing features (custom nodes) can be installed with a single button click from the GUI. Essential.

3. Model Placement

• Checkpoints → models/checkpoints/
• VAE → models/vae/
• LoRA → models/loras/

Place the trained LoRA file in models/loras/ and ComfyUI can load it.

References

• AICU media (Note) — Great coverage of the latest image generation AI techniques and ComfyUI workflow explanations

Plans for Part 2

• Results of actually setting up on the Mac mini M4 Pro
• Whether Kohya_ss actually runs
• How long training takes
• Results of generating with ComfyUI

Trying it all when it arrives.