Optimizing Stable Diffusion LoRA Training for Consistent Character Features

Character consistency in LoRA training usually falls apart because of "overfitting to the background" or "feature bleeding" from the base model. If your character's hair color changes based on the outfit in the training set, you aren't training a character; you're training a specific set of photos.

The biggest breakthrough in my workflow was switching to a strict "differential captioning" strategy. Instead of just tagging the character, you have to explicitly tag everything that isn't the character. If your character is always wearing a red hoodie in your dataset, and you don't tag red hoodie in every single image, the AI assumes the red hoodie is part of the character's skin or identity.

Here is the specific pipeline I use with Kohya_ss to lock in features:

Dataset Preparation
Use 20-30 high-quality images. More isn't always better; it just introduces more noise.
Captioning: Use WD14 tagger but manually prune. If the character has a unique scar, do NOT tag the scar. If you tag scar on cheek, the AI learns that "scar" is a separate concept it can toggle on or off. By leaving it untagged, the AI associates that visual feature with your unique trigger word.
Resolution: Train at 512x512 or 768x768 depending on the base model (SD1.5 vs SDXL), but ensure you use "buckets" to avoid cropping out essential head/body features.

Hyperparameter Tuning
I've found that the default learning rates are often too aggressive, causing the "deep fried" look (over-saturation) after a few epochs. I stick to a Cosine scheduler with these settings:

# Recommended for SDXL Character LoRA
Learning Rate: 1e-4
UNet Learning Rate: 1e-4
Text Encoder Learning Rate: 5e-5
Network Dim/Alpha: 64 / 32
Optimizer: Adafactor or AdamW8bit
Max Train Epochs: 10-15 (Save every 2 epochs to find the sweet spot)

The "Overfitting" Trap
The most common mistake is training for too many steps. The "sweet spot" for character consistency is usually just before the model starts perfectly recreating your training images. If you prompt for a new pose and the character looks exactly like one of your training photos, you've overfit.

To fix this, I use a regularization set. If I'm training a specific girl, I include 100-200 generic images of "a girl" or "a woman" in a reg folder. This tells the model: "Keep the general concept of a human female, but apply these specific deviations to my trigger word."

Testing Workflow
Don't just rely on the final epoch. I use the XYZ plot in Automatic1111 to test different weights of the LoRA (from 0.5 to 1.0) against different prompts.

Prompt: <lora:MyCharacter:0.8>, a woman standing in a futuristic city, cinematic lighting, 8k

If the character's face drifts at 0.6 but the background looks great, and the face is perfect at 1.0 but the image is grainy, your learning rate was likely too high or your dataset was too small.

One final gotcha: ensure your trigger word is unique. Don't use Girl or Woman. Use something like SKS_Character or a nonsense string like Zylora. This prevents the LoRA from fighting with the base model's existing knowledge of those common words.