Training a Stable Diffusion V1.5 LoRA

The Hugging Face document provides complete guidance on training a LoRA by calling a pre-defined script [2] provided by Diffusers. However, we don’t want to stop at “using” the script. The training code from Diffusers includes a lot of edge-case handling and additional code that is hard to read and learn. In this section, we will write up each line of the training code to fully understand what happens in each step.

In the following sample, we will use eight images with associated captions to train a LoRA. The image and image captions are provided in the train_data folder of the code for this chapter.

Our training code structure will be like this:

# import packages
import torch
from accelerate import utils
from accelerate import Accelerator
from diffusers import DDPMScheduler,StableDiffusionPipeline
from peft import LoraConfig
from peft.utils import get_peft_model_state_dict
from datasets import load_dataset
from torchvision...