Diffusion Model Transfer Learning
This book is mainly focused on using Stable Diffusion with Python, and when doing so, we will need to fine-tune a model for our specific needs. As we discussed in previous chapters, there are many ways to customize the model, such as the following:
- Unlocking UNet to fine-tune all parameters
- Training a textual inversion to add new keyword embeddings
- Locking UNet and training a LoRA model for customized styles
- Training a ControlNet model to guide image generation with control guidance
- Training an adaptor to use the image as one of the guidance embeddings
It is impossible to cover all the model training topics in simply one chapter. Another book would be needed to discuss the details of model training.
Nevertheless, we still want to use this chapter to drill down to the core concepts of model training. Instead of listing sample code on how to fine-tune a diffusion model, or using the scripts from the Diffusers package...
