Summary

In this chapter, we discussed how to enhance the Stable Diffusion model using LoRA, understood what LoRA is, and why it is good for fine-tuning and inference.

Then, we began loading LoRA using the experimental functions from the Diffusers package and provided LoRA weights through a custom implementation. We used simple code to quickly understand what LoRA can bring to the table.

Then, we dived into the internal structure of a LoRA model, walked through the detailed steps to extract LoRA weights, and understood how to merge those weights into the checkpoint model.

Further, we implemented a function in Python that can load a LoRA safetensors file and perform weight merges.

Finally, we briefly explored why LoRA works, based on the most recent papers from researchers.

In the next chapter, we are going to explore another powerful technique – textual inversion – to teach a model new “words,” and then use the pre-trained “words”...