Diving into the internal structure of LoRA

Understanding how LoRA works internally will help us to implement our own LoRA-related features based on specific needs. In this section, we are going to dive into the internals of LoRA’s structure and its weights schema, and then manually load the LoRA model into the Stable Diffusion model step by step.

As we discussed at the beginning of the chapter, applying LoRA is as simple as the following:

W′= W + αΔW

And ΔW can be broken down into A and B:

ΔW = A B T

So, the overall idea of merging LoRA weights to the checkpoint model works like this:

1. Find the A and B weight matrix from the LoRA file.
2. Match the LoRA module layer name to the checkpoint module layer name so that we know which matrix to merge.
3. Produce ΔW = A B T.
4. Update the checkpoint model weights.

If you have prior experience training a LoRA model, you might be aware that a hyperparameter...