Summary

In this chapter, we introduced a way to precisely control image generation using SD ControlNets. From the detailed samples we have provided, you can start using one or multiple ControlNet models with SD v1.5 and also SDXL.

We also drilled down into the internals of ControlNet, explaining how it works in a nutshell.

We can use ControlNet in lots of applications, including applying a style to an image, applying a shape to an image, merging two images into one, and generating a human body using a posed image. It is powerful and amazingly useful in many ways. Our imagination is the only limitation.

However, there is one other limitation: it is hard to align the background and overall context between two generations (with different seeds). You may want to use ControlNet to generate a video from the extracted frames from a source video, but the results are still not ideal.

In the next chapter, we will cover a solution to generate video and animation using SD.