Using multiple ControlNets in one pipeline

In this section, we will initialize one more ControlNet, NormalBAE, and then feed the Canny and NormalBAE ControlNet models together to form a pipeline.

Let’s generate a Normal BAE as one additional control image. Normal BAE is a model that’s used to estimate a normal map using the normal uncertainty method [4] proposed by Bae et al:

from controlnet_aux import NormalBaeDetector
normal_bae = \
    NormalBaeDetector.from_pretrained("lllyasviel/Annotators")
image_canny = normal_bae(image)
image_canny

This code will generate the original image’s Normal BAE map, as shown in Figure 13.4:

Figure 13.4: Normal BAE image of the generated cat

Now, let’s initialize two ControlNet models for one pipeline: one Canny ControlNet model, and another NormalBae ControlNet model:

from diffusers import ControlNetModel
canny_controlnet = ControlNetModel.from_pretrained...