Generative Fill with Adobe Firefly (Part I)

Adobe Firefly AI Overview 

Adobe Firefly is a new set of generative AI tools that can be accessed via https://firefly.adobe.com/ by anyone with an Adobe ID. To learn more about Firefly… have a look at their FAQ.   

Image 1: Adobe Firefly 

For more information about the usage of Firefly to generate images, text effects, and more… have a look at the previous articles in this series:  

• Animating Adobe Firefly Content with Adobe Animate  
• Exploring Text to Image with Adobe Firefly  
• Generating Text Effects with Adobe Firefly  
• Adobe Firefly Feature Deep Dive 

In the next two articles, we’ll continue our exploration of Firefly with the Generative fill module. We’ll begin with an overview of accessing Generative fill from a generated image and then explore how to use the module on our own personal images.  

Recall from a previous article Exploring Text to Image with Adobe Firefly that when you hover your mouse cursor over a generated image – overlay controls will appear. 

Image 2: Generative fill overlay control from Text to image  

One of the controls in the upper right of the image frame will invoke the Generative fill module and pass the generated image into that view.  

Image 3: The generated image is sent to the Generative fill module 

Within the Generative fill module, you can use any of the tools and workflows that are available when invoking Generative fill from the Firefly website. The only difference is that you are passing in a generated image rather than uploading an image from your local hard drive.  

Keep this in mind as we continue to explore the basics of Generative fill in Firefly – as we’ll begin the process from scratch. 

Generative Fill 

When you first enter the Firefly web experience, you will be presented with the various workflows available.  

These appear as UI cards and present a sample image, the name of the procedure, a procedure description, and either a button to begin the process or a label stating that it is “in exploration”. Those which are in exploration are not yet available to general users. 

We want to locate the Generative fill module and click the Generate button to enter the experience.  

Image 4: The Generative fill module card 

From there, you’ll be taken to a view that prompts you to upload an image into the module. Firefly also presents a set of sample images you can load into the experience.   

Image 5: Generative fill getting started promptly 

Clicking the Upload image button summons a file browser for you to locate the file you want to use Generative fill on. In my example, I’ll be using a photograph of my cat, Poe. You can download the photograph of Poe [[ NOTE – LINK TO FILE Poe.jpg ]] to work with as well.  

 Image 6: The photograph of Poe, a cat 

Once the image file has been uploaded into Firefly, you will be taken to the Generative fill user experience and the photograph will be visible. Note that this is exactly the same experience as when entering Generative fill from a prompt-generated image as we saw above. The only real difference is how we get to this point.  

Image 7: The photograph is loaded into Generative fill 

You will note that there are two sets of tools available within the experience. One set is along the left side of the screen and includes Insert, Remove, and Pan tools.  

Image 8: Insert, Remove, and Pan 

Switching between the Insert and Remove tools changes the function of the current process. The Pan tool allows you to pan the image around the view.  

Along the bottom of the screen is the second set of tools – which are focused on selections. This set contains the Add and Subtract tools, access to Brush Settings, a Background removal process, and a selection Invert toggle.  

Image 9: Add, Subtract, Brush Settings, Background removal, and selection Invert 

Let’s perform some Generative fill work on the photograph of Poe.  

1. In the larger overlay along the bottom of the view, locate and click the Background option. This is an automated process that will detect and remove the background from the image loaded into Firefly.  

Image 10: The background is removed from the selected photograph 

2. A prompt input appears directly beneath the photograph. Type in the following prompt: “a quiet jungle at night with lots of mist and moonlight” 

Image 11: Entering a prompt into the prompt input control 

3. If desired, you can view and adjust the settings for the generative AI by clicking the Settings icon in the prompt input control. This summons the Settings overlay. 

Image 12: The generative AI Settings overlay 

Within the Settings overlay, you will find there are three items that can be adjusted to influence the AI:  

• Match shape: You have two choices here – freeform or conform.  
• Preserve content: A slider that can be set to include more of the original content or produce new content. 
• Guidance strength: A slider that can be set to provide more strength to the original image or the given prompt. 

I suggest leaving these at the default setting for now. 

4. Click the Settings icon again to dismiss the overlay. 

5. Click the Generate button to generate a background based upon the entered prompt. 

A new background is generated from our prompt, and it now appears as though Poe is visiting a lush jungle at night.  

Image 13: Poe enjoying the jungle at night 

Note that the original photograph included a set of electric outlets exposed within the wall. When we removed the background, Firefly recognized that they were distinct from the general background and so retained them. The AI has taken them into account when generating the new background and has interestingly propped them up with a couple of sticks. It also has gone through and rendered a realistic shadow cast by Poe.  

1. Before moving on, click the Cancel button to bring the transparent background back. Clicking the Keep button will commit the changes – and we do not want that as we wish to continue exploring other options. 
2. Clear out the prompt you previously wrote within the prompt input control so that there is no longer any prompt present.  

Image 14: Click the Generate button with no prompt present 

3. Click the Generate button without a text prompt in place. 

The photograph receives a different background from the one generated with a text prompt. When clicking the Generate button with no text prompt, you are basically allowing the Firefly AI to make all the decisions based solely on the visual properties of the image.  

Image 15: A set of backgrounds is generated based on the remaining pixels present 

You can select any of the four variations that were generated from the set of preview thumbnails beneath the photograph. If you’d like Firefly to generate more variations – click the More button. Select the one you like best and click the Keep button. 

Okay! That’s pretty good but we are not done with Generative fill yet. We haven’t even touched the Insert and Remove functions… and there are Brush Settings to manipulate… and much more. 

In the next article, we’ll explore the remaining Generative fill tools and options to further manipulate the photograph of Poe.  

Author Bio

Joseph Labrecque is a Teaching Assistant Professor, Instructor of Technology, University of Colorado Boulder / Adobe Education Leader / Partner by Design

Joseph is a creative developer, designer, and educator with nearly two decades of experience creating expressive web, desktop, and mobile solutions. He joined the University of Colorado Boulder College of Media, Communication, and Information as faculty with the Department of Advertising, Public Relations, and Media Design in Autumn 2019. His teaching focuses on creative software, digital workflows, user interaction, and design principles and concepts. Before joining the faculty at CU Boulder, he was associated with the University of Denver as adjunct faculty and as a senior interactive software engineer, user interface developer, and digital media designer.

Labrecque has authored a number of books and video course publications on design and development technologies, tools, and concepts through publishers which include LinkedIn Learning (Lynda.com), Peachpit Press, and Adobe. He has spoken at large design and technology conferences such as Adobe MAX and for a variety of smaller creative communities. He is also the founder of Fractured Vision Media, LLC; a digital media production studio and distribution vehicle for a variety of creative works.

Joseph is an Adobe Education Leader and member of Adobe Partners by Design. He holds a bachelor’s degree in communication from Worcester State University and a master’s degree in digital media studies from the University of Denver.

Author of the book: Mastering Adobe Animate 2023