Transferring resources between queue families

In this recipe, we will demonstrate how to transfer resources between queue families by uploading textures to a device from the CPU using a transfer queue and generating mip-level data in a graphics queue. Generating mip levels needs a graphics queue because it utilizes vkCmdBlitImage, supported only by graphics queues.

Getting ready

An example is provided in the repository in chapter2/mainMultiDrawIndirect.cpp, which uses the EngineCore::AsyncDataUploader class to perform texture upload and mipmap generation on different queues.

How to do it…

In the following diagram, we illustrate the procedure of uploading texture through a transfer queue, followed by the utilization of a graphics queue for mip generation:

Figure 2.14 – Recoding and submitting commands from different threads and transferring a resource between queues from different families

The process can be summarized as follows:

1. Record the commands to upload the texture to the device and add a barrier to release the texture from the transfer queue using the VkDependencyInfo and VkImageMemoryBarrier2 structures, specifying the source queue family as the family of the transfer queue and the destination queue family as the family of the graphics queue.
2. Create a semaphore and use it to signal when the command buffer finishes, and attach it to the submission of the command buffer.
3. Create a command buffer for generating mip levels and add a barrier to acquire the texture from the transfer queue into the graphics queue using the VkDependencyInfo and VkImageMemoryBarrier2 structures.
4. Attach the semaphore created in step 2 to the SubmitInfo structure when submitting the command buffer for processing. The semaphore will be signaled when the first command buffer has completed, allowing the mip-level-generation command buffer to start.

   Two auxiliary methods will help us create acquire and release barriers for a texture. They exist in the VulkanCore::Texture class. The first one creates an acquire barrier:

   void Texture::addAcquireBarrier(
       VkCommandBuffer cmdBuffer,
       uint32_t srcQueueFamilyIndex,
       uint32_t dstQueueFamilyIndex) {
     VkImageMemoryBarrier2 acquireBarrier = {
         .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2,
         .dstStageMask =
             VK_PIPELINE_STAGE_2_FRAGMENT_SHADER_BIT,
         .dstAccessMask = VK_ACCESS_2_MEMORY_READ_BIT,
         .srcQueueFamilyIndex = srcQueueFamilyIndex,
         .dstQueueFamilyIndex = dstQueueFamilyIndex,
         .image = image_,
         .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT,
                              0, mipLevels_, 0, 1},
     };
     VkDependencyInfo dependency_info{
         .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO,
         .imageMemoryBarrierCount = 1,
         .pImageMemoryBarriers = &acquireBarrier,
     };
     vkCmdPipelineBarrier2(cmdBuffer, &dependency_info);
   }

   Besides the command buffer, this function requires the indices of the source and destination family queues. It also assumes a few things, such as the subresource range spanning the entire image.

5. Another method records the release barrier:

   void Texture::addReleaseBarrier(
       VkCommandBuffer cmdBuffer,
       uint32_t srcQueueFamilyIndex,
       uint32_t dstQueueFamilyIndex) {
     VkImageMemoryBarrier2 releaseBarrier = {
         .sType = VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER_2,
         .srcStageMask = VK_PIPELINE_STAGE_2_TRANSFER_BIT,
         .srcAccessMask = VK_ACCESS_TRANSFER_WRITE_BIT,
         .dstAccessMask = VK_ACCESS_SHADER_READ_BIT,
         .srcQueueFamilyIndex = srcQueueFamilyIndex,
         .dstQueueFamilyIndex = dstQueueFamilyIndex,
         .image = image_,
         .subresourceRange = {VK_IMAGE_ASPECT_COLOR_BIT,
                              0, mipLevels_, 0, 1},
     };
     VkDependencyInfo dependency_info{
         .sType = VK_STRUCTURE_TYPE_DEPENDENCY_INFO,
         .imageMemoryBarrierCount = 1,
         .pImageMemoryBarriers = &releaseBarrier,
     };
     vkCmdPipelineBarrier2(cmdBuffer, &dependency_info);
   }

   This method makes the same assumptions as the previous one. The main differences are the source and destination stages and access masks.

6. To perform the upload and mipmap generation, we create two instances of VulkanCore::CommandQueueManager, one for the transfer queue and another for the graphics queue:

   auto transferQueueMgr =
       context.createTransferCommandQueue(
           1, 1, "transfer queue");
   auto graphicsQueueMgr =
       context.createGraphicsCommandQueue(
           1, 1, "graphics queue");

7. With valid VulkanCore::Context and VulkanCore::Texture instances in hand, we can upload the texture by retrieving a command buffer from the transfer family. We also create a staging buffer for transferring the texture data to device-local memory:

   VulkanCore::Context context;  // Valid Context
   std::shared_ptr<VulkanCore::Texture>
       texture;        // Valid Texture
   void* textureData;  // Valid texture data
   // Upload texture
   auto textureUploadStagingBuffer =
       context.createStagingBuffer(
           texture->vkDeviceSize(),
           VK_BUFFER_USAGE_TRANSFER_SRC_BIT,
           "texture upload staging buffer");
   const auto commandBuffer =
       transferQueueMgr.getCmdBufferToBegin();
   texture->uploadOnly(commandBuffer,
                       textureUploadStagingBuffer.get(),
                       textureData);
   texture->addReleaseBarrier(
       commandBuffer,
       transferQueueMgr.queueFamilyIndex(),
       graphicsQueueMgr.queueFamilyIndex());
   transferQueueMgr.endCmdBuffer(commandBuffer);
   transferQueueMgr.disposeWhenSubmitCompletes(
       std::move(textureUploadStagingBuffer));

8. For submitting the command buffer for processing, we create a semaphore to synchronize the upload command buffer and the one used for generating mipmaps:

   VkSemaphore graphicsSemaphore;
   const VkSemaphoreCreateInfo semaphoreInfo{
       .sType = VK_STRUCTURE_TYPE_SEMAPHORE_CREATE_INFO,
   };
   VK_CHECK(vkCreateSemaphore(context.device(),
                               &semaphoreInfo, nullptr,
                               &graphicsSemaphore));
   VkPipelineStageFlags flags =
       VK_PIPELINE_STAGE_TRANSFER_BIT;
   auto submitInfo =
       context.swapchain()->createSubmitInfo(
           &commandBuffer, &flags, false, false);
   submitInfo.signalSemaphoreCount = 1;
   submitInfo.pSignalSemaphores = &graphicsSemaphore;
   transferQueueMgr.submit(&submitInfo);

9. The next step is to acquire a new command buffer from the graphics queue family for generating mipmaps. We also create an acquire barrier and reuse the semaphore from the previous command buffer submission:

   // Generate mip levels
   auto commandBuffer =
       graphicsQueueMgr.getCmdBufferToBegin();
   texture->addAcquireBarrier(
       commandBuffer,
       transferCommandQueueMgr_.queueFamilyIndex(),
       graphicsQueueMgr.queueFamilyIndex());
   texture->generateMips(commandBuffer);
   graphicsQueueMgr.endCmdBuffer(commandBuffer);
   VkPipelineStageFlags flags =
       VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT;
   auto submitInfo =
       context_.swapchain()->createSubmitInfo(
           &commandBuffer, &flags, false, false);
   submitInfo.pWaitSemaphores = &graphicsSemaphore;
   submitInfo.waitSemaphoreCount = 1;

In this chapter, we have navigated the complex landscape of advanced Vulkan programming, building upon the foundational concepts introduced earlier. Our journey encompassed a diverse range of topics, each contributing crucial insights to the realm of high-performance graphics applications. From mastering Vulkan’s intricate memory model and efficient allocation techniques to harnessing the power of the VMA library, we’ve equipped ourselves with the tools to optimize memory management. We explored the creation and manipulation of buffers and images, uncovering strategies for seamless data uploads, staging buffers, and ring-buffer implementations that circumvent data races. The utilization of pipeline barriers to synchronize data access was demystified, while techniques for rendering pipelines, shader customization via specialization constants, and cutting-edge rendering methodologies such as PVP and MDI were embraced. Additionally, we ventured into dynamic rendering approaches without relying on render passes and addressed the intricacies of resource handling across multiple threads and queues. With these profound understandings, you are primed to create graphics applications that harmonize technical prowess with artistic vision using the Vulkan API.