Getting Vulkan Ready For VR

Introduction

Authors: James Jones, Mathias Schott

It’s a great time to be working in the field of graphics these days as there are a slew of new exciting technologies emerging, like Vulkan and VR. Naturally, the question frequently arises: how to combine those? (And not just because they share the capital letter.) However, as with many emerging technologies, there is often the “chicken-and-egg” problem between application developers and platform providers. For Vulkan and VR, this means that the VR HMD runtimes need to support Vulkan to make them interesting for Vulkan application developers. On the other hand, VR HMD vendors will only focus on adding Vulkan support to their HMD runtimes if there is application demand. Additional complexities arise because an HMD runtime is written in one API, but applications are written in any of a number of APIs. E.g., if the HMD runtime is written in Direct3D 11 and the application is written in Vulkan, then interoperability between Vulkan and Direct3D 11 is required.

Naturally, anything related to graphics APIs involves graphics-drivers, and this is where NVIDIA is contributing by collaborating with our partners in the VR ecosystem. We specifically explored the design space of interoperability between Vulkan and other APIs as well as sharing resources between the application process and the process the HMD runtime is running in. We are now shipping this functionality as a set of Vulkan extensions that allow cross-process resource sharing and interoperability between Vulkan and other APIs.

This solves both immediate needs of our partners, but more importantly is a first step towards a possible standardization via the established Khronos process, which preferably starts with practical experience from a wide range of ISVs and IHVs. This functionality has been standardized as the VK_KHR_external_* extension framework since the original publication of this article. See NVIDIA Vulkan Developer Driver for Khronos Vulkan Spec update 1.0.54for details.

The VK_NV_external_memory Extension Framework

These extensions enable developers to create applications which import Vulkan memory from another processes, as well as import memory from Direct3D images into Vulkan. Vulkan images can then be created using the imported memory, enabling both processes and APIs to access the same texture or render target. The extensions themselves are grouped into two categories. A set of platform-independent extensions define the core mechanisms. Platform-dependent extensions are then used to integrate those mechanisms with specific operating systems.

Note: Some of those are instance extensions and as such require support from the Vulkan loader.

Sample Code - Importing a Direct3D 11 Texture

The following sections highlight the key steps required for a Vulkan application to import an existing Direct3D 11 texture into a Vulkan instance for use as a render target.

Querying for Device Support

The first step would be to query the Vulkan driver to determine whether it supports the Direct3D 11 image handle type with the desired image properties.

 // Query the Vulkan driver for Direct3D image support.
 VkExternalMemoryHandleTypeFlagsNV handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_IMAGE_BIT_NV;
 VkResult res = vkGetPhysicalDeviceExternalImageFormatPropertiesNV(
    physicalDevice,
    VK_FORMAT_R8G8B8A8_UNORM, VK_IMAGE_TYPE_2D, VK_IMAGE_TILING_OPTIMAL, VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT,
    handleType,
    &extProperties);

Some platforms might not support an NT-handle for Direct3D sharing. In this case, try again with VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_IMAGE_KMT_BIT_NV.

 if (res == VK_ERROR_FORMAT_NOT_SUPPORTED) {
    handleType = VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_IMAGE_KMT_BIT_NV;
    VkResult res = vkGetPhysicalDeviceExternalImageFormatPropertiesNV(...);
    if (res == VK_ERROR_FORMAT_NOT_SUPPORTED) {
        FAIL();
    }
 }

After checking for basic support, the application should check the values in VkExternalImageFormatPropertiesNV to determine whether the desired operations are supported for the requested handle type. In this case, the application wants to import an existing object.

 if (!(extProperties.externalMemoryFeatures & VK_EXTERNAL_MEMORY_FEATURE_IMPORTABLE_BIT_NV)) {
    FAIL();
 }

Creating the Direct3D Export Texture

At some point, the application will need to create a texture and export it from Direct3D. This can be done in the same process as the Vulkan code, or in another process as long as the handle is properly shared between the processes.

 D3D11_TEXTURE2D_DESC descColor;
 memset(&descColor, 0, sizeof(descColor));
 descColor.Width = 512;
 descColor.Height = 512;
 descColor.MipLevels = 1;
 descColor.ArraySize = 1;
 descColor.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
 descColor.SampleDesc.Count = 1;
 descColor.SampleDesc.Quality = 0;
 descColor.Usage = D3D11_USAGE_DEFAULT;
 descColor.BindFlags = D3D11_BIND_RENDER_TARGET;
 descColor.CPUAccessFlags = 0;

 if (handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_IMAGE_BIT_NV) {
     descColor.MiscFlags = D3D11_RESOURCE_MISC_SHARED_NTHANDLE | D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
 } else {
     descColor.MiscFlags = D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX;
 }

 d3dDevice->CreateTexture2D(&descColor, 0, &d3dTexture);

 d3dTextureSharedHandle = 0;
 if (handleType == VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_IMAGE_BIT_NV) {
     IDXGIResource1* tempResource = NULL;
     d3dTexture->QueryInterface(__uuidof(IDXGIResource1), (void**)&tempResource);
     tempResource->CreateSharedHandle(NULL,
                                      DXGI_SHARED_RESOURCE_READ | DXGI_SHARED_RESOURCE_WRITE,
                                      NULL,
                                      &d3dTextureSharedHandle);
     tempResource->Release();
 } else {
     IDXGIResource* tempResource = NULL;
     d3dTexture->QueryInterface(__uuidof(IDXGIResource), (void**)&tempResource);
     tempResource->GetSharedHandle(&d3dTextureSharedHandle);
     tempResource->Release();
 }

Creating the Vulkan Import Image

After verifying the necessary driver capabilities are available, the application can create a Vulkan image with parameters corresponding to those of the Direct3D texture, and bind it to a Vulkan memory object created from the shared handle of the Direct3D texture. 

 VkExternalMemoryImageCreateInfoNV extMemoryImageInfo = {
     VK_STRUCTURE_TYPE_EXTERNAL_MEMORY_IMAGE_CREATE_INFO_NV, // sType
     0, // pNext
     handleType, // handleTypes
 };

Some implementations may require use of the VK_NV_dedicated_allocation extension when creating images from some handle types, as indicated by VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT_NV.

 VkDedicatedAllocationImageCreateInfoNV dedicatedImageCreateInfo = {
     VK_STRUCTURE_TYPE_DEDICATED_ALLOCATION_IMAGE_CREATE_INFO_NV, // sType
     0, // pNext
     VK_FALSE // dedicatedAllocation
 };

 if (extProperties.externalMemoryFeatures & VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT_NV) {
     externalMemoryImageInfo.pNext = &dedicatedImageCreateInfo;
     dedicatedImageCreateInfo.dedicatedAllocation = VK_TRUE;
 }

 VkImageCreateInfo imageCreateInfo = {
     VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO, // sType
     &externalMemoryImageInfo // pNext
 };

 imageCreateInfo.imageType = VK_IMAGE_TYPE_2D;
 imageCreateInfo.format = VK_FORMAT_R8G8B8A8_UNORM;
 iamgeCreateInfo.extent.width = 512;
 imageCreateInfo.extent.height = 512;
 imageCreateInfo.extent.depth = 1;
 imageCreateInfo.mipLevels = 1;
 imageCreateInfo.arrayLayers = 1;
 imageCreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
 imageCreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
 imageCreateInfo.usage = VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT;
 imageCreateInfo.flags = 0;
 imageCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;


 vkCreateImage(device, &imageCreateInfo, NULL, &image);

Binding Memory to the Vulkan Image

After creating the image, its memory requirements must be queried as usual to determine the supported memory types. The application can then use one of the supported memory types when importing the external handle to a Vulkan memory object. If a VK_NV_dedicated_allocation is not used, the memory must be bound to an image as usual.

 VkMemoryRequirements memReqs;
 vkGetImageMemoryRequirements(m_device, m_buffer[i].image, &memReqs);

 uint32_t memoryTypeIndex = getMemoryTypeIndex(memReqs.memoryTypeBits, VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT);

 VkImportMemoryWin32HandleInfoNV importMemInfoNV = { 
     VK_STRUCTURE_TYPE_IMPORT_MEMORY_WIN32_HANDLE_INFO_NV, // sType
     0, // pNext
     handleType, // handleTypes
     d3dTextureSharedHandle // handle
 };

 VkMemoryAllocateInfo memInfo = {
     VK_STRUCTURE_TYPE_MEMORY_ALLOCATE_INFO, // sType
     &importMemInfoNV, // pNext
     memReqs.size, // allocationSize
     memoryTypeIndex, // memoryTypeIndex
 };

 VkDedicatedAllocationMemoryAllocateInfoNV dedicatedAllocationInfo = {
     VK_STRUCTURE_TYPE_DEDICATED_ALLOCATION_MEMORY_ALLOCATE_INFO_NV // sType
 };

 if (extProperties.externalMemoryFeatures & VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT_NV) {
     dedicatedAllocationInfo.image = image;
     importMemInfo.pNext = &dedicatedAllocationInfo;
 }

 vkAllocateMemory(device, &memInfo, 0, &mem);

 if (!(extProperties.externalMemoryFeatures & VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT_NV)) {
     vkBindImageMemory(device, image, mem, 0);
 }

Note: The memory types may differ from those reported when an external handle type is not specified.

Synchronizing Vulkan Queue Access via Keyed Mutex

Finally, when using the image, Vulkan queue operations can be synchronized using the keyed mutex associated with the Direct3D shared texture. The keyed mutex is referenced using the Vulkan handle of the memory object imported from the Direct3D shared texture:

 VkSubmitInfo submitInfo = {
     VK_STRUCTURE_TYPE_SUBMIT_INFO, // sType
     0 // pNext
 };

 submitInfo.commandBufferCount = 1;
 submitInfo.pCommandBuffers = cmdBuf;


 VkWin32KeyedMutexAcquireReleaseInfoNV keyedMutex = { 
     VK_STRUCTURE_TYPE_WIN32_KEYED_MUTEX_ACQUIRE_RELEASE_INFO_NV, // sType
     0, // pNext
     1, // acquireCount
     &mem, // pAcquireSyncs
     &acquireKey, // pAcquireKeys
     &timeout, // pAcquireTimeoutMilliseconds
     1, // releaseCount
     &mem, // pReleaseSyncs
     &releaseKey // pReleaseKeys
 };

 submitInfo.pNext = &keyedMutex;
 vkQueueSubmit(queue, 1, &submitInfo, fence);

References

Software

Specifications And Documentation