Texture memory import API

[zmerp]

Is your feature request related to a problem? Please describe.
In my personal project I need to import texture memory into wgpu (Vulkan backend) from file descriptors on Linux, HANDLEs on Windows and AHardwareBuffer on Android. fd's and HANDLEs are for cross-process interop, AHardwareBuffer is for decoder interop.

Describe the solution you'd like
I have not designed the API yet.

Describe alternatives you've considered
Use the raw handles Vulkan API (initially authored by me). Having a texture memory import API integrated directly into wgpu would remove a lot of boilerplate.

[i509VCB]

I'd be willing to help implement this.

My expertise here is on Linux, I don't know how the situation would look on Android or Windows. On Linux the texture memory import/export object is a dmabuf.

I help develop Smithay, a library for writing Wayland compositors. Smithay heavily relies on texture memory import from dmabufs to implement fast hardware accelerated rendering. I would like to use wgpu as a renderer, but this is one of the big blockers that would make a wgpu renderer something that could be useful in a high performance setting.

Regarding this API, I think it would also make sense to support texture memory export as well.

Edit 1: should the user be responsible for format conversions or should wgpu handle drm format conversions?

API additions

In my opinion the texture memory import/export functionality is generally going to be platform dependent. I don't see any nice way to avoid this. This probably means each platform will need it's own codepath to support these features.

If there are any conventions for platform specific apis, please do mention these.

Technical details

Dmabuf import/export can be implemented in both EGL and Vulkan given that the requisite extensions are available:

For Vulkan the following extensions are required:

• VK_EXT_external_memory_dma_buf
• VK_EXT_image_drm_format_modifier (to get a list of valid DRM modifiers)

These Vulkan extensions are supported in the latest NVIDIA beta driver (515 at the time of writing) and generally in Mesa (anv, radv, v3dv (Vulkan 1.1)).

For EGL the following extensions are required:

• GL_OES_EGL_image (this would make import/export for an EGLImage possible if anyone would desire such behavior)
• EGL_EXT_image_dma_buf_import
• EGL_EXT_image_dma_buf_import_modifiers
• EGL_MESA_image_dma_buf_export

I believe NVIDIA implements these extensions from what I've heard. (I will edit the message when get a response from someone to run eglinfo for me Yes nvidia does support these extensions).

For Linux with Dmabufs I would expose an API something like the following (specifics pending bikeshed and testing the api):

pub struct PlaneDescriptor {
    pub fd: RawFd,
    pub offset: u32,
    pub stride: u32,
    pub modifier: u64,
}

pub struct DmabufDescriptor {
    plane_0: PlaneDescriptor,
    plane_1: Option<PlaneDescriptor>,
    // up to plane_3 (since a dmabuf has a max of 4 planes)
}

impl Device {
    // This should be able to fail, safety being a concern regarding plane file descriptors.
    // Also could just not be supported by the driver.
    pub unsafe fn import_dmabuf_texture(&self, &DmabufDescriptor) -> Result<Texture, ImportDmabufError> {
        todo!()
    }
}

And on Texture for export (could be Device if any architectural constraints prevent the code below):

impl Texture {
    // Similar failure reasons as above, but safe because we create the file descriptors.
    // Also a texture may have a format that cannot be represented as a dmabuf.
    pub fn export_dmabuf(&self) -> Result<Dmabuf, ExportDmabufError> {
        todo!()
    }
}

[i509VCB]

Texture export is going to be quite interesting to implement:

In Vulkan an image's underlying memory must be created with a flag that states what types of memory objects the texture may be exported as. (By extending VkAllocateMemoryInfo with VkExportMemoryAllocateInfo). When exporting an image in vulkan, we export the underlying memory of the texture. Therefore we need to state ahead of time the intention of exporting a texture.

There are a few instances of this requirement in the "Valid Usage" sections of the specification:

• VUID-VkMemoryGetFdInfoKHR-handleType-00671 (Linux Dmabuf export)
• VUID-VkMemoryGetWin32HandleInfoKHR-handleType-00662 (Win32 HANDLE export)
• VUID-VkMemoryGetAndroidHardwareBufferInfoANDROID-handleTypes-01882 (Android AHardwareBuffer export)

To properly support texture export we probably need one of the following:

1. If export is supported, create every texture with the capability to be exported. This has the problem of limiting what formats may be used since Android's format mappings are a subset of the supported Vulkan formats.
2. Allow specifying that a texture may be exported to a platform's memory handle at creation time. (I guess adding a new variant to TextureFormatFeatureFlags comes to mind)

I'd prefer the latter for the sake of being explicit.

For EGL with EGL_MESA_image_dma_buf_export whether a texture may be exported is entirely implementation dependent (no really the specification states nothing about this (https://www.khronos.org/registry/EGL/extensions/MESA/EGL_MESA_image_dma_buf_export.txt). The best we could do here is probably to try to export the image by getting the properties of the texture using ExportDMABUFImageQueryMESA before returning the Texture to the user.

I haven't seen any extensions outside of Linux for EGL and exporting texture memory, so EGL would only apply on Linux.

DX12 mentions something about creating the texture's memory in a shared heap, creating a shared handle (CreateSharedHandle) and opening the shared handle (OpenSharedHandleByName) either in another graphics API or process.

DX11 appears to be similar in that case as well.

If wgpu wants to support texture export as a first class part of the API, then we will need to consider the above.

[zmerp]

@i509VCB In case you want to export the memory of a texture I think the best route would be to create a separate method for creating the texture with export capabilities, and then export the memory with a method marked as unsafe, and probably exposing the wgpu-hal layer of abstraction first (like my raw Vulkan API) so that some code can be shared if the same functionality is to be implemented in other backends. I think what is important is to make those extension methods distinct from the rest of the safe API because they are not part of the WebGpu standard.

[mahkoh]

VK_EXT_image_drm_format_modifier

This extension is not available for AMD Polaris (GFX8) cards and below which are still widely used: https://gitlab.freedesktop.org/mesa/mesa/-/issues/5882

[i509VCB]

Further information, when creating an external image that is backed by a dmabuf and we did not create the underlying memory object, we need to transition the image from VK_QUEUE_FAMILY_FOREIGN_EXT to the wgpu queue and then back to VK_QUEUE_FAMILY_FOREIGN_EXT all foreign imported images.

This acquire and release is needed to transfer ownership of the memory object (likely via a DMA acquire in kernel) so Vulkan can use the memory object.

VK_QUEUE_FAMILY_EXTERNAL_KHR could be used but you have to make it unsafe since the caller would have to garuntee the image is from a device in EGL or Vulkan with the same deviceUUID and driverUUID.

I believe the simple solution would be to be to add additional arguments to allow specifying acquire and release barriers that could be recorded when a foreign image is used.

[i509VCB]

Progress update

To summary the work done, things are falling into place for some platforms to support texture memory import, assuming you import the memory for wgpu.

GL

#3046 exposes what is needed to create external textures for the GL backend on native platforms from GL texture and renderbuffer ids..

Linux

With this and the previous pull requests to expose internals, there should be enough to import EGLImages into the gles hal and then create a GL texture name and bind the EGLImage to it for non
YUV formats.

On Linux, this should all work once the pull request is merged. On Linux with Dmabufs, TEXTURE_EXTERNAL_OES is required for YUV formats.

On Mesa, even desktop GL supports this use case as GL_OES_EGL_image is enabled on Desktop GL by Mesa which could be useful in the future if the gles hal allows selecting GL 4.6 or whatever in the future.

For more info consult this Mesa issue:

• https://gitlab.freedesktop.org/mesa/mesa/-/issues/2119#note_315982

Android

Android however REQUIRES TEXTURE_EXTERNAL_OES. This is a work in progress task that needs to be implemented in Naga and then managed in wgpu-hal.

Vulkan

Once #3019 is merged, everything should be in place for the Vulkan hal to use externally imported textures. For Vulkan specifically, you will need to manually create the Instance and Device with the required extensions and the memory import extensions, and then create the wgpu::Instance and wgpu::Device.

This should in theory work on every platform that has Vulkan and external memory assuming you import the memory for wgpu.

[mahkoh]

TEXTURE_EXTERNAL_OES is required if you want to sample directly from YUV buffers, e.g. from buffers produced by vaapi. Otherwise you have to import every plane of the buffer as a separate texture and write a shader that uses a format-specific algorithm to combine those planes into an RGB image.

[i509VCB]

TEXTURE_EXTERNAL_OES is required if you want to sample directly from YUV buffers, e.g. from buffers produced by vaapi. Otherwise you have to import every plane of the buffer as a separate texture and write a shader that uses a format-specific algorithm to combine those planes into an RGB image.

Well Firefox on Linux is using TEXTURE_2D for YUV and I can't seem to find bugs from that.

[mahkoh]

Then firefox is using such a shader either directly or via a library.

The only exception is when importing directly as YUV. In that case, many users (e.g. Weston) still avoid using TEXTURE_EXTERNAL: instead of importing as YUV, they import the luma & chroma planes separately as R/RG/ARGB images bound to TEXTURE_2D and convert in a fragment shader.

From the very mesa issue you linked.

[i509VCB]

Ah, so YUV needs the special sampler then. I checked and firefox does indeed decompose the images into parts.

[jgcodes2020]

Came up on this issue; is it still planned or is it currently waiting on someone to implement?