Handle_cast not working using mResourceAllocator; offscreen rendering and VkImage, VkImageView

[gamerx1221]

I am attempting to get Skia and Filament working together; I have been modifying the source code slightly here and there and making progress to understand how the modules and mechanisms are engineered. Super appreciate the interesting programming, lots of great security practices!

I was able to get something working with swapchain custom backend context being passed in, though ran into an issue with accessing Semaphores of drawing completion on filament end. I did a search and decided to try offscreen render textures as that seems a great long term and simplified solution.

I am able to create a render target and assign it a texture. What is the best way to get the underlying VkImage or resource ... originally I thought downcast<> would solve the issue, though it still requires an addition getHandle to get to the Vulkan implementation of that resource. I modified some source to get access to the mResourceAllocator to handle_cast<> as I seen in the DriverApi, VulkanDriver implementation.

The goal being to render the offscreen render target using a separate external vulkan implementation. Like I mentioned previously I have been able to have it working when rendering directly to the swapchains that are being passed using a custom backend context, though would like to learn as much as I can.

VulkanTexture* vulkanTexture = vulkan_driver->mResourceAllocator.handle_cast<VulkanTexture*>(texture_->getHwHandle());

This returns a vulkanTexture incorrectly; I also tried HwTexture*. I verified the vulkan_driver is correct and so is the mResourceAllocator it points to.

Is there a system way of using a handle_cast?? Also, my understanding of how mResourceAllocator is working with the handle id, led me to verify that the VulkanResources in this case, VulkanTextures are being created in VulkanDriver.

any help is appreciated, I hope I can learn enough to start contributing down the road, so excuse me if my questions are out of line or not clear.

[gamerx1221]

I also implemented a function directly in the VulkanDriver.cpp file using DriverApi.inc definition in case the stream command processor had some background mechanisms or whatnot, like it needed to be used in order to get the object pointer from handle ID.

`void VulkanDriver::getNativeTextureFromHandle(HwTexture* nativeTexture, backend::TextureHandle textureHandle) {

VulkanTexture* returnTexture = mResourceAllocator.handle_cast<VulkanTexture*>(textureHandle);
nativeTexture = (HwTexture*)returnTexture;
printf("get texture\n");

}
`
please ignore the "debugging" function design, the 1st line returns a not correct VulkanTexture pointer. how am I supposed to use that correctly? does the the engine have to be in a specific state, like in between a begin() or end() API call, which I have tried as well to same effect

[gamerx1221]

currently i'm keeping a map of textures in VulkanDriver.cpp that I can access correctly from the application facing side, using the handleId; I"m guessing this is what mResourceAllocator accomplishes, but wasn't getting the correct texture between beginFrame, endFrame
if resources are created and deallocated every frame, then will have to update that process.

[poweifeng]

I am attempting to get Skia and Filament working together; I have been modifying the source code slightly here and there and making progress to understand how the modules and mechanisms are engineered. Super appreciate the interesting programming, lots of great security practices!

I was able to get something working with swapchain custom backend context being passed in, though ran into an issue with accessing Semaphores of drawing completion on filament end. I did a search and decided to try offscreen render textures as that seems a great long term and simplified solution.

I am able to create a render target and assign it a texture. What is the best way to get the underlying VkImage or resource ... originally I thought downcast<> would solve the issue, though it still requires an addition getHandle to get to the Vulkan implementation of that resource. I modified some source to get access to the mResourceAllocator to handle_cast<> as I seen in the DriverApi, VulkanDriver implementation.

I think the custom "platform/swapchain" is your best bet here. We don't really expose the VkImage to the "frontend" (i.e. client API).

What issue did you run into with the semaphores? It could be something we could address. (Whereas passing VkImage through to the client is much less likely to be a formal API).

[gamerx1221]

well the issue was there wasn't access to the semaphore when drawing complete by filament. I think by implementing the "import" functionality in Vulkan, that would be useful also, because I think having access to the render target texture would be easier to implement for applications across platforms opengl, vulkan, dx, or metal, ideally having access to the semaphore and underlying render target texture would be best. I would love to help with the effort. Currently i'm just modifying library for private project use. do you plan on implementing the "import" function in the vulkan render target feature, or is that API headed in a different direction?

I have the external swapchain working, like I was saying I just need access to the final submit semaphore. any help is appreciated.

[gamerx1221]

I remember when I first started trying to get the semaphore, it was connected to the VulkanCommands object... has something to do with gc and wait(), is that the best way to get access to it? perhaps develop an API around that for use with the external swapchain and vulkan backend context.

[poweifeng]

Have you tried override VulkanPlatform::present:

https://github.com/google/filament/blob/main/filament/backend/include/backend/platforms/VulkanPlatform.h#L193

It's called with a semaphore that indicates when the drawing completes.

[gamerx1221]

I haven't thought to override that functionality yet, as I was looking for an out of box way of doing it... that looks like a good idea! My original approach was to use an external vulkan present, like neither filament or skia, though it might be better to use one of them for the present. Each API has its own present() function, so I was using my own and trying to do things that way. I'll try to override the filament present() in the VulkanPlatform like you mentioned.

[nmfisher]

@poweifeng I'd like to ask a similar question - do you plan to add Vulkan support for Texture::Builder::import?

@aguerrieri82 provided a sample implementation at #7617, but there was never any follow up. Is this feasible?

[poweifeng]

There is currently a plan to support Texture::Builder::import for AHardwareBuffer (the Android generic hardware buffer), but there's no plan beyond that (e.g. no plans for import directx images into filament). Would you mind specifying what kind of external textures you'd like to import?

[nmfisher]

This is specifically to use external textures as render targets. To integrate with Flutter, currently on Android/iOS/MacOS I am creating a hardware-backed texture (glTexture/CVMetalTexture/etc) that can be imported into Filament and used as the color attachment for a RenderTarget. The texture is then imported as-is into Flutter for compositing with other layers.

This doesn't work for OpenGL on Windows, because Flutter on Windows uses EGL/D3D. However, I can use Vulkan to render into the texture by creating a VkImage that wraps the D3D texture (using VK_EXTERNAL_MEMORY_HANDLE_TYPE_D3D11_TEXTURE_BIT). Currently, though, there's no way of importing that into Filament as a render target. It seems the only options are (a) create a custom Vulkan Platform implementation that does somethign similar, or (b) create a headless swapchain manually and blit every frame to the D3D texture. If there's another alternative, please let me know.

[aguerrieri82]

Yep is feasible. I can help you out to integrate it in the current base code, as well help in build the library. Unfortunately I never received a feedback, and I hate to branch out and maintain my version. But it is what it is :)