Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux texture support #24916

Merged
merged 27 commits into from
Sep 23, 2021
Merged

Conversation

huanghongxun
Copy link
Contributor

@huanghongxun huanghongxun commented Mar 10, 2021

Add support for external textures to the Linux platform shell.

Related issues

Fixes flutter/flutter#64188.

Design

For API, I introduce an abstract class FlTexture for abstration of different OpenGL textures, allowing any other textures to be supported in the future.
Subclass FlPixelBufferTexture support loading textures from given pixel buffers with RGBA format. For other formats, or textures not from pixel buffers, developer may implement their own texture class.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
  • I listed at least one issue that this PR fixes in the description above.
  • I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on
    writing and running engine tests.
  • I updated/added relevant documentation (doc comments with ///).
  • I signed the CLA.
  • All existing and new tests are passing.
  • The reviewer has submitted any presubmit flakes in this PR using the engine presubmit flakes form before re-triggering the failure.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@google-cla google-cla bot added the cla: yes label Mar 10, 2021
@huanghongxun huanghongxun mentioned this pull request Mar 10, 2021
10 tasks
@stuartmorgan stuartmorgan requested review from robert-ancell, stuartmorgan and cbracken and removed request for robert-ancell and stuartmorgan March 10, 2021 15:50
@stuartmorgan
Copy link
Contributor

@cbracken See #20714 and #19405 for design discussion context.

@anirudhb
Copy link
Contributor

Has anything changed from the original PR?

@huanghongxun
Copy link
Contributor Author

Has anything changed from the original PR?

In one word, FlExternalTextureGL has been replaced with an abstract class FlTexture for developer customization and future support, as well as an official implementation FlPixelBufferTexture.

Also code is ported to be compatible with master branch. Comments are refined.

@dnsoumik
Copy link

This is great hope this time it will get merged 👍 I Appreciate all the hard work everyone is doing :)

@escamoteur
Copy link

@huanghongxun Amazing to see progress here. I currently work on https://github.com/FlutterGL/flutter_web_gl
And am waiting for Linux support. Could you tell me how your Testure support works? I use OpenGL ES from the Angle framework internally if there would be a way to avaoid an additional copying of the image data this would be amazing

@huanghongxun
Copy link
Contributor Author

huanghongxun commented Mar 11, 2021

@escamoteur
FlTexture::populate_texture returns a texture id. So subclassing FlTexture and rewriting populate_texture to return your texture id should prevent from copying data. Please be careful of OpenGL context. You may find a way to share Flutters' context.

@escamoteur
Copy link

But that's inside the Flutter Engine

@huanghongxun
Copy link
Contributor Author

huanghongxun commented Mar 11, 2021

@escamoteur egl provides eglGetCurrentContext which can be used for creating shared context.

GdkWindow* window = gtk_widget_get_parent_window(#FlView)
GdkGLContext* context = gdk_window_create_gl_context(window, error);

above code should create a context shared with the one Flutter is using.

@escamoteur
Copy link

@huanghongxun lets have a more detailed chat when this is merged and on master :-)

@RossComputerGuy
Copy link
Contributor

Will this allow transparent windows (or at least a transparent FlView)?

@huanghongxun
Copy link
Contributor Author

Will this allow transparent windows (or at least a transparent FlView)?

Already supported, but not in this PR. The only thing to do is to set app paintable for FlView.

@TimWhiting
Copy link
Contributor

Hooray! Thanks @huanghongxun for the patience and dedication and thanks for the reviews and merge flutter team!

@HedgeHao
Copy link

Is there any tutorial about how to use this texture on Linux? Does that mean I can write a native OpenGL function render the image then display in flutter texture widget?

@huanghongxun
Copy link
Contributor Author

Is there any tutorial about how to use this texture on Linux? Does that mean I can write a native OpenGL function render the image then display in flutter texture widget?

You can render your texture in Flutter's OpenGL context with hardware acceleration.
We have detailed implementation guide in comments. Please read them.
Also, an example: https://github.com/lucaszanella/external_texture_flutter_linux/blob/master/external_texture/linux/external_texture_plugin.cc#L40

@lattice0
Copy link

@HedgeHao the repo linked by @huanghongxun is mine. It uses a version that is a little older than the one merged, but it should amost work. If you get it working, I'd appreciate a PR, as I have no time right now

@HedgeHao
Copy link

HedgeHao commented Oct 23, 2021

I follow this repo and try to use PixelBufferTexture on my own but texture widget cannot render the data in buffer correctly.

Here is my version.

else if (strcmp(method, "registerTexture") == 0)
  {
    self->myTextxure = MY_TEXTURE(g_object_new(my_texture_get_type(), nullptr));
    FL_PIXEL_BUFFER_TEXTURE_GET_CLASS(self->myTextxure)->copy_pixels = my_texture_copy_pixels;
    fl_texture_registrar_register_texture(self->texture_registrar, FL_TEXTURE(self->myTextxure));
    MY_TEXTURE_GET_CLASS(self->myTextxure)->texture_id = reinterpret_cast<int64_t>(FL_TEXTURE(self->myTextxure));
    fl_texture_registrar_mark_texture_frame_available(self->texture_registrar, FL_TEXTURE(self->myTextxure));
    response = FL_METHOD_RESPONSE(fl_method_success_response_new(fl_value_new_int(MY_TEXTURE_GET_CLASS(self->myTextxure)->texture_id)));
  }
  else if (strcmp(method, "nativeClick") == 0)
  {
    int len = 300 * 300 * 4;
    uint8_t buffer[len];
    for (int i = 0; i < len; i++)
    {
      buffer[i] = 127;
    }
    MY_TEXTURE_GET_CLASS(self->myTextxure)->buffer = buffer;
    fl_texture_registrar_mark_texture_frame_available(self->texture_registrar, FL_TEXTURE(self->myTextxure));
    response = FL_METHOD_RESPONSE(fl_method_success_response_new(nullptr));
  }

I except after registerTexture, each time when I call nativeClick some data will set to the buffer and texture widget in Flutter will refresh. Here's a screenshot. I set all the bytes in buffer to constant 127. There's some color dots at the bottom of the widget and it did change a little bit when I click every time. But it's obviously wrong since it should all be 127.

Can someone please help? Thanks a lot!

@alexmercerind dart_vlc is such a great work! Wonder if you could help me out with this.

@alexmercerind
Copy link

alexmercerind commented Oct 23, 2021

@HedgeHao I'm happy dart_vlc could serve as an example, only difference is that I used G_DECLARE_DERIVABLE_TYPE instead of G_DECLARE_FINAL_TYPE (as in the tests of this PR) since I wanted to store some additional data like dimensions & id (and wanted to access it within frame callbacks aswell) etc.

I can't help out much, however I think that the buffer (whose pointer you are assigning) gets deallocated as soon as the flutter_hello_plugin_handle_method_call call ends. Maybe store it somewhere else or pass the ownership (std::move on std::unique_ptr? I don't know much about GObject style but standard C++).

@lattice0
Copy link

I'm willing to try the merged code, but I couldn't compile my code in the dev channel. Anyone knows when this will land on any of the channels?

@HedgeHao
Copy link

@LucasZanella I open a PR. It should work on the latest flutter version on master channel

Then I have a further question. Instead of create a pixel buffer directly. I try to use openGL to draw a triangle and get the pixel buffer data then create a texture with that buffer. I thought I can have a triangle render in flutter texture widget but it doesn't work. The code is here. The code work great with GLUT window but not in Flutter. Any ideas?

@lattice0
Copy link

lattice0 commented Oct 24, 2021

Thanks @HedgeHao, PR merged.

@lattice0
Copy link

Anyone knows if direct GPU texture is supported yet? Or does it still copy from GPU to CPU and to GPU back again?

@charafau
Copy link
Contributor

@lattice0 someone please correct me if I'm wrong but I think it still copies from GPU to CPU and to GPU. Engine doesn't support that yet but there is an issue for direct GPU texture support

@alexmercerind
Copy link

Anyone knows if direct GPU texture is supported yet? Or does it still copy from GPU to CPU and to GPU back again?

I kindly request the answer to the same question (same for macOS).

Hardware accelerated rendering is a big concern. Pixel buffers are just not good.

If the GL texture in this PR also copies the frame to an intermediate CPU RAM buffer internally, what's the point???

The decodeImageFromPixels from dart:ui already supports rendering such pixel buffers without any platform specific care. Besides a performant zero-copy open-gl (for Linux) & METAL (for macOS) texture API, there's no point. It's a must.

Playing a 1080p video through pixel buffers will cramp the CPU, redundantly drain the battery.
4K videos won't even play due to buffer copy latency. All this while knowing that a better solution exists.

Please consider. Thankyou!

@knopp
Copy link
Member

knopp commented Dec 1, 2022

I haven't tested but the linux texture takes GL Texture name and target. Why would you assume it's copied to CPU and then GPU?

On macOS the texture is backed by IOSurface, which is a GPU buffer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes waiting for tree to go green This PR is approved and tested, but waiting for the tree to be green to land.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Texture widget in Linux embedding