-
Notifications
You must be signed in to change notification settings - Fork 6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add texture support for macOS shell. #8507
Merged
Merged
Changes from 37 commits
Commits
Show all changes
40 commits
Select commit
Hold shift + click to select a range
5302cf8
Merge pull request #1 from flutter/master
cloudwebrtc 2953e13
Merge remote-tracking branch 'upstream/master'
cloudwebrtc 4b422b3
Add texture support for macOS shell.
cloudwebrtc fa3f8b0
code format.
cloudwebrtc a222a4a
Add missing file.
cloudwebrtc 668e656
Update FLETexture.h
cloudwebrtc 97f1e91
Update FLEPluginRegistrar.h
cloudwebrtc cd636b4
Restore header file order.
cloudwebrtc 799e9c1
Update code styles and comments.
cloudwebrtc 52b1417
Rename textureId to textureID.
cloudwebrtc e49c549
Update comment for FLEExternalTextureGL.h.
cloudwebrtc 7589f46
Update comments and partial code.
cloudwebrtc 2c60631
Rename the texture in some methods to disambiguate.
cloudwebrtc 42b7502
Merge remote-tracking branch 'upstream/master'
cloudwebrtc dbcdf16
Merge branch 'master' into macos_texture
cloudwebrtc 9b15427
Fix errors caused by merge master conflicts.
cloudwebrtc e4ff652
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc ba2efa3
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc 9c19bac
clang format.
cloudwebrtc 6ff0a76
gn format.
cloudwebrtc ec8083a
Add files to licenses_flutter.
cloudwebrtc ff49e14
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc ee09e69
clang-format.
cloudwebrtc 0f06004
Update.
cloudwebrtc 09156bf
Use shared FlutterTexture.h.
cloudwebrtc 235314a
clang-format.
cloudwebrtc 650c00a
Fixed FlutterTexture.h import path for ios.
cloudwebrtc d4c0be8
Update.
cloudwebrtc cda947c
clang-format.
cloudwebrtc e78d2e1
Change _pixelBuffer to a local variable.
cloudwebrtc eb3bf39
Add missing static on a local constant
stuartmorgan d622f01
Fix variable naming inconsistency
stuartmorgan ec70d4a
Merge branch 'master' into macos_texture
cloudwebrtc 2871956
Use drawImageRect to draw textures of undesired size.
cloudwebrtc 9b63c65
Merge branch 'macos_texture' of https://github.com/cloudwebrtc/flutte…
cloudwebrtc 4962f91
clang-format.
cloudwebrtc b20be2c
Fixed CI errors.
cloudwebrtc 067e8a1
update.
cloudwebrtc 27c6f94
clang-format.
cloudwebrtc 5f6833f
Merge remote-tracking branch 'upstream/master' into macos_texture
cloudwebrtc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
36 changes: 36 additions & 0 deletions
36
shell/platform/darwin/macos/framework/Source/FlutterExternalTextureGL.h
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| // Copyright 2013 The Flutter Authors. All rights reserved. | ||
| // Use of this source code is governed by a BSD-style license that can be | ||
| // found in the LICENSE file. | ||
|
|
||
| #import <Foundation/Foundation.h> | ||
|
|
||
| #import "flutter/shell/platform/darwin/common/framework/Headers/FlutterTexture.h" | ||
| #import "flutter/shell/platform/embedder/embedder.h" | ||
|
|
||
| /** | ||
| * Used to bridge FlutterTexture object and handle the texture copy request the | ||
| * Flutter engine. | ||
| */ | ||
| @interface FlutterExternalTextureGL : NSObject | ||
|
|
||
| /** | ||
| * Initializes a texture adapter with |texture| return a instance. | ||
| */ | ||
| - (nonnull instancetype)initWithFlutterTexture:(nonnull id<FlutterTexture>)texture; | ||
|
|
||
| /** | ||
| * Accepts texture buffer copy request from the Flutter engine. | ||
| * When the user side marks the textureID as available, the Flutter engine will | ||
| * callback to this method and ask for populate the |openGLTexture| object, | ||
| * such as the texture type and the format of the pixel buffer and the texture object. | ||
| */ | ||
| - (BOOL)populateTextureWithWidth:(size_t)width | ||
| height:(size_t)height | ||
| openGLTexture:(nonnull FlutterOpenGLTexture*)openGLTexture; | ||
|
|
||
| /** | ||
| * Returns the ID for the FlutterTexture instance. | ||
| */ | ||
| - (int64_t)textureID; | ||
|
|
||
| @end |
87 changes: 87 additions & 0 deletions
87
shell/platform/darwin/macos/framework/Source/FlutterExternalTextureGL.mm
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| // Copyright 2013 The Flutter Authors. All rights reserved. | ||
| // Use of this source code is governed by a BSD-style license that can be | ||
| // found in the LICENSE file. | ||
|
|
||
| #import "flutter/shell/platform/darwin/macos/framework/Source/FlutterExternalTextureGL.h" | ||
|
|
||
| #import <AppKit/AppKit.h> | ||
| #import <CoreVideo/CoreVideo.h> | ||
| #import <OpenGL/gl.h> | ||
|
|
||
| static void OnCVOpenGLTextureRelease(CVOpenGLTextureRef cvOpenGLTexture) { | ||
| CVOpenGLTextureRelease(cvOpenGLTexture); | ||
| } | ||
|
|
||
| @implementation FlutterExternalTextureGL { | ||
| /** | ||
| * OpenGL texture cache. | ||
| */ | ||
| CVOpenGLTextureCacheRef _openGLTextureCache; | ||
| /** | ||
| * User side texture object, used to copy pixel buffer. | ||
| */ | ||
| id<FlutterTexture> _texture; | ||
| } | ||
|
|
||
| - (instancetype)initWithFlutterTexture:(id<FlutterTexture>)texture { | ||
| self = [super init]; | ||
| if (self) { | ||
| _texture = texture; | ||
| } | ||
| return self; | ||
| } | ||
|
|
||
| - (int64_t)textureID { | ||
| return reinterpret_cast<int64_t>(self); | ||
| } | ||
|
|
||
| - (BOOL)populateTextureWithWidth:(size_t)width | ||
| height:(size_t)height | ||
| openGLTexture:(FlutterOpenGLTexture*)openGLTexture { | ||
| // Copy the pixel buffer from the FlutterTexture instance implemented on the user side. | ||
| CVPixelBufferRef pixelBuffer = [_texture copyPixelBuffer]; | ||
|
|
||
| if (!pixelBuffer) { | ||
| return NO; | ||
| } | ||
|
|
||
| // Create the opengl texture cache if necessary. | ||
| if (!_openGLTextureCache) { | ||
| CGLContextObj context = [NSOpenGLContext currentContext].CGLContextObj; | ||
| CGLPixelFormatObj format = CGLGetPixelFormat(context); | ||
| if (CVOpenGLTextureCacheCreate(kCFAllocatorDefault, NULL, context, format, NULL, | ||
| &_openGLTextureCache) != kCVReturnSuccess) { | ||
| NSLog(@"Could not create texture cache."); | ||
| CVPixelBufferRelease(pixelBuffer); | ||
| return NO; | ||
| } | ||
| } | ||
|
|
||
| // Try to clear the cache of OpenGL textures to save memory. | ||
| CVOpenGLTextureCacheFlush(_openGLTextureCache, 0); | ||
|
|
||
| CVOpenGLTextureRef cvOpenGLTexture = NULL; | ||
| if (CVOpenGLTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _openGLTextureCache, | ||
| pixelBuffer, NULL, | ||
| &cvOpenGLTexture) != kCVReturnSuccess) { | ||
| CVPixelBufferRelease(pixelBuffer); | ||
| return NO; | ||
| } | ||
|
|
||
| openGLTexture->target = static_cast<uint32_t>(CVOpenGLTextureGetTarget(cvOpenGLTexture)); | ||
| openGLTexture->name = static_cast<uint32_t>(CVOpenGLTextureGetName(cvOpenGLTexture)); | ||
| openGLTexture->format = static_cast<uint32_t>(GL_RGBA8); | ||
| openGLTexture->destruction_callback = (VoidCallback)OnCVOpenGLTextureRelease; | ||
| openGLTexture->user_data = cvOpenGLTexture; | ||
| openGLTexture->width = CVPixelBufferGetWidth(pixelBuffer); | ||
| openGLTexture->height = CVPixelBufferGetHeight(pixelBuffer); | ||
|
|
||
| CVPixelBufferRelease(pixelBuffer); | ||
| return YES; | ||
| } | ||
|
|
||
| - (void)dealloc { | ||
| CVOpenGLTextureCacheRelease(_openGLTextureCache); | ||
| } | ||
|
|
||
| @end | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are not using the
widthand theheightargument in this method at all!The way this API works it that the engine tells the embedder what size it needs the texture to be. If the texture is not of that exact size, the engine will attempt to clamp or resize the texture to get something usable. The clamping behavior is what you seem to be running into.
There is information being thrown away in this method. That is, when you ask the texture to
copyPixelBuffer, you are not giving it any indication about what the size of the pixel buffer should be. This method acts as if size does not matter. When, in fact, the engine care about the size and it is asking you for a texture of that size. This method is just not using it.So now you have a texture of some unspecified size that you need to give to the engine because you didn't use the dimensions the engine asked of the embedder. To solve this issue, we now need to resort to the expensive device to host transfer so we can resize on the CPU before transferring that information back to the device.
If we don't throw away the information the engine gives us, we can be a whole lot faster. This can be done in one of the following ways:
copyPixelBufferto take thewidthand theheight. This way, plugins will know what texture the engine expects and can create texture of the right size to begin with.FlutterOpenGLTexturetake awidthandheightparameter (subject to ABI restrictions I mentioned below). That way, we give the plugins the best chance to match texture sizes. If however, the size the engine expects is not the same as the size the plugin gives back to us, instead of implementing the resize via a software render pass, we can achieve the same by applying a transformation to the draw on the GPU. This can be modified by pushing a CTM on the canvas here.Whatever you choose, let do as many operations on the GPU as possible by utilizing all information the engine give to the embedder.