Migrate CameraX from SurfaceTexture to SurfaceProducer. #6462
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Due to a release hiccup, the |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@johnmccutchan @camsim99 PTAL |
Co-authored-by: Camille Simon <43054281+camsim99@users.noreply.github.com>
…ce-producer-camerax
jonahwilliams
added
the
autosubmit
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/flutter
that referenced
this pull request
May 30, 2024
auto-submit bot
pushed a commit
to flutter/flutter
that referenced
this pull request
May 30, 2024
flutter/packages@31d3329...910fabb 2024-05-29 58529443+srujzs@users.noreply.github.com Amend package:web tweaks to allow package:web roll (flutter/packages#6793) 2024-05-29 matanlurey@users.noreply.github.com Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6456) 2024-05-29 joonas.kerttula@codemate.com [google_maps_flutter] Undeprecate BitmapDescriptor methods (flutter/packages#6832) 2024-05-29 matanlurey@users.noreply.github.com Migrate CameraX from SurfaceTexture to SurfaceProducer. (flutter/packages#6462) 2024-05-29 matanlurey@users.noreply.github.com Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6461) 2024-05-29 katelovett@google.com [dynamic_layouts] Remove the dynamic_layouts package (flutter/packages#6830) 2024-05-29 43054281+camsim99@users.noreply.github.com [camerax] Add notes about Android permissions (flutter/packages#6741) 2024-05-29 43054281+camsim99@users.noreply.github.com [Re-land] Bump legacy all_packages project AGP version to 7.0.0, Gradle version to 7.0.2 (flutter/packages#6742) 2024-05-29 engine-flutter-autoroll@skia.org Roll Flutter from a1a33e6 to c85fa6a (20 revisions) (flutter/packages#6829) 2024-05-29 katelovett@google.com [rfw] Migrate deprecated doc references (flutter/packages#6744) 2024-05-29 katelovett@google.com [flutter_adaptive_scaffold] Migrate MaterialStateProperty to WidgetStateProperty (flutter/packages#6743) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
jonahwilliams
added a commit
that referenced
this pull request
May 30, 2024
This reverts commit 47d5eb2.
victorsanni
pushed a commit
to victorsanni/flutter
that referenced
this pull request
May 31, 2024
flutter/packages@31d3329...910fabb 2024-05-29 58529443+srujzs@users.noreply.github.com Amend package:web tweaks to allow package:web roll (flutter/packages#6793) 2024-05-29 matanlurey@users.noreply.github.com Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6456) 2024-05-29 joonas.kerttula@codemate.com [google_maps_flutter] Undeprecate BitmapDescriptor methods (flutter/packages#6832) 2024-05-29 matanlurey@users.noreply.github.com Migrate CameraX from SurfaceTexture to SurfaceProducer. (flutter/packages#6462) 2024-05-29 matanlurey@users.noreply.github.com Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6461) 2024-05-29 katelovett@google.com [dynamic_layouts] Remove the dynamic_layouts package (flutter/packages#6830) 2024-05-29 43054281+camsim99@users.noreply.github.com [camerax] Add notes about Android permissions (flutter/packages#6741) 2024-05-29 43054281+camsim99@users.noreply.github.com [Re-land] Bump legacy all_packages project AGP version to 7.0.0, Gradle version to 7.0.2 (flutter/packages#6742) 2024-05-29 engine-flutter-autoroll@skia.org Roll Flutter from a1a33e6 to c85fa6a (20 revisions) (flutter/packages#6829) 2024-05-29 katelovett@google.com [rfw] Migrate deprecated doc references (flutter/packages#6744) 2024-05-29 katelovett@google.com [flutter_adaptive_scaffold] Migrate MaterialStateProperty to WidgetStateProperty (flutter/packages#6743) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
11 tasks
arc-yong
pushed a commit
to Arctuition/packages-arc
that referenced
this pull request
Jun 14, 2024
_**WIP**: We do not plan to land this PR until the next stable release (>= April 3rd 2024)_. Work towards flutter/flutter#145930. ## Details Migrates uses of `createSurfaceTexture` to `createSurfaceProducer`, which is intended to have no change in behavior, but _does_ change the backend rendering path, so it will require more testing (and we're also open to minor API renames or changes before it becomes stable). ## Background Android plugins previously requested a `SurfaceTexture` from the Android embedder, and used that to produce a `Surface` to render external textures on (i.e. `video_player`). This worked because 100% of Flutter applications on Android used OpenGLES (via our Skia backend), and `SurfaceTexture` is actually an (opaque) OpenGLES-texture. Starting soon (roughly ~Q3, this is not a guarantee and just an estimate), Flutter on Android will start to use our new Impeller graphics backend, which on newer devices (`>= API_VERSION_28`), will default to the Vulkan, _not_ OpenGLES. In other words, `SurfaceTexture` will cease to work (it is possible, but non-trivial, to map an OpenGLES texture over to Vulkan). After consultation with the Android team, they helped us understand that vending `SurfaceTexture` (the _consumer-side_ API) was never the right abstraction, and we should have been vending the _producer-side_ API, or `Surface` directly. The new `SurfaceProducer` API is exactly that - it generates a `Surface`, and similar to our platform view strategy, picks the "right" _consumer-side_ implementation details _for_ the user/plugin packages. The new `SurfaceProducer` API has 2 possible rendering types (as an implementation detail): - `SurfaceTexture`, for older OpenGLES devices, which works exactly as it does today. - `ImageReader`, for newer OpenGLES _or_ Vulkan devices. These are some subtle nuances in how these two APIs work differently (one example: flutter/flutter#144407), but our theory at this point is we don't expect these changes to be observed by any users, and we have other ideas if necessary. > [!NOTE] > These invariants are [tested on CI in `flutter/engine`](https://github.com/flutter/engine/tree/main/testing/scenario_app/android#ci-configuration). Points of contact: - @matanlurey or @jonahwilliams (Flutter Engine) - @johnmccutchan or @reidbaker (Flutter on Android)
arc-yong
pushed a commit
to Arctuition/packages-arc
that referenced
this pull request
Jun 14, 2024
11 tasks
auto-submit bot
pushed a commit
that referenced
this pull request
Aug 19, 2024
…ew rotation (#6856) Re-lands #6462 with the following change to `buildPreview`: #### Switches on a fix for correctly rotating the camera preview This fix is required for `Surface`s not backed by a `SurfaceTexture` because they don't contain the transformation information needed to correctly rotate the camera preview. In that case, we use the logic described in https://developer.android.com/media/camera/camera2/camera-preview#orientation_calculation. This logic was added in a previous PR (#7044); this simply enables it for devices where `Surface`s are not backed by a `SurfaceTexture` (see [how this is determined](https://github.com/flutter/packages/blob/f118119cf830fe369b34a04ed63c9ed66c647985/packages/camera/camera_android_camerax/android/src/main/java/io/flutter/plugins/camerax/SystemServicesHostApiImpl.java#L107-L120)). Fixes flutter/flutter#149294.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP: We do not plan to land this PR until the next stable release (>= April 3rd 2024).
Work towards flutter/flutter#145930.
Details
Migrates uses of
createSurfaceTexturetocreateSurfaceProducer, which is intended to have no change in behavior, but does change the backend rendering path, so it will require more testing (and we're also open to minor API renames or changes before it becomes stable).Background
Android plugins previously requested a
SurfaceTexturefrom the Android embedder, and used that to produce aSurfaceto render external textures on (i.e.video_player). This worked because 100% of Flutter applications on Android used OpenGLES (via our Skia backend), andSurfaceTextureis actually an (opaque) OpenGLES-texture.Starting soon (roughly ~Q3, this is not a guarantee and just an estimate), Flutter on Android will start to use our new Impeller graphics backend, which on newer devices (
>= API_VERSION_28), will default to the Vulkan, not OpenGLES. In other words,SurfaceTexturewill cease to work (it is possible, but non-trivial, to map an OpenGLES texture over to Vulkan).After consultation with the Android team, they helped us understand that vending
SurfaceTexture(the consumer-side API) was never the right abstraction, and we should have been vending the producer-side API, orSurfacedirectly. The newSurfaceProducerAPI is exactly that - it generates aSurface, and similar to our platform view strategy, picks the "right" consumer-side implementation details for the user/plugin packages.The new
SurfaceProducerAPI has 2 possible rendering types (as an implementation detail):SurfaceTexture, for older OpenGLES devices, which works exactly as it does today.ImageReader, for newer OpenGLES or Vulkan devices.These are some subtle nuances in how these two APIs work differently (one example: flutter/flutter#144407), but our theory at this point is we don't expect these changes to be observed by any users, and we have other ideas if necessary.
Note
These invariants are tested on CI in
flutter/engine.Points of contact: