feat: ✨ VisionCamera V4 ✨ (mrousavy#2623)

* feat: Give V4 🫴

* Update to compile SDK 34

* fix kotlin error

* fix ui thread

* Run all on Main Thread

* Update lifecycle

* Take Photo?

* Type Camera Errors

* Unbind before starting bind

* Create Recording base?

* Add `photoQualityBalance` to iOS

* feat: Add `takeSnapshot()` to `PreviewView`

* Add new API

* Low Light Boost throttles

* feat: Add CodeScanner pipeline (`ImageAnalysis`)

* Format

* Remove outputs

* Update CameraSession.kt

* Call back with Video result

* Delete `RecordingSession`

* Configure zoom and torch

* Only update torch, zoom or exposure if needed

* Create `takeSnapshot()`

* Use `compressionQuality`

* Format

* Catch CameraErrors directly

* Use new API for Snapshot

* Throw `.fileError`

* fix temp filename

* fix: Implement `takeSnaphot` on iOS

* Update Camera.tsx

* Don't re-initialize outputs if only inputId changes

* Update CameraSession.kt

* Update CameraSession.kt

* Use photo/video size selectors

* fix: Use runOnUiThread only if needed

* Code org

* Update CameraSession.kt

* feat: Use HDR or NIGHT extensions if available

* fix: Check for ImageAnalysis support

* feat: Shutter sound and stabilization

* Update CameraSession.kt

* restructure Threading a bit

* Restructure Threading in CameraSession, configure is UI Thread

* Update CameraSession.kt

* Update CameraSession.kt

* Better `mainExecutor`

* Implement `focus()`

* Use file type

* Make `VideoPipeline` a CameraX use-case

* Update CameraQueues.kt

* Update VideoPipeline.kt

* feat: Implement native HardwareBuffer rendering for `VideoPipeline`

* Try use VideoPipeline

* Try extending ImageAnalysis

* Create `FrameProcessorEffect` as camera middleman

* Use PREVIEW output

* fix not running FP for PREVIEW

* Update FrameProcessorEffect.kt

* feat: Allow setting custom format in ImageWriter

* Restructure code a bit

* Make everything synchronized

* Make `FrameProcessorEffect` closeable

* chore: Clean up Codebase

* Don't re-create Recorder if actively recording

* docs: Update docs for `qualityBalance`

* Update CameraProps.ts

* Remove OpenGL VideoPipeline (now CameraX does that)

* Properly disable CameraSession when `isActive=false`

* Remove a few Errors

* chore: release 4.0.0-beta.0

* chore: release 4.0.0-beta.1

* fix: Rename to `qualityBalance`

* Update package.json

* Small changes

* Update CameraView.kt

* Throw focus-canceled

* Implement `onStarted` and `onStopped` for Android again

* Always activate Camera lifecycle, ignore `isAttachedToWindow`

* feat: Add `onShutter` event

* chore: release 4.0.0-beta.2

* fix: Fix optional delegate

* Update CameraError.kt

* feat: Add `cancelRecording`

* chore: release 4.0.0-beta.3

* Add `Preview` docs

* feat: Add `androidPreviewViewType` prop

* feat: Use CameraX for device details (mrousavy#2624)

* feat: Use CameraX device details

* Remove `maxZoom` from format

* More

* feat: Implement the rest (some Camera2 fallbacks)

* Properly await ProcessCameraProvider and ExtensionsManager

* Initialize earlier

* Update CameraDeviceDetails.kt

* feat: Properly get orientation and isMirrored in FP

* fix: Properly close `ImageWriter`

* fix: Fix transform matrix

* fix: Use image transformation info from input surface

* fix: Capture directly in lambda

* fix: Fix focus not running on UI Thread

* chore: release 4.0.0-beta.4

* fix: Fix calling `sendAvailableDevicesChangedEvent` before module is initialized

* fix: Allow re-creating if session changed from outside

* fix: Don't play shutter sound unless explicitly enabled

* fix: Allow catching errors in `.await()`

* feat: Remove `enableAutoStabilization` prop as this was deprecated in iOS 13

* fix: Export `Snapshot.ts`

* feat: Add `type` to `onShutter` (`photo` or `snapshot`)

* fix: Only unbind current outputs, not all

* chore: Lint/Fix

* chore: release 4.0.0-beta.5

* fix: Use dummy metering factory instead of display-oriented

* fix: Remove unneeded context

* fix: Fix rotation getting

* Update Podfile.lock

* fix: Check format requirements in outputs

* fix: Remove `not-compatible-with-outputs` error on Android since CameraX supports StreamSharing

* fix: Remove `didSessionChangeFromOutside`

* fix: Catch Objective-C errors in iOS Frame Processor Plugins

* feat: Catch Objective-C init errors on plugin init

* fix: Fix `TimeoutException` by using `STARTED` lifecycle

* chore: release 4.0.0-beta.6

* add todo about hdr extension bug

* fix: Remove focus listeners after resetting

* fix: Fix capture session configuration error by manually layouting PreviewView

* feat: Write proper photo metadata (orientation & isMirrored) (mrousavy#2660)

* feat: Write proper photo metadata (orientation & isMirrored)

* Format

* fix: Fix Camera not starting because PreviewView isn't laid out

* fix: Use `CREATED` as default lifecycle state

* fix: Fix 10-bit HDR in combination with HDR Extension

* Update CameraSession.kt

* fix: Throw correct errors

* fix: Properly check SDR

* fix: Update outputs if video stabilization or low-light-boost changes

* fix: Also dont enable if other extension is enabled

* chore: release 4.0.0-beta.7

* Update package.json

* feat: Basic read-only Frame Processors (`ImageAnalysis` Use-Case) (mrousavy#2664)

* feat: Basic read-only Frame Processors (`ImageAnalysis` Use-Case)

* feat: Set target resolution

* Update CameraSession.kt

* Refactor

* fix: Fix Frame Processor not being unbound

* fix: Properly set `maxPhotoDimensions` on PhotoOutput (mrousavy#2659)

* fix: Properly set `maxPhotoDimensions` on PhotoOutput

* fix: Remove `maxPhotoDimensions` since it is the highest value by default anyways I think

* fix: Remove `enableHighQualityPhotos` entirely

* fix: Update RN Worklets

* Revert "fix: Remove `enableHighQualityPhotos` entirely"

This reverts commit 7110b76.

* fix: Use high res capture / maxPhotoDimensions always.

* Update CameraSession+Configuration.swift

* chore: release 4.0.0-beta.8

* Update CameraSession+Photo.swift

* fix: Fix focus on Android by using PX instead of DP (mrousavy#2666)

* fix: focus on Android (v4)

* format

---------

Co-authored-by: Marc Rousavy <me@mrousavy.com>

* Optimised images with calibre/image-actions

* feat: Add GPS Location EXIF tagging for photos and videos (`enableLocation={true}`) (mrousavy#2665)

* feat: Add GPS Location EXIF tagging for photos and videos (`enableLocation={true}`)

* feat: Also add location tag to video

* fix: Fix location tagging for videos

* fix: Write proper location to video

* Create VisionCamera branding

* fix: Write branding

* Use quicktime location instead of common location

* Finish metadata track

* Update Podfile.lock

* Try writing metadata

* Write video metadata

* Throw error with cause

* fix: Properly write metadata at start

* Properly convert to ISO 6709 String

* Use quicktime metadata location again

* fix: Put branding into QuickTime information field

* Format

* Update MetadataProvider.swift

* Only stream location if camera is active

* fix: Set location

* Format

* fix: Fix changing object reference in `usePermission`

* fix: Use proper location

* Update MetadataProvider.kt

* fix: Use last location as default

* Add some logs

* Only update location once every 5s

* fix: Also allow coarse location

* feat: Add location docs and `enableLocation` to expo config plugin

* Optimised images with calibre/image-actions

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Optimised images with calibre/image-actions

* fix: Remove HardwareBufferUtils

* fix: Fix CodeScanner not working

* chore: release 4.0.0-beta.9

* fix: Fix location permission not resolving on iOS (mrousavy#2672)

* fix: Fix location permission not resolving on iOS

* fix: Remove unused code

* fix: Ignore first call

* Use `GlobalReferenceHolder`

* chore: Remove RN Fast Image (we dont have web images anyways)

* Update package.json

* chore: release 4.0.0-beta.10

* fix: Fix wrong orientation enum being send to ImageCapture (mrousavy#2683)

* perf: Add `@FastNative` flags to critical Frame Processor methods

* fix: Fix `RCTDeviceEventEmitter` import for RN 0.71.X (mrousavy#2692)

fix: build for rn 0.71.X

* chore: release 4.0.0-beta.11

* feat: Disable shutter sound if in silent

* fix: Remove `enablePrecapture`

* chore: release 4.0.0-beta.12

* fix: Remove `mustPlayShutterSound()` check

* Update Podfile.lock

* chore: release 4.0.0-beta.13

* fix: Always support `rgb` and `yuv`, remove `native`

All Pixel Formats (`rgb` and `yuv`) are supported now, always.
On Android this sometimes uses a conversion step.

* fix: Use `ResolutionSelector`s with fallback values

* fix: Fall-back to H.264 if custom codec is not supported

* Update Camera.tsx

* feat: Add `hardware-cost-too-high` error for iOS 16

* chore: release 4.0.0-beta.14

* fix: Use 1280x720 for QR scanner

* feat: Skia for V4! 🥳  (mrousavy#2727)

* feat: Add Skia support (`toSkImage()`)

* feat: Add `frame.getPlaformBuffer()`

* Try use getPlatformBuffer

* Update Frame.ts

* fix: Fix `getPlatformBuffer()`

* fix: Fix buffer access

* fix: Always support `rgb` and `yuv`, remove `native`

All Pixel Formats (`rgb` and `yuv`) are supported now, always.
On Android this sometimes uses a conversion step.

* feat: Build Skia example renderer

* fix: Render a bit better

* fix: Use atomic updates now

* fix: Remove SK flags

* fix: Clear and copy Frame

* fix: Use bigint

* fix: Use `ImageProxy` and close it to avoid stalling

* Only do copy on iOS

* fix: Add throw method signature

* feat: Box into `DrawableFrame`

* feat: Create `SkiaCameraCanvas`

* feat: Allow disabling `preview` use-case

* feat: Catch some errors

* Update FrameHostObject.cpp

* fix: Use proper layouts

* Update useSkiaFrameProcessor.ts

* fix: Use main Coroutine Scope

* fix: Fix wrong changes because we now properly abort changes

* chore: Format

* fix: Upgrade react-native-worklets-core

Now includes shift()

* fix: Add `focusRequiresPreview` error to iOS

* chore: Use latest react-native-worklets-core

* fix: Fix buffer orientation on iOS (it's currently always `.up`

* fix: Only pass function into native, no object

* fix: Format code

* fix: Fix styling

* fix: Fix pointer-events

* Update Camera.tsx

* Update useSkiaFrameProcessor.ts

* Update Worklets

* Draw radial face blur

* circle that spins

* fix: Proxify

* fix: Also use worklets proxy lazily

* fix: Try multiple error handlers

* fix: Also use Skia Proxy

* Add docs to the objects

* fix: Use `createModuleProxy`

* fix: Also re-create surface if height is different

* perf: Remove unneeded mutex locks as Frame only locks refCount

* Update FRAME_PROCESSORS_CREATE_OVERVIEW.mdx

* fix: Remove unneeded CPU copy

* fix: Catch PixelFormat error and fallback to default format

* fix: Throw proper PixelFormat error with more info

* Update CameraError.swift

* Update Frame.java

* fix: Use `jsi::Scope`

In Hermes, this will probably not do anything.

* chore: Update react-native-worklets-core

* fix: Dispose previous Surface

* Update useSkiaFrameProcessor.ts

* fix: Fix surface re-creating each time

* chore: Bump to latest RN Skia with the features we need

* feat: Clean everything on unmount

* Add CPU copy again. On iOS, we need it.

* fix: Fix Android build

* fix: Fix Android build again

* Update ModuleProxy.ts

* fix: Use type imports if possible

* fix: Add RN Skia patch

* Add react-native-fast-tflite

* fix: Configure for tflite files as well

* Revert "fix: Configure for tflite files as well"

This reverts commit 37db0f5.

* Revert "Add react-native-fast-tflite"

This reverts commit 264dd4b.

* Update @Shopify+react-native-skia+1.2.0.patch

* Add `useSkiaFrameProcessor` to ESLint

* Update yarn.lock

* fix: Upgrade to latest Skia to use NativeBuffer API now

* fix: Use `CVPixelBuffer` instead of `CMSampleBuffer`

* fix: Fix package.json

* chore: Update to latest react-native-worklets-core

* fix: Use thread-local storage for SkSurface

* Update Podfile.lock

* fix: Migrate over to new Worklets API

* Update FRAME_PROCESSORS.mdx

* Update useSkiaFrameProcessor.ts

* fix: Run destroy in interaction manager

* fix: Safely remove surfaces

* fix: Update RNWC to fix terminated threads using dead `thread_local` storage

* fix: Install hierarchy fitter in `CameraView`

* chore: Remove Skia example from example app

* Update CameraPage.tsx

* fix: Simplify default pixel format

* fix: Rename `ReactLogger` to `VisionLogger`

* chore: Format

* fix: Fix ResolutionSelector `difference` abs (mrousavy#2746)

* chore: release 4.0.0-beta.15

* Update CameraPage.tsx

* chore: Fix `type` imports in TS

* fix: Remove `enableGpuBuffers`

* fix: Make dependencies (`ModuleProxy`) actually optional (mrousavy#2750)

fix: Make dependencies actually optional

* feat: Optionally compile out Location APIs (mrousavy#2751)

* fix: Rename `disableFrameProcessors` to `enableFrameProcessors`

* add podfile writer

* Create vc_example_cocoapod_utils.rb

* feat: Make Location compile optionally

* fix: Only compile out location related permissio

* Update Podfile

* Use `$(inherited)`

* fix: Fix `minFps` being larger in `Range` than `maxFps` (mrousavy#2755)

* fix: Fix `minFps` being larger in `Range` than `maxFps`

* import

* fix: Also override `onProviderDisabled` to prevent abstract method crash

* fix: Clean up Skia resources on Worklet context to avoid race condition (mrousavy#2754)

* fix: Clean up Skia resources on Worklet context to avoid race condition

* Update FrameProcessorPlugins.ts

* Update useSkiaFrameProcessor.ts

* docs: Update docs for V4 (mrousavy#2747)

* fix: Update prism-react-renderer

* fix: Fix docs

* fix: Remove `enableGpuBuffers`

* feat: Update Docusaurus to latest 3

* Update docusaurus.config.js

* update languages

* update languages

* rename getting_started

* Update some docs

* Update CODE_SCANNING.mdx

* update some docs

* Update docusaurus.config.js

* files

* starting

* fix: Properly display video

* lots of updates

* more docs

* fix: Fix errors causing memory-leaks by not closing Frame

* Add example

* more

* almost

* fix broken links

* update skia fp docs

* Update FRAME_PROCESSORS_SKIA.mdx

* Update FRAME_PROCESSORS_SKIA.mdx

* fix broken link

* chore: release 4.0.0-beta.16

* fix: Fix `Promise` resolve wrong name

* fix: Fix build for Android because of Worklet usage

* feat: Separate VisionCamera into two subspecs (mrousavy#2761)

* feat: Separate VisionCamera into two subspecs

* Swift

* update podspec

* fix: Refactor

* some fixes

* hmm

* fix: Only export ObjC/Swift visible headers

* fix: Also drop C++ sources

* fix: Move to correct folder

* fix: Fix any remaining C++ includes

* chore: Format

* fix: Fix missingg Swift include

* fix: Add PluginRegistry Swift include

* Update FrameProcessorPluginRegistry.h

* fix: Disable location

* fix: Use `public_header_files` instead of `preserve_paths`

* Disable FP if needed

* fix Podfile

* fix: Fix `dispatch_queue_t` in Swift

* fix: Disable FPs if not built

* Revert "fix: Disable FPs if not built"

This reverts commit 82cb560.

* Remove Example Plugins from source code

* feat: Separate VisionCamera into `React`, `Core` and `FrameProcessors` subspecs (mrousavy#2764)

* feat: Separate VisionCamera into `React`, `Core` and `FrameProcessors` subspecs

* Split utils

* fix: Refactor Android codebase into `core`,  `react` and `frameprocessors` (mrousavy#2765)

* Add example pod helper again

* feat: Also separate Android into `react`, `core` and `frameprocessor`

* Log similar to iOS

* Update build.gradle

* fix: Fix weird refactor mistakes

* fix: Refactor a bit

* fix: `frameprocessor` -> `frameprocessors`

* duplicate clang format with ln

* chore: release 4.0.0-beta.17

* chore: Cleanup C++

* chore: Cleanup TS codebase and fix Skia Preview View style/layout issue (mrousavy#2767)

* chore: Cleanup TS codebase

* chore: More refactors

* fix: Put Skia Canvas into CameraView

* Update Camera.tsx

* fix: Fix JNI types

* chore: Update to RN Skia 1.2.3

* fix: Simplify `cpp/` codebase into `android`/`ios` folders (mrousavy#2766)

* `Frame Processor` -> `FrameProcessors`

* fix: Remove almost empty `cpp/` folder

* fix: Add wrap ctor to `MutableRawBuffer`

* fix: Add `.clang-format` file again

* fix: Remove `cpp` folder references from everywhere

* fix: Fix `already bound to a different lifecycle` error when flipping camera (mrousavy#2768)

* fix: Fix `already bound to a different lifecycle` error when flipping camera

* Update CameraSession.kt

* Delete .clang-format

* feat: Rebuild FPS Graph as JS component (mrousavy#2769)

* feat: Rebuild FPS Graph as JS component

* feat: Properly style graph

* chore: Lint

* fix: Fix FPS Graph not displaying on Android

* chore: release 4.0.0-beta.18

* fix: Fix `ImageAnalysis is not supported when Extension is enabled` error (mrousavy#2770)

* feat: Upgrade to CameraX alpha5 (mrousavy#2771)

* fix: Separate `FpsSampleCollectorDelegate`

* Downgrade to CameraX alpha04 because 05 fails to build

* fix: Make `FpsSampleCollectorDelegate` weak

* fix: Fix `delegate` guard

* fix: Fix Android import path due to new subfolder

* fix: Fix `Timer is already canceled`

* chore: release 4.0.0-beta.19

---------

Co-authored-by: Kirill Zyusko <zyusko.kirik@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Rodrigo Gomes <rodgomesc@gmail.com>
Co-authored-by: Versus <3686499+versusvoid@users.noreply.github.com>

package/.eslintrc.js

@@ -10,7 +10,7 @@ module.exports = {
     ecmaVersion: 2018,
     sourceType: 'module',
   },
-  ignorePatterns: ['scripts', 'lib', 'docs', 'example', 'app.plugin.js'],
+  ignorePatterns: ['scripts', 'lib', 'docs', 'example'],
   plugins: ['@typescript-eslint'],
   extends: ['plugin:@typescript-eslint/recommended', '@react-native'],
   rules: {

package/tsconfig.json

@@ -23,6 +23,7 @@
   },
   "include": [
     "src",
+    "app.plugin.js",
     ".eslintrc.js",
     "babel.config.js",
     "react-native.config.js",