feat: ✨ V3 ✨ (#1466)

See #1376 ## Breaking Changes * Frame Processors are now **synchronous**. Previously they ran on a separate Thread. If you want to run something on a separate Thread now, use `runAsync` inside a Frame Processor * Frame Processor Plugins are no longer in the global object with the `__` prefix, but rather stored directly in the `FrameProcessorPlugins` object exported by react-native-vision-camera. (e.g. replace `__scanQRCodes(frame)` with `FrameProcessorPlugins.scanQRCodes(frame)`) * `frameProcessorFps` no longer exists. Use `runAtTargetFps` inside a Frame Processor to throttle some calls. * `onFrameProcessorPerformanceSuggestionAvailable` no longer exists. Use the FPS display (`enableFpsGraph={true}`) to see how your Frame Processor performs over time. This is more in-line with how React Native works (Dev Tools / Perf Monitor) * VisionCamera V3 will not work on RN 0.70 or below. You need to use RN 0.71. This is because the build script got way simpler and smaller, making it faster to build and way less error prone. Backwards compatibility is just too complex here. * Reanimated is no longer used as a Worklet Runtime. Instead, VisionCamera now uses [react-native-worklets-core](https://github.com/margelo/react-native-worklets-core). ## Progress You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets: ```sh yarn add react-native-vision-camera@3.0.0-rc.5 yarn add react-native-worklets-core yarn add @shopify/react-native-skia ``` Things to test: * TensorFlow Lite plugin to load any `.tflite` model!! ✨ (see [this PR for more info](#1633), will be a separate library soon) * Drawing onto a Frame using Skia!! 🎉 * Using `frame.toArrayBuffer()` to get the Frame's byte content in JS * New Android build script. This should drastically speed up the build time! 💨 * New Worklet library. This replaces Reanimated Worklets. Should be faster and more stable :) * New synchronous Frame Processors. Should be faster :) * `runAtTargetFps` and `runAsync` in Frame Processors * Using HostObjects or HostFunctions (like models from PyTorch) inside a Frame Processor. This will probably require a few native bindings on PyTorch's end to make the integration work (cc @raedle) Overall V3 is close to completion. I have a few things to do the coming days so not sure how much work I can put into this. **If anyone wants to support the development of v3, I'd appreciate donations / sponsors: https://github.com/sponsors/mrousavy** ❤️ :) ## Related issues features - resolves #1376 - fixes #281 - resolves #211 - resolves #130 - resolves #117 - fixes #76 - resolves #75 - resolves #562 - resolves #565 - fixes #570 - fixes #287 - resolves #311 - fixes #315 - resolves #323 - fixes #340 - fixes #354 - resolves #420 - fixes #434 - fixes #452 - fixes #496 - fixes #497 - resolves #499 - fixes #516 - fixes #527 - fixes #542 - fixes #548 - fixes #561 - fixes #740 - fixes #770 ...and then pretty much every Android issue lol - fixes #1675 (**maybe**, please test @PrernaBudhraja) - fixes #1671 .. maybe also (not tested): - fixes #1698 - fixes #1687 - fixes #1685 - fixes #1681 - fixes #1650 - fixes #1646 - fixes #1635 - fixes #1631 - fixes #1621 - fixes #1615 - fixes #1612 - fixes #1605 - fixes #1599 - fixes #1585 - fixes #1581 - fixes #1569 - fixes #1568 - fixes #1565 - fixes #1561 - fixes #1558 - fixes #1554 - fixes #1551 - fixes #1547 - fixes #1543 - fixes #1538 - fixes #1536 - fixes #1534 - fixes #1528 - fixes #1520 - fixes #1498 - fixes #1489 - fixes #1477 - fixes #1474 - fixes #1463 - fixes #1462 - fixes #1449 - fixes #1443 - fixes #1437 - fixes #1431 - fixes #1429 - fixes #1427 - fixes #1423 - fixes #1416 - fixes #1407 - fixes #1403 - fixes #1402 - fixes #1398 - fixes #1396 - fixes #1395 - fixes #1379 - fixes #1377 - fixes #1374 - fixes #1373 - fixes #1365 - fixes #1356 - fixes #1353 - fixes #1352 - fixes #1351 - fixes #1343 - fixes #1340 - fixes #1334 - fixes #1330 - fixes #1322 - fixes #1296 - fixes #1283 - fixes #1260 - fixes #1253 - fixes #1251 - fixes #1245 - fixes #1238 - fixes #1227 - fixes #1226 - fixes #1225 - fixes #1222 - fixes #1211 - fixes #1208 - fixes #1193 - fixes #1191 - fixes #1184 - fixes #1164 - fixes #1143 - fixes #1128 - fixes #1122 - fixes #1120 - fixes #1110 - fixes #1097 - fixes #1081 - fixes #1080 - fixes #1064 - fixes #1053 - fixes #1047 - fixes #1044 - fixes #1032 - fixes #1026 - fixes #1023 - fixes #1015 - fixes #1012 - fixes #997 - fixes #960 - fixes #959 - fixes #954 - fixes #946 - fixes #945 - fixes #922 - fixes #908 - fixes #907 - fixes #868 - fixes #855 - fixes #834 - fixes #793 - fixes #779 - fixes #746 - fixes #740 - fixes #727 - fixes #671 - fixes #613 - fixes #595 - fixes #588 - fixes #570 - fixes #569 - fixes #542 - fixes #516 - fixes #515 - fixes #434 - fixes #354 - fixes #323 - fixes #315 - fixes #281 - fixes #211 - fixes #76
mrousavy · Sep 1, 2023 · 02a0371 · 02a0371 · vercel · Sep 1, 2023
2 parents c62b5de + 74b5898
commit 02a0371
Show file tree

Hide file tree

Showing 466 changed files with 29,070 additions and 22,651 deletions.
diff --git a/.github/ISSUE_TEMPLATE/BUG_REPORT.yml b/.github/ISSUE_TEMPLATE/BUG_REPORT.yml
@@ -5,28 +5,26 @@ labels: [🐛 bug]
 body:
   - type: textarea
     attributes:
-      label: What were you trying to do?
-      description: Explain what you are trying to do.
-      placeholder: I wanted to take a picture.
+      label: What's happening?
+      description: Explain what you are trying to do and what happened instead. Be as precise as possible, I can't help you if I don't understand your issue.
+      placeholder: I wanted to take a picture, but the method failed with this error "[capture/photo-not-enabled] Failed to take photo, photo is not enabled!"
     validations:
       required: true
   - type: textarea
     attributes:
       label: Reproduceable Code
       description: Share a small reproduceable code snippet here (or the entire file if necessary). This will be automatically formatted into code, so no need for backticks.
       render: tsx
-  - type: textarea
-    attributes:
-      label: What happened instead?
-      description: Explain what happened instead of the desired outcome. Did something crash?
-      placeholder: The app crashes with an `InvalidPhotoCodec` error.
-    validations:
-      required: true
   - type: textarea
     attributes:
       label: Relevant log output
       description: Please copy and paste any relevant log output (Xcode Logs/Android Studio Logcat). This will be automatically formatted into code, so no need for backticks.
       render: shell
+  - type: textarea
+    attributes:
+      label: Camera Device
+      description: Please paste the JSON Camera `device` that was used here. (`console.log(JSON.stringify(device, null, 2))`) This will be automatically formatted into code, so no need for backticks.
+      render: shell
   - type: input
     attributes:
       label: Device
@@ -38,15 +36,22 @@ body:
     attributes:
       label: VisionCamera Version
       description: Which version of react-native-vision-camera are you using?
-      placeholder: ex. 2.0.1-beta.1
+      placeholder: ex. 3.1.6
     validations:
       required: true
+  - type: checkboxes
+    attributes:
+      label: Can you reproduce this issue in the VisionCamera Example app?
+      description: Run the example app (`package/example/`) and see if the issue is reproduceable here.
+      options:
+      - label: I can reproduce the issue in the VisionCamera Example app.
   - type: checkboxes
     attributes:
       label: Additional information
       description: Please check all the boxes that apply
       options:
         - label: I am using Expo
+        - label: I have enabled Frame Processors (react-native-worklets-core)
         - label: I have read the [Troubleshooting Guide](https://react-native-vision-camera.com/docs/guides/troubleshooting)
           required: true
         - label: I agree to follow this project's [Code of Conduct](https://github.com/mrousavy/react-native-vision-camera/blob/main/CODE_OF_CONDUCT.md)

diff --git a/.github/ISSUE_TEMPLATE/BUILD_ERROR.yml b/.github/ISSUE_TEMPLATE/BUILD_ERROR.yml
@@ -0,0 +1,64 @@
+name: 🔧 Build Error
+description: File a build error bug report
+title: "🔧 "
+labels: [🔧 build error]
+body:
+  - type: textarea
+    attributes:
+      label: How were you trying to build the app?
+      description: Explain how you tried to build the app, through Xcode, `yarn ios`, a CI, or other. Be as precise as possible, I can't help you if I don't understand your issue.
+      placeholder: I tried to build my app with react-native-vision-camera using the `yarn ios` command, and it failed.
+    validations:
+      required: true
+  - type: textarea
+    attributes:
+      label: Full build logs
+      description: Share the full build logs that appear in the console. Make sure you don't just paste the last few lines here, but rather everything from start to end.
+      render: tsx
+  - type: textarea
+    attributes:
+      label: Project dependencies
+      description: Share all of your project's dependencies including their versions from `package.json`. This is useful if there are any other conflicting libraries.
+      render: tsx
+    validations:
+      required: true
+  - type: dropdown
+    attributes:
+      label: Target platforms
+      description: Select the platforms where the build error occurs.
+      multiple: true
+      options:
+        - iOS
+        - Android
+    validations:
+      required: true
+  - type: dropdown
+    attributes:
+      label: Operating system
+      description: Select your operating system that you are trying to build on.
+      multiple: true
+      options:
+        - MacOS
+        - Windows
+        - Linux
+    validations:
+      required: true
+  - type: checkboxes
+    attributes:
+      label: Can you build the VisionCamera Example app?
+      description: Try to build the example app (`package/example/`) and see if the issue is reproduceable here.
+      options:
+      - label: I can build the VisionCamera Example app.
+  - type: checkboxes
+    attributes:
+      label: Additional information
+      description: Please check all the boxes that apply
+      options:
+        - label: I am using Expo
+        - label: I have enabled Frame Processors (react-native-worklets-core)
+        - label: I have read the [Troubleshooting Guide](https://react-native-vision-camera.com/docs/guides/troubleshooting)
+          required: true
+        - label: I agree to follow this project's [Code of Conduct](https://github.com/mrousavy/react-native-vision-camera/blob/main/CODE_OF_CONDUCT.md)
+          required: true
+        - label: I searched for [similar issues in this repository](https://github.com/mrousavy/react-native-vision-camera/issues) and found none.
+          required: true
diff --git a/.github/workflows/build-android.yml b/.github/workflows/build-android.yml
@@ -6,29 +6,34 @@ on:
       - main
     paths:
       - '.github/workflows/build-android.yml'
+      - 'cpp/**'
       - 'android/**'
       - 'example/android/**'
       - 'yarn.lock'
       - 'example/yarn.lock'
   pull_request:
     paths:
       - '.github/workflows/build-android.yml'
+      - 'cpp/**'
       - 'android/**'
       - 'example/android/**'
       - 'yarn.lock'
       - 'example/yarn.lock'
 
 jobs:
-  build_example:
+  build:
     name: Build Android Example App
     runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ./package
     steps:
       - uses: actions/checkout@v2
 
-      - name: Setup JDK 1.8
+      - name: Setup JDK 11
         uses: actions/setup-java@v1
         with:
-          java-version: 1.8
+          java-version: 11
 
       - name: Get yarn cache directory path
         id: yarn-cache-dir-path
@@ -55,7 +60,49 @@ jobs:
           key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
           restore-keys: |
             ${{ runner.os }}-gradle-
-      - name: Run Gradle Build for android/
-        run: cd android && ./gradlew assembleDebug --build-cache && cd ..
+      - name: Run Gradle Build for example/android/
+        run: cd example/android && ./gradlew assembleDebug --build-cache && cd ../..
+
+  build-no-frame-processors:
+    name: Build Android Example App (without Frame Processors)
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: ./package
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Setup JDK 11
+        uses: actions/setup-java@v1
+        with:
+          java-version: 11
+
+      - name: Get yarn cache directory path
+        id: yarn-cache-dir-path
+        run: echo "::set-output name=dir::$(yarn cache dir)"
+      - name: Restore node_modules from cache
+        uses: actions/cache@v2
+        id: yarn-cache
+        with:
+          path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
+          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-yarn-
+      - name: Install node_modules
+        run: yarn install --frozen-lockfile
+      - name: Install node_modules for example/
+        run: yarn install --frozen-lockfile --cwd example
+      - name: Remove react-native-worklets-core
+        run: yarn remove react-native-worklets-core --cwd example
+
+      - name: Restore Gradle cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.gradle/caches
+            ~/.gradle/wrapper
+          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
       - name: Run Gradle Build for example/android/
         run: cd example/android && ./gradlew assembleDebug --build-cache && cd ../..
diff --git a/.github/workflows/build-ios.yml b/.github/workflows/build-ios.yml
@@ -6,12 +6,14 @@ on:
       - main
     paths:
       - '.github/workflows/build-ios.yml'
+      - 'cpp/**'
       - 'ios/**'
       - '*.podspec'
       - 'example/ios/**'
   pull_request:
     paths:
       - '.github/workflows/build-ios.yml'
+      - 'cpp/**'
       - 'ios/**'
       - '*.podspec'
       - 'example/ios/**'
@@ -22,7 +24,7 @@ jobs:
     runs-on: macOS-latest
     defaults:
       run:
-        working-directory: example/ios
+        working-directory: package/example/ios
     steps:
       - uses: actions/checkout@v2
 
@@ -47,9 +49,8 @@ jobs:
       - name: Setup Ruby (bundle)
         uses: ruby/setup-ruby@v1
         with:
-          ruby-version: 2.6
+          ruby-version: 2.6.10
           bundler-cache: true
-          working-directory: example/ios
 
       - name: Restore Pods cache
         uses: actions/cache@v2
@@ -62,7 +63,68 @@ jobs:
           restore-keys: |
             ${{ runner.os }}-pods-
       - name: Install Pods
-        run: bundle exec pod check || bundle exec pod install
+        run: pod install
+      - name: Install xcpretty
+        run: gem install xcpretty
+      - name: Build App
+        run: "set -o pipefail && xcodebuild \
+          CC=clang CPLUSPLUS=clang++ LD=clang LDPLUSPLUS=clang++ \
+          -derivedDataPath build -UseModernBuildSystem=YES \
+          -workspace VisionCameraExample.xcworkspace \
+          -scheme VisionCameraExample \
+          -sdk iphonesimulator \
+          -configuration Debug \
+          -destination 'platform=iOS Simulator,name=iPhone 11 Pro' \
+          build \
+          CODE_SIGNING_ALLOWED=NO | xcpretty"
+
+  build-no-frame-processors:
+    name: Build iOS Example App without Frame Processors
+    runs-on: macOS-latest
+    defaults:
+      run:
+        working-directory: package/example/ios
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Get yarn cache directory path
+        id: yarn-cache-dir-path
+        run: echo "::set-output name=dir::$(yarn cache dir)"
+      - name: Restore node_modules from cache
+        uses: actions/cache@v2
+        id: yarn-cache
+        with:
+          path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
+          key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-yarn-
+      - name: Install node_modules for example/
+        run: yarn install --frozen-lockfile --cwd ..
+      - name: Remove react-native-worklets-core
+        run: yarn remove react-native-worklets-core --cwd ..
+
+      - name: Restore buildcache
+        uses: mikehardy/buildcache-action@v1
+        continue-on-error: true
+
+      - name: Setup Ruby (bundle)
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: 2.6.10
+          bundler-cache: true
+
+      - name: Restore Pods cache
+        uses: actions/cache@v2
+        with:
+          path: |
+            example/ios/Pods
+            ~/Library/Caches/CocoaPods
+            ~/.cocoapods
+          key: ${{ runner.os }}-pods-${{ hashFiles('**/Podfile.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-pods-
+      - name: Install Pods
+        run: pod install
       - name: Install xcpretty
         run: gem install xcpretty
       - name: Build App

diff --git a/.github/workflows/notice-yarn-changes.yml b/.github/workflows/notice-yarn-changes.yml
diff --git a/.github/workflows/validate-android.yml b/.github/workflows/validate-android.yml
@@ -20,13 +20,13 @@ jobs:
     runs-on: ubuntu-latest
     defaults:
       run:
-        working-directory: ./android
+        working-directory: ./package/android
     steps:
       - uses: actions/checkout@v2
-      - name: Setup JDK 1.8
+      - name: Setup JDK 11
         uses: actions/setup-java@v1
         with:
-          java-version: 1.8
+          java-version: 11
 
       - name: Get yarn cache directory path
         id: yarn-cache-dir-path