‼️‼️‼️‼️ ✨ VisionCamera V3 ‼️‼️‼️‼️‼️

[mrousavy]

We at Margelo are planning the next major version for react-native-vision-camera: VisionCamera V3 ✨

For VisionCamera V3 we target one major feature and a ton of stability and performance improvements:

1. Write-back frame processors. We are introducing a new feature where you can simply draw on frame in a Frame Processor using RN Skia. This allows you to draw face masks, filters, overlays, color shadings, shaders, Metal, etc..)
   • Uses a hardware accelerated Skia layer for showing the Preview
   • Some cool examples like inverted colors shader filter, VHS filter (inspired by Snapchat's VHS + distortion filter), and a realtime text/bounding box overlay
   • Realtime face blurring or license plate blurring
   • Easy to write color correction, beauty filters
   • All in simple JS (RN Skia) - no native code and hot reload while still maintaining pretty much native performance!
2. Sync Frame Processors. Frame Processors will now be fully synchronous and run on the same Thread as the Camera is running.
   • Pretty much on-par with native performance now.
   • Run frame processing without any delay - everything until your function returns is the latest data.
   • Use runAtTargetFps(fps, ...) to run code at a throttled FPS rate inside your Frame Processor
   • Use runAsync(...) to run code on a separate thread for background processing inside your Frame Processor. This can take longer without blocking the Camera.
3. Migrate VisionCamera to RN 0.71. Benefits:
   • Much simpler build setup. The CMakeLists/build.gradle files will be simplified as we will use prefabs, and a ton of annoying build errors should be fixed.
   • Up to date with latest React Native version
   • Prefabs support on Android
   • No more Boost/Glog/Folly downloading/extracting
4. Completely redesigned declarative API for device/format selection (resolution, fps, low-light, ...)
   • Control exactly how much FPS you want to record in
   • Know exactly if a desired format is supported and be able to fall back to a different one
   • Control the exact resolution and know what is supported (e.g. higher than 1080, but no higher than 4k, ...)
   • Control settings like low-light mode, compression, recording format H.264 or H.265, etc.
   • Add reactive API for getAvailableCameraDevices() so external devices can become plugged in/out during runtime
   • Add zero-shutter lag API for CameraX
5. Rewrite the native Android part from CameraX to Camera2
   • Much more stability as CameraX just isn't mature enough yet
   • Much more flexibility with devices/formats
   • Slow-motion / 240 FPS recording on Android
6. Use a custom Worklet Runtime instead of Reanimated
   • Fixes a ton of crashes and stability issues in Frame Processors/Plugins
   • Improves compilation time as we don't need to extract Reanimated anymore
   • Doesn't break with a new Reanimated version
   • Doesn't require > Reanimated v2 anymore
7. ML Models straight from JavaScript. With the custom Worklet Runtime, you can use outside HostObjects and HostFunctions. This allows you to just use things like TensorFlow Lite or PyTorch Live in a Frame Processor and run ML Models fully from JS without touching native code! (See proof of concept PR: feat: Add VisionCamera integration (imageFromFrame) facebookresearch/playtorch#199 and working PR for Tensorflow: feat: V3 tensorflow plugin #1633)
8. Improve Performance of Frame Processors by caching FrameHostObject instance
9. Improve error handling by using default JS error handler instead of console.error (mContext.handleException(..))
10. More access to the Frame in Frame Processors:
    • toByteArray(): Gets the Frame data as a byte array. The type is Uint8Array (TypedArray/ArrayBuffer). Keep in mind that Frame buffers are usually allocated on the GPU, so this comes with a performance cost of a GPU -> CPU copy operation. I've optimized it a bit to run pretty fast :)
    • orientation: The orientation of the Frame. e.g. "portrait"
    • isMirrored: Whether the Frame is mirrored (eg in selfie cams)
    • timestamp: The presentation timestamp of the Frame

Of course we can't just put weeks of effort into this project for free. This is why we are looking for 5-10 partners who are interested in seeing this become reality by funding the development of those features.
Ontop of seeing this become reality, we also create a sponsors section for your company logo in the VisionCamera documentation/README, and we will test the new VisionCamera V3 version in your app to ensure it's compatibility for your use-case.
If you are interested in that, reach out to me over Twitter: https://twitter.com/mrousavy or email: me@mrousavy.com

---

Demo

Here's the current proof of concept we built in 3 hours:

const runtimeEffect = Skia.RuntimeEffect.Make(`
  uniform shader image;
  half4 main(vec2 pos) {
    vec4 color = image.eval(pos);
    return vec4((1.0 - color).rgb, 1.0);
  }
`);

const paint = paintWithRuntimeEffect(runtimeEffect)

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    frame.drawPaint(paint)
  }, [])

  return <Camera frameProcessor={frameProcessor} />
}

---

Progress

Currently, I spent around ~60 hours to improve that proof of concept and created the above demos. I also refined the iOS part a bit and created some fixes, did some research and improved the Skia handling.

Here is the current Draft PR: #1345

Here's a TODO list:

• iOS
  • Set up Skia Preview
  • Pass Skia Canvas to JS Frame Processor
  • Make sure we use high performance Skia drawing operations
  • Make sure Frames are not out of sync of the screen refresh rate (60Hz / 120Hz)
  • Do some performance profiling and see if we can improve something
  • Make sure everything continues to work when not using Skia
  • Swap the REA Runtime with the custom Worklet Runtime
  • Implement synchronous Frame Processors
  • Implement runAtTargetFps
  • Implement runAsync
  • Implement toByteArray(), orientation, isMirrored and timestamp on Frame
  • Add orientation to Frame
  • Convert it to a TurboModule/Fabric
  • Rewrite to new simple & declarative API
• Android
  • Set up Skia Preview
  • Pass Skia Canvas to JS Frame Processor
  • Make sure we use high performance Skia drawing operations
  • Make sure Frames are not out of sync of the screen refresh rate (60Hz / 120Hz)
  • Do some performance profiling and see if we can improve something
  • Make sure everything continues to work when not using Skia
  • Swap the REA Runtime with the custom Worklet Runtime
  • Implement synchronous Frame Processors
  • Implement runAtTargetFps
  • Implement runAsync
  • Implement toByteArray(), orientation, isMirrored and timestamp on Frame
  • Add orientation to Frame
  • Convert it to a TurboModule/Fabric
  • Rewrite from Camera2 to CameraX
  • Rewrite to new simple & declarative API
• Documentation
  • Create documentation for write-back/Skia Frame Processors
  • Create documentation for synchronous Frame Processors
  • Create documentation for runAtTargetFps
  • Create documentation for runAsync
  • Create a realtime face blurring example
  • Create a realtime license plate blurring example
  • Create a realtime text recognition/overlay example
  • Create a realtime color grading/beauty filter example
  • Create a realtime face outline/landmarks detector example

I reckon this will be around 500 hours of effort in total.

Update 15.2.2023: I just started working on this here: feat: ✨ V3 ✨ #1466. No one is paying me for that so I am doing all this in my free time. I decided to just ignore issues/backlash so that I can work as productive as I can. If someone is complaining, they should either offer a fix (PR) or pay me. If I listen to all issues the library will never get better :)

[mrousavy]

Write-Back Frame Processors will shape the future of realtime image processing on mobile. For reference, let's compare this example, w/ native iOS/Android vs w/ RN VisionCamera:

Imagine how you'd do that in a native app:

• ~700 lines of code across ~5 files to set up the Camera accordingly
• ~300 lines of very low-level C-style Metal code across ~3 files to set up Metal
• ~40 lines of Metal Shader code to draw the Frame from a Texture (with the box) to the Layer
• ~30 lines of code to set up the Face Detector Module
• Then do the same thing again for Android.

In VisionCamera, it's just:

• ~4 lines of code to set up the <Camera>
• ~35 lines of code to set up the Face Detector Frame Processor Plugin
• ~13 lines of code to set up the Frame Processor that detects faces and draws on the view

Plus it already works on both platforms, you have way more flexibility (third party FP Plugins on npm) and can draw more (entire Skia API is available for drawing stuff), and you write basic JS (Command + S to instantly see your changes appear on your device, no need to rebuild).

[todorone]

if VisionCamera will become Fabric only, it will reduce amount potential users drastically, say 10x

[mrousavy]

Making it Fabric only will make a lot of things way simpler for me, especially talking about the C++/native code buildscripts. For legacy architecture support it requires a bunch of code, making it harder for me to maintain VisionCamera - and VisionCamera is a huge repository that I maintain myself in my free time.

Also, all older versions of VisionCamera still work on the legacy architecture. :)

EDIT: VisionCamera V3 will support the oldarch (Paper) too, so not only Fabric! It will require RN 0.71+ tho :)

[todorone]

Making it Fabric only will make a lot of things way simpler for me, especially talking about the C++/native code buildscripts. For legacy architecture support it requires a bunch of code, making it harder for me to maintain VisionCamera - and VisionCamera is a huge repository that I maintain myself in my free time.

Also, all older versions of VisionCamera still work on the legacy architecture. :)

Sure, i totally understand that it's quite a burden to support both archs, thanx a lot for such an amazing library!
I just wanted to mention, that it's already obvious that migration of real-world existing production apps to Fabric is the question of pretty far future w/o any intention to put request or pressure on You.🐱

[mrousavy]

I just wanted to mention, that it's already obvious that migration of real-world existing production apps to Fabric is the question of pretty far future w/o any intention to put request or pressure on You.

I disagree, a lot of our clients at Margelo are running Fabric in production. Facebook is too. We can help with the migration of your app if you want -> hello@margelo.io ;)

[patrickkabwe]

@mrousavy I really appreciate the effort you and the team at margelo.io are putting into this package. i think it would be great to also add barcode/qrcode scanning by default, because some of the community packages that have this implementation do not maintain them any more for example this.

Thanks

[kesha-antonov]

I just wanted to mention, that it's already obvious that migration of real-world existing production apps to Fabric is the question of pretty far future w/o any intention to put request or pressure on You.

I disagree, a lot of our clients at Margelo are running Fabric in production. Facebook is too. We can help with the migration of your app if you want -> hello@margelo.io ;)

Hi
I guess many projects can't migrate to Fabric because they depend on libraries that aren't Fabric compatible yet

Here's a table of those libs
reactwg/react-native-new-architecture#6

[mrousavy]

i think it would be great to also add barcode/qrcode scanning by default

I have received hundreds of requests like this seems like a much wanted feature haha... It definitely has to be a Frame Processor Plugin the way that VisionCamera is built. But how about this; if we manage to get enough partners/companies to sponsor the VisionCamera V3 project, I'll build a QR Code scanner plugin into VisionCamera or as a separate package maintained by me :)

[jonxssc]

Very nice of you to maintain this library, i am a big fan but from using this library i have some improvements that should also be looked at:

• Flash (When using the takesnapshot method the flash i think fires to early and the picture gets ruined)
• Zoom (It would be way easier to take a look at how expo camera did this by just adding a simple option true/false)
• Performance (The overall performance on Android is pretty bad, expo camera package is about 0,5 seconds faster)
• Documentation (The Documentation is not really god, to understand or even find much functions you have to take a look deep in the examples to understand most of it and see all functions/features)

To be clear, that is absolutely no hate and i am a big fan of this package and would love to use it in my apps, but for now i stay at expo camera, because it is faster and the flash is working properly. But i would love to switch, because expo camera has not much features and also has its problems like mirrored images on the front camera :)

[mrousavy]

Hi @jonxssc,

When using the takesnapshot method the flash i think fires to early and the picture gets ruined

takeSnapshot does not have a flash. It is literally just taking a snapshot of the view.

It would be way easier to take a look at how expo camera did this by just adding a simple option true/false

What? How does zoom={true} or zoom={false} make sense?

The overall performance on Android is pretty bad, expo camera package is about 0,5 seconds faster

The performance is pretty optimized, this is what VisionCamera is about. On iOS, it's really fast and optimized especially for streaming camera frames and initial launch time.
What exactly is 0,5 seconds faster in expo-camera? Open time? Photo time? Can you create a repo to reproduce this?
Remember that it's hard to compare Camera speeds if you're using a different format (e.g. if you use a higher-res camera device/format, it will obviously take longer to open the Camera).

The Documentation is not really god, to understand or even find much functions you have to take a look deep in the examples to understand most of it and see all functions/features

What functions do you mean specifically?

[Bamorem]

I agree for the QR code scanner

[obasille]

Thank you for working on this library! It saved me a ton of work :)
Looking forward to V3!

[mrousavy]

Thank you for working on this library! It saved me a ton of work :)
Looking forward to V3!

Thank you! Happy to hear it helps :)

[rocket13011]

Both Skia and Worklets are not mandatory at the moment.
I can remove them from peerDependencies, not sure why that'd be a a problem though.
I'm thinking about making RN Worklets required though as that greatly simplifies the build setup.

it's not a worry but you get a warning when you yarn ;)

PR > #1728

[rocket13011]

can you update the frameprocessor examples on the V3 part because they don't match the documentation at all. as a result, it's impossible to update my MLKit module and offer it to the community.

Thank you.

Ok the rc8 doesn't have the same callback function as the latest v3 version ^^.
I've been fooled!

https://github.com/mrousavy/react-native-vision-camera/blob/v3/android/src/main/java/com/mrousavy/camera/frameprocessor/FrameProcessorPlugin.java#L1C1-L25C2

[mrousavy]

Oh my god, finally. I got the OpenGL pipeline working. #1721

This was by far the most difficult project I ever worked on in my programming career. By far.

Previously, the setup was:

graph TD;
    Camera-->PreviewView;
    Camera-->PhotoCapture;
    Camera-->FrameProcessor;
    Camera-->VideoCapture;

Loading

• Camera streams 4k Frames
• PreviewView: a 720p SurfaceView
• FrameProcessor: streams low-resolution frames for processing (640x480)
• VideoCapture: writes the original full 4k Frames to file
• PhotoCapture: receives special 8k Frames once, no stream

This had a few limitations:

1. It was technically impossible to attach all 4 outputs at the same time, since Cameras on Android only allow attaching 3 outputs at the same time. This is why the supportsParallelVideoProcessing prop existed, if this was false, you cannot attach FrameProcessor and VideoCapture at the same time - which was obviously stupid if you wanted to do advanced processing.
2. The FrameProcessor never received the full original 4k frames, which is something you might want (eg for video chats) - it only received 640x480 Frames.
3. The FrameProcessor never ran synchronously, meaning we couldn't implement drawing to the Frame (e.g. a dog mask like on snapchat) and save that drawing to the video recording. It was just impossible
4. The VideoCapture was very abstracted - we couldn't rotate/mirror the Frames.
5. The VideoCapture was very abstracted - we couldn't flip the Camera while recording a Video. (like on Snap/Insta)

On iOS I had exactly the same problem with AVCaptureMovieFileOutput, but it was much simpler to fix since I just used the AVCaptureVideoDataOutput to receive the Frames as CMSampleBuffers, from there on I just called the Frame Processor with it and wrote them to a video file - easy.

On Android, as always, this isn't that simple. Advanced stuff like this is insanely difficult and I spent the last 3 weeks planning and implementing this, going back and forth from one approach to another, and finally I solved it by writing a fully custom OpenGL pipeline.

This is the Camera structure for native preview:

graph TD;
    Camera-->PreviewView;
    Camera-->PhotoCapture;
    Camera-->OpenGLPipeline;
    OpenGLPipeline-->FrameProcessor;
    OpenGLPipeline-->VideoCapture;

Loading

And this will be the Camera structure if you have a Skia preview (needs to be implemented first, but that's gonna be easy):

graph TD;
    Camera-->PhotoCapture;
    Camera-->OpenGLPipeline;
    OpenGLPipeline-->FrameProcessor;
    FrameProcessor-->VideoCapture;
    FrameProcessor--> SkiaPreviewView;

Loading

And this is how the OpenGL pipeline is built:

graph TD;
    Camera-->OffscreenTexture;
    OffscreenTexture-->PreviewView;
    OffscreenTexture-->VideoCapture;
    OffscreenTexture-->ImageReader-->FrameProcessor;

Loading

I think I can replace the ImageReader with an ImageWriter here to call the Frame Processor synchronously. Right now it's kinda async.

The most difficult things were just getting started with OpenGL on Android. There's virtually no documentation on this, it is a C-based API so everything you do is global (eglMakeCurrent(..), glBindTexture(..), ...), it is unclear on what can actually be shared across contexts, you always need target surfaces, you cannot share it across threads and you need to make sure everything is properly synchronized.

I'm using one OpenGL context for everything, the Camera writes frames to one offscreen-texture, and from there on I just render that to all different surfaces (automatically downsizing whenever needed).

It is much faster than before. I can:
i. Call the JS Frame Processor with the 4k Video Frame
ii. Downscale and render the 4k Frame to a 1080p Preview
iii. Record a Video + Audio and write the 4k Frames to a mp4 file
...all at the same time in just 6ms. 🤯

Note: I don't think I need a MediaCodec now. MediaRecorder should be enough I think.

In total, we now have a minimum amount of Surfaces, and can do a ton of advanced stuff in realtime while using pure GPU/hardware-accelerated operations. In other words; this is fast asf

Things left to do:

• Implement Skia rendering on Android (should be easy now, before it was just not possible lol)
• Fix Orientation (perf: Improve pixelFormat and add CameraDevice.sensorOrientation #1729)
• Fix Flash not working (feat: Flash #1717)
• Test everything

[Kypsis]

Nice job on the OpenGL pipeline. And sadly indeed there is very little info out there how to do this low level stuff. Years ago this guys repo helped me out a lot to get started: https://github.com/sixo/overlay

[mrousavy]

I'm stuck on the last part of V3. Skia for Android.

I need someone who has experience with OpenGL/Skia, I'll pay $500 to whoever can fix the issue. ‼️‼️📢 Please share this if you know someone who might know this

My Camera renders frames into a SurfaceTexture, which is a GL_TEXTURE_EXTERNAL_OES texture in OpenGL.

I can then simply render this OpenGL texture to all outputs (1920x1080 Preview EGLSurface, 4000x2000 Video Recorder EGLSurface) using a simple pass-through shader.

graph TD;
    Camera-->GL_TEXTURE_EXTERNAL_OES;
    GL_TEXTURE_EXTERNAL_OES -- PassThroughShader -->PO[Preview Output EGLSurface];
    GL_TEXTURE_EXTERNAL_OES -- PassThroughShader -->VO[Video Recorder Output EGLSurface];

Loading

Now I want to introduce Skia into this, which allows me to render onto the Camera Frame before passing it along to my outputs (e.g. to draw a red box onto the Frame). Since I can't directly render onto the same GL_TEXTURE_EXTERNAL_OES again, I created a separate offscreen texture (GL_TEXTURE_2D), and a separate offscreen frame buffer (FBO1) and attached them.

Now, when I render to the FBO1, the offscreen texture GL_TEXTURE_2D gets updated, and I want to pass that GL_TEXTURE_2D to my outputs then:

graph TD;
    Camera-->GL_TEXTURE_EXTERNAL_OES;
    GL_TEXTURE_EXTERNAL_OES -- Skia to FBO1 + drawing a red box --> GL_TEXTURE_2D;
    GL_TEXTURE_2D -- PassThroughShader -->PO[Preview Output EGLSurface];
    GL_TEXTURE_2D -- PassThroughShader -->VO[Video Recorder Output EGLSurface];

Loading

But for some reason, this draws the first frame only, then gets stuck and renders weird glitching artefacts on the screen.

IMG_7857.mp4

I'll pay $500 to whoever can fix the issue

If you want to take a look at this;

1. Clone this PR/branch: feat: Skia for Android #1731
2. Run yarn && cd example && yarn
3. Open example/android in Android Studio

The relevant files are:

1. onFrame(..): Called once the Camera put a new Frame into _inputTexture (GL_TEXTURE_EXTERNAL_OES).
2. SkiaRenderer::renderFrame(..): Called to render the Frame onto an offscreen texture (GL_TEXTURE_2D) and add some additional Skia commands (draw a red rectangle).
3. OpenGLRenderer::renderTextureToSurface(..): Called with the newly rendered to offscreen texture (GL_TEXTURE_2D) that contains the frame and the red rectangle. This will render the Frame to the output EGLSurface.
4. PassThroughShader::draw(..): Actually doing the pass-through rendering of the 2D texture that contains my Skia drawing. This is the shader that it uses. (sampler2D instead of samplerExternalOES, right?)

I'm not sure if Skia is doing some texture/buffer binding that I should undo, if there is a memory issue, if the pass-through shader is wrong, or if the transformation matrix also needs to be applied to the Skia Shader.

Any help appreciated!

[mrousavy]

Created a stackoverflow question as well this should have more details: https://stackoverflow.com/questions/77010619.

Logging off for today, spent way too much time debugging this

[rocket13011]

Hello,

Thanks for the rc9.

but Frame has lost getImage? how can I get it back?

Thank you.

[mrousavy]

@rocket13011 Yea that's the thing with Skia Frame Processors. See #1740 - i reverted all and will add it back

[mrousavy]

Hey all - just made the decision to remove Skia from VisionCamera after spending more than 300 hours on it. The integration is fun and cool for some demos, but it just significantly increases the complexity of the VisionCamera codebase, making it impossible for me to maintain it all by myself. See this PR for more details: #1740

VisionCamera V3 still has tons of new features and improvements and I'll try to launch it later today 😄

[jthure]

Hey all - just made the decision to remove Skia from VisionCamera after spending more than 300 hours on it. The integration is fun and cool for some demos, but it just significantly increases the complexity of the VisionCamera codebase, making it impossible for me to maintain it all by myself. See this PR for more details: #1740

VisionCamera V3 still has tons of new features and improvements and I'll try to launch it later today 😄

Great ambition though, and awesome that you "tried it out" (put hundreds of hours into it). Appreciate all your hard work!

[jlsneto]

Great! Thanks 🤩

[Crashtor]

Congratulations on this, been following your pain closely over the last months. Dying to try it out!

[melhotiby]

i think it would be great to also add barcode/qrcode scanning by default

I have received hundreds of requests like this seems like a much wanted feature haha... It definitely has to be a Frame Processor Plugin the way that VisionCamera is built. But how about this; if we manage to get enough partners/companies to sponsor the VisionCamera V3 project, I'll build a QR Code scanner plugin into VisionCamera or as a separate package maintained by me :)

@mrousavy - Were you ever able get barcode into v3 or is that for a later date?

[druti]

Will V3 have a "onRecordingStarted" callback?

[druti]

@mrousavy when can we expect react-native-vision-camera to return a promise from startRecording ? As per the comment here

react-native-vision-camera/package/example/src/views/CaptureButton.tsx

Line 119 in f69e3df

// TODO: wait until startRecording returns to actually find out if the recording has successfully started

[ombogdanJoinToIt]

Both Skia and Worklets are not mandatory at the moment.

I can remove them from peerDependencies, not sure why that'd be a a problem though.

I'm thinking about making RN Worklets required though as that greatly simplifies the build setup.

they are mandatory. without react-native-worklets-core for example I get the error [VisionCamera] react-native-worklets-core found, Frame Processors enabled!

[mrousavy]

@melhotiby yes, V3 now has a CodeScanner API! 🥳

@ombogdanJoinToIt they are not mandatory. On both iOS and Android I have a CI pipeline that explicitly tests the build without Worklets/Frame Processors.
I know for a fact that this worked on iOS last week as I tested it, not sure if that works on Android but if the CI is green it should work.

[alexvazquez]

Can I start using V3 on my iOS and Android devices? what branch is the stable or latest and how can I import it?

[mrousavy]

V3 stable is on current main here and on npm under version 3 (latest).

The Skia stuff is not yet part of V3, but I've been doing some interesting experiments lately which I'll announce soon.

[karventhan13]

Hi @mrousavy i want to create face detection with face blur, while capturing the video so, can you help me with the compatible version for react native vision camera and related library?

[timotismjntk]

Hi @mrousavy i want to create face detection with face blur, while capturing the video so, can you help me with the compatible version for react native vision camera and related library?

Is that only for android?

[karventhan13]

Hi @timotismjntk No for both android and ios