Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithreading with worklets #1561

Closed
wants to merge 6 commits into from
Closed

Conversation

karol-bisztyga
Copy link
Contributor

Description

This idea has been postponed due to low priority.
The solution here is dirty, it's more like a POC for the future if we decide to focus on this topic again.
The goal is to be able to run worklets on multiple threads different than JS and UI.

Use cases

  • downloading data from the Internet
  • processing video/audio/images
  • obtaining a contact list from the device
  • in fact, processing any data in the background in order to avoid blocking UI or JS.

Examples

example code
import React, { useState } from 'react';
import {
  View,
  Button,
  Text,
} from 'react-native';
import {
  useSharedValue,
  spawnThread,
} from 'react-native-reanimated';


const Child = () => {
  const sv = useSharedValue(0)

  spawnThread(() => {
    'worklet';
    sv.value = 1.1;
    console.log('here thread #1 begin', sv.value, _WORKLET);
    for (let i = 0; i < 3000000000; ++i) {}
    sv.value = 1.2;
    console.log('here thread #1 end', sv.value, _WORKLET);
  });

  spawnThread(() => {
    'worklet';
    sv.value = 2.1;
    console.log('here thread #2 begin', sv.value, _WORKLET);
    for (let i = 0; i < 3000000000; ++i) {}
    sv.value = 2.2;
    console.log('here thread #2 end', sv.value, _WORKLET);
  });

  return (
    <View>
      <Text>Child</Text>
      <Button title="check responsiveness" onPress={() => {
        console.log('UI responsive', sv.value, Math.random())
      }} />
    </View>
  );
}

export default function App() {
  const [s, setS] = useState(0);

  return (<View>
    <Button title="change state" onPress={()=>{ setS(s+1) }} />

    { s%2 === 1 ? <Child /> : <Text>EMPTY</Text>}
  </View>)
}

@mrousavy
Copy link
Contributor

mrousavy commented Dec 28, 2020

Hi @karol-bisztyga! Thanks for taking time to look into this! I'm currently writing a Camera TurboModule, and I'm planning on creating an API where the user can process frames using JS.

So for example:

const processFrame = useWorkletCallback((frame: CameraFrame) => {
  // process frame data in this worklet, for example to detect faces, render a square ontop of a QR code, or whatever.
}, []);


return <Camera {...props} processFrame={processFrame} />;

The benefits of this are hopefully immediately clear:

  1. The user has full control over the video frames of the Camera and doesn't have to go native for stuff like object detection.
  2. The processFrame function does not block the UI or JS thread.
  3. There is no bridge-spam.

I'm currently halfway through implementing the Camera base stuff (format selection, capturing videos, capturing photo, ...), but I'm already excited to have something working in the near future.

Just leaving this here to make it clear that multithreaded worklets do indeed have many use cases, and perhaps we can work together to extend the proof of concept to a fully functional use case with the Camera process frame API 👍

I'll be posting status updates on my Twitter, #1, #2

@mrousavy
Copy link
Contributor

mrousavy commented Feb 26, 2021

To those interested: I've published the react-native-vision-camera library (🎉) and am now working on the Frame Processor thing. All progress is public at: mrousavy/react-native-vision-camera#2

EDIT

I've finished the Frame Processors API for iOS and Android!! 🎉🎉🎉 - This is an API where you can run multithreaded JS code on every camera frame in realtime (endless possibilities for video frame processing such as Face filters, QR code scanners, WebRTC and more). Take a look at the docs: VisionCamera - Frame Processors

Also, I created a library for running JS code on a separate thread, basically exactly what this PR attempted to do (Multithreading with Worklets for running heavy code in there in parallel) - Check it out: mrousavy/react-native-multithreading

@likern
Copy link
Contributor

likern commented Mar 1, 2021

It's exactly what I'm looking for and dreamed of. It allows not touching native as much as possible.

I need this for video recorder app.
And it would be cool to use another thread to work with sqlite database, not polluting main and UI thread (where reanimated animations might run).

While I can't help, eagerly waiting for this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants