`FaceControls` #1461

abernier · 2023-05-22T13:06:12Z

Note

Merged, see <FaceControls> doc

Why / What

DEMO: https://abernier.github.io/fld/ as well as the story

facecontrols-videotexture_720p.mov

Control the camera with your head/eyes:

<FaceControls />

or a custom camera, with your eyes:

<FaceControls camera={cam01} eyes />

This PR basically consists in:

an updated Facemesh (fully backward-compatible with current version)
a FaceLandmarker provider to bring @mediapipe/tasks-vision
and a FaceControls

Checklist

Documentation updated (example)
Storybook entry added (example)
Ready to be merged

vercel · 2023-05-22T13:06:17Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
drei	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 4, 2023 4:13pm

codesandbox-ci · 2023-05-22T13:07:14Z

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit e3f4365:

Sandbox	Source
magical-rain-8dur7g	Configuration
Ground reflections and video textures	Configuration
arc-x-pmndrs-colors	Configuration

CodyJasonBennett · 2023-06-02T21:25:32Z

src/core/FaceLandmarker.tsx

+    FilesetResolver.forVisionTasks(basePath)
+      .then((vision) => FaceLandmarkerImpl.createFromOptions(vision, options))
+      .then((faceLandmarker) => setFaceLandmarker(faceLandmarker))
+      .catch((err) => console.error('error while creating faceLandmarker', err))


I'd consider using suspense here.

~~but how to deal with ret.close() cleanup with suspend?~~

Ok, following @drcmda additional advices on that, I've now converted it to suspend, but I let you check my impl is correct

CodyJasonBennett · 2023-06-02T21:26:09Z

src/core/FaceLandmarker.tsx

+    return () => void ret?.close()
+  }, [basePath, options])


If options is an object, this will break reference every rerender. Either destructure or use an expression that is immune like JSON.stringify -- not ideal but it works.

Ok, I've set opts = JSON.stringify(options) and set my dep with

src/core/FaceControls.tsx

CodyJasonBennett · 2023-06-02T21:29:46Z

src/core/FaceControls.tsx

+  const [stream, setStream] = useState<MediaStream>()
+
+  const videoTextureApiRef = useRef<VideoTextureApi>(null)
+
+  const faceControls = useFaceControls()
+  useEffect(() => {
+    let strm: MediaStream
+
+    if (!videoTextureSrc) {
+      navigator.mediaDevices
+        .getUserMedia({
+          audio: false,
+          video: { facingMode: 'user' },
+        })
+        .then((s) => {
+          strm = s
+          faceControls.dispatchEvent({ type: 'stream', stream: strm })
+          setStream(strm)
+        })
+        .catch(console.error)
+    }
+
+    return () => strm?.getTracks().forEach((track) => track.stop())
+  }, [faceControls, videoTextureSrc])


Suspense would also be helpful here. I'm pedantic about this now as it's breaking later.

ok now done ;) I let you check my impl

abernier · 2023-06-02T23:02:46Z

thanks for the review @CodyJasonBennett, will integrate them tomorrow ;)

abernier · 2023-06-03T22:24:00Z

.storybook/stories/FaceControls.stories.tsx

+      <color attach="background" args={['#303030']} />
+      <axesHelper />
+
+      <React.Suspense fallback={null}>


I guess React.Suspense is now required here, since <FaceLandmarker> provider is now suspended?

I'm not sure

abernier · 2023-06-03T23:29:31Z

src/core/FaceControls.tsx

@@ -284,7 +285,11 @@ export const FaceControls = forwardRef<FaceControlsApi, FaceControlsProps>(
    const faceBlendshapes = faces?.faceBlendshapes?.[0]
    return (
      <FaceControlsContext.Provider value={api}>
-        {webcam && <Webcam ref={webcamApiRef} autostart={autostart} videoTextureSrc={webcamVideoTextureSrc} />}
+        {webcam && (
+          <Suspense fallback={null}>


same here: is Suspense really required? I'm not sure

Your call whether suspense should go here or in user-land around the component.

abernier · 2023-06-04T15:56:29Z

@CodyJasonBennett I think I've now addressed your precious feedbacks. If you have no other ones, I think I would be ready to merge

github-actions · 2023-06-05T22:47:18Z

🎉 This PR is included in version 9.74.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

markhughes · 2023-12-11T11:34:00Z

FYI this commit breaks react native builds.

Use <= v9.73.4 for react native.

CodyJasonBennett · 2023-12-11T12:05:59Z

@markhughes, can you create an issue? I think we'll have to remove it from the native target since iOS doesn't support JIT/WASM.

vercel bot deployed to Preview May 22, 2023 13:06 View deployment

vercel bot deployed to Preview May 24, 2023 15:26 View deployment

abernier changed the title ~~Facemesh v2.1~~ FaceControls May 25, 2023

vercel bot deployed to Preview June 2, 2023 10:09 View deployment

vercel bot deployed to Preview June 2, 2023 11:10 View deployment

abernier force-pushed the abernier/facemesh-v2.1 branch from 5fa25c9 to 2d39bdd Compare June 2, 2023 11:18

vercel bot deployed to Preview June 2, 2023 11:21 View deployment

abernier requested review from drcmda and CodyJasonBennett June 2, 2023 11:32

abernier marked this pull request as ready for review June 2, 2023 11:33

abernier added 4 commits June 2, 2023 23:18

feat(Facemesh): eyes and @mediapipe/tasks-vision facemesh v2

426642d

chore: simplifying with visionTasks transformationMatrix and blendshapes

e9f34e2

chore: facemesh final

a81c2c1

feat: facecontrols component

eef118c

abernier force-pushed the abernier/facemesh-v2.1 branch from 2d39bdd to eef118c Compare June 2, 2023 21:22

vercel bot deployed to Preview June 2, 2023 21:24 View deployment

CodyJasonBennett reviewed Jun 2, 2023

View reviewed changes

src/core/FaceControls.tsx Outdated Show resolved Hide resolved

CodyJasonBennett reviewed Jun 2, 2023

View reviewed changes

fix: facelandmarker suspend + JSON.stringify options dep

8a8747b

vercel bot deployed to Preview June 3, 2023 22:20 View deployment

abernier commented Jun 3, 2023

View reviewed changes

fix: webcam suspend

3963d42

vercel bot deployed to Preview June 3, 2023 22:47 View deployment

abernier requested a review from CodyJasonBennett June 3, 2023 22:54

abernier commented Jun 3, 2023

View reviewed changes

fix: missing suspend dep

06c02e2

vercel bot deployed to Preview June 4, 2023 08:00 View deployment

chore: passing args

6fdc861

vercel bot deployed to Preview June 4, 2023 08:32 View deployment

fix: missing dep

d39e0d1

vercel bot deployed to Preview June 4, 2023 09:16 View deployment

docs: adding csb ex

a950eb3

vercel bot deployed to Preview June 4, 2023 12:46 View deployment

docs: update thumb

4bad468

vercel bot deployed to Preview June 4, 2023 12:57 View deployment

abernier added 2 commits June 4, 2023 17:37

docs: facelandmarker README entry

6ef9207

chore: removing unused vars in story

0ceaf73

vercel bot deployed to Preview June 4, 2023 15:39 View deployment

fix: remove circular dep

084e769

vercel bot deployed to Preview June 4, 2023 15:45 View deployment

docs: wording

0f6b51e

vercel bot deployed to Preview June 4, 2023 15:47 View deployment

docs: rVFC polyfill note

e3f4365

vercel bot deployed to Preview June 4, 2023 16:13 View deployment

abernier merged commit 1254b90 into master Jun 5, 2023

github-actions bot added the released label Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`FaceControls` #1461

`FaceControls` #1461

abernier commented May 22, 2023 •

edited

Loading

vercel bot commented May 22, 2023 •

edited

Loading

codesandbox-ci bot commented May 22, 2023 •

edited

Loading

CodyJasonBennett Jun 2, 2023

abernier Jun 3, 2023 •

edited

Loading

abernier Jun 3, 2023 •

edited

Loading

CodyJasonBennett Jun 2, 2023 •

edited

Loading

abernier Jun 3, 2023

CodyJasonBennett Jun 2, 2023

abernier Jun 3, 2023

abernier commented Jun 2, 2023

abernier Jun 3, 2023 •

edited

Loading

abernier Jun 3, 2023

CodyJasonBennett Jun 4, 2023

abernier commented Jun 4, 2023 •

edited

Loading

github-actions bot commented Jun 5, 2023

markhughes commented Dec 11, 2023

CodyJasonBennett commented Dec 11, 2023

FaceControls #1461

FaceControls #1461

Conversation

abernier commented May 22, 2023 • edited Loading

Why / What

Checklist

vercel bot commented May 22, 2023 • edited Loading

codesandbox-ci bot commented May 22, 2023 • edited Loading

CodyJasonBennett Jun 2, 2023

Choose a reason for hiding this comment

abernier Jun 3, 2023 • edited Loading

Choose a reason for hiding this comment

abernier Jun 3, 2023 • edited Loading

Choose a reason for hiding this comment

CodyJasonBennett Jun 2, 2023 • edited Loading

Choose a reason for hiding this comment

abernier Jun 3, 2023

Choose a reason for hiding this comment

CodyJasonBennett Jun 2, 2023

Choose a reason for hiding this comment

abernier Jun 3, 2023

Choose a reason for hiding this comment

abernier commented Jun 2, 2023

abernier Jun 3, 2023 • edited Loading

Choose a reason for hiding this comment

abernier Jun 3, 2023

Choose a reason for hiding this comment

CodyJasonBennett Jun 4, 2023

Choose a reason for hiding this comment

abernier commented Jun 4, 2023 • edited Loading

github-actions bot commented Jun 5, 2023

markhughes commented Dec 11, 2023

CodyJasonBennett commented Dec 11, 2023

`FaceControls` #1461

`FaceControls` #1461

abernier commented May 22, 2023 •

edited

Loading

vercel bot commented May 22, 2023 •

edited

Loading

codesandbox-ci bot commented May 22, 2023 •

edited

Loading

abernier Jun 3, 2023 •

edited

Loading

abernier Jun 3, 2023 •

edited

Loading

CodyJasonBennett Jun 2, 2023 •

edited

Loading

abernier Jun 3, 2023 •

edited

Loading

abernier commented Jun 4, 2023 •

edited

Loading