version 2 of the classic garbage openvtuber, this time with more terrible technology powered by tensorflow
the new version is much better suited to bad lighting or webcams and has basic support for upper-body pose estimation through posenet
This apps starts a PeerJS server, enabling it to be recorded by an OBS browser capture.
You must run it through Express, not by opening index.html.
Install node.js (tested with v12), open a command prompt in the downloaded directory and run npm install followed by npm start.
Open the app by going to http://localhost:9000/
This app was tested on a Ryzen APU and took 60-80% of the GPU running all the models at once.
"Minimum" environment for good detection
- Head, mouth: Pretty much anything will work
- Arms, hands: To avoid awkward positioning, try to place the camera so only your shoulders and above are showing - camera quality doesn't matter much
- Eye tracking, blinking: Webcam quality should be good enough to stream with your real face
Generally, performance will degrade significantly with more than one model (facemesh, posenet, handpose) running at once
Using a window capture is impossible because the app is based on a graphics-accelerated Electron instance. You can use display capture and crop it to record the model.
A less finicky way to record the model is to create a "video call" through PeerJS.
- Add the peerjs.html to a browser capture while the Electron app is running.
- Right click the browser capture and click Interact
- Copy the key shown in the browser capture, paste it into the "PeerJS Connection ID" field in the main application, and click Call.
- Chroma key out the green background
- tensorflow.js + prebuilt models - facemesh, posenet, handpose
- three.js + GLTFLoader + OrbitControls
- three-vrm + their default model
- tween.js
- canvasfilters (filters.js)
- express + peerjs
- ml5
- JSZip, FileSaver.js
- stats.js
- lploc.js
PoseNet solution using atan2 was inspired by atskimura