Greenscreen background image

[RascalTwo]

• Please check if the PR fulfills these requirements

• You've followed the contributing guidelines
• You've adheared the code of conduct

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

• Bug Fix
• New Feature
• Documentation Update
• Other

If bug fixes or new features

• What is the current behavior? (You can also link to an open issue here)

You can only publish a raw video source, closes #5.

• What is the new behavior (if this is a feature change)?

You can now publish a video source with colors chroma keyed and replaced with a background image.

The user can upload a image of their choice, and once uploaded, customize the X/Y position, width, and height of the image.

To assist with positioning, a Overlay option is provided to draw the image on top of the video.

Additionally, the user may select a low and high colors as their chroma key color if they wish to use a non-green color.

Although it's not the most user friendly due to using the native color picker, which uses various color representations - HSV, RGB, etc - while the actual color representation used for calculations is HSL.

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No

• Other information:

In order to have better performance, the image manipulation is done in a Worker.

I attempted to make it reusable by exposing a getUserStream method, so the user can be given another chance to provide a video source by calling this method again.

---

I couldn't come up with a simple way to directly return the native stream when no background image is in use, but have it replaceable with the canvas one when the background image is in use.

Therefore even when there is no background image, the video is still being passed through a canvas.

---

I avoided actual styling of the form aside from making it collapsible with the Open/Close button.

---

While requestAnimationFrame() is used, it's limited to the reported frame rate of the user input video.

---

I had intended to pass the ImageData by reference to the Worker, but I ran out of time - doing this I imagine would further increase performance, with only modifications to sending/receiving portions of the code being required.

---

My attempt to use the implementation described here did not have positive results in what testing I did - the inability to use a Worker resulted in decreased performance, and this was before I was even able to implement all the existing customization features - you can see it here if you're interested.

[netlify]

Deploy request for build-a-thing-video pending review.

Review with commit 9f823a8

https://app.netlify.com/sites/build-a-thing-video/deploys

[phazonoverload]

Thank you so very much for this @RascalTwo! This is a great contribution, and it's good to see you free up the main thread with a worker. However, when testing, I wasn't able to get the output from the canvas into the publisher - instead it looked like this:

Any idea why this might be the case?

[phazonoverload]

I'm using Chromium 86 in Brave 1.15 😄

[RascalTwo]

After testing, I'm quite confident it was due to the fact that I had disabled alpha in the canvas contexts, so hopefully this has resolved it for you also.

Only in hindsight I realized that since the fake device is all green, the greenscreen effect isn't actually visible above - so the below gif actually displays the effect working.

---

Additionally, I've brought it up to date with the latest main commit - 685da9b - and added some basic styles that should match with the rest of the page.

Unrelated, but turns out my errors in chromium browsers with opentok were due to the low quality of my USB webcam, hence I used a fake device/video to demonstrate it in Brave.

[phazonoverload]

Hey! I tried this but still couldn't get it to work. This time, I can't see anything except the overlay image, and my chroma min and max are sensible. This is still an excellent piece of work, and I'll mark it as valid for the sake of Hacktoberfest.

[phazonoverload]

If you want to jump on a call to try and debug this let me know. I'll close this for now unless you want to do any further work 💖

[RascalTwo]

I'm glad the initial error was truly solved, wasn't able to be there live but I did watch your VOD.

The issue you had with pulling my PR was likely due to the fact that I rebased my changes to match the updated styles at the time, so this time I didn't do that so you shouldn't encounter the same issue again.

As for the chroma-keying not working, it appears that's due to the rigidity/inaccuracy of the algorithm I choose, so I've swapped it out for a different one that I can get working on just about any greenscreen video - even yours, even at the low resolution I was able to scrape from the VOD.

After seeing how much performance really drops in the VOD, I also decided to make performance a priority, so along with various micro optimizations, it's now much faster - probably as fast as I personally can make it without getting WebAssembly/WebGL involved.

Just to be safe though, in case performance is still lackluster, I've added options to allow for customization of the output - turning down the output resolution allows for higher frame rates.

---

Now I'm no Vue expert, so while I've got it all working, It's likely I've broken some Vue rules or missed some things that Vue offers to make certain things much easier, but hopefully not too many though.

---

I personally follow these steps to get each background working:

• Upload image
• Enable Preview Overlay
• Adjust background Width/Height as needed
• Disable Preview Overlay
• Pick the Darkest and Lightest chromas
  • Pausing if needed
• If needed, adjust Tolerance
  • Decrease if the person is being replaced
  • Increase if there is a significant chroma border around the person, or not all chroma is being detected

I was only able capture your webcam from the VOD at 320x180, hence the low quality, but as you see with the second example - with a resolution of 1920x1080 - it does work when the quality is higher.

Lost some quality to get the .gif under 10 MB for the upload limit, here's the higher quality video if you're interested.

---

UI Clarification

I've also attempted to make the UI clearer, so hopefully the options are more obvious, but just in case here's a quick rundown:

Background

The Pause/Play button pauses or plays the input video - useful to fine tune the settings without having to deal with new frames or that one pixel you're trying to pick jumping around.

• Preview Overlay
  • Places the background image on top no matter the chroma
  • I thought it'd be useful to see the position without guessing,
    • Especially since the default chroma values will probably make the image invisible to most
• X/Y
  • Image Offsets from the top-left corner
  • Higher X is right
  • Higher Y is down
• Width/Height
  • The dimensions of the image
  • New images are automatically shrunk (aspect ratio preserved) when larger then the input video
• Chrome Dark/Chroma Light
  • The darkest/lightest color to replace with background image
  • Pick button
    • Makes it so whatever pixel you click on in the canvas will be used for this chroma
• Tolerance
  • How close outside of the chrome colors the color can be get replaced
  • Higher means pixels further outside the dark->light range are replaced
  • Lower means only pixels within the dark->light range are replaced
  • I recommend only adjusting this after picking chroma values, although the default 5% seems to work for most greenscreens

Output

• Width/Height
  • The dimensions of the output canvas/video
  • If there is high frame loss, decreasing these values should decrease the frame loss
• FPS
  • The target Frames Per Second to render at

With all the inputs being instantly reactive thanks to Vue, changes are now visible instantly instead of having to wait for focus to leave the input (the change event), so it's now easier and quicker to fine tune all these settings.

---

I appreciate the offer to jump into a call, and the time you've taken for this single issue, perhaps if these latest changes don't work as expected I'll take you up on that offer.