Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Azure Kinect support #33

Closed
SirLynix opened this issue Oct 15, 2020 · 32 comments
Closed

Azure Kinect support #33

SirLynix opened this issue Oct 15, 2020 · 32 comments
Assignees
@SirLynix
Labels
enhancement New feature or request

Comments

@SirLynix
Copy link
Owner

Good news everyone.

Microsoft contacted me about this plugin and is willing to help me support the Azure Kinect by sending one to me so I can add support for it!

This also means I will work on the plugin again, after not touching it for a few months due to other projects.

I'll use this thread to keep you informed about azure kinect support in obs-kinect.
@SirLynix SirLynix added the enhancement New feature or request label Oct 15, 2020
@SirLynix SirLynix self-assigned this Oct 15, 2020
@SirLynix
Copy link
Owner Author

Just received the beast!
image

@cefoot
Copy link

cefoot commented Oct 17, 2020

I'm looking forward to this! If I can assist you (dev or test) let me know!

@SirLynix
Copy link
Owner Author

Initial Azure Kinect support is done (color, depth, infrared and depth to color mapping which means the greenscreen works).

image

I had to put a hat on because Azure Kinect has some troubles with my hair, but it flickers a lot less than v2.
Body tracking support is on its way but it's gonna be a little bit more complex than with Kinect v1/v2 because it's a separate SDK/lib.

Shouldn't take long

@SirLynix
Copy link
Owner Author

image

Shouldn't take long before a test version.
Since I never properly released 0.3, this will be part of this release!

@SirLynix SirLynix mentioned this issue Oct 20, 2020
Open
@cefoot
Copy link

cefoot commented Oct 21, 2020

Hi looking really forward to this.. so much that I tried to build it after I added some missing parts to the conflig.lua I was able to build unfortunately it craches when I try to add the kinect Source to a scene.
But since this is my first try with building an obs plugin the problem is probably on my side ;)

@SirLynix
Copy link
Owner Author

You're probably missing a few plugins files, but don't worry, packages are coming very soon!

@SirLynix
Copy link
Owner Author

SirLynix commented Oct 21, 2020 •
edited
Loading

Here's a test package of obs-kinect with Azure Kinect support (64bits only for now):
obs-kinect-azure-test-fixed.zip

Unpack the obs-studio folder to your obs-studio folder.
Please note this version doesn't include body filtering files for Azure Kinect! It will try to load them at runtime but will disable body filtering if it fails.

I've updated the README for Azure Kinect if you want to test body filtering.

Please let me know how it works (or not) for you, and let me know of any bug you encounter!

@SirLynix SirLynix mentioned this issue Oct 21, 2020
Open
@amourao
Copy link

amourao commented Oct 21, 2020

@SirLynix Thank you for taking the time and adding support the Azure Kinect so quickly.

I have an additional request and offer to help: as you may know, the Azure SDK is also available for Linux (tested on Ubuntu 18.04).
How hard would it be to try and compile it on Linux?

This is also sort of related to #9

@SirLynix
Copy link
Owner Author

Actually I planned already to release the next version of obs-kinect on Linux (with Azure Kinect support only for now, I hope to bring Kinect V1 and Kinect V2 support next), don't worry!

It shouldn't be difficult, I've been thinking of supporting Linux (and macOS) since I started working on 0.3, at this point I think the code is close to ready (it doesn't rely on Windows specific code, which only exists in Sdk10 and Sdk20 backends).
I'll give it a try soon!

@cefoot
Copy link

cefoot commented Oct 21, 2020 •
edited
Loading

I changed the folder in the zip from obs-studio\bin\64bits to obs-studio\bin\64bit (without s) and everything is working fine! :)
kinect_v2

just a short first test for now ;)

@SirLynix
Copy link
Owner Author

You're right, I messed up the folder name. I uploaded a fixed package in my original message.

Glad to see it's working! It looks great! Did you try body tracking yet? you'll have to install Azure Kinect Body Tracking Sdk dependencies for that.

And some new cool effects are coming as well #35

@Funkcorner
Copy link

Does the Azure Kinect cause problems with its very high resolution video? 12MP - that's a lot more for the GPU to think about...

@SirLynix
Copy link
Owner Author

4K video is pretty fine on my computer, even with green screen enabled (taking about 20% of GPU). Except for some lag (probably due to depth to color mapping which is pretty compute-heavy especially on 4K).

@reneschulte
Copy link

Awesome work! Gave it a try and it works really well, also Body masking, though I disable it as it's too heavy on my GPU.
I noticed the typical holes that can appear with depth data. Are you thinking about some extra image processing to mitigate that like dilation and erosion filter combo?

@SirLynix
Copy link
Owner Author

SirLynix commented Dec 4, 2020

Hello, and thank you!

Yes I'm thinking of using post-processing to improve the filtering. I'm also trying to see if that can be improved at the Azure Kinect level (see here).

@reneschulte
Copy link

Makes a ton of sense @SirLynix that it should be improved by the AK itself.
We both have the same problem with hair color! 😁
https://youtu.be/WhT3ZpQrvf0

Great work and glad Microsoft donated that Azure Kinect and you got it quickly implemted. 😉

@SirLynix
Copy link
Owner Author

SirLynix commented Dec 4, 2020

Thanks a lot for the video! Glad my plugin is useful to you!

I have a lot of work to do for other projects right now but I will get back to it soon and try to release the (hopefully) stable version asap!

@dotMorten
Copy link

Does this support more than one kinect camera, like the "official" green screen example? https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen#why-would-more-than-1-camera-be-used

@SirLynix
Copy link
Owner Author

SirLynix commented Dec 4, 2020

Does this support more than one kinect camera, like the "official" green screen example? https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen#why-would-more-than-1-camera-be-used

It does support multiple Kinect when using multiple sources but doesn't support combining multiple Azure Kinect into one result yet.
However that's something I'm thinking about, it could be cool to support it (I won't be able to test it myself however).

@dotMorten
Copy link

Time for Microsoft to send you a second one :-)

It would help with the shadow effects I see here:
image

@SirLynix
Copy link
Owner Author

SirLynix commented Dec 5, 2020

It would help with the shadow effects I see here:

Yes indeed, but I will soon try to work on a post-process solution (see #32 )

@cefoot
Copy link

cefoot commented Dec 5, 2020

this could also help I guess
https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen

@SirLynix
Copy link
Owner Author

SirLynix commented Dec 5, 2020

this could also help I guess
https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen

Yes I did check the code. Nothing too complex since it relies on a computer vision library to get the transformation between the two cameras. I think I can do something similar but I'll need someone with two azure kinect to test it.

@cefoot
Copy link

cefoot commented Dec 8, 2020

thx again! I had so much fun in a meeting today!
https://youtu.be/PgZNRoZY8rM

@SirLynix
Copy link
Owner Author

The latest build can be found at the end of #7 with color camera settings for Azure Kinect!

@Ahmed-KASAPBASI
Copy link

I need to record videos (RGB, depth and skeleton) with voice at the same time (synchronously) from 3 different Azure kinect which are connected to one PC.

How I can do that?
can OBS do that, what are the possible formats of the recorded videos ?
Please, can you help me?

@SirLynix
Copy link
Owner Author

Hi.

I doubt obs and obs-kinect can really help you, you should try using the Kinect SDK directly: https://github.com/microsoft/Azure-Kinect-Sensor-SDK

@Ahmed-KASAPBASI
Copy link

Hi.

I doubt obs and obs-kinect can really help you, you should try using the Kinect SDK directly: https://github.com/microsoft/Azure-Kinect-Sensor-SDK

Thank you so much for your reply.
I searched a lot about this topic but I could not get an answer.
When I searched, I found that Azure kinect SDK can not do that.
Do you have an other advice, please. OR do you know any person can help me?

@SirLynix
Copy link
Owner Author

Of course the SDK can do that, it has its own recorder: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/tools/k4arecorder

@Ahmed-KASAPBASI
Copy link

Of course the SDK can do that, it has its own recorder: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/tools/k4arecorder

Thank you a gain.
Yes, you can record RGB videos without the voice but in my issue I need to record 3 types of videos (RGB, depth and Skeleton) with voice at the same time.

@SirLynix
Copy link
Owner Author

SirLynix commented Nov 22, 2021 •
edited
Loading

I understand, I'm not saying k4arecorder is the solution to all your problems, but it shows how you can record color and depth using the SDK, adding skeleton and voice shouldn't be a big deal (see k4aviewer which records audio).

For skeleton tracking, see https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download

See also other examples for synchronisation between Azure Kinect.

I don't think there already is a tool which does everything you need, you'll have to code it yourself, feel free to look how I use Azure Kinect in obs-kinect if that helps)

@Ahmed-KASAPBASI
Copy link

I understand, I'm not saying k4arecorder is the solution to all your problems, but it shows how you can record color and depth using the SDK, adding skeleton and voice shouldn't be a big deal (see k4aviewer which records audio).

For skeleton tracking, see https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download

See also other examples for synchronisation between Azure Kinect.

I don't think there already is a tool which does everything you need, you'll have to code it yourself, feel free to look how I use Azure Kinect in obs-kinect if that helps)

I appreciate your patience.
Final question, do you know any program or software that can help me to get Skeleton videos from RGB videos?
If there is, I can solve the Skeleton videos problem instead of recording the Skeleton videos separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SirLynix SirLynix

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dotMorten @amourao @SirLynix @reneschulte @cefoot @Funkcorner @Ahmed-KASAPBASI