Skip to content
This repository has been archived by the owner on Aug 22, 2024. It is now read-only.

System lock up with multiple devices #1905

Open
DrMaxP opened this issue Mar 16, 2023 · 11 comments
Open

System lock up with multiple devices #1905

DrMaxP opened this issue Mar 16, 2023 · 11 comments
Labels
Bug Something isn't working Triage Needed The Issue still needs to be reviewed by Azure Kinect team members.

Comments

@DrMaxP
Copy link

DrMaxP commented Mar 16, 2023 •
edited
Loading

I am using the latest SDK with multiple cameras connected and powered by USB C only. All cameras have the latest firmware. I see consistent whole level system hangs (all I can do is reboot) when using serval devices at once - e.g. using multiple instances of the Kinect viewer app or 3rd party solutions. There is nothing in the event log to suggest what happens. What else could I do to investigate the cause of the system lock up? This is on Window 11. The lock up occurs consistently, within a few minutes and normally within a free seconds.

I still need to look at the Azure Kinect logs, I've enabled the registry keys but can't locate the logs.

Not sure if this is a USB issue or a low level driver issue, given that the system is hanging and has to be rebooted. Seems to happen irrespective of which USB C ports / controllers I use.
@DrMaxP DrMaxP added Bug Something isn't working Triage Needed The Issue still needs to be reviewed by Azure Kinect team members. labels Mar 16, 2023
@DrMaxP DrMaxP changed the title I am using the latest SDK with multiple cameras connected and powered by USB C only. All cameras have the latest firmware. I see consistent whole level system hangs (all I can do is reboot) when using serval devices at once - e.g. using multiple instances of the Kinect viewer app or 3rd party solutions. There is nothing in the event log to suggest what happens. What else could I do to investigate the cause of the system lock up? This is on Window 11. The lock up occurs consistently, within a few minutes and normally within a free seconds. System lock up with multiple devices Mar 17, 2023
@DrMaxP
Copy link
Author

DrMaxP commented Mar 28, 2023 •
edited
Loading

There's nothing obvious in the logs. Individual cameras seem to work fine. The more cameras I start up the more likely the system is to hang. This happens even at relatively low CPU usage. I did think it was a thermal issue but I've seen it happen when cores temps are still not very elevated. Does anyone else see similar behavior with multiple cameras? I have 8 connected, but once I start 4 the likelihood of a system lock up rises rapidly.

@DrMaxP
Copy link
Author

DrMaxP commented Mar 28, 2023 •
edited
Loading

RAM, GPU usage, GPU temp all seem fine when the crash happens.

image

The screenshot shows the system locked up when 7 Kinects were streaming using the Kinect Viewer app.

@DrMaxP
Copy link
Author

DrMaxP commented Apr 16, 2023 •
edited
Loading

So the system is more stable now, in that I can connect 8 devices and they'll stream from Azure Kinect viewer for much longer than they used to. I've shifted some of the cameras to a 4 port StarTech card which are now connected via USB-A->USB-C + power rather than all being powered by USB-C. It's certainly not reliable - I still see system crashes, but they are not immediate (i.e. within 1 minute). Yesterday they all streamed for >1hr, which they would never do before.

My conclusion is that the instability was coming from multiple USB stacks for different parts of the system (motherboard controllers, add in cards, TB4 cards, USB-C hub) running at the same time.

I have one Kinect that's somewhat unreliable in terms of connecting / errors.

@DrMaxP
Copy link
Author

DrMaxP commented Apr 16, 2023

Upload.from.GitHub.for.iOS.MOV

@moversevlad
Copy link

Hello! For clarification, can you please share the model of the USB hub you're using? We also deal with those kind of problems and we seek to find a solution. Also, do you know at what FPS you're getting data? Are there any framedrops?
Thank you

@DrMaxP
Copy link
Author

DrMaxP commented Apr 27, 2023 via email

@DrMaxP
Copy link
Author

DrMaxP commented May 12, 2023 via email

@TomHVR
Copy link

TomHVR commented Sep 19, 2023 •
edited
Loading

I think I am encountering a similar issue #1950

edit: I have a really stupid work around I discovered just for a laugh running the sensors each through a usb 3.0 hub, and then later USB 3.0 active extensions.

It seems being at 3.1 or 3.2 speed, at least on my host controllers (and possibly yours) causes some hardware fault that locks up the system. The active repeaters in the hub/cables forces the data to be at 3.0 speeds, thus not triggering the fault.

Extremely stupid but I guess it works.

@HM281099
Copy link

how to solve it, I had a similar problem, thanks

@TomHVR
Copy link

TomHVR commented Mar 14, 2024

What worked for me was to connect the sensors with USB 3.0 extensions or use a 3.0 hub that downgraded the connection to the NUC.

@HM281099
Copy link

thanks for your reply , best wishes @TomHVR

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Bug Something isn't working Triage Needed The Issue still needs to be reviewed by Azure Kinect team members.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@TomHVR @HM281099 @DrMaxP @moversevlad