USB cam bandwidth requirements causing CAN bus timeouts

[dflemstr]

What happened

On my modded Voron 2.4, when having cameras enabled with high resolution (in my case, a chamber and a nozzle camera, each running at 720p), any devices that use CAN bus for communication regularly time out. For example, Z or X homing times out, or doing bed probing/quad gantry leveling/...

Disabling the crowsnest service during homing/probing of the printer makes the operation succeed every time. Turning crowsnest back on once the printer has started printing doesn't cause any issues, and print jobs can complete successfully.

I'm using "modern" v4 crowsnest with camera-streamer and webrtc, but this issue also used to occur with v3, or mjpeg streaming.

What did you expect to happen

I would expect homing/probing to succeed without having to disable the crowsnest service.

How to reproduce

1. Use a Raspberry Pi 4B 8GB (probably works with others, this is what I use) for running Klipper with MainsailOS
2. Connect cameras to the Pi over USB; I use a 3DO nozzle camera and a Logitech C920 chamber camera
3. Configure these cameras in crowsnest; I use this config:

   [crowsnest]
   log_path: /home/pi/printer_data/logs/crowsnest.log
   log_level: verbose                      # Valid Options are quiet/verbose/debug
   delete_log: false                       # Deletes log on every restart, if set to true
   no_proxy: false
   
   [cam 1]
   mode: camera-streamer                   # ustreamer - Provides mjpg and snapshots. (All devices)
                                           # camera-streamer - Provides webrtc, mjpg and snapshots. (rpi + Raspi OS based only)
   enable_rtsp: true                       # If camera-streamer is used, this enables also usage of an rtsp server
   rtsp_port: 8554                         # Set different ports for each device!
   port: 8080                              # HTTP/MJPG Stream/Snapshot Port
   device: /dev/v4l/by-id/usb-046d_HD_Pro_Webcam_C920-video-index0
   resolution: 1280x720                    # widthxheight format
   max_fps: 30                             # If Hardware Supports this it will be forced, otherwise ignored/coerced.
   #custom_flags:                          # You can run the Stream Services with custom flags.
   #v4l2ctl:                               # Add v4l2-ctl parameters to setup your camera, see Log what your cam is capable of.
   
   [cam 2]
   mode: camera-streamer                   # ustreamer - Provides mjpg and snapshots. (All devices)
                                           # camera-streamer - Provides webrtc, mjpg and snapshots. (rpi + Raspi OS based only)
   enable_rtsp: true                       # If camera-streamer is used, this enables also usage of an rtsp server
   rtsp_port: 8555                         # Set different ports for each device!
   port: 8081                              # HTTP/MJPG Stream/Snapshot Port
   device: /dev/v4l/by-id/usb-XCG-230315-J_3DO_NOZZLE_CAMERA_4K_01.00.00-video-index0
   resolution: 1280x720                    # widthxheight format
   max_fps: 30                             # If Hardware Supports this it will be forced, otherwise ignored/coerced.
   #custom_flags:                          # You can run the Stream Services with custom flags.
   #v4l2ctl:                               # Add v4l2-ctl parameters to setup your camera, see Log what your cam is capable of.

4. Use a toolhead board over CAN bus; I use an EBB SB2209 toolhead board with a MKS Canable Pro 2.0 as the USB-to-CANbus transceiver, but I have also experienced this issue with an EBB U2C or similar transceivers, and other toolhead boards that use CAN bus. I run the bus at 1Mbit/sec. The bus doesn't report any dropped or error packets.
5. Start Mainstail + Klipper + Crowsnest + ... with the two cameras streaming to a browser window.
6. Start a homing/quad gantry leveling operation.
7. The operation will be cancelled with a message like Communication timeout while homing Z

Additional information

One possible solution could be for crowsnest to somehow control the priority of USB traffic when interacting with webcams, but I'm not sure if that's possible with current Linux APIs?

Another option could be to have crowsnest detect timing sensitive operations like homing/bed probing in Klipper and pause streaming/reducing resolution/... during that time

A third option could be to add crowsnest g-code accessible commands that lets e.g. a PRINT_START macro pause crowsnest during homing; similar to how the current timelapse support is working.

[mryel00]

Hey thank for the issue and the provided ideas,
to explain my decision why to remove the bug label and add an enhancement label:
First of all this is no bug this is, as you already noticed yourself, just a hardware limitation of the USB port of the Pi. If you want to run that much devices over USB you should consider upgrading your Klipper Host. A Pi 5 might be better for this usecase. Reducing the FPS could also help. Just have a look at the log for available options.

One possible solution could be for crowsnest to somehow control the priority of USB traffic when interacting with webcams, but I'm not sure if that's possible with current Linux APIs?

A small Google search gave me some results of a few possibilities to reduce the requested bandwidth. If you want to read yourself into it and make a PR, go for it. I will not implement such a thing myself at this point of time. Especially as there will be a rewrite of Crowsnest in Python first.

Another option could be to have crowsnest detect timing sensitive operations like homing/bed probing in Klipper and pause streaming/reducing resolution/... during that time

As I already mentioned the rewrite, currently Crowsnest is written in bash. Trust me if I say to you, that you don't want to implement such a thing in bash. If we have a good working Python implementation this might be something we would have to think about again, but atm this doesn't sound like something that is even possible.
As a small side note: reducing the resolution would end up in stopping and restarting the stream, same goes for pausing it at least with the current options of streamer we provide.

A third option could be to add Crowsnest g-code accessible commands that lets e.g. a PRINT_START macro pause Crowsnest during homing; similar to how the current timelapse support is working.

This sounds like the best option out of those three. Currently there won't be such a thing like the timelapse support for the same reason as above. Currently you could add a macro with the gcode_shell_command script. Then just run systemctl stop crowsnest or systemctl start crowsnest to do as you wish. With a rewrite in Python such macros could get implemented a lot better.

So for now this won't be something we will add, as it's just not feasible for us to do so.

[dflemstr]

Thanks for the pointers, and your reasoning makes a lot of sense to me. I will use the stop-gap solutions you suggested in the interim, and then hold off on doing anything else until the rewrite is complete and then consider if something could be done in crowsnest itself to handle this sort of situation more automatically!

[czechbol]

Hi, I would like to report the same issue.
Curiously, my setup has been working for over a month with no issues and this had started happening last week, which is when I performed an update. Has there been any change during the last month or so that could have caused this?

[mryel00]

@czechbol The release notes from the release of last week.
The only change affecting it could be this.
We added the parameter --camera-force_active=1 as default for camera-streamer as a lot of users have reported issues with their stream cutting out after a while. Add --camera-force_active=0 to your custom_flags inside the crowsnest.conf and if that helps, it's something about that parameter. That parameter should only assure that the camera will stay active and if it goes to sleep or something like that, it should just restart the stream.
If it doesn't help to reset that parameter to 0, then it might be a Klipper, Moonraker or some other update you have done.

[czechbol]

This seems to have done the trick!

I have been able to successfully perform a few full probing sequences without any issues.

[mryel00]

@czechbol Can you describe your setup further? How many cams? The used Pi? What USB devices are connected beside the cams? Which USB port is each device connected (2.0/3.0)?
The problem about this topic is that it's hard to decide how we should handle this then. The parameter at 1 fixes 2 issues but introduces a new one....
I will try to reproduce this somehow and maybe I can find a good solution for everyone.

@dflemstr please try to set custom_flags: --camera-force_active=0 and see if that fixes it for you too

[czechbol]

@mryel00 Sure! My situation is very similar.

I also have a Voron 2.4, with a Pi 4 2GB. I also use the EBB SB2209 CAN bus board on my toolhead.
It is connected like so: Pi4 <USB 3> BTT U2C <CAN> BTT EBB SB2209. Other than that I only have the BTT Octopus connected to the Pi, I don't recall which port it is but I would assume it's USB 2

I only use a single Raspberry Pi Camera 3 Wide connected via the ribbon cable.

I should probably also note that I don't run Mainsail OS, I flashed the official Raspberry Pi OS Lite 64bit and installed everything else with KIAUH, and I use Fluidd, but that shouldn't make any difference. I'm pretty confident in my linuxing skills after a few years of using it on my desktop.

[zellneralex]

My recommendation get the BTT U2C and the octopus on the 2 USB2 ports and the cams on the USB3 port(s)

[czechbol]

Thanks, I will take a look at it during the week and report back later.

[dflemstr]

Anecdotally the issue hasn't happened any more since enabling --camera-force_active=0, but I had also already made some changes like switching to lower resolutions/frame rates. I currently have a long (multi day) print running but I hope to be able to run a more scientific test after it is done.

[mryel00]

I'm readding the bug label. We already got confirmation of 2 other users with a Voron 2.4. So it's most likely something about that parameter and most likely something about the stepper motor count. We will revert that parameter soon, as it's a bit harder to see the connection between the timer too close and the cams than something affecting the stream directly.

[LAP87]

RPI4B with Pi Cam v2 over interface flat cable also giving me grief since the update where you added some stuff.. Right now running blind since i can't home or do anything on the V2.4 (u2c, ebb36, 2x skr1.4's) with crowsnest active.
Also, you introduced some glorious timeskip lag effect, when the printer is idling it's constantly skipping frames back n forth, was really hilarious on halloween but not so much when trying to use it for intended purpose.

[mryel00]

since the update where you added some stuff

Like I wrote before, the only thing affecting anything is this one parameter. Everything else in the changelogs is just QOL and installer stuff.

Also, you introduced some glorious timeskip lag effect, when the printer is idling it's constantly skipping frames back n forth

@LAP87 Please elaborate on that further. I think this doesn't only happen with the idling printer as this would sound pretty weird.
Does this also appear with setting --camera-force_active=0 in the custom_flags? What OS version are you running (Bullseye/Bookworm)? What service are you using (webrtc/mjpg/adaptive mjpg)? Does this also appear with ustreamer?

To give some idea of how we test updates. I'm testing this stuff on 3 different Pis. The Pi3B, Pi4B and the Pi0.2 with a picam v3, picam v2 and/or multiple USB cams. I use the updates for quite some time to identify issues that could happen. Besides me some members of the mainsail-crew are testing these changes too, especially something like that parameter change. This change also got tested from 3 or more members of the community. No one of us run into any problems.
The current state of this problem is that we only know of the Voron 2.4 that is affected by this, most likely because of the 4 z-Steppers and CAN. This is something you can only catch, if you got a setup like that. Using e.g. 3 USB cams and a pi cam module v3 simultanously only results in unusable streams of the USB cams => USB bandwidth
So we and OP thought this might be just some overload of the USB chip, as this is something quite common especially with some faster printers like the Voron. Only with the mention of @czechbol that he has it since roughly a week (exactly the date the update came) and with the module v3, we could lead it to that parameter.
We are by far too small to have a big enough group for beta testing for such updates. So we are dependent on people like you and the others in this post to give us exact information on when an issue appeared and what exactly someone already tried to fix it.

[zellneralex]

To clearly some here. In a multi home scenario run time of the signals are important.
We would need infos like CAN speed, USB configuration (what is plugged where), type of cam

As I already mentioned on a PI4 (and honestly any V2 running multi board homing via CAN should use a PI4 and CAN setup set to 1MHz) should connect both mcu (your printer board and you und board) at the 2 USB2 ports and any cam on the USB3 port.
I am currently re configure my V2 to that setup. Until now I ran my probe separately down to avoid any issue while homing.

[zellneralex]

@dflemstr is my assumption correct that you connect your x and z end stop/probe to the toolhead board?

[dflemstr]

@dflemstr is my assumption correct that you connect your x and z end stop/probe to the toolhead board?

Yes

[LAP87]

@LAP87 Please elaborate on that further. I think this doesn't only happen with the idling printer as this would sound pretty weird. Does this also appear with setting --camera-force_active=0 in the custom_flags? What OS version are you running (Bullseye/Bookworm)? What service are you using (webrtc/mjpg/adaptive mjpg)? Does this also appear with ustreamer?

Video clip of the stream in action, it behaved exactly the same before and after adding the custom flag (rebooted klipper, firmware, host, no changes)
https://www.youtube.com/watch?v=t1-tMv7j3Ag

Host(aarch64, 64bit)
Version: v0.12.0-0-g0d67d9c4-dirty
OS: Debian GNU/Linux 11 (bullseye)
Distro: MainsailOS 1.1.1 (bullseye)

I have not used ustreamer since crowsnest camera-streamer was released, i do not know if it behaves the same.
Do you suggest i have to reflash mainsail to the newer bookworm release to make this work as intended?
Tbh, i would rather try downgrading crowsnest to a working version instead

[mryel00]

You can just go back to an earlier version if you want. Go into the directory and use the git reset --hard command to go to a specifc commit. As we don't know what version worked for you, I cannot say how far you have to go back. You can look into the commit history to see what changed and search for a commit that is working for you.
Like I already said, the last update only changed this parameter. If it's not connected to the last update, you have some other problem.
Updating to the latest OS shouldn't help btw. I asked because we are currently using two different versions of camera-streamer. But it seems like there shouldn't be any differences for that.

[mryel00]

About the video:
I don't think it's possible to get the images of WebRTC in the wrong order, so this looks like the frames get scrambled before the encoding to WebRTC happens. This would mean some problems with camera-streamer directly and a different issue than this. camera-streamer didn't get any update for Crowsnest, so maybe there changed something different in your setup or maybe network. For any further debugging of this and your other issue, I recommend to join our Discord that we don't spam this issue with maybe something unrelated.

[dflemstr]

Ran some more "scientific" experiments now. I'll do a full home, then QGL, then 15x15 bed mesh, just to see how far it gets. I would really need many samples for each experiment to get anything representative, but maybe this is some indication at least.

Each run below uses the same config as in the original issue description above (2 cameras 720p 30fps) unless otherwise noted. I have one browser window open showing both streams. I already had my cameras connected to the USB 3 ports, and controllers (CAN bus, BTT octopus) connected to the USB 2 ports, prior to creating this issue ticket.

1. No changes: Homing succeeds, timeout during QGL
2. Stopping the crowsnest service entirely: Homing succeeds, QGL succeeds, bed mesh actually fails (!) after almost completing, ie with only ~20 probes left. Probably a fluke but it shows that CANbus probing in general might not be super reliable.
3. Adding custom_flags: --camera-force_active=0: Homing succeeds, timeout during QGL
4. Significantly lowering resolution to 480x320 for both cameras, leaving frame rate the same: Homing succeeds, QGL succeeds, bed mesh fails after ~100 probes, ~10 minutes.
5. Reducing max_fps to 10 (still using 720p resolution): Homing succeeds, QGL succeeds, bed mesh fails after ~50 probes, ~5 minutes.
6. Reducing max_fps to 10 and also lowering resolution to 480x320: Homing succeeds, QGL succeeds, bed mesh fails after ~160 probes, ~13 minutes.
7. Changing TRSYNC_TIMEOUT from 0.025 to 0.05 in /home/pi/klipper/klippy/mcu.py: Homing succeeds, QGL succeeds, bed mesh succeeds 🎉

[gknops]

CrowsNest 2 seems to be contributing, I did have very few problems with V1 but since updating to V2 homing fails almost always. Not viewing any stream seems to reduce the load (checked with top), homing works reliable then. While printing timing seems less critical, usually OK to turn on the camera view then.

With V1 I regularly was having video streams visible on a couple devices without issues (laptop next to the printer, desktop in the other room). Pi4, wired.

[meteyou]

@gknops pls doublecheck your crowsnest version number... we are right now in version 4.

[gknops]

Of course. Until a couple months ago I had been on an ancient version due to the OS requirements of more recent builds. Just updated from 4.0.2 (IIRC) to the latest, will check if that makes a difference.

[SebastianMusser]

Hello guys, I just stumbled over this thread.
I am having similar/ the same issues.
My setup is a ratrig Vcore 3.1 running Ratos on a Raspberry 4 with an Octopus 1.1 and a EBB42 toolboard that I use with USB, not CAN. And I use a logitech C920 webcam.

My issues started when I ran a full system update a few weeks ago, and as I didnt find a fix I first bought a new raspberry, then a new EBB42 and last but not least a new usb cable. Nothing helped.

Unlike to the other reports here, my issues were not a 100% reproducible, but very sporadic. Sometimes I could print for 2-3 days without issues, then it would fail with a timeout during probing again.

In the klipper discord, I found the tip to add "custom_flags: --camera-force_active=0". Will try that tomorrow.

Please let me know if I can contribute to your analysis somehow.

Regards!

[meteyou]

@SebastianMusser pls double-check your Crowsnest version number or update Crowsnest. This setting was revent to 0 in version v4.1.0.

[SebastianMusser]

I was on 4.1.0.1 and just ran an update to 4.1...
So maybe I have another issue. Will keep an eye on it and report back here!

[daTobi1]

Hello i have the same problem with crowsnest v4. I get an overload by printing or timeout by probing. If i stop the crowsnest service i have no problems and the printer runs fine.

Setup:
Voron 2.4r2
Pi 3+ 64bit
Octopusborad --> connected via USB
BTT u2c --> connected via USB
USB Webcam --> connected via USB
BTT SB2240 --> connected via Can
Mellow ERCF board--> connected via Can

[mryel00]

@daTobi1 An exact version of Crowsnest would be good. You can find it at the top of your crowsnest.log. The main problem of this issue is already fixed, so if you don't use v4.1.0 and didn't update yet, we cannot do anything about it right now.

As an overall note and not targeted at you. It doesn't help us, if people don't bring new informations to the table.
This issue was more about the problem with the parameter --camera-force_active=1. I let this open for now, as I would really like to add that parameter in a future update, after it got some changes.

Also I might add a feature to conveniently stop crowsnest for the probing, but currently it's not possible and I would have to think about a good solution for such a feature.

[feirouu]

my case

I used the USB to CAN bus bridge to connect my Raspberry Pi 4 and spider2.3, the camera had frame loss problems.
Recently I used waveshare's canhat, now everything is working fine

[meteyou]

@JunyuMu FYI: the CANHAT is not recommended by Klipper because of the small buffer.

[dflemstr]

Hey, I switched to a Raspberry Pi 5 now, and anecdotally I haven't had any issues with CANbus timeouts yet. On rpi4 I had to run with:

Changing TRSYNC_TIMEOUT from 0.025 to 0.05 in /home/pi/klipper/klippy/mcu.py

... as mentioned above, but so far I haven't needed that change on rpi5.

Only unfortunate thing is that camera-streamer doesn't support rpi5 yet, but I'm happy using MJPEG and a working printer compared to WebRTC and a non-working printer :)

[gknops]

I switched the CAN-hat to a Canable 1.0 board via USB and also had no issues since (with limited testing).

[dflemstr]

For the record I use Canable Pro 2.0 as mentioned in the original post, and in my experience switching CAN transceiver board hasn't been helpful for me.

[M3HNGRY]

I am having this issue as well. Crowsnest v4.1.4-1-gba0ac8fb. Changing TRSYNC_TIMEOUT from 0.025 to 0.05 in /home/pi/klipper/klippy/mcu.py cause my klipper install to be dirty and mainsail makes me soft or hard recover before updating. Any fix for this yet?

Setup:
Voron 2.4r2
Pi4 4gb 64bit
Octopusboard --> connected via USB
BTT u2c --> connected via USB
USB Webcam --> connected via USB
BTT SB2240 --> connected via Can

[czechbol]

Hey, I'm just checking in.
I've been dealing with the CAN timeout issues since November and after checking everything a million times, ordering a new SB2209 with no different result, I just swapped my Pi4 2GB for a Pi5 8GB.

AND IT WORKS!

Now I need to stress this - I'm sure this is some kind of HW issue and the CAN retransmits (which cause the timeouts) are still happening because I still see insane jumps in my klipper load.

My theory is simply that the Pi 5 is so much faster at computing that it handles it well even with the retransmits. My Pi 4 would kill klipper whenever the klipper load jump would exceed 400% (4 CPU threads maxed out), which didn't happen yet on my Pi 5.

So in the end, I absolutely hate that this bandaid solution works, I want to get to the bottom of this and resolve it once and for all, but at least it seems that I can now print and I do not think this is an inherent issue with crowsnest, just the CAN in general.

PS. I don't know what prints are causing you the most problems but for me, anything with a lot of arcs was a nightmare. Prints with straight paths printed fine, but cylinders, tree supports and similar features were hell.

[mryel00]

@M3HNGRY

Any fix for this yet?

No, we don't provide features to stop the cam during your homing. You can see the status of this issue at the top of your page. As long as it says opened, we didn't implement that feature. I'm already thinking of locking this post down and open a new FR, as the initial issue got already fixed. The issue about a specific parameter. Everything else is nothing we can fix, we can only provide workarounds that people can use. But we will still need a long time before this can work, as it's impossible with the current codebase.

@czechbol

I'm sure this is some kind of HW issue

Exactly. We cannot provide fixes for limited HW capabilities. We also can only provide bandaid fixes, like stopping the cam during homing (the initial issue). This bandaid fix won't come in the near future.

My Pi 4 would kill klipper whenever the klipper load jump would exceed 400%
anything with a lot of arcs was a nightmare.

Your issue seems to be related to processing power and not USB bandwidth and that's nothing we can fix either ofc. You shouldn't set arcs too low as this will create a lot of load. Default is a resolution: 1.0 and I think you shouldn't go below 0.2.

Overall a Pi5 isn't a good alternative for camera streaming as it is missing H264 HW encoding. Therefore you won't get WebRTC support. CSI cam support isn't possible too atm, but might come in a future update.

As an overall reminder:
ANY TIMEOUTS ARE NOT AN ISSUE CREATED BY CROWSNEST! THIS IS A HARDWARE LIMITATION!
We can only provide a band aid fix as mentioned above and for that we still need a lot of other stuff first.

[czechbol]

@mryel00 you were right, it turns out I didn't understand what the arc resolution means and out of fear of seeing resolution artefacts in my prints I set it way too low.

There was nothing wrong with Crowsnest or CAN bus in my case

[M3HNGRY]

@M3HNGRY

Any fix for this yet?

No, we don't provide features to stop the cam during your homing. You can see the status of this issue at the top of your page. As long as it says opened, we didn't implement that feature. I'm already thinking of locking this post down and open a new FR, as the initial issue got already fixed. The issue about a specific parameter. Everything else is nothing we can fix, we can only provide workarounds that people can use. But we will still need a long time before this can work, as it's impossible with the current codebase.

@czechbol

I'm sure this is some kind of HW issue

Exactly. We cannot provide fixes for limited HW capabilities. We also can only provide bandaid fixes, like stopping the cam during homing (the initial issue). This bandaid fix won't come in the near future.

My Pi 4 would kill klipper whenever the klipper load jump would exceed 400%
anything with a lot of arcs was a nightmare.

Your issue seems to be related to processing power and not USB bandwidth and that's nothing we can fix either ofc. You shouldn't set arcs too low as this will create a lot of load. Default is a resolution: 1.0 and I think you shouldn't go below 0.2.

Overall a Pi5 isn't a good alternative for camera streaming as it is missing H264 HW encoding. Therefore you won't get WebRTC support. CSI cam support isn't possible too atm, but might come in a future update.

As an overall reminder: ANY TIMEOUTS ARE NOT AN ISSUE CREATED BY CROWSNEST! THIS IS A HARDWARE LIMITATION! We can only provide a band aid fix as mentioned above and for that we still need a lot of other stuff first.

Since it sounds like you are not recommending Pi5 what should we get to have this running correctly?

Thanks

[mryel00]

Like I said, it's not a good alternative for camera streaming.
Overall I can only say, if it doesn't work with your current Pi4 without a camera, you might want to seek help to improve your setup, as there is a different problem then.
alexz tested a bit such setups and shared his results internally. With his setup he couldn't reproduce any issues with a cam. He has also by far more knowledge about this than me, so you might want to check his answer. But here is ofc the wrong place for such kind of support.

You can ofc use a Pi5 for your printer, but I cannot say anything about it, if it will be good or bad for the printing use case, especially on those setups. I'm only certain of the fact that the Pi5 is by far overspecced for the printing use case.
An alternative for the streaming would be to use a second Pi for crowsnest, like a Pi Zero 2w. This would work ofc for a Pi4 and Pi5 setup.

[M3HNGRY]

Like I said, it's not a good alternative for camera streaming. Overall I can only say, if it doesn't work with your current Pi4 without a camera, you might want to seek help to improve your setup, as there is a different problem then. alexz tested a bit such setups and shared his results internally. With his setup he couldn't reproduce any issues with a cam. He has also by far more knowledge about this than me, so you might want to check his answer. But here is ofc the wrong place for such kind of support.

You can ofc use a Pi5 for your printer, but I cannot say anything about it, if it will be good or bad for the printing use case, especially on those setups. I'm only certain of the fact that the Pi5 is by far overspecced for the printing use case. An alternative for the streaming would be to use a second Pi for crowsnest, like a Pi Zero 2w. This would work ofc for a Pi4 and Pi5 setup.

So when I uninstall Crowsnest it works just fine. No issues. I will read into alexz tests. Thanks

[cfoo]

@mryel00 We can probably close this issue so that folks don't go through and try all the other silly suggestions that do not work. Simply, the solution suggested by Alexz works. What you actually want to do is limit the cameras from saturating the USB channel. On the pi4 there is a built in USB 3.0 hub that forwards both USB 2.0 and 3.0 traffic. The USB 2.0 ports share a common root port via an integrated USB 2.0 4-port hub, so the total USB 2.0 bandwidth across all the ports is 480Mbps. You can use this bandwidth limit to prevent the cameras from saturating the upstream USB 3.0 hub by keep all your cameras on the USB 2.0 ports. Using a USB 2.0 hub connected to the USB 3.0 port works the same. Better yet, get an old USB 2.0 hub and you'll be able to connect it to any ports. See the known pi USB issues.

I ran 4 15x15 bed mesh torture tests with 3 cameras running at 1920x1080. The timeout went away if you isolate the cameras away from the CAN. The easiest thing to do is connect all your USB cameras to a USB 2.0 hub and free up all the other ports for your other devices. Check your USB tree setup with lsusb -t my is below.

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=gs_usb, 12M
        |__ Port 1: Dev 3, If 1, Class=Application Specific Interface, Driver=, 12M
        |__ Port 2: Dev 4, If 1, Class=CDC Data, Driver=cdc_acm, 12M
        |__ Port 2: Dev 4, If 0, Class=Communications, Driver=cdc_acm, 12M
        |__ Port 3: Dev 5, If 0, Class=Communications, Driver=cdc_acm, 12M
        |__ Port 3: Dev 5, If 1, Class=CDC Data, Driver=cdc_acm, 12M
        |__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 7, If 1, Class=Video, Driver=uvcvideo, 480M
            |__ Port 4: Dev 7, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 4: Dev 7, If 0, Class=Video, Driver=uvcvideo, 480M
            |__ Port 4: Dev 7, If 3, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 1: Dev 8, If 3, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 1: Dev 8, If 1, Class=Video, Driver=uvcvideo, 480M
            |__ Port 1: Dev 8, If 2, Class=Audio, Driver=snd-usb-audio, 480M
            |__ Port 1: Dev 8, If 0, Class=Video, Driver=uvcvideo, 480M

[mryel00]

Thanks for the long explanation. As I said earlier I wanted to keep it up as some tracker for some other feature, but after this wonderful explanation, I will close it without locking it for now.
I might also go through the post and make sure that statements from me and others, that are confusing or wrong will be hidden.

Thanks again for your time and research on this.