It is a known issue with RTSP streams and time-consuming algorithms such as deep learning frameworks. You're getting out of sync if individual frames take longer than your stream's frame rate to process. There is an increasing delay between reality and the captured images.
There are two possible solutions to this problem.
- You could determine the missed images and skipping them before grabbing a new frame. This solution is implemented here.
- Or you can continuously grabbing images in a seperated thread. This way, it is possible to connect more than one camera to your application.
The code speaks for itself. You can either use GStreamer of FFmpeg to open the stream.
The number of cameras connected is not limited. It could be even just one camera.
However, the more cameras, the more CPU time is needed to visit all seperate threads.
To run the application, you have to:
Create build path and build project with cmake as follows:
rm -rf build && mkdir -p build && cd build
cmake .. && make
Alternatively, you can also build the project using Code::Blocks to which you need to install it ($ sudo apt-get install codeblocks).
Use only a wired Ethernet connection for your RTSP stream. Wi-Fi can be unstable.
Because the RTSP protocol is sensitive to even just one missing frame, the stream can easily crash.
If you are using the stream in a deep learning app, adjust your resolution and frame rate to the requirements of the deep learning model.
It is not a good idea to send a 1280x960 stream at 30 FPS if your model has a 416x416 input and takes 200 mSec to process a single frame. It only costs extra memory and processing power.
Make sure all cameras have the same resolution. In the example, the OpenCV concatenate algorithm is sensitive to different resolutions. It throws an exception when it faces different widths or heights.
Please note that the cameras are not synchronized with each other by the threads. They may have slightly different lags.
If you want to stream UDP or TCP, are sure the streams work with the command line prompt beforehand. If not, they certainly won't work in OpenCV. Often errors are caused by the coding in the pipeline, the addresses or missing modules. If you need to install additional GStreamer modules, you'll need to rebuild your OpenCV also! For more information, see our website.
To run the application load the project file MultiThread.cbp in Code::Blocks.
If you are using a Jetson Nano, you have to change the location where OpenCV has stored its header files to /usr/include/opencv4
At line 23 in main.cpp the stream is defined.
Grb1->Init("rtsp://192.168.178.129:8554/test/");
Grb1->Init("udpsrc port=5200 ! application/x-rtp, media=video, clock-rate=90000, payload=96 ! rtpjpegdepay ! jpegdec ! videoconvert ! appsink", cv::CAP_GSTREAMER);
Sender: RaspiCam with Raspberry Pi Buster OS
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=-1 ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! jpegenc ! rtpjpegpay ! udpsink host=192.168.178.84 port=5200
Sender: RaspiCam with Raspberry Pi Bullseye OS
gst-launch-1.0 -v libcamerasrc ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! jpegenc ! rtpjpegpay ! udpsink host=192.168.178.84 port=5200
Note, host=192.168.178.84 being the IP address of the receiver.
Grb1->Init("tcpclientsrc host=192.168.178.129 port=5000 ! jpegdec ! videoconvert ! appsink", cv::CAP_GSTREAMER);
Sender: RaspiCam with Raspberry Pi Buster OS
gst-launch-1.0 -v v4l2src device=/dev/video0 num-buffers=-1 ! video/x-raw,width=640,height=480, framerate=30/1 ! videoconvert ! jpegenc ! tcpserversink host=192.168.178.32 port=5000
Sender: RaspiCam with Raspberry Pi Bullseye OS
gst-launch-1.0 -v libcamerasrc ! video/x-raw,width=640,height=480, framerate=30/1 ! videoconvert ! jpegenc ! tcpserversink host=192.168.178.32 port=5000
Note, host=192.168.178.32 being the IP address of the sender.
Grb1->Init("libcamerasrc ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! videoscale ! video/x-raw, width=640, height=480 ! appsink", cv::CAP_GSTREAMER);
Grb1->Init("v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! videoscale ! video/x-raw, width=640, height=480 ! appsink", cv::CAP_GSTREAMER);
Grb1->Init(0);
Grb1->Init(0); //if RaspiCam is not connected
Grb2->Init(1); //if RaspiCam is connected
