Can the triangulation process be done in real-time directly from the camera streams?

[johndim21]

Hello @mprib,

First of all, I would like to congratulate you for your work. I am a software developer and research associate working on a European Union funded research project about developing an open-source platform for interactive installations in museums and open spaces, and one of our tasks is to find a cost effective solution for calibrating multiple cameras. So your repository was an oasis in the desert when I was looking for possible solutions!

I have tested it with 3 webcams in multiple spaces and the results are pretty well. What I would like to ask, is if it is possible to do the triangulation process in real-time directly from the camera streams, instead of capturing synchronized videos and loading them to the recordings folder. I would appreciate it if you could tell me if this is possible and guide me on which files I have to look up to implement it.

[mprib]

Hello! Thank you for your encouraging words. I'm glad to hear that it has been successful for the camera calibration.

Regarding real time triangulation: I originally constructed caliscope with the intention of performing real time calculations with more tightly integrated camera management. In an older version of it I was able to achieve real time triangulation with 2 cameras running at a rather clunky frame rate.

This is some rough footage I made way back when just to memorialize that it is possible: https://youtu.be/S8Vqo_Vd-r4?si=37bsWRLLKXyQ0Law

So my reply to your inquiry is a bit murky.... while possible, the processing overhead of running multiple cameras + landmark detection + triangulation all through a single machine is substantial. I think this limits the overall effectiveness of the system.

If you still have an interest in this after those qualifications, please let me know and I'll try to describe the flow of processing that might allow you to duplicate that functionality on you own.

In the long run, my dream is a system composed of multiple cheap camera servers (raspberry pi, etc) that can record high resolution/frame rate from a single camera module and do local frame processing, with packets of points pushed to a central client for triangulation. I think that is what is needed for a truly scalable, low-cost system. But that is quite far out on the horizon!

[mprib]

@johndim21,

As a follow up in the morning after some sleep, I realize you are asking about the triangulation calibration [edit: In a cursory re-reading of your message this morning I thought you were referring to the calibration data capture] process and not the motion capture itself. The answer to that is essentially the same: yes, that real time feedback is how I originally put things together. I would display to the user where each pair of cameras had picked up common points so that they could get good coverage. I come from a background of using a conventional vicon system that similarly provides feedback with blinking lights to let you know about the data collection process and thought that was the best way to go about things

I discovered at that time how incredibly difficult it was to manage webcams across multiple operating systems, as well as the limitations of how many pixels a standard computer can pull down/webcam motion blur/etc. I moved to a post-processing exclusive framework for simplicity.

Still, the real-time workflow is part of my end-game on this project, so if this is something you would like to discuss more, please let me know! It would then probably also be helpful to understand what kind of hardware/OS configurations you are looking at.

edit: looking back at this I think I got my words and interpretation all mixed up. I presented my dissertation proposal on Wednesday and my brain is currently recovering from some sleep debt and distraction from catching up on loose ends. The long story short here is: real time is possible, but very much constrained by hardware.

[johndim21]

Thanks for your answers! For the calibration I used 3 webcams connected to a Lenovo Legion 5 Pro laptop, with AMD Ryzen 7 5800H, NVIDIA GeForce RTX 3060, 16 GB RAM and Windows 11.

The ultimate goal of this task would be to calibrate the cameras, then run the YOLOv8 pose model on each camera stream to detect multiple people's keypoints and perform the triangulation to have the keypoints in the 3D space. I know it is far-fetched to make it work, especially in real-time for multiple people, not only because of the hardware constraints, but also because I will have to find a way to associate the multiple people keypoints detected from each camera, in order to triangulate the correct keypoints for each person.

So the first step would be to try and do the triangulation for one person in real-time with MediaPipe. I also have a powerful PC with an Intel i9-9900K CPU @ 3.60GHz, 64 BG of RAM and NVIDIA GeForce RTX 4090. I have thought to send the 3 webcam streams from the laptop to the PC and perform the keypoints detection and triangulation on the PC. The setup I am thinking of, is having the 3 webcams connected on the laptop and calibrated with caliscope, send the 3 webcam streams to the PC, do the keypoints detection for each camera stream, and then do the triangulation using the config file produced by caliscope (I will also have the config file on the PC for the triangulation). Do you think that would be possible, and if so, could you tell me which files should I look in order to try and implement the real-time detection and triangulation on the PC side?

[mprib]

What you are describing (multiple computers distributing the frame capture/image processing/triangulation load) is definitely the long term direction I've had in my mind so I will say that I believe it is possible, though certainly will require some work.

To give a sense of how this might be achieved I will describe the general process caliscope uses.

There are a few processing nodes that each consume and then produce data packets. The stream of data flows from one node to another via queues. I think what you would want to do is replace the queues with network sockets to pass the data packets. Here is an overview of the current data processing pipeline:

The stream consumes frames from a video file, and passes them to a Tracker which returns a PointPacket

The stream then rolls up the PointPacket with the frame and time stamp into a FramePacket and pushes those FramePackets out to a Queue.

The packets on those queues are consumed by the Sychronizer which groups FramePackets based on their time stamps into SyncPackets. These are then pushed out to a Queue.

That queue of SyncPackets is consumed by the SyncPacketTriangulator which performs the triangulation and can push out XYZ_Packets to a queue or just roll up the data to save the points in a .csv file.

Using sockets instead of queues seems like the way to do what you are looking to do.

---

Aside: my understanding is that the Network Time Protocol can enable microsecond accuracy for clock synchronization so I think it might be feasible to place the socket layer between the streams and the synchronizer rather than between the synchronizer and the triangulator. That way each stream can be processed on its own machine. With that setup, I expect that the system would be much easier to scale up and you could use cheap refurbished NUCs for the "spokes" that do frame capture and mediapipe processing on a single video stream, and then something more powerful as the hub of the system that does the triangulation.

Please let me know if you have any questions or just want to talk through anything!