Trouble with Camera Calibration for Motion Capture System

[oursland]

I am developing a multi-camera motion capture system to be open sourced later this summer when it is functional. The pain point I am experiencing right now is due to CheiralityException when attempting to perform the initial intrinsic calibration.

The file examples/SelfCalibrationExample.cpp has been used as a reference. In this example all landmarks are visible through all poses, but that is not true for my calibration, which I believe is the source of the issue.

The system uses ROS2 for communication, AprilTags as the calibration pattern, and GTSAM for optimization. Attached is the source to the intrinsic calibration file: mocap_intrinsic_calibration.cpp.txt The source for the project can be found at: https://github.com/oursland/camera-calibration-gtsam

The camera is stationary throughout the calibration. Each AprilTag and it's corners are uniquely identified, which makes landmark association straightforward. A ros2 bag file is played back in which around 30 seconds of moving the calibration pattern is performed resulting in the following log lines at calibration:

...
[component_container-1] [INFO] [1718829282.800061657] [feynman.mocap_intrinsic_calibration_node]: frames: 1390, poses: 701, total # tags found: 5769
[component_container-1] [INFO] [1718829347.113950259] [feynman.mocap_intrinsic_calibration_node]: Calibrating!
[component_container-1] [INFO] [1718829347.114066760] [feynman.mocap_intrinsic_calibration_node]: Number of factors: 23079
...
[component_container-1] CheiralityException: Landmark l41 behind Camera x630
[component_container-1] CheiralityException: Landmark l7 behind Camera x374
[component_container-1] CheiralityException: Landmark l42 behind Camera x630
[component_container-1] CheiralityException: Landmark l43 behind Camera x630
[component_container-1] CheiralityException: Landmark l9 behind Camera x374
[component_container-1] CheiralityException: Landmark l10 behind Camera x374
[component_container-1] CheiralityException: Landmark l29 behind Camera x630
[component_container-1] CheiralityException: Landmark l28 behind Camera x630
[component_container-1] CheiralityException: Landmark l8 behind Camera x374
[component_container-1] libc++abi: terminating due to uncaught exception of type gtsam::IndeterminantLinearSystemException:
[component_container-1] Indeterminant linear system detected while working near variable
[component_container-1] 8646911284551352622 (Symbol: x302).
[component_container-1]
[component_container-1] Thrown when a linear system is ill-posed.  The most common cause for this
[component_container-1] error is having underconstrained variables.  Mathematically, the system is
[component_container-1] underdetermined.  See the GTSAM Doxygen documentation at
[component_container-1] http://borg.cc.gatech.edu/ on gtsam::IndeterminantLinearSystemException for
[component_container-1] more information.

[oursland]

I have uploaded a complete example with instructions to build and run in Docker along with calibration dataset at the following repo: https://github.com/oursland/camera-calibration-gtsam

[ProfFan]

If something is not visible in a camera it should not be added to the factor. More likely is that you are not initializing your estimates correctly...

[oursland]

Yes, this was the issue. I had put all of the AprilTags corners at z=0 and default constructed the camera, resulting in all components being coplanar and not within the view of the camera. Adjusting the camera initialization corrected the issue.

I am now working on the extrinsics calibration of a motion capture rig that has 6 cameras pointing towards the center of a room which contains a large AprilTags target. I had intended to use the SmartProjectionRigFactor, unless another factor would be better suited for estimating the extrinsics for the cameras.