Deep Learning based SFM

[mazy1998]

When experimenting with the orthomosaicing, I noticed it was using classic computer vision techniques to find the camera's position. Would it be possible to use differentiable techniques such as ACE Zero https://nianticlabs.github.io/acezero/ or MASt3R https://github.com/naver/mast3r?tab=readme-ov-file which can handle pose estimation for thousands of images.

[pierotofy]

It's an interesting area of research; I'm my experience most deep learning methods currently fall short because of one or more of these reasons:

• They require a GPU and hefty amounts of VRAM to work (unacceptable for our project)
• Don't generalize well to aerial datasets, because they are trained on ground images
• Are academic code and don't run well on all of our supported platforms

But that doesn't mean it's impossible! We'd welcome contributions toward this. Interested in helping out?

[mazy1998]

I'd love to help the best I can. Let me address some of your concerns first, referencing this (https://x.com/eric_brachmann/status/1782637300901458102?s=46&t=kxRtkTKlzVCPl064MP8lfA ):

1. They claim to run pose estimation on thousands of images in one hour using one V100 GPU. I believe consumer-grade GPUs can still handle a significant amount, but this is extremely impressive regardless.
2. ACE Zero does not use a pretraining dataset, unlike MASt3r and DUSt3R. However, from my initial impressions, they were still more robust than OpenCV feature detection. You can try a mini demo (https://huggingface.co/spaces/pablovela5620/mini-dust3r).
3. Both codebases appear well-maintained and reliable.

This area of research is advancing rapidly, thanks to innovations like NeRF and Gaussian Splatting.

[pierotofy]

Forgot one more reason:

• A lot of this "research" code is licensed under proprietary licenses and cannot be used in WebODM.

We've explored gaussian splatting a bit, and that's a very promising area, see https://community.opendronemap.org/t/opensplat-open-source-gaussian-splatting/19371

https://antimatter15.com/splat/?url=https://splat.uav4geo.com/sand_key.splat

[mazy1998]

True I understand your concerns, but would it be possible to implement a non-commercial version of WebODM, one only students and researchers can use that is still much faster? Also since this area of research is hot right now I suspect an open-licensed variant of these ideas will drop soon, so maybe some implementation into it wouldn't be a bad idea. This would give WebODM a head start when the open-licensed versions do come out.

[pierotofy]

would it be possible to implement a non-commercial version of WebODM

No.

an open-licensed variant of these ideas will drop soon

I wouldn't count on it, but we'll see! When they do we could certainly consider them.

[mazy1998]

Ok thank you for the support, I'll send you an email if something worthwhile drops soon. :)

[mazy1998]

Hi Piero, a new much faster open source sfm algorithm just dropped, it a new updated version of colmap.

https://github.com/colmap/glomap?tab=readme-ov-file

[pierotofy]

I'll close this for now; future improvements to the SfM pipeline can be discussed on the community forum and pull requests are welcome in the ODM repository. https://github.com/OpenDroneMap/ODM 🙏