Adding in memory support for xyz files

[samypr100]

Type

• Bug fix (non-breaking change which fixes an issue): Fixes #
• New feature (non-breaking change which adds functionality). Resolves Support reading and writing file-like objects #1560 (partially)
• Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

The ability to read/write formats in memory is a typical ask due to a variety of use cases. Open3D has some in memory options for file formats such as PNG/JPG (e.g. ReadPNGFromMemory) so it's useful if other file formats had this kind of support.

Checklist:

• I have run python util/check_style.py --apply to apply Open3D code style
  to my code.
• This PR changes Open3D behavior or adds new functionality.
  • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
    updated accordingly.
  • I have added or updated C++ and / or Python unit tests OR included test
    results (e.g. screenshots or numbers) here.
• I will follow up and update the code if CI fails.
• For fork PRs, I have selected Allow edits from maintainers.

Description

The changes in this PR doesn't fully close #1560 since I believe there's other formats to still support such as xyzn, xyzrgb, pts, pcd, and maybe others.

This PR only adds full "in memory" support for xyz file format in order to keep the PR small, simple, and reviewable. It's also early so if somethings needs to change with the approach it's easier rather than trying to retro-fit to all formats.

Since we're reading contents from memory, "auto" would not quite work as there's no magic headers for all these files, so some changes were made so that format is specified for both Reading/Writing Point Clouds. I'm using a prefix of mem::{format} for the new in memory formats.

It simplifies the Python API when temporary files are used to achieve similar behavior like shown below.

>>> import open3d as o3d
>>> xyz_bytes = b"x y z\nx y z\n"
>>> pcr = o3d.io.read_point_cloud_from_bytes(xyz_bytes, "mem::xyz")
>>> pcb = o3d.io.write_point_cloud_to_bytes(pc, "mem::xyz")
>>> pcb.decode("utf8")
"x y z\nx y z\n"

The issue asked to mimic something like the file-like interface in BytesIO(), but Pybind currently doesn't "officially" offer that functionality despite being discussed in pybind/pybind11#1477.

There's a solution to it from that issue, but that would mean bundling a custom pybind extension into the build process, which we could explore. The benefit of it is that we'd be able to retain a file-like interface rather than sending a buffer around, although I'm not sure if it will work with FILE though and some of the other more advanced File System IO that's done through the codebase.

Laslty, with the current approach it's possible there'll be a decent amount of code duplication (similar to FileJPG.cpp, and FilePNG.cpp). The existing code seems tightly coupled with the File IO, so I'm open to suggestions/recommendations/alternatives.

---

This change is 

[update-docs]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

[samypr100]

Please retrigger workflows when possible, thank you 😅

[samypr100]

@yxlao I've made some updates, thanks

[samypr100]

@ssheorey Please re-run workflows, I've pushed a change that hopefully fixes windows build, thanks

[samypr100]

Any pointers as to why I'd be getting these errors on the windows pipelines are welcome. Short of trying to setup a development environment on windows myself (mainly use osx and linux), I'm not sure 100% how this is working on osx/linux but not on Windows unless I missed something obvious during building. I haven't tried to touch the code under t and maybe that's the issue?

D:\a\Open3D\Open3D\cpp\tests\t\io\PointCloudIO.cpp(423,1): error C2440: '<function-style-cast>': cannot convert from 'initializer list' to 'open3d::io::WritePointCloudOption' [C:\Open3D\build\cpp\tests\tests.vcxproj]
D:\a\Open3D\Open3D\cpp\tests\t\io\PointCloudIO.cpp(426,1): message : No constructor could take the source type, or constructor overload resolution was ambiguous [C:\Open3D\build\cpp\tests\tests.vcxproj]
D:\a\Open3D\Open3D\cpp\tests\t\io\PointCloudIO.cpp(440,1): error C2440: '<function-style-cast>': cannot convert from 'initializer list' to 'open3d::io::WritePointCloudOption' [C:\Open3D\build\cpp\tests\tests.vcxproj]
D:\a\Open3D\Open3D\cpp\tests\t\io\PointCloudIO.cpp(443,1): message : No constructor could take the source type, or constructor overload resolution was ambiguous [C:\Open3D\build\cpp\tests\tests.vcxproj]

Edit: Fixed in 4284a20

[samypr100]

@ssheorey workflow run please :), I tested on windows seems to build now 🙌 (also rebased w/ latest master for consistency)

[ssheorey]

Hi @yxlao can you take another look? Thanks.

[ssheorey]

Hi @yxlao can you take another look at this PR? Thanks.

[Chen-Suyi]

Thanks for your contributions!

Is this feature accessible now? It would be really helpful for me, but I cannot find any API descriptions in the documents.

[charangodwin]

I could help review this PR.

[charangodwin]

Reviewable status: 0 of 11 files reviewed, 9 unresolved discussions (waiting on @samypr100, @ssheorey, and @yxlao)

---

cpp/open3d/io/PointCloudIO.h line 95 at r4 (raw file):

            // Attention: when you update the defaults, update the docstrings in
            // pybind/io/class_io.cpp
            std::string format = "auto",

ES.23: Prefer the {}-initializer syntax

---

cpp/open3d/io/file_format/FileXYZ.cpp line 67 at r5 (raw file):

        reporter.SetTotal(static_cast<int64_t>(length));

        std::string content(reinterpret_cast<const char *>(buffer), length);

This create another copy of the buffer as std::string deep copy the input buffer. Also in the class_io.cpp we use std::memcpy.
Is there a way to process the buffer directly using char*? Maybe using std::istrstream.

[samypr100]

@charangodwin re istratream, see #5866 (comment) about some history on how changing may affect benchmarks significantly.

[nsaiapova]

The PR looks good to me with just a couple of minor suggestions.

[ssheorey]

Thanks @samypr100 for this very useful contribution and sorry for the delay in getting this merged - we'll try to merge faster in the future....

Minor comment about other formats (xyzn, xyzi, xyzrgb) - these are better supported in the more generic Tensor based datatypes (open3d::t::io namespace). Functions for reading from a file are already available there.