Skip to content

Commit

Permalink
copc-lib v2 (update to latest spec, incl. extents and prdf 6-8) (#51)
Browse files Browse the repository at this point in the history 
* FEAT-437: Tackle some TODOs.
- Fix static function calls in python
- Update operator definitions

* FEAT-437: Implement LasBase Class.

* FEAT-437: Add ToString functions to LasHeader and LasConfig.

* FEAT-437: Fix post-merge issues.

* FEAT-437: Add python implicit conversion of VoxelKey to list and tuple.

* FEAT-431: Setup scaffolding for copc extents VLR.

* FEAT-431: Implement WriterInternal::WriteExtents.

* FEAT-431: Implement extents after lazperf update, bug in reading extents.

* FEAT-431: Fix remaining bug.

* FEAT-431: Add python bindings.

* FEAT-431: Add writer extents initialization and tests for extent size.

* FEAT-431: Add CopcStats. Buggy.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* FEAT-431: Fix bug and integrate CopcExtents class to code. Buggy.

* FEAT-431: Move and update CopcExtent.

* FEAT-431: Clang and re-organize extents.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* FEAT-431: Fix remaining issues with extents.

* FEAT-431: Add python bindings.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* FEAT-431: Add extents to example-writer.cpp.

* FEAT-434: Implement name changes for new Info VLR.

* FEAT-434: Move WKT and Extents as VLR.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* FEAT-434: First round of addressing reviews, more to come.

* FEAT-434: Fix bugs after testing with new autzen file.

- Make python las header etc as properties instead of functions.
- Update values of test based on test file to match new file

* FEAT-434: Remove parentheses for def_static python functions.

* FEAT-434: Make Extents a vector of shared pointers.

* FEAT-434: Make VLR class to merge lazperf VLR and EVLRs.

* FEAT-434: Remove debug comment.

* fix headers

* fix pathes and cmakefile

* add updated autzen test file temporarily

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove temporary test file, update test file link

* FEAT-434: Updated tests for new hobu test file.

* FEAT-434: Update CI tests to use laz-perf master branch

* FEAT-434: Fix CI changes.

* FEAT-434: Fix CI changes.

* FEAT-434: Fix CI changes.

* FEAT-434: Fix CI changes.

* FEAT-434: Implement CopcConfig.

* FEAT-434: Fix typo.

* FEAT-434: Add Extents to Reader tests.

* FEAT-434: Addressing PR comments.

- Pin Autzen test file commit
- Address extents TODOs
- Update CI to fix windows
- Update VLR reading functions
- Add Extent implicit conversion to tuple

* FEAT-434: Fix CI tests

* fix changelog history

* FEAT-434: Update EbByteSize and NumEbItems for clarity.

* FEAT-434: More renaming of EBs.

* FEAT-434: Update lazperf version in README.md.

* FEAT-434: Remove wave_offset setting.

* FEAT-434: Remove PRDF setter from Extents.

* FEAT-434: Remove X,Y,Z from COPC extents.

* update test data

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* FEAT-434: Update Point and Points to support only PRDF 6-8.

* FEAT-434: Update Point and Points to support only PRDF 6-8.

* FEAT-434: Update writer_node_test.cpp with new data.

* FEAT-434: Fix writer_node_test.cpp.

* FEAT-434: Add handle for wrong file paths for FileReader and FileWriter.

* FEAT-434: Update CHANGELOG.md.

* FEAT-434: Update use of VoxelKey to implicit tuple.

* Remove windows tests.

* FEAT-434: Fix extents constructor.

* add lazperf build instructions

* default copc-lib to release mode build

* FEAT-434: Initial changes.

* FEAT-434: More progress.

* FEAT-434: Cpp side working and test pass.

* FEAT-434: Fix python side.

* FEAT-434: Add tests to writer_test.cpp to update cfg values.

* FEAT-434: Add tests to writer_test.py to update cfg values.

* FEAT-434: Address PR reviews.

* FEAT-434: Add TODO.

* FEAT-434: Naming Convention Start.

* FEAT-434: Naming Convention Fix-up.

* FEAT-434: Naming Convention Fix-up.

* FEAT-434: Naming Convention Fix-up.

* FEAT-434: PR Review pt1.

* FEAT-434: PR Review pt2.

- CopcConfig tests
- Copy constructors
- Rename Point bit fields members
- Check WKT bit
- Make WKT VLR optional

* FEAT-434: PR Review pt2.

- Clean-up
- CHANGELOG.md update

* FEAT-434: Add extended stats (mean/var) to Extents.

* FEAT-434: Update extended stats VLR header.

* FEAT-434: Update extended stats VLR header.

* FEAT-434: CHANGELOG.md and README.md updates.

* FEAT-434: Name changes.

* FEAT-434: Accronyms name changes.

* FEAT-434: Merge with main.

* FEAT-434: Rename ScanAngleFloat to ScanAngleDegrees.

* FEAT-434: Add optional flag for extended stats.

* FEAT-434: Add extents to examples files.

* python name changes and readme update

* final renaming

* update changelog

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Christopher Lee <csl170000@utdallas.edu>
  • Loading branch information
@leo-stan @pre-commit-ci @CCInc
3 people authored Oct 28, 2021
1 parent 1fae509 commit 6b8e92c
Show file tree
Hide file tree
Showing 76 changed files with 3,884 additions and 3,334 deletions.
25 changes: 23 additions & 2 deletions .github/workflows/run_tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-18.04, macos-10.15, windows-latest]
os: [ubuntu-18.04, macos-10.15] #, windows-latest]
python-version: [3.6, 3.7, 3.8, 3.9]

steps:
Expand All @@ -36,12 +36,33 @@ jobs:
- name: Install dependencies
run: |
conda install -c conda-forge "laz-perf>=2.1.0" Catch2 pybind11 --yes
conda install -c conda-forge Catch2 pybind11 --yes
python -m pip install --upgrade pip
pip install pytest wheel
shell:
bash -l {0}

- name: Checkout Lazperf
uses: actions/checkout@v2
with:
path: lazperf
repository: hobu/laz-perf
ref: 4819611b279cb791508a0ac0cedd913f8c1d2103

- name: Build lazperf
shell: bash -l {0}
working-directory: lazperf
run: |
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}} -Dgtest_force_shared_crt=ON
cmake --build . --config ${{env.BUILD_TYPE}}
if [ "$RUNNER_OS" == "Windows" ]; then
cmake --install .
else
sudo cmake --install .
fi
- name: Build Copclib Python Bindings
run: |
pip install .
Expand Down
53 changes: 53 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **\[Python/C++\]** Add `CopcExtent` and `CopcExtents` classes
- **\[Python/C++\]** Add extended stats (mean/var) to `CopcExtent` using an additional Lazperf extents VLR (see README.md for more info).
- **\[Python/C++\]** Add `GetCopcExtents` function to `CopcConfig`
- **\[Python/C++\]** Add `GetCopcExtents` and `SetCopcExtents` functions to `Writer`
- **\[Python\]** Add `reader.extents` property
- **\[Python/C++\]** Add `CopcInfo` class
- **\[Python/C++\]** Add vector(c++)/list(python) constructor to `VoxelKey`
- **\[Python/C++\]** Add `CopcConfig` class
- **\[Python/C++\]** Add `GetCopcConfig` function to `BaseIO`
- **\[Python/C++\]** Support COPC Extents, WKT, and Extra Bytes as VLR or EVLR
- **\[Python/C++\]** Add `SetCopcExtents` and `SetCopcInfo` to `Writer`
- **\[Python/C++\]** Add `LasHeaderBase` class
- **\[Python/C++\]** Add `ToString` to `LasHeader` class
- **\[Python/C++\]** Add `VlrHeader` class
- **\[Python/C++\]** Add a check for `WKT` byte in `LasHeader::FromLazPerf`

### Changed

- **\[Python/C++\]** Supported LAS point formats are now strictly 6 to 8 throughout the library
- **\[Python/C++\]** Remove `point_count_14`, `points_by_return_14`, `header_size_`, `wave_offset`, and `version` from `LasHeader`
- **\[Python/C++\]** Change lazperf version requirement to > 2.1.0
- **\[Python/C++\]** Update autzen-classified.copc.laz test file to be latest official
- **\[Python/C++\]** Change use of `las::CopcVlr::span` to `CopcInfo::spacing`
- **\[Python/C++\]** Change `CopcConfig::GetWkt` return type from `las::WktVlr` to `std::string`
- **\[Python/C++\]** Rename `CopcConfig::GetCopc` to `CopcConfig::GetCopcInfo` and now returns a `CopcInfo` class
- **\[Python/C++\]** Rename `Reader::GetCopcHeader` to `Reader::GetCopcInfo` and now returns a `CopcInfo` class
- **\[Python/C++\]** `VoxelKey::Resolution` and `VoxelKey::GetResolutionAtDepth` now take `CopcInfo` argument instead of `las::CopcVlr`
- **\[Python/C++\]** Remove `Writer::LasConfig`, now using `CopcConfig` instead.
- **\[Python/C++\]** `Writer` and `FileWriter` constructors now takes a `CopcConfig` class as argument
- **\[Python/C++\]** Rename `NumExtraBytes` to `EbByteSize`
- **\[Python\]** Change `reader.GetCopcHeader()` function to `reader.copc_info` property
- **\[Python\]** Change `reader.CopcConfig().LasHeader()` function to `reader.las_header` property
- **\[Python\]** Change `reader.CopcConfig().Wkt()` function to `reader.wkt` property
- **\[Python\]** Change `reader.CopcConfig().ExtraBytesVlr()` function to `reader.extra_bytes_vlr` property
- **\[C++\]** Add `create_test_data.py` and `create_test_data.sh` to create data for `writer_node_test.cpp`
- **\[Python\]** Make `VoxelKey.BaseKey` and `VoxelKey.InvalidKey` static functions
- **\[Python\]** Update `Point` and `Points` property names from `CamelCase` to `under_score`
- **\[Python/C++\]** Rename `Reader::GetAllChildren(VoxelKey)` to `Reader::GetAllChildrenOfPage(VoxelKey)`
- **\[Python/C++\]** Rename `Reader::GetAllChildren()` to `Reader::GetAllNodes()`
- **\[C++\]** Rename `PointFormatID` to `PointFormatId`
- **\[C++\]** Rename `PointSourceID` to `PointSourceId`
- **\[C++\]** Rename `HasRGB` and `HasNIR` to `HasRgb` and `HasNir` respectively
- **\[C++\]** Rename `NIR` to `Nir`
- **\[Python/C++\]** Rename `ScanAngleFloat` to `ScanAngleDegrees`
- **\[Python\]** Rename `unscaled_x`, `unscaled_y`, `unscaled_z` to `x`, `y`, `z`
- **\[General\]** Rename CMake flag `WITH_TESTS_AND_EXAMPLES` to `WITH_TESTS`
- **\[Python/C++\]** Rename `Box::ZeroBox` to `Box::EmptyBox`
- **\[Python/C++\]** Rename `VoxelKey::BaseKey` to `VoxelKey::RootKey`

## [1.3.1] - 2021-10-19

### Added
Expand All @@ -28,6 +79,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **\[Python/C++\]** Change order of arguments in `VoxelKey` spatial functions `Intersects`, `Contains`, and `Within`
- **\[Python/C++\]** Add optional `resolution` argument to `Reader` spatial query functions `GetNodesWithinBox`, `GetNodesIntersectBox`, `GetPointsWithinBox`, and `GetAllPoints` . `resolution` can be used to limit the resolution during spatial queries
- **\[Python/C++\]** Update `span` of `autzen-classified.copc.laz` test file from 0 to 128
- **\[Python/C++\]** Rename `ExtendedReturnsBitFields` to `ReturnsBitField` and `ExtendedFlagsBitFields` to `FlagsBitField` in `Point` class
- **\[Python/C++\]** Make WKT VLR optional

### Fixed

Expand Down
16 changes: 12 additions & 4 deletions CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ include(GNUInstallDirs)
# Shared/Dynamic or Static library?
option(BUILD_SHARED_LIBS "Build libraries as shared as opposed to static" ON)

option(WITH_TESTS_AND_EXAMPLES "Build test and example files." OFF)
option(WITH_TESTS "Build test and example files." OFF)
option(WITH_PYTHON "Build python bindings." OFF)
option(ONLY_PYTHON "Build only python bindings, for setup.py env" OFF)

if (ONLY_PYTHON)
set(WITH_PYTHON ON)
set(WITH_TESTS_AND_EXAMPLES OFF)
set(WITH_TESTS OFF)
set(BUILD_SHARED_LIBS ON)
endif()

Expand Down Expand Up @@ -76,7 +76,7 @@ set(CMAKE_CXX_EXTENSIONS ON)
# find_package(YCM REQUIRED)
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake)

if(WITH_TESTS_AND_EXAMPLES)
if(WITH_TESTS)
enable_testing()
endif()

Expand All @@ -90,6 +90,14 @@ add_install_rpath_support(BIN_DIRS "${CMAKE_INSTALL_FULL_BINDIR}"
INSTALL_NAME_DIR "${CMAKE_INSTALL_FULL_LIBDIR}"
USE_LINK_PATH)

# Encourage user to specify a build type (e.g. Release, Debug, etc.), otherwise set it to Release.
if(NOT CMAKE_CONFIGURATION_TYPES)
if(NOT CMAKE_BUILD_TYPE)
message(STATUS "Setting build type to 'Release' as none was specified.")
set_property(CACHE CMAKE_BUILD_TYPE PROPERTY VALUE "Release")
endif()
endif()

### Compile- and install-related commands.
add_subdirectory(cpp)

Expand Down Expand Up @@ -121,7 +129,7 @@ install_basic_package_files(${PROJECT_NAME}
include(AddUninstallTarget)

# Build Submodules
if (WITH_TESTS_AND_EXAMPLES)
if (WITH_TESTS)
add_subdirectory(test)
add_subdirectory(example)
endif()
Expand Down
103 changes: 100 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,23 @@ copc-lib is a library which provides an easy-to-use reader and writer interface

copc-lib has the following dependencies:

- [laz-perf](https://github.com/hobu/laz-perf) >= 2.1.0
- laz-perf >= [commit 4819611](https://github.com/hobu/laz-perf/commits/4819611b279cb791508a0ac0cedd913f8c1d2103)
- Catch2
- Pybind11

To build the latest version of laz-perf:

```bash
git clone https://github.com/hobu/laz-perf.git
cd laz-perf
git checkout 4819611b279cb791508a0ac0cedd913f8c1d2103
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .
sudo cmake --install .
```

### C++

```bash
Expand Down Expand Up @@ -56,7 +69,7 @@ To build the copc-lib examples and unit tests along with the main library, you m

```bash
mkdir build && cd build
cmake .. -DWITH_TESTS_AND_EXAMPLES=ON
cmake .. -DWITH_TESTS=ON
cmake --build .
```

Expand Down Expand Up @@ -97,8 +110,30 @@ for point in points.Get():
- \[x\] Add writer for COPC data
- \[x\] Python bindings
- \[x\] JavaScript bindings (not planned, see below)
- \[x\] Spatial querying for nodes (given spatial coordinates, retrieve the appropriate Entry object)
- \[ \] Conda and pip packages
- \[ \] Spatial querying for nodes (given spatial coordinates, retrieve the appropriate Entry object)

## Conformity to Spec

This version of copc-lib is pinned to a draft version of COPC respective of the state at [COPC.io](https://github.com/copcio/copcio.github.io/tree/a6e8654f65db7c7d438ebea90993bd7a8d59091a).

### ``extended stats`` VLR

| User ID | Record ID |
| -------------------------- | ---------------- |
| ``rock_robotic`` | ``10001`` |

We additionally add an ``extended stats extents`` VLR to store mean and (population) variance values for each dimension. This VLR can be parsed in the same way as the ``extents`` VLR defined by the COPC spec.

struct CopcExtentExtended
{
double mean; // replaces minimum
double var; // replaces maximum
}

This VLR is optional to process existing COPC files. If present, mean/variance are set appropriately for each dimension in `CopcExtents`; if not, `CopcExtents` will have default values of `mean=0` and `var=1`.

This VLR is only written by the `Writer` if the flag `has_extended_stats` is true in `CopcConfigWriter::CopcExtents`.

## Helpful Links

Expand All @@ -112,6 +147,68 @@ Pull requests are welcome. For major changes, please open an issue first to disc

Please make sure to update tests as appropriate.

### Naming Convention

#### C++

We mostly use Google's [Style Guide](https://google.github.io/styleguide/cppguide.html).
```c++

namespace name
{
class ClassName
{
public:
// Default constructor
ClassName() = default;
ClassName(int public_variable, int private_variable, bool private_read_only)
: public_variable(public_variable), private_variable_(private_variable),
private_read_only_(private_read_only){};

int public_variable{};

// Getters and Setters
void PrivateVariable(int private_variable) { private_variable_ = private_variable; }
int PrivateVariable() const { return private_variable_; }
bool PrivateReadOnly() const { return private_read_only_; }

// Any other function
int PrivateVariablePlusOne() const { return private_variable_ + 1; }
int SumPublicAndPrivate() const { return public_variable + private_variable_; }
static std::string ReturnEmptyString() { return {}; }

private:
int private_variable_{};
bool private_read_only_{false};
};
} // namespace name
```
#### Python
```python
test_class = ClassName(public_variable=1,private_variable=2,private_read_only=True)
# using pybind .def_readwrite
test_class.public_variable = 4
assert test_class.public_variable == 4
# using pybind .def_property
test_class.private_variable = 5
assert test_class.private_variable == 5
# using pybind .def_property_readonly
assert test_class.private_read_only == True
# using pybind .def
assert test_class.PrivateVariablePlusOne() == 6
assert test_class.SumPublicAndPrivate() == 9
# using pybind .def_static
assert test_class.ReturnEmptyString == ""
```

Note that dimension names for points follow the [laspy naming scheme](https://laspy.readthedocs.io/en/latest/intro.html#point-format-6), with the exception of `scan_angle`.

## License

Please see [LICENSE.md](LICENSE.md)
Expand Down
2 changes: 1 addition & 1 deletion cmake/DownloadExampleFiles.cmake
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ macro(download_test_files OUT_PATH)
else()
message(STATUS "Downloading test files to ${OUT_PATH}")
file (DOWNLOAD
"https://cloudfront.rockrobotic.com/public/laz-examples/copc/drafts/span/autzen-classified.copc.laz"
"https://github.com/PDAL/data/raw/a3d2a351ca1002c7ea4daa96b5c5fcb0fafeaa6f/autzen/autzen-classified.copc.laz"
${OUT_PATH}
)

Expand Down
9 changes: 7 additions & 2 deletions cpp/CMakeLists.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ set(LIBRARY_TARGET_NAME copc-lib)

# Only public header files go here.
set(${LIBRARY_TARGET_NAME}_HDR
include/${LIBRARY_TARGET_NAME}/copc/file.hpp
include/${LIBRARY_TARGET_NAME}/copc/info.hpp
include/${LIBRARY_TARGET_NAME}/copc/extents.hpp
include/${LIBRARY_TARGET_NAME}/copc/config.hpp
include/${LIBRARY_TARGET_NAME}/geometry/box.hpp
include/${LIBRARY_TARGET_NAME}/geometry/vector3.hpp
include/${LIBRARY_TARGET_NAME}/hierarchy/entry.hpp
Expand All @@ -12,7 +14,6 @@ set(${LIBRARY_TARGET_NAME}_HDR
include/${LIBRARY_TARGET_NAME}/io/base_io.hpp
include/${LIBRARY_TARGET_NAME}/io/reader.hpp
include/${LIBRARY_TARGET_NAME}/io/writer.hpp
include/${LIBRARY_TARGET_NAME}/las/file.hpp
include/${LIBRARY_TARGET_NAME}/las/header.hpp
include/${LIBRARY_TARGET_NAME}/las/point.hpp
include/${LIBRARY_TARGET_NAME}/las/points.hpp
Expand All @@ -27,6 +28,9 @@ set(${LIBRARY_TARGET_NAME}_SRC
include/${LIBRARY_TARGET_NAME}/hierarchy/internal/page.hpp
include/${LIBRARY_TARGET_NAME}/hierarchy/internal/hierarchy.hpp
include/${LIBRARY_TARGET_NAME}/io/internal/writer_internal.hpp
src/copc/info.cpp
src/copc/extents.cpp
src/copc/config.cpp
src/geometry/box.cpp
src/hierarchy/key.cpp
src/hierarchy/page.cpp
Expand All @@ -38,6 +42,7 @@ set(${LIBRARY_TARGET_NAME}_SRC
src/las/point.cpp
src/las/points.cpp
src/las/utils.cpp
src/las/vlr.cpp
)

add_library(${LIBRARY_TARGET_NAME} ${${LIBRARY_TARGET_NAME}_SRC} ${${LIBRARY_TARGET_NAME}_HDR})
Expand Down
Loading

0 comments on commit 6b8e92c

Please sign in to comment.