Pointrix V1.0

A differentiable point-based rendering framework: Pointrix V1.0
Document🏠 | 中文文档🏠 | Paper(Comming soon)📄 | Msplat Backend🌐 | 教程视频🔗

Pointrix is a differentiable point-based rendering framework which has following properties:

• Highly Extensible:
  • Python API
  • Modular design for both researchers and beginners
  • Implementing your own method without touching CUDA
• Powerful Backend:
  • CUDA Backend
  • Forward Anything: rendering image, depth, normal, optical flow, etc.
  • Backward Anything: optimizing even intrinsics and extrinsics.
• Rich Features:
  • Support camera parameters optimization.
  • Support Dynmamic scene reconstruction task and Generation task (WIP).
  • Support mesh extraction and different type of initialization.

Quickstart

Installation

Clone pointrix:

git clone https://github.com/pointrix-project/pointrix.git  --recursive
cd pointrix

Create a new conda environment with pytorch:

conda create -n pointrix python=3.9
conda activate pointrix
conda install pytorch==2.1.1 torchvision==0.16.1 pytorch-cuda=12.1 -c pytorch -c nvidia

Install Pointrix and MSplat:

cd msplat
pip install .

cd ..
pip install -r requirements.txt
pip install -e .

(Optional) You can also install gsplat or diff-gaussian-rasterization:

pip install gsplat

git clone https://github.com/graphdeco-inria/diff-gaussian-rasterization.git
cd diff-gaussian-rasterization
python setup.py install
pip install .

Train Your First 3D Gaussian

Tanks and Temples Dataset Demo (Colmap format dataset)

Download the demo truck scene data and run:

cd examples/gaussian_splatting
# For Tanks and Temples data which have high-res images and need to downsample.
python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.5 trainer.output_path=your_log_path

# you can also use GaussianSplatting renderer or GSplat renderer
python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.5 trainer.output_path=your_log_path trainer.model.renderer.name=GaussianSplattingRender

python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.5 trainer.output_path=your_log_path trainer.controller.normalize_grad=True trainer.model.renderer.name=GsplatRender

The scale should be set as 0.25 for mipnerf 360 datasets.

For other colmap dataset which do not need to downsample:

python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=1.0 trainer.output_path=your_log_path

if you want test your model:

cd examples/gaussian_splatting
# For Tanks and Temples data which have high-res images and need to downsample.
python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=0.25 trainer.output_path=your_log_path trainer.training=False trainer.test_model_path=your_model_path

NeRF-Lego (NeRF-Synthetic format dataset)

Download the lego data:

wget http://cseweb.ucsd.edu/~viscomp/projects/LF/papers/ECCV20/nerf/nerf_example_data.zip

Run the following (with adjusted data path):

cd examples/gaussian_splatting
python launch.py --config ./configs/nerf.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.output_path=your_log_path

if you want to test the model:

python launch.py --config ./configs/nerf.yaml trainer.training=False trainer.datapipeline.dataset.data_path=your_data_path trainer.test_model_path=your_model_path

Advanced Approaches

Camera optimization

To enable camera optimization, you should set trainer.model.camera_model.enable_training=True and trainer.optimizer.optimizer_1.camera_params.lr=1e-3:
The renderer must be setted as MsplatRender.

python launch.py --config ./configs/colmap.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.datapipeline.dataset.scale=1.0 trainer.output_path=your_log_path trainer.model.renderer.name=MsplatRender trainer.model.camera_model.enable_training=True trainer.optimizer.optimizer_1.camera_params.lr=1e-3

Post-Processing Results Extraction (Metric, Mesh, Video)

Pointrix uses exporters to obtain desired post-processing results, such as mesh and video. The relevant configuration is as follows:

trainer:
    exporter:
        exporter_a:
            type: MetricExporter
        exporter_b:
            type: TSDFFusion
            extra_cfg:
                voxel_size: 0.02
                sdf_trunc: 0.08
                total_points: 8_000_000 
        exporter_c:
            type: VideoExporter

Users can specify multiple exporters to obtain various post-processing results. For example, with the above configuration, users can get Metric and Mesh extraction results as well as Video post-processing results.
Mesh is obtained using the TSDF fusion method by default.
The renderer must be set as MsplatRender or GsplatRender. You need to set trainer.model.renderer.render_depth as True to enable TSDFFusion.

cd pointrix/projects/gaussian_splatting
python launch.py --config ./configs/nerf.yaml trainer.training=False trainer.datapipeline.dataset.data_path=your_data_path trainer.test_model_path=your_model_path trainer.model.renderer.render_depth=True

Dust3r initialization (Beta)

1. Switch to the Beta branch.

2. Download Dust3r to examples/dust3r_init and follow the installation instructions.

3. Move convert_dust3r.py to the examples/dust3r_init/dust3r folder.

4. Navigate to examples/dust3r_init/dust3r, and then use Dust3r to extract point cloud priors and camera priors:

python convert_dust3r.py --model_path your_dust3r_weights --filelist your_image_path

5. Run the program

python launch.py --config config.yaml trainer.datapipeline.dataset.data_path=your_data_path trainer.output_path=your_log_path

Release Plans

• Nerf_synthetic dataset (this week).
• Dust3r initialization (this week).
• Mesh exstraction (this week).
• Introduction video (this week)
• reformat the document (this week)
• Dynamic Gaussian Project(next week).

Welcome to discuss with us and submit PR on new ideas and methods.

Acknowledgment

Thanks to the developers and contributors of the following open-source repositories, whose invaluable work has greatly inspire our project:

• 3D Gaussian Splatting: 3D Gaussian Splatting for Real-Time Radiance Field Rendering.
• Threestudio: A unified framework for 3D content creation
• OmegaConf: Flexible Python configuration system.
• SSIM: pytorch SSIM loss implemetation.
• GSplat: An open-source library for CUDA accelerated rasterization of gaussians with python bindings.
• detectron2: Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art det...