Gaussian Heritage: 3D Digitization of Cultural Heritage with Integrated Object Segmentation

ECCV 2024 VISART Workshop Project Page | Paper

Overview

This repository contains the implementation of "Gaussian Heritage", a pipeline for 3D digitization of cultural heritage objects using only RGB images. Our method leverages advancements in novel view synthesis and Gaussian Splatting to create 3D replicas of scenes and extract models for individual items of interest.

Key Features

• Generate 3D replicas of scenes using only multi-view RGB images (e.g., photos from a museum)
• Extract individual 3D models for items of interest within the scene
• Deployable as a Docker container for easy setup and use

Installation

Requirements

• Host machine with at least one NVIDIA GPU/CUDA support and installed drivers
• Docker
• Ubuntu 22.04

Installation

1. Check that Docker and Docker Compose are installed on your host machine:

   docker --version
   docker-compose --version

2. Check that you have an NVIDIA driver installed on your host machine:

   nvidia-smi

3. Setup the NVIDIA Container Toolkit on your host machine: Follow the instructions bellow or from NVIDIA's official documentation.

   Configure Production Repository:

    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
    && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
        sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

   Optionally, configure the repository to use experimental packages:

   sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

   Update the packages list from the repository:

   sudo apt-get update

   Install the NVIDIA Container Toolkit packages:

   sudo apt-get install -y nvidia-container-toolkit

   Configure the container runtime by using the nvidia-ctk command:

   sudo nvidia-ctk runtime configure --runtime=docker

   Restart the Docker daemon:

   sudo systemctl restart docker

4. Check that you have CUDA installed on your host machine:

   nvcc --version

   If CUDA is not installed on your host machine, Install CUDA by executing the following command:

   sudo apt install nvidia-cuda-toolkit

Building and Running

1. Clone this repository to your local machine:

   git clone https://github.com/mahtaabdn/Gaussian-Heritage.git

2. Navigate to the project directory:

   cd Gaussian-Heritage

3. Build and start the Docker containers:

   docker-compose up

Accessing the Application

If everything works fine, you should be able to open a browser and connect to http://127.0.0.1:5000.

Usage

1. Collect a set of multi-view RGB images of the scene you want to digitize
2. Use our web interface to upload images to the local server
3. The system will process the images to generate:
   • 2D instance segmentation masks
   • A sparse 3D model
   • A 3D model capturing appearances and 3D segmentation of the scene
4. Extract individual 3D models for objects of interest

A step-by-step guide with examples will be added soon.

Citation

If you use this work in your research, please cite our paper:

@article{dahaghin2024gaussian,
  title={Gaussian Heritage: 3D Digitization of Cultural Heritage with Integrated Object Segmentation},
  author={Dahaghin, Mahtab and Castillo, Myrna and Riahidehkordi, Kourosh and Toso, Matteo and Del Bue, Alessio},
  journal={arXiv preprint arXiv:2409.19039},
  year={2024}
}