Skip to content

Latest commit

 

History

History
296 lines (176 loc) · 12.2 KB

README.md

File metadata and controls

296 lines (176 loc) · 12.2 KB
Raw

XVERSE 3D-GS UE Plugin

License

English | 中文

Introduction

XVERSE 3D Gaussian Splatting UE plugin（XVERSE 3D-GS UE Plugin）is an Unreal Engine 5 (UE5) based plugin developed by XVERSE Technology Inc. (Zhenshen, China), aiming to provide real-time visualization, management, editing, and scalable hybrid rendering of Gaussian Splatting model, which is a recently arised technique for reconstructing 3D scenes from multiple photos, more details can refer to here.

Currently, our plugin is implemented based on UE5 Niagara, fully exploiting the advantage of the features provided by UE5 to realize efficient rendering and managing for Guassian Splatting models.

Since we are at early access, current accessible features are summarized below:

  • Niagara-Based High-quality real-time visualizing and rendering for 3D Gaussian Splatting
  • Easily importing and converting from the original Gaussian Splatting scene (.ply file) to ours
  • Hybrid rendering with other UE assets
  • RTS and volume based crop
  • Crop a pure niagara for VFX
  • Support dynamic illumination of Gaussian Splatting scene
  • Automatic lod generation, supporting over 200,000 point clouds in niagara
  • Training tools in Windows system (XV3DTools)： train a given mp4 video to GaussianSplatting ply file

We will actively release new features in this repo, please stay tuned. Some future updates will contain:

  • Dynamic Lod Rendering
  • Automatic collision generation
  • Support preview window
  • Interactively Edit 3D-GS assets
  • 3D-GS assets compression

Getting Started

Video Tutorial

Youtube（English）：

B站（中文）：

System Requirements

  • Windows 10 or 11
  • Unreal Engine 5.1.x / 5.2.x / 5.3.x
  • cuda11 or higher (if using the XV3DTools)

Plugin download

  1. Open github Release page

  1. Download the zip corresponding to the UE version you are using

  1. Unzip and move folder XV3dGS to folder Plugins

Demo Data Download Links:

We have provided two sets of test data available for download at:

Included in these downloads are demo_fox_1080p.mp4 and demo_office_1080p.mp4, serving as two sample MP4 files for testing XV3DTools' local training capabilities.

demo_fox_gs.ply and demo_office_gs.ply are 3DGS models trained by XV3DTools, which can be imported into Unreal Engine (UE) using this plugin for real-time rendering.

Open demo project

This repo contains a demo project with an example scene and level

  1. download using git clone
git clone https://github.com/xverse-engine/XV3DGS-UEPlugin.git
  1. Open XV3DGS.uproject to start UE
  2. Open demo level

Import your guassian splatting model

After v1.1.3, this plugin supports dragging multiple ply files directly into the Content Browser and importing them.

Click on the 'Load Assets' button on the navigation bar or navigate to Window/XV3dgs.

This action will bring up an integrated tool page by default set to load the PLY page. Click on the 'Load PLY' button.

This will trigger a resource selection window to appear.

Choose your Gaussian model file (in .ply format), then it will automaticly generate LOD and loaded into your Content Browser as a blueprint.

Drag the blueprint into Level Viewport and you can see your Gaussian scene.

Local Training on Windows Platform

We develop the XV3DTools to train MP4 videos into Gaussian Splatting ply.

XV3DGS-UEPlugin Version XV3DTools Version
1.0.0 ~ 1.1.4 1.1.1
1.1.5 1.1.2

Please Note: Installation of CUDA 11 or higher is required. CUDA Download Link

XV3DTools Usage Steps:

  1. Download XV3DTools tool from the cloud drive to your local machine:
XV3DTools Version Download Link Preview
1.1.1 Baidu Pan, Google Drive
1.1.2 Baidu Pan, Google Drive
  1. Click on the icon to open the interface and switch to the 'training' page.

  1. Click button to set the parameter "XV3DTools Folder" to the local path where XV3DTools tool is located. The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  2. Click button to select the MP4 file for processing and set it as the parameter "Input MP4 Path" (requires video frame rate > 20). The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  3. Click button to choose the output folder and set it as the parameter "Output Folder". The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  4. click Start Training button.

  1. Training takes time. Once completed, locate the trained ply file in the output folder.

Training Speed Reference:

  • Test Machine Configuration: Windows 10, CUDA 11.8, GeForce RTX 2080 (8GB VRAM), i9-9900 processor.
  • demo_fox.mp4: Video duration of 10 seconds, training time of 5 minutes( extract image with 5fps).
  • demo_office.mp4: Video duration of 430 seconds, training time of 40 minutes( extract image with 1fps).

Packaging UE level for Windows Using Our Plugins

You can refer to the following tutorial to package your own UE levels into exe files that can be run on windows System.

packing tutorial

We provide a packaged Windows executable demo, you can download pack_win_exe.zip from release, simply double-click the exe file to start it.

Feature Introduction

RTS (Rotation, Translation, Scaling) Transform

Select the 3D-GS asset and drag it into the scene. then you can Transform it.

You can also select a 3D-GS actor in the level viewport and transform it using GIZMO.

Model Clipping by Gaussian model

  1. Click to select your Gaussian model blueprint, then click to select the "Clip Volume Component", and open the "Visible" box in the "Clip Volume Component".

  1. With the "Clip Volume Component" selected, you can see a crop volume in the window. You can modify the rotation, translation, and scale of the volume by adjusting the Transform attribute. Alternatively, use the keyboard shortcuts WER to visually adjust the GIZMO for rotation, translation, and scale.

  1. Keep the "Clip Volume Component" selected, and choose different cropping options in the Crop Type. The default state is no cropping, but you can change it to crop the inside or outside of the volume. The example below shows cropping the outside of the volume.

  1. Click the "Crop GSPoints" button to clip the specified portion and obtain a clean Niagara.

  1. This clean Niagara can be used for VFX. For more details, please refer to our VFX tutorial (Coming soon).

Dynamic illumination

Dynamic lighting is not default opened.

you can enable it by turning on the withRelighting button.

Supports all the UE light types

The light source can be moved at will

Change color

Modify the AlbedoTint parameter to change the overall color.

LOD

For imported point clouds, LOD is automatically calculated, with the default being to explicitly display the maximum LOD within the limit of 2 million points for Niagara. The parameter "VisWhichLOD" can be used to force the use of a specific LOD.

Dynamic switching of LOD based on the view camera will be supported in future updates.

Release Note

v1.1.5

  • Support user-defined settings of training parameters
  • Modify 3DGS densification strategy to focus more on important areas
  • Support post-processing strategies for object reconstruction, which can effectively remove floaters
  • Support Gaussian models with spherical harmonic coefficients up to the 3rd degree.

v1.1.4

  • Support UE5.4

v1.1.3

  • Support dragging multiple ply files into the content directory
  • Fix the crash issue when the buffer asset position is incorrect
  • Fix bug in XV3DTools while using wrong path

v1.1.2

  • Supports more types of ply header
  • Fixed flickers when moving objects
  • Update UI

v1.1.1

  • New Model Clipping allowing the creation of a clean Niagara for VFX.
  • XV3DTools v1.1.1: Users can now adjust the training iteration.

v1.1.0

  • Training tools in Windows system (XV3DTools v1.0) to train a given mp4 video to GaussianSplatting ply file
  • Automatic lod generation, supporting over 200,000 point clouds in niagara
  • fix bug in transform

v1.0.1

  • Fix bug in v1.0.0：Error while packaging for Windows in UE5.2 and UE5.3

v1.0.0

  • Niagara-Based High-quality real-time visualizing and rendering for 3D Gaussian Splatting
  • Easily importing and converting from the original Gaussian Splatting scene (.ply file) to ours
  • Hybrid rendering with other UE assets
  • RTS and Geometry Editing
  • Apply VFX effect to Gaussian Splatting scene
  • Support dynamic illumination of Gaussian Splatting scene
  • Automatic enhancement of Gaussian Splatting scene

Contributors