V3 and Gaussian Splatting

.env.example

@@ -20,3 +20,6 @@ RABBITMQ_DEFAULT_PASS="your_password"
 RABBITMQ_IP="rabbitmq"
 
 # Any changes to Database or RabbitMQ ip address should be in configs/docker_out.json
+
+# Use GPU acceleration on sfm reconstruction (good for large resolution inputs)
+SFM_USE_GPU = 0

.gitignore

@@ -6,4 +6,6 @@ Pipfile
 # local env files
 # Do not commit any .env files to git, except for the .env.example file.
 .env
-.env*.local
+.env*.local
+
+*.log

.gitmodules

@@ -1,3 +1,9 @@
-[submodule "NeRF/TensoRF"]
-	path = NeRF/TensoRF
-	url = git@github.com:NeRF-or-Nothing/TensoRF.git
+[submodule "go-web-server"]
+	path = go-web-server
+	url = https://github.com/NeRF-or-Nothing/go-web-server.git
+[submodule "nerf-worker"]
+	path = nerf-worker
+	url = https://github.com/NeRF-or-Nothing/nerf-worker.git			
+[submodule "sfm-worker"]
+	path = sfm-worker
+	url = https://github.com/NeRF-or-Nothing/sfm-worker.git

README.md

@@ -13,28 +13,28 @@
     <img src="pics/logo.png" alt="Logo" width="80" height="80">
   </a>
 
-<h3 align="center">NeRF or Nothing core repository</h3>
+<h3 align="center">NeRF or Nothing backend core repository</h3>
 
   <p align="center">
     A micro-services based project in rendering novel perspectives of input videos
     utilizing neural radiance fields.
     <br />
-    <a href="https://github.com/NeRF-or-Nothing/vidtonerf/wiki/Learning-Resources">
+    <a href="https://github.com/NeRF-or-Nothing/backend/wiki/Learning-Resources">
         <strong>Learn more about NeRFs »</strong>
     </a>
     <br />
     <br />
-    <a href="https://github.com/NeRF-or-Nothing/vidtonerf">View Demo</a>
+    <a href="https://github.com/NeRF-or-Nothing/backend">View Demo</a>
     ·
-    <a href="https://github.com/NeRF-or-Nothing/vidtonerf/issues">Report Bug</a>
+    <a href="https://github.com/NeRF-or-Nothing/backend/issues">Report Bug</a>
     ·
-    <a href="https://github.com/NeRF-or-Nothing/vidtonerf/issues">Request Feature</a>
+    <a href="https://github.com/NeRF-or-Nothing/backend/issues">Request Feature</a>
   </p>
 </div>
 
 ## About The Project
 
-This repository contains the backend for the NeRf (Neural Radiance Fields) or Nothing
+This repository contains the backend for the (Neural Radiance Fields) NeRF-or-Nothing
 web application that takes raw user video and renders a novel realistic view of the
 scene they captured. Neural Radiance Fields are a new technique in novel view synthesis
 that has recently reached state of the art results.
@@ -59,6 +59,49 @@ the locations for each image are needed in order to train a NeRF, we get this da
 from running structure from motion (using COLMAP) on the input video. To learn more
 please visit the learning resources in the wiki.
 
+## Gaussian Splatting Background
+Gaussian splatting is a novel approach to neural scene representation that offers significant 
+improvements over traditional Neural Radiance Fields (NeRFs) in terms of rendering speed and
+visual quality. Like NeRFs, gaussian splatting starts with a set of input images capturing 
+different perspectives of the same scene, along with their corresponding camera positions and orientations.
+
+The key difference lies in how the scene is represented and rendered:
+
+1. **Scene Representation**: Instead of using a neural network to model the entire scene, gaussian
+   splatting represents the scene as a collection of 3D Gaussian primitives. Each Gaussian
+   is defined by its position, covariance matrix (which determines its shape and orientation), and 
+   appearance attributes (color and opacity).
+
+4. **Initialization**: The process begins by running structure from motion (using tools like COLMAP) on 
+   the input images to obtain initial camera parameters and a sparse point cloud. This point
+   cloud is used to initialize the Gaussian primitives.
+
+5. **Training**: The system then optimizes these Gaussians to best reproduce the input images. This 
+   involves adjusting the Gaussians' positions, shapes, and appearance attributes. 
+   The training process is typically faster than NeRF training and can be done end-to-end using gradient descent.
+
+6. **Rendering**: To generate a new view, the Gaussians are projected onto the image plane of the 
+virtual camera. Each Gaussian splat contributes to the final image based on its projected size, shape, 
+and appearance. This process is highly parallelizable and can be efficiently implemented on GPUs, 
+resulting in real-time or near-real-time rendering speeds.
+
+7. **View-dependent Effects**: Gaussian splatting can model view-dependent effects by incorporating 
+additional parameters for each Gaussian, allowing for realistic 
+representation of specular highlights and reflections. If you want to take advantage of this, use .ply files, and for quick reflectionless
+rendering, use .splat files.
+
+The resulting representation is compact, efficient to render, and capable of producing high-quality novel views. 
+Importantly, like NeRFs, gaussian splatting requires accurate camera positions for the input images,
+ which are typically obtained through structure from motion techniques.
+
+Gaussian splatting offers several advantages over traditional NeRFs:
+- Faster training times
+- Real-time or near-real-time rendering of novel views
+- Better preservation of fine details and sharp edges
+- More compact scene representation
+
+To learn more about gaussian splatting and its implementation details, please refer to the learning resources in the wiki.
+
 ### General Pipeline:
 
 1. Run Structure from motion on input video (using COLMAP implementation) to
@@ -81,20 +124,36 @@ aforementioned folders.
 
 ## Getting Started
 
-To run the project install and run the web-server, the nerf worker, and the
-colmap worker in any order by running their respective installations in their
-READMEs. Once these are running the front-end can be started by visiting the
-[front end repo](https://github.com/NeRF-or-Nothing/web-app). Once everything is
-running the website should be available at `localhost:3000` and a video can
-be uploaded to test the application.
-
 ### Prerequisites
 
 1. Have [Docker](https://www.docker.com/) installed locally
-2. Install [COLMAP](https://colmap.github.io/)
-3. Install [ffmpeg](https://ffmpeg.org/)
-4. If you intend to run the NeRF and COLMAP workers locally ensure you have
-NVIDIA GPUS with atleast 6GB of vram as these are resource intensive applications
+2. Have a CUDA 11.7+ Nvidia GPU (To run training) 
+3. Follow the service prerequisites:
+   - [go-web-server]()
+   - [sfm-worker]()
+   - [nerf-worker]() **IMPORTANT READ**  
+
+### Instalation
+
+The project should be be easy to install/run once you have completed the respective prerequisites.
+The files `./docker-compose-go.yml` and `docker-compose-flask.yml` handle the setup given that you want to run
+V3 or V2 of the api, respectively. 
+
+1. Clone this repository
+  ```
+  git clone https://github.com/NeRF-or-Nothing/backend.git
+  ```
+
+2. Compose the backend. View indepth [instructions]()
+  ```
+  docker compose -f <chosen_compose_file>.yml up -d
+  ```
+
+3. Follow the [frontend](https://github.com/NeRF-or-Nothing/frontend) installation.
+
+Once everything is running the website should be available at `localhost:5173` and a video can
+be uploaded to test the application.
+
 
 ## Output Example
 
@@ -105,15 +164,26 @@ dataset lego example to a video then running vidtonerf produces the following re
 
 ## Roadmap
 
-TODO
+- **Deployment**: The team has been mixing with the idea of deploying for numerous years now. In order to do so we need to get production ready. 
+    1. More request verification
+    2. Reverse proxy
+    3. TSL/SSL frontend
+    4. Lockdown communication
+- **Colmap**: Colmap is notoriously hard to please, and we should investigate how to make it more tolerant of user videos. See [Colmap Brainstorming]() to get started.
+- **Expand functionality**: We could possible expand into a more general purpose Deep Learning powered video app. Some possibilites:
+    1. Stylized Text-to-Scene: Recent research for Text-Based Scene generation has shown crazy progress on stylized/themed scene generation
+- **Testing and Cleanup**: We can always improve our codebase by implementing further testing.
+- **CI/CD Pipelines**: Upon successful deployment we could set up dedicated testing pipelines. This would be a big stretch. For now, we could create
+  workflows to ensure code quality, security, and testing coverage for lighter parts of the system.
+- **Docker Hub Image Generation**: Setting up image generation would allow for users to easily start their own instance without build hassles.
 
 ## Contributing
 
 Contributions are what make the open source community such an amazing place to
 learn, inspire, and create. Any contributions you make are **greatly appreciated**.
 
 If you have a suggestion that would make this better, please fork the repo and
-create a pull request.
+create a pull request. Please go the the relevant repository and follow this process.
 
 1. Fork the Project
 2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)
@@ -136,16 +206,17 @@ Or, inquire at: `nerf@quicktechtime.com`
 ## Acknowledgments
 
 * [TensoRF project](https://github.com/apchenstu/TensoRF)
+* [Gaussian Splatting](https://github.com/graphdeco-inria/gaussian-splatting)
 * [Original NeRF](https://github.com/bmild/nerf)
 * [COLMAP](https://colmap.github.io/)
 
 <!-- MARKDOWN LINKS & IMAGES -->
 <!-- https://www.markdownguide.org/basic-syntax/#reference-style-links -->
-[contributors-shield]: https://img.shields.io/github/contributors/NeRF-or-Nothing/vidtonerf.svg?style=for-the-badge
-[contributors-url]: https://github.com/NeRF-or-Nothing/vidtonerf/graphs/contributors
-[forks-shield]: https://img.shields.io/github/forks/NeRF-or-Nothing/vidtonerf.svg?style=for-the-badge
-[forks-url]: https://github.com/NeRF-or-Nothing/vidtonerf/network/members
-[issues-shield]: https://img.shields.io/github/issues/NeRF-or-Nothing/vidtonerf.svg?style=for-the-badge
-[issues-url]: https://github.com/NeRF-or-Nothing/vidtonerf/issues
-[license-shield]: https://img.shields.io/github/license/NeRF-or-Nothing/vidtonerf.svg?style=for-the-badge
-[license-url]: https://github.com/NeRF-or-Nothing/vidtonerf/blob/master/LICENSE.txt
+[contributors-shield]: https://img.shields.io/github/contributors/NeRF-or-Nothing/backend.svg?style=for-the-badge
+[contributors-url]: https://github.com/NeRF-or-Nothing/backend/graphs/contributors
+[forks-shield]: https://img.shields.io/github/forks/NeRF-or-Nothing/backend.svg?style=for-the-badge
+[forks-url]: https://github.com/NeRF-or-Nothing/backend/network/members
+[issues-shield]: https://img.shields.io/github/issues/NeRF-or-Nothing/backend.svg?style=for-the-badge
+[issues-url]: https://github.com/NeRF-or-Nothing/backend/issues
+[license-shield]: https://img.shields.io/github/license/NeRF-or-Nothing/backend.svg?style=for-the-badge
+[license-url]: https://github.com/NeRF-or-Nothing/backend/blob/master/LICENSE.txt