Skip to content

Commit

Permalink
Implementation of 3dgs-mcmc
Browse files Browse the repository at this point in the history
  • Loading branch information
Shakiba Kheradmand authored and Shakiba Kheradmand committed Jun 17, 2024
1 parent 472689c commit b8b4671
Show file tree
Hide file tree
Showing 28 changed files with 2,197 additions and 561 deletions.
7 changes: 4 additions & 3 deletions .gitmodules
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
[submodule "submodules/simple-knn"]
path = submodules/simple-knn
url = https://gitlab.inria.fr/bkerbl/simple-knn.git
[submodule "submodules/diff-gaussian-rasterization"]
path = submodules/diff-gaussian-rasterization
url = https://github.com/graphdeco-inria/diff-gaussian-rasterization
[submodule "SIBR_viewers"]
path = SIBR_viewers
url = https://gitlab.inria.fr/sibr/sibr_core.git
[submodule "submodules/diff-gaussian-rasterization"]
path = submodules/diff-gaussian-rasterization
url = git@github.com:shakibakh/diff-gaussian-rasterization.git
branch = gs-mcmc
519 changes: 16 additions & 503 deletions README.md
View file

Large diffs are not rendered by default.

11 changes: 8 additions & 3 deletions arguments/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,13 +47,15 @@ def extract(self, args):
class ModelParams(ParamGroup):
def __init__(self, parser, sentinel=False):
self.sh_degree = 3
self._source_path = ""
self._source_path = "../../data/360_v2/counter"
self._model_path = ""
self._images = "images"
self._resolution = -1
self._white_background = False
self.data_device = "cuda"
self.eval = False
self.eval = True
self.cap_max = -1
self.init_type = "random"
super().__init__(parser, "Loading Parameters", sentinel)

def extract(self, args):
Expand Down Expand Up @@ -84,9 +86,12 @@ def __init__(self, parser):
self.densification_interval = 100
self.opacity_reset_interval = 3000
self.densify_from_iter = 500
self.densify_until_iter = 15_000
self.densify_until_iter = 25_000
self.densify_grad_threshold = 0.0002
self.random_background = False
self.noise_lr = 5e5
self.scale_reg = 0.01
self.opacity_reg = 0.01
super().__init__(parser, "Optimization Parameters")

def get_combined_args(parser : ArgumentParser):
Expand Down
Binary file added docs/.DS_Store
View file
Binary file not shown.
287 changes: 287 additions & 0 deletions docs/index.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,287 @@
<html lang="en">
<head>
<link rel="stylesheet" href="./resources/css/bulma.min.css" />
<link rel="stylesheet" href="./resources/css/slide.css" />
<link rel="stylesheet" href="./resources/css/bulma-carousel.min.css" />
<link rel="stylesheet" href="./resources/css/bulma-slider.min.css" />
<link rel="stylesheet" href="./resources/css/fontawesome.all.min.css" />
<link rel="stylesheet" href="./resources/css/carasoul.css" />
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"
/>
<link rel="stylesheet" href="./resources/css/index.css" />
<link rel="icon" href="./resources/images/favicon.svg" />
<link
rel="stylesheet"
href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css"
/>
<link
rel="stylesheet"
type="text/css"
href="./resources/css/style.css"
media="screen"
/>
<link rel="stylesheet" href="resources/css/dics.original.css" />

<title>3D Gaussian Splatting as Markov Chain Monte Carlo</title>
<meta
property="og:title"
content="3D Gaussian Splatting as Markov Chain Monte Carlo"
/>
<meta name="viewport" content="width=device-width, initial-scale=1.0" />

<style>
body {
text-align: center;
background-color: #f0f0f0; /* Optional: Set a background color for better visibility */
}
.larger-arrow {
font-size: 2em; /* Adjust the font size as needed */
letter-spacing: 4em; /* Adjust the letter spacing as needed */
display: inline-block; /* Ensures text-align works */
}
.tbl_video {
margin-bottom: 40px;
}
</style>
<script src="resources/js/video_comparison.js"></script>
<script src="resources/js/dics.original.js"></script>
</head>

<body>
<div class="container">
<div class="title">
3D Gaussian Splatting as Markov Chain Monte Carlo
</div>

<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<div class="is-size-5 publication-authors">
<span class="author-block">
<a href="https://shakibakh.github.io/">Shakiba Kheradmand</a
><sup>1</sup>,</span
>
<span class="author-block">
<a href="http://drebain.com/"> Daniel Rebain</a
><sup>1</sup>,</span
>
<span class="author-block">
<a href="https://hippogriff.github.io/"> Gopal Sharma</a
><sup>1</sup>,</span
>
<span class="author-block">
<a href="http://www.hossamisack.com/">Hossam Isack</a
><sup>2</sup>,
</span>
<span class="author-block">
<a href="https://abhishekkar.info/">Abhishek Kar</a><sup>2</sup>
</span>
<br />
<span class="author-block">
<a href="https://taiya.github.io/">Andrea Tagliasacchi</a
><sup>3, 4, 5</sup>
</span>
<span class="author-block">
<a href="https://www.cs.ubc.ca/~kmyi/">Kwang Moo Yi</a
><sup>1</sup>
</span>
</div>

<div class="is-size-5 publication-authors">
<span class="author-block"
><sup>1</sup>University of British Columbia</span
>
<span class="author-block"><sup>2</sup>Google Research</span>
<span class="author-block"><sup>3</sup>Google DeepMind</span>
<br />
<span class="author-block"
><sup>4</sup>Simon Fraser University</span
>
<span class="author-block"
><sup>5</sup>University of Toronto</span
>
</div>
</div>
</div>
</div>

<div class="column has-text-centered">
<div class="publication-links">
<!-- Paper Link. -->
<span class="link-block">
<!-- <a href="https://arxiv.org/abs/2404.09591" class="external-link button is-normal is-rounded is-dark"> -->
<a
href="paper.pdf"
class="external-link button is-normal is-rounded is-dark"
>
<span class="icon">
<i class="fa fa-file-o"></i>
</span>
<span>Paper</span>
</a>
</span>
<!-- Code Link. -->
<span class="link-block">
<a
href="https://github.com/ubc-vision/3dgs-mcmc"
class="external-link button is-normal is-rounded is-dark"
>
<span class="icon">
<svg
class="svg-inline--fa fa-github fa-w-16"
aria-hidden="true"
focusable="false"
data-prefix="fab"
data-icon="github"
role="img"
xmlns="http://www.w3.org/2000/svg"
viewBox="0 0 496 512"
data-fa-i2svg=""
>
<path
fill="currentColor"
d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"
></path></svg
><!-- <i class="fab fa-github"></i> Font Awesome fontawesome.com -->
</span>
<span>Code</span>
</a>
</span>
</div>
</div>

<div class="parent-video-compare-container">
<hr />
</div>

<div class="parent-video-compare-container">
<div class="video-compare-container" id="materialsDiv">
<video
class="video"
id="materials"
loop
playsinline
autoplay
muted
src="./resources/training_rand_compare/bicycle_both-rand.mp4"
onplay="resizeAndPlay(this)"
></video>
<canvas height="0" class="videoMerge" id="materialsMerge"></canvas>
</div>
<br>
<p class="justified">
Novel view reconstructions for <strong>(right) our method</strong>
and <strong>(left) conventional</strong> 3D Gaussian Splatting with
random initializations. Our method, even with random initialization,
faithfully reconstructs the scene (e.g.. buildings at the back and
the ground texture) providing much higher quality renderings.
</p>
</div>

<div class="parent-video-compare-container">
<hr />
</div>

<h1>Abstract</h1>
<div class="parent-video-compare-container">
<p class="justified">
While 3D Gaussian Splatting has recently become popular for neural
rendering, current methods rely on carefully engineered cloning and
splitting strategies for placing Gaussians, which can lead to
poor-quality renderings, and reliance on a good initialization. In
this work, we rethink the set of 3D Gaussians as a random sample
drawn from an underlying probability distribution describing the
physical representation of the scene---in other words, Markov Chain
Monte Carlo (MCMC) samples. Under this view, we show that the 3D
Gaussian updates can be converted as Stochastic Gradient Langevin
Dynamics (SGLD) update by simply introducing noise. We then rewrite
the densification and pruning strategies in 3D Gaussian Splatting as
simply a deterministic state transition of MCMC samples, removing
these heuristics from the framework. To do so, we revise the
`cloning' of Gaussians into a relocalization scheme that
approximately preserves sample probability. To encourage efficient
use of Gaussians, we introduce a regularizer that promotes the
removal of unused Gaussians. On various standard evaluation scenes,
we show that our method provides improved rendering quality, easy
control over the number of Gaussians, and robustness to
initialization.
</p>
</div>

<div class="parent-video-compare-container">
<hr />
</div>

<h1>More Results</h1>

<div class="parent-video-compare-container">
<table class="tbl_video" style="width:100%;"">
<tr>
<td colspan="4" style="background-color: #d1c4ce; font-size: 20px">
'10' sequence from OMMO dataset
</td>
</tr>
<tr>
<td width="50%" style="font-size: 18px">3DGS-Random</td>
<td width="50%" style="font-size: 18px">3DGS</td>
</tr>
<tr>
<td colspan="2">
<video
class="kitti360"
id="00"
width="95%"
preload="auto"
playsinline
webkit-playsinline
loop
autoplay
muted
>
<source src="resources/10/10.mp4" type="video/mp4" />
</video>
</td>
</tr>
<tr>
<td width="50%" style="font-size: 18px">Ours-Random</td>
<td width="50%" style="font-size: 18px">Ours</td>
</tr>

<table class="tbl_video" style="width:100%;"">
<tr>
<td colspan="4" style="background-color: #d1c4ce; font-size: 20px">
'Stump' sequence from the MipNeRF360 dataset (pay attention to the
details between the leaves)
</td>
</tr>
<tr>
<td width="50%" style="font-size: 18px">3DGS-Random</td>
<td width="50%" style="font-size: 18px">3DGS</td>
</tr>
<tr>
<td colspan="2">
<video
class="kitti360"
id="00"
width="95%"
preload="auto"
playsinline
webkit-playsinline
loop
autoplay
muted
>
<source src="resources/stump/stump.mp4" type="video/mp4" />
</video>
</td>
</tr>
<tr>
<td width="50%" style="font-size: 18px">Ours-Random</td>
<td width="50%" style="font-size: 18px">Ours</td>
</tr>
</div>
</div>
</body>
</html>
Binary file added docs/paper.pdf
View file
Binary file not shown.
Binary file added docs/resources/10/10.mp4
View file
Binary file not shown.
1 change: 1 addition & 0 deletions docs/resources/css/bulma-carousel.min.css
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit b8b4671

Please sign in to comment.