Transparency / Depth

[timothyallan]

Hey Mark, I see in the readme that having this work with transparency in three is "challenging". I'm using this library to import splats into existing three scenes, where almost everything else has transparency=true set in the materials. Of course this means that those materials/shapes are visible at all times even when they should be occluded by the splat. People can move objects around in the scene, so raycasting and transparency are both needed for my use case. I've got raycasting going with both splats and objects, but am stuck on proper occlusion.

Being as math is not my strong suit, but having seen other splat viewers where there is proper occluding with other objects with transparency, I'm wondering a few things:

• Is integration with three.js depth/transparency on the roadmap at all?
• If not, is it viable for a math normie like myself to fork this and hack together something, or are there really good reasons why it's not implemented with this project?

Edit: I saw that the a-frame folks have transparency enabled in their implementation https://github.com/quadjr/aframe-gaussian-splatting/blob/main/index.js, but there are still discussions around multiple splats:quadjr/aframe-gaussian-splatting#20 ... although this demo seems to handle multi splat occlusion without issue: https://splats.vercel.app/ https://github.com/drcmda/splats

[mkkellogg]

Sorry for the late reply, which viewers have you seen achieve proper occluding with transparent objects? The viewer by drcmda seems to have issues at several positions and orientations where objects are clearly not rendered in the correct order. He attempts to get good results in most cases by sorting the objects in the scene by distance from the camera and rendering back to front, but it doesn't always work. There is also an open PR in the AFrame gaussian splatting repo that adds depth buffer writes to the splat renderer, the but the results don't look quite right: quadjr/aframe-gaussian-splatting#25 (comment) The problem there is that you can't just write to the depth buffer when rendering a gaussian, only the center position (mean) of the gaussian has any real physical meaning.

For now proper co-existence of splat scenes with other transparent objects is not planned. However there are research projects that have managed to derive depth information during the training process and include that information with the final output (e.g SuGaR). I have thought about eventually supporting something like that, which would allow more correct depth buffer rendering, and compatibility with other transparent objects in the scene.

[timothyallan]

All makes sense, thanks for getting back. I thought the drcmda one was pretty good, but obviously didn't spend enough time playing around with it to see the kinks. And while the side by side in the the A-Frame one in #25 looks passable while still, it might fall apart when actually playing around with it in use.

The SuGaR thing is interesting, I'm going to see what's involved in setting up their demo from the github https://github.com/Anttwo/SuGaR

Did you see the SuGaR demo gives you a shout out for your viewer? :)

[mkkellogg]

I actually had no idea they used my viewer in one of their updates! Thanks for pointing that out :)