WebXRManager: Added depth sensing support #27154

cabanier · 2023-11-09T01:02:32Z

First pass at adding support for WebXR GPU based depth sensing.
Also updated the XR samples to request depth.

We can tweak shaders and parameters later.

This contribution is funded by Meta

github-actions · 2023-11-09T01:05:33Z

📦 Bundle size

Full ESM build, minified and gzipped.

Filesize `dev`	Filesize PR	Diff
668.2 kB (165.8 kB)	674.9 kB (167 kB)	+6.71 kB

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

Filesize `dev`	Filesize PR	Diff
449.4 kB (108.9 kB)	455.7 kB (109.9 kB)	+6.37 kB

CodyJasonBennett · 2023-11-09T01:26:27Z

src/renderers/webgl/WebGLProgram.js

+			parameters.occlusion ? '#define USE_OCCLUSION' : '#define USE_OCCLUSION',
+


Both sides are equal; the right-hand side should be ''.

I know :-)
That is why it's WIP and still a draft PR

@CodyJasonBennett fixed!

hybridherbst · 2023-11-09T05:09:36Z

Super cool!

Out of curiosity, have you done a perf/quality comparison between:

passing depth into each material as in this PR
drawing a fullscreen pass that renders depth only and sets z buffer values so others test against that, for example right from the background material

I’d think writing the z buffer directly loses the fuzziness but should be faster (can use early z discard, no extra texture fetch per fragment, etc)

cabanier · 2023-11-09T05:12:42Z

Super cool!

Out of curiosity, have you done a perf/quality comparison between:

passing depth into each material as in this PR

drawing a fullscreen pass that renders depth only and sets z buffer values so others test against that, for example right from the background material

I’d think writing the z buffer directly loses the fuzziness but should be faster (can use early z discard, no extra texture fetch per fragment, etc)

I have not done a perf comparison. My naive way seems to integrate nicely with three, although it's a bit special since it's a renderer feature and not per material or scene.
We can revisit more optimal approaches later. I can also investigate how this is done in unity and unreal.

mrdoob · 2023-11-09T08:21:09Z

@hybridherbst

drawing a fullscreen pass that renders depth only and sets z buffer values so others test against that, for example right from the background material

I feel like this approach is much more straight forward and easier to maintain, regardless of the performance implications.

I think I would try implementing this approach first before modifying all the shaders for all materials.

cabanier · 2023-11-09T14:10:25Z

src/renderers/webxr/WebXRManager.js

+
+						if (( depthData.depthNear != session.renderState.depthNear ) || ( depthData.depthFar != session.renderState.depthFar )) {
+
+							session.updateRenderState( {depthNear: depthData.depthNear, depthFar: ( depthData.depthFar > 10000 ? 10000 : depthData.depthFar ) } );


this seems like a WebXR bug

Spec looks like it allows infinity:

If activeState’s depthFar is greater than session’s maximum far clip plane set activeState’s depthFar to session’s maximum far clip plane.

And

The maximum far clip plane SHOULD be greater than 1000.0 (and MAY be infinite).

So if it's not working then it's a Chromium implementation issue. Idea is to allow depthFar: Infinity, right?

So if it's not working then it's a Chromium implementation issue. Idea is to allow depthFar: Infinity, right?

Yes. If I pass infinity, I get an error saying the value is non-finite

cabanier · 2023-11-10T22:41:20Z

src/renderers/WebGLRenderer.js

@@ -1600,7 +1615,7 @@ class WebGLRenderer {

 			const lightsStateVersion = lights.state.version;

-			const parameters = programCache.getParameters( material, lights.state, shadowsArray, scene, object );
+			const parameters = programCache.getParameters( material, lights.state, shadowsArray, scene, object, _occlusion );


@mrdoob is there a better way than passing in a renderer variable?

cabanier · 2023-11-10T22:46:30Z

@mrdoob @Mugen87 would you mind taking a look?

src/renderers/shaders/ShaderChunk/occlusion_fragment.glsl.js

snowzurfer · 2023-11-11T07:50:27Z

fwiw it looks like on Unity, at least for custom shaders, they suggest modifying the shaders and calculating the occlusion in the fragment shader:

half4 frag(v2f i) {
   UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(i);

   // this is something your shader will return without occlusions
   half4 fragmentShaderResult = someColor;

   // calculate UV for the depth texture lookup for occlusions
   float2 uv = i.positionNDC.xy / i.positionNDC.w;

   // pass UV and the current depth of the texel
   float occlusionValue = CalculateEnvironmentDepthOcclusion(uv, i.vertex.z);

   // consider early rejection to not write to depth if it's an opaque shader
   if (occlusionValue < 0.01) {
       discard;
   }

   // premultiply color and alpha by occlusion value
   // when it's 1 - color is not affected - virtual covers real
   // when it's 0 - texel is invisible - virtual is under real
   // when it's in between - texel is semi transparent
   fragmentShaderResult *= occlusionValue;

   return result;
}

cabanier · 2023-11-11T15:23:44Z

fwiw it looks like on Unity, at least for custom shaders, they suggest modifying the shaders and calculating the occlusion in the fragment shader

We can play with the shaders in follow up patches. It might even be interesting to vary which depth values are honored, or provide a color to overlay real world objects

* Linting, add maxGeometryCount to BatchedMesh, remove undocumented functions * BatchedMesh: Update documentation

* Introduction to fragmentNode * revision

…#27233) * Fix .clear() using RenderTarget * webgpu_rtt: improve a little * cleanup * cleanup * revisions * revision * wip flipY * update example * cleanup

* WebGPURenderer: Depth Pixel & Logarithmic Depth Buffer * Update puppeteer.js

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

…were possible (mrdoob#27551) * remove redundant code * use cached descriptor properties for clear() * remove unused import --------- Co-authored-by: aardgoose <angus.sawyer@email.com>

…rdoob#27544) * WebXR: add onLoad callback for XRHandMeshModel * add onLoad for XRControllerModelFactory

…ackend` (mrdoob#27567)

cabanier · 2024-01-16T17:27:18Z

@mrdoob @Mugen87 depth sensing is now on by default in the Quest browser. Can we merge this PR?

* add lengthSq to math nodes * forgot lengthSq doesn't exist in glsl * cleanup ---------

* WebGPURenderer: PassNode + PostProcessing * update sky color * cleanup * a bit more realistic

* WebGPURenderer improve copyTextureToBuffer * remove unecessary comment * correct comment

* camera proj inverse was missing * apply mat4 to projectionMatrixInverse

hybridherbst · 2024-01-17T23:45:00Z

@cabanier I was able to try this out (finally!) but still had to turn the option on on v60. Looks very jaggy/pixelated even on Quest 3 but for some experiences this might be acceptable. Is there already a newer version than v60 that has it enabled by default?

Also, some questions:

can the depth transition "softness" be controlled per object?
can the depth effect be turned off per object/per material? From the code it looks like this would just be material.occlusion = false;, right?

cabanier · 2024-01-18T00:23:01Z

@cabanier I was able to try this out (finally!) but still had to turn the option on on v60. Looks very jaggy/pixelated even on Quest 3 but for some experiences this might be acceptable.

True. There are better shaders than the simple ones I added. I was hoping that we experiment with them after landing the fundamentals.

Is there already a newer version than v60 that has it enabled by default?

It should be enabled by default in 31.1. Is that not the version you have?
OS v60 has support for depth sensing.

can the depth transition "softness" be controlled per object?

Not at the moment

can the depth effect be turned off per object/per material? From the code it looks like this would just be material.occlusion = false;, right?

Yes, for some content it might be desired that it's not occluded. (ie hands or the controller)

Mugen87 · 2024-02-02T12:03:07Z

Closing in favor of #27586.

hybridherbst · 2024-02-02T22:17:03Z

@Mugen87 I'm not sure if this issue should be closed, the topic of "how to allow the shader-based version in a sample" is still very much open. Many effects can't be achieved with the v2 implementation and vice versa, both have their place.

Mugen87 · 2024-02-03T10:14:22Z

At least at the moment the project is not willing to support both paths in WebGLRenderer, especially since this PR required in-depth changes in the material system. Unless @mrdoob does not decide different, three.js will only offer what has been implement in #27586 for now.

My hope is it will be easier to support changes like this with WebGPURenderer.

hybridherbst · 2024-02-03T20:44:40Z

I mean like this – v2 in core, v1 as sample:
#27586 (comment)

Of course @cabanier or @mrdoob can open separate PRs for a sample that uses custom shaders, I just think it may get lost when this PR here is closed.

cabanier · 2024-02-06T16:03:01Z

I mean like this – v2 in core, v1 as sample: #27586 (comment)

Of course @cabanier or @mrdoob can open separate PRs for a sample that uses custom shaders, I just think it may get lost when this PR here is closed.

I will make an attempt to extend the WebXRManager and write a test that uses a custom shader.

CodyJasonBennett reviewed Nov 9, 2023

View reviewed changes

cabanier commented Nov 9, 2023

View reviewed changes

cabanier force-pushed the occlusion branch 7 times, most recently from 546c1af to e0b2926 Compare November 10, 2023 22:16

cabanier commented Nov 10, 2023

View reviewed changes

cabanier marked this pull request as ready for review November 10, 2023 22:44

snowzurfer reviewed Nov 11, 2023

View reviewed changes

src/renderers/shaders/ShaderChunk/occlusion_fragment.glsl.js Show resolved Hide resolved

cabanier force-pushed the occlusion branch 3 times, most recently from 5c10730 to f0b6a7c Compare November 17, 2023 17:59

gkjohnson and others added 8 commits November 23, 2023 18:16

BatchedMesh: cleanup, add maxGeometryCount member (mrdoob#27231)

1310548

* Linting, add maxGeometryCount to BatchedMesh, remove undocumented functions * BatchedMesh: Update documentation

AudioLoader: Fix uncaught DOMException in promise. (mrdoob#27237)

b87e54f

Updated builds.

6f36242

Rename .colorSpace -> .colorSpaced (mrdoob#27238)

3b2deb6

TSL: Introduction to fragmentNode (mrdoob#27239)

1ab4a8a

* Introduction to fragmentNode * revision

WebGPURenderer: Fix .clear() using RenderTarget and revisions (mrdoob…

0f94eea

…#27233) * Fix .clear() using RenderTarget * webgpu_rtt: improve a little * cleanup * cleanup * revisions * revision * wip flipY * update example * cleanup

Cleanup from mrdoob#27233

1eaa18e

WebGPURenderer: Depth Pixel & Logarithmic Depth Buffer (mrdoob#27243)

549a2e3

* WebGPURenderer: Depth Pixel & Logarithmic Depth Buffer * Update puppeteer.js

renovate bot and others added 8 commits January 14, 2024 19:42

Update dependency rollup to v4.9.5 (mrdoob#27562)

a1be53d

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

show which shader caused compilation error (mrdoob#27553)

329f9f0

WebGLProgram: Improve formatting of error log. (mrdoob#27565)

2ba547d

WebGPURenderer: clear() - use cached descriptor properties and views …

c6c81de

…were possible (mrdoob#27551) * remove redundant code * use cached descriptor properties for clear() * remove unused import --------- Co-authored-by: aardgoose <angus.sawyer@email.com>

WebXR: optional onLoad callback when loading hands or controllers (m…

d25f427

…rdoob#27544) * WebXR: add onLoad callback for XRHandMeshModel * add onLoad for XRControllerModelFactory

Updated builds.

7a47f0e

WebGPURenderer: Fix BachedMesh coordinate system in case of `WebGLB…

0b91656

…ackend` (mrdoob#27567)

Nodes: Fix viewportCoordinate build (mrdoob#27566)

534737d

RenaudRohlinger and others added 9 commits January 16, 2024 20:57

Nodes: Add lengthSq to MathNodes (mrdoob#27568)

112f812

* add lengthSq to math nodes * forgot lengthSq doesn't exist in glsl * cleanup ---------

WebGPURenderer: PassNode + PostProcessing - WebGPUBackend (mrdoob#27537)

6a3efed

* WebGPURenderer: PassNode + PostProcessing * update sky color * cleanup * a bit more realistic

WebGPURenderer: Improve copyTextureToBuffer support (mrdoob#27569)

96cb1b5

* WebGPURenderer improve copyTextureToBuffer * remove unecessary comment * correct comment

WebGPURenderer: Add renderer parameter in Textures

24b8c20

Example webgpu_portal: cleanup

beecde0

camera proj inverse was missing (mrdoob#27578)

21b6b50

Nodes: Add mat4 format to cameraProjectionMatrixInverse (mrdoob#27579)

b8b82d9

* camera proj inverse was missing * apply mat4 to projectionMatrixInverse

add support for depth sensing

b6ee92b

Merge branch 'occlusion' of github.com:cabanier/three.js into occlusion

7be54b5

Mugen87 mentioned this pull request Jan 19, 2024

WebXRManager: Added depth sensing support (v2). #27586

Merged

mrdoob modified the milestones: r161, r162 Jan 31, 2024

Mugen87 removed this from the r162 milestone Feb 2, 2024

Mugen87 closed this Feb 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebXRManager: Added depth sensing support #27154

WebXRManager: Added depth sensing support #27154

cabanier commented Nov 9, 2023 •

edited

Loading

github-actions bot commented Nov 9, 2023 •

edited

Loading

CodyJasonBennett Nov 9, 2023

cabanier Nov 9, 2023

cabanier Nov 10, 2023

hybridherbst commented Nov 9, 2023 •

edited

Loading

cabanier commented Nov 9, 2023 •

edited

Loading

mrdoob commented Nov 9, 2023

cabanier Nov 9, 2023

toji Nov 9, 2023

cabanier Nov 9, 2023

cabanier Nov 10, 2023

cabanier commented Nov 10, 2023

snowzurfer commented Nov 11, 2023

cabanier commented Nov 11, 2023

cabanier commented Jan 16, 2024

hybridherbst commented Jan 17, 2024

cabanier commented Jan 18, 2024

Mugen87 commented Feb 2, 2024

hybridherbst commented Feb 2, 2024

Mugen87 commented Feb 3, 2024 •

edited

Loading

hybridherbst commented Feb 3, 2024 •

edited

Loading

cabanier commented Feb 6, 2024

		parameters.occlusion ? '#define USE_OCCLUSION' : '#define USE_OCCLUSION',


		if (( depthData.depthNear != session.renderState.depthNear ) \|\| ( depthData.depthFar != session.renderState.depthFar )) {

		session.updateRenderState( {depthNear: depthData.depthNear, depthFar: ( depthData.depthFar > 10000 ? 10000 : depthData.depthFar ) } );

WebXRManager: Added depth sensing support #27154

WebXRManager: Added depth sensing support #27154

Conversation

cabanier commented Nov 9, 2023 • edited Loading

github-actions bot commented Nov 9, 2023 • edited Loading

📦 Bundle size

🌳 Bundle size after tree-shaking

CodyJasonBennett Nov 9, 2023

Choose a reason for hiding this comment

cabanier Nov 9, 2023

Choose a reason for hiding this comment

cabanier Nov 10, 2023

Choose a reason for hiding this comment

hybridherbst commented Nov 9, 2023 • edited Loading

cabanier commented Nov 9, 2023 • edited Loading

mrdoob commented Nov 9, 2023

cabanier Nov 9, 2023

Choose a reason for hiding this comment

toji Nov 9, 2023

Choose a reason for hiding this comment

cabanier Nov 9, 2023

Choose a reason for hiding this comment

cabanier Nov 10, 2023

Choose a reason for hiding this comment

cabanier commented Nov 10, 2023

snowzurfer commented Nov 11, 2023

cabanier commented Nov 11, 2023

cabanier commented Jan 16, 2024

hybridherbst commented Jan 17, 2024

cabanier commented Jan 18, 2024

Mugen87 commented Feb 2, 2024

hybridherbst commented Feb 2, 2024

Mugen87 commented Feb 3, 2024 • edited Loading

hybridherbst commented Feb 3, 2024 • edited Loading

cabanier commented Feb 6, 2024

cabanier commented Nov 9, 2023 •

edited

Loading

github-actions bot commented Nov 9, 2023 •

edited

Loading

hybridherbst commented Nov 9, 2023 •

edited

Loading

cabanier commented Nov 9, 2023 •

edited

Loading

Mugen87 commented Feb 3, 2024 •

edited

Loading

hybridherbst commented Feb 3, 2024 •

edited

Loading