Quick KHR_materials_dispersion implementation

[MiiBond]

Here's a go at implementing dispersion in the renderer as well as support for the glTF KHR_materials_dispersion extension.

I'm not completely satisfied with the implementation in the shader. It's a bit ugly but creating another function was the only way I could see to avoid duplicating lots of code.
To mimic the dispersion effect, I compute the refraction coords 3 times with slightly different IOR's and then sample the environment refraction with each of these. The final refraction is a combination of R from the first, G from the second and B from the last.

[bjsplat]

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

[bjsplat]

Snapshot stored with reference name:
refs/pull/14501/merge

Test environment:
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/index.html

To test a playground add it to the URL, for example:

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/index.html#WGZLGJ#4600

Links to test babylon tools with this snapshot:

https://playground.babylonjs.com/?snapshot=refs/pull/14501/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/14501/merge
https://gui.babylonjs.com/?snapshot=refs/pull/14501/merge
https://nme.babylonjs.com/?snapshot=refs/pull/14501/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/14501/merge#BCU1XR#0

[bjsplat]

Visualization tests for webgl1 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl1/index.html

If tests were successful afterwards, this report might not be available anymore.

[bjsplat]

Visualization tests for webgl2 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl2/index.html

If tests were successful afterwards, this report might not be available anymore.

[RaananW]

Wanted to comment about the failed vis tests - it seems like this PR breaks the node material PBR playground. You can see it here (using the build snapshot of this PR)

https://playground.babylonjs.com/?snapshot=refs/pull/14501/merge#D8AK3Z#17

[bjsplat]

Visualization tests for webgl1 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl1/index.html

If tests were successful afterwards, this report might not be available anymore.

[bjsplat]

Visualization tests for webgl2 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl2/index.html

If tests were successful afterwards, this report might not be available anymore.

[RaananW]

Seems to be now related to a shader compilation issue. Some object is incorrectly deserialized:

This can be seen in the same example I pasted in the last message.

[Popov72]

Waiting for the CI to run without errors before reviewing!

[bghgary]

I only reviewed the glTF parts. I'll leave the shader parts to @Popov72.

[bjsplat]

Visualization tests for webgl1 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl1/index.html

If tests were successful afterwards, this report might not be available anymore.

[bjsplat]

Visualization tests for webgl2 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl2/index.html

If tests were successful afterwards, this report might not be available anymore.

[bjsplat]

Visualization tests for webgl1 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl1/index.html

If tests were successful afterwards, this report might not be available anymore.

[bjsplat]

Visualization tests for webgl2 have failed. If some tests failed because the snapshots do not match, the report can be found at

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge/testResults/webgl2/index.html

If tests were successful afterwards, this report might not be available anymore.

[emackey]

I have a draft sample model in the works: KhronosGroup/glTF#2340 (comment)

[sebavan]

Can not wait this is way too cool to see all of you here :-)

[MiiBond]

Sorry, I'm just getting back to this. Is there a way that I can run the failing tests locally? Or see some better output?
I don't see "Visualization" test listed as a debug target in VSCode.

[RaananW]

Sorry, I'm just getting back to this. Is there a way that I can run the failing tests locally? Or see some better output? I don't see "Visualization" test listed as a debug target in VSCode.

yes, you can run visualization tests locally or against the snapshot of this PR.

If you want to run locally, first run npm run start to start the dev server. Afterwards, run npm run test:visualization -- -i "webgl2" -t "name of test you want to run" . This will run the visualization test locally and will display any console error in your console. You can also try running it with the new playwright ui (still experimental) using npm run test:playwright -w @tools/tests -- --project=webgl2 --ui

If you want to test against the pull request (no need for the local server) you can change the environment variable CDN_BASE_URL to https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14501/merge. But you have have the dev enrionment running it is simpler to use the dev server, because then you can make changes to the code and run the test again.

[Popov72]

Looking good!

[deltakosh]

Can we merge that one?

[deltakosh]

@bghgary ? Or else should we move it to draft

[sebavan]

one only tiny comment but great job on the PR @MiiBond !!!!

[bghgary]

LGTM for the glTF parts

[echadwick-artist]

Sample asset of the Khronie Award, suggested by @emackey. I couldn't get it to drag-n-drop into the Babylon.js demo above, guess I need to use a Playground?

Anyhow here's the asset so far, screenshot from the main Sandbox.

Also in Dassault Systemes' PBR Sample Renderer:

  "materials": [
    {
      "name": "KhronieAward_Clear_Glass",
      "extensions": {
        "KHR_materials_transmission": {
          "transmissionFactor": 1.0
        },
        "KHR_materials_volume": {
          "attenuationColor": [
            1.0,
            1.0,
            1.0
          ],
          "thicknessFactor": 0.01
        },
        "KHR_materials_dispersion": {
          "dispersion": 0.33
        }
      },
      "pbrMetallicRoughness": {
        "metallicFactor": 0.0,
        "roughnessFactor": 0.0
      }
    },
    {
      "name": "KhronieAward_Etched_Text",
      "alphaMode": "BLEND",
      "pbrMetallicRoughness": {
        "baseColorTexture": {
          "index": 0
        },
        "baseColorFactor": [
          1,
          1,
          1,
          0.4
        ],
        "metallicFactor": 0.0,
        "roughnessFactor": 0.7
      },
      "normalTexture": {
        "index": 1,
        "scale": 1
      }
    },
    {
      "name": "KhronieAward_Rubber_Feet",
      "extensions": {
        "KHR_materials_transmission": {
          "transmissionFactor": 1.0
        },
        "KHR_materials_volume": {
          "attenuationColor": [
            1.0,
            1.0,
            1.0
          ],
          "thicknessFactor": 0.01
        }
      },
      "pbrMetallicRoughness": {
        "metallicFactor": 0.0,
        "roughnessFactor": 0.4
      }
    },
    {
      "name": "KhronieAward_Smoked_Glass",
      "extensions": {
        "KHR_materials_transmission": {
          "transmissionFactor": 1.0
        },
        "KHR_materials_volume": {
          "attenuationColor": [
            1.0,
            1.0,
            1.0
          ],
          "thicknessFactor": 0.01
        },
        "KHR_materials_dispersion": {
          "dispersion": 0.33
        }
      },
      "pbrMetallicRoughness": {
        "baseColorFactor": [
          0.3,
          0.3,
          0.3019610047340393,
          1.0
        ],
        "metallicFactor": 0.0,
        "roughnessFactor": 0.0
      }
    }
  ],

KhronieAward.zip

[echadwick-artist]

Improved the asset a little.

1. Front-facing text is now alphaMode:Mask without Transmission, so rasterizers can render it behind the transmissive glass. Babylon.js can render the Blend material behind a Tranmission material, but Three.js and Filament will only do this if Alpha is set to Mask.

2. Rear-facing text is now alphaMode:Blend with Transmission, to prevent rasterizers from rendering a duplicate within the transmission of the glass.

Also here's a V-Ray render with the corresponding Abbe value:

KhronieAward_dispersion.zip

[echadwick-artist]

I've adjusted the materials for the text a bit more, so the rear-facing text is a better match to the look of the front-facing text.
KhronieAward-glTF.zip

I'm also able to see dispersion rendering in the Sandbox. However it only seems to appear when I disable "Use Thickness as Depth". This asset is modeled to scale, and the glass is 1cm thick, so shouldn't we be seeing more dispersion here with it enabled?

materials": [
    {
      "name": "KhronieAward_Clear_Glass",
      "extensions": {
        "KHR_materials_transmission": {
          "transmissionFactor": 1.0
        },
        "KHR_materials_volume": {
          "thicknessFactor": 0.01
        },
        "KHR_materials_dispersion": {
          "dispersion": 0.33
        }
      },
      "pbrMetallicRoughness": {
        "metallicFactor": 0.0,
        "roughnessFactor": 0.0
      }
    },

Use Thickness as Depth = Enabled

Use Thickness as Depth = Disabled

Photo of the real object.

[echadwick-artist]

FYI, submitted the KhronieAward asset to the glTF Sample Assets repo: KhronosGroup/glTF-Sample-Assets#99

[Popov72]

When the "use thickness as depth" option is ticked, the effect is not visible because the thickness used is too low: this is the "Max thickness value", which is 0.01 by default. If you increase this value/increase the intensity of dispersion, you should start to see something.

[emackey]

@Popov72 Thanks. Actually we had some interesting discussion of this effect on this week's PBR call, and I've investigated a little bit further since then. My current thoughts are:

• The thickness of 0.01 (meters) represents a real-world thickness of 1 cm, which is about as thick as the glass really is. The wording is etched into the back side of the glass, so when viewing it from the front, the words sit at exactly the thickness depth, no more and no less. Turning on "use thickness as depth" makes the words (but not the background) behave almost exactly as the real thing, because the assumption of depth is correct for the words alone. The background, however, is much further away than 1cm, and makes the glass look too thin and incorporeal.

• Turning off "use thickness as depth" improves the background a lot! The glass distorts the background substantially, and looks like the real object. But the words "swim around" and get clipped and so on. They're too deep now.

But now the bad news: There's no fix for only 1 opaque pass. Consider the following screenshot with "use thickness as depth" turned off:

The background isn't too exciting, but I want to call out a particular tan spot on the ground, behind the swirly letter O in the middle of the Khronos logo. This same tan spot on the ground can be seen in 3 different places: You can see it through the main piece of glass, you can see it again looking through the top bevel that angles your line of sight down to the spot behind the glass, and you can see the same tan spot a third time by looking through the left-edge bevel in the glass which angles your line of sight to the right and finds the same spot behind the glass.

So there are 3 lines of sight to the same background spot. But the single opaque-pass render has plastered the swirly letter O on top of the one spot, so now that logo is visible in all three locations.

Long story short, I don't think there's any easy fix for this using only a single opaque pass. But it's interesting to study where the limitations are for this type of rendering (and someday this may become moot when realtime raytracing is common).