Adopting a Progressive Photorealistic Global Illumination in Three.JS

[bhouston]

There has recently been a bunch of work towards progressive photo-realistic global illumination in WebGL. The best example I've seen of this is https://cl3ver.com renderer. There is some overlap between this and the high end rendering that Octane and other photorealistic GPU renderers do. I think it is time to add this to Three.JS.

This is a meta-task that describes what needs to be done (this is in the spirit of the PBR meta task from 2015: #5847)

The main components of this approach would be:

Temporal Reprojection Anti-Aliasing
#14050

Progressive Global Illumination via Instant Radiosity/Virtual Point Lights
#14047

Progressive Soft Shadows via Light Source Sampling
#14048

Screen-space Reflections
#8248

[gkjohnson]

Activision's Ground Truth Ambient Occlusion and Unigine's SSRTGI both look really impressive and would be great to have available.

Is the assumption that these would come after something like multiple render textures or deferred rendering is available in THREE?

[donmccurdy]

I'm surprised not to see your previous suggestion for light probes, #6251, in this list. Do you see these as complementary? The features here appear to me (without much background) more appropriate for viewing static models photo-realistically, whereas light probes are more of a GI approximation for dynamic objects in realtime scenes?

[bhouston]

Real-time light probes can be either diffuse or specular. Diffuse light probes are made obsolete by instant radiosity. Specular light probes are made obsolete via SSR.

That said, this solution is for progressive and very high quality result. If you want an instantaneous and a bit lower quality solution you should use light probes.

[EliasHasle]

How about this? https://github.com/erichlof/THREE.js-PathTracing-Renderer (Demos: https://erichlof.github.io/THREE.js-PathTracing-Renderer/)

[elfrank]

Hi,

At HOVER, we recently open-sourced our Ray Tracing Renderer. It's a drop-in replacement for the WebGLRenderer in three.js.

Links

• Repository
• Demo: WebGLRenderer vs RayTracingRenderer

Note

• You need WebGL 2.

[EliasHasle]

@elfrank Nice demo! It takes a while to get a nicer render with the raytracer than with the WebGLRenderer, though. Also, the WebGLRenderer is initialized without antialiasing, which would significantly improve the result.

[mrdoob]

@elfrank That looks great!

Things render pretty glitchy on ChromeOS/Pixelbook though 😶

Kinda cool though! 😁

[looeee]

That said, this solution is for progressive and very high quality result.

Does progressive in this context mean something like what sketchfab does? It takes maybe 0.5 seconds or less there to see the high quality result after moving the camera. That's something I'd love to see made possible in three.js.

Or is more like the raytracer (or maybe a bit faster) where it takes several seconds to see a good result even on fast hardware?

[mrdoob]

Does progressive in this context mean something like what sketchfab does? It takes maybe 0.5 seconds or less there to see the high quality result after moving the camera.

That's what this example does (minus the improved shadows):
https://threejs.org/examples/#webgl_postprocessing_taa

[looeee]

I think sketchfab does more than just disable AA while the camera is moving though. It seems like the reduce the resolution and possibly reduce texture quality too, and perhaps some other things that are less obvious (maybe the use progressive shadows and lighting like @bhouston is suggesting here).

I tried a quick tests to reduce resolution while OrbitControls is engaged but I couldn't get decent results or any noticeable performance gain, even in a fill rate limited scene.

[gkjohnson]

I think sketchfab does more than just disable AA while the camera is moving though.

Some of this has already been mentioned but here's what I noticed with sketchfabs renderer if it's worth noting:

It's most pronounced in this bacteria model but you can see that it renders transparency using some kind of noise distribution that then resolves to a smooth look. Presumably this is to help alleviate artifacts associated with transparent object render order.

It also looks like some kind of sharpen filter gets applied to the image before the accumulation kicks in but I'm not sure why. It's pretty easy to see in this car model when you zoom in and out (notice the light outlines around the dark edges of the model). Maybe someone else has an idea as to what this is for?

As was mentioned shadow map camera is subpixel jittered (much like the TAA approach, I think) to smooth shadow resolution artifacts. This can be seen pretty easily in this cyborg boy model if you pause the animation and zoom into the shirt on the right arm.

Of course there's some really nice high quality post processing going on, too.

I'm curious to what the plan should be for adding these types features into the library. As far as I'm concerned it's already possible to achieve all of this but maybe it's worth adding some facilities to make it easier to achieve? And an example that glues all the necessary pieces together?

[bhouston]

I've analyzed Sketchfab. It has a series of options that you can turn on and off so exactly what it does on a specific scene is dependent upon that. But it has these capabilities:

• Sketchfab supports a series of optional and parameter post processing filters like sharpen, color correct, vignetting, sepia, chromatic aberration, etc.

• TRAA - temporal reproduction anti-aliasing. It uses a velocity buffer of the scene to allow one to to TAA (https://threejs.org/examples/#webgl_postprocessing_taa) like effects but with a moving camera or deforming object. I have code for this and I want our team here (ThreeKit) to contribute it back to Three.js, just it is lower priority than contributing back enhanced PBR materials (e.g. Intent to implement/contribute Enterprise PBR improvements #16977 )

• Shadow camera jitter when the scene is static. This smoothes out the shadows. One should jitter the shadow cameras within the light volume for correct soft shadows. When moving, one should position the shadow camera at the center of the light volume or at least the functional equivalent.

• Stochastic transparency. This is what we see in the bacteria model I believe: http://luebke.us/publications/StochTransp-slides.pdf I think Stochastic transparency can resolve to a good solution, but it looks like crap in dynamic scenes. I am unsure of its value.

(An aside: Alternatively, we (Threekit) have implemented order independent transparency based on the McGuire model, http://casual-effects.blogspot.com/2014/03/weighted-blended-order-independent.html But it is also highly problematic - it is dependent upon a lot of parameter tweaking and it is slow and needs floating point buffers. I would recommend against implement this algorithm in Three.JS as well.

I am unsure of the best approach to OIT - both Stochastic and the McGuire OIT method have significant downsides.)

A suggested plan forward:

• I think we should add shadow camera jitter within the shadow emission volumes. It should be something one can turn on and then on each frame it jitters within that region. Then users can enable it during a TAA refinement phase. It works exceptionally well with the TAA post effect I wrote.

• We should add TRAA and we can contribute that code. Maybe we can contribute broken code ripped out of our custom Three.JS implementation and a hero can adapt it to the mainstream Three.JS? (Interesting fact, you can not easily combine jitter shadow cameras with TRAA because TRAA uses a luminance discriminator to determine if things will blend across frames, and well shadow boundaries have significant luminance contrast -- but maybe someone can figure out a solution to this.)

Some people have combined TRAA with stochastic transparency. https://casual-effects.com/research/Wyman2017Improved/Wyman2017Improved.pdf https://developer.download.nvidia.com/assets/gameworks/downloads/regular/GDC17/RealTimeRenderingAdvances_HashedAlphaTesting_GDC2017_FINAL.pdf Maybe the game INSIDE used this approach -- one slide says stochastic everywhere: http://twvideo01.ubm-us.net/o1/vault/gdc2016/Presentations/Pedersen_LasseJonFuglsang_TemporalReprojectionAntiAliasing.pdf

[elfrank]

@elfrank Nice demo! It takes a while to get a nicer render with the raytracer than with the WebGLRenderer, though.

Yes, we use path tracing, which could be broadly bucketed as a "progressive photorealistic Global Illumination" technique. We are looking into faster convergence. Path tracing is expensive but there is a good amount of recent research on the matter.

Also, the WebGLRenderer is initialized without antialiasing, which would significantly improve the result.

@EliasHasle Noted! Thanks.

Things render pretty glitchy on ChromeOS/Pixelbook though 😶

That's unfortunate @mrdoob . We have experienced a few driver bugs/glitches in some devices. It's nearly possible to account for all scenarios without having access to the hardware. With that said, Pull requests are welcomed! Thoughts @jaxry ?

Or is more like the raytracer (or maybe a bit faster) where it takes several seconds to see a good result even on fast hardware?

@looeee I highly recommend this video on Path Tracing: https://www.youtube.com/watch?v=frLwRLS_ZR0&t=11s
As you pointed out, the technique is inherently expensive. Hardware keeps getting faster and that helps a lot.

[zalo]

I've been implementing Approximate Radiosity using Stochastic Depth Buffering, which is a version of the recent sampled Shadows PR that tunnels through surfaces to cast rays throughout the scene. This should enable reasonably fast, physically correct global illumination and area lighting (without path tracing or acceleration structures).

Here is an example scene that sort of demonstrates basic colored reflections and area lighting:
https://zalo.github.io/three.js/examples/#webgl_shadowmap_progressive

[mrdoob]

@zalo that runs pretty well on my cheap Android phone 😮

[EliasHasle]

My cheap Android phone shows it like this (with FPS eyeballed to ~10):

Do you have a reference screenshot for how it should look?

[zalo]

Took this on a Pixel 4 (the banding isn’t visible on Desktops with full float precision buffers... though I’d like to fix that with additional dithering...). @EliasHasle is there any chance you could check the console for errors and report them to me? As far as I know, the most exotic things I’m doing are float buffers and writing depth in the fragment shader...

If the latter is the issue... it may be possible to jitter depth in vertex space instead...

Performance-wise, the nice thing about this effect is that it can be turned off after a few seconds (once it’s converged) and it’s basically free from then on (as a “baked lightmap”).

EDIT:

I just added a check for the frag depth breakage (which was the issue on iOS). This looks alright in this scene, but will appear more and more incorrect as the scene complexity increases. Unfortunately I can’t think of a good alternative off the top of my head.... only bad alternatives 🙃

[mrdoob]

@zalo That looks good to me. Should that become another example?

[bhouston]

Looks broken on Safari on M1 in WebGL2 mode.

[mrdoob]

@bhouston Could you try if it's also broken on Chrome on M1?

[bhouston]

It works fine in Chrome on M1:

[mrdoob]

@bhouston Thanks for testing. Would you like to report the issue to Apple?

[gkjohnson]

This looks great -- here's an artifact I see in Chrome on 2017 Mac with a Radeon Pro 560 that I don't see in @bhouston's screenshot (notice the hard edge in the lighting):

[carstenschwede]

Results for Macbook Pro 2020, Radeon 5600M. Hard edge in Chrome and Firefox, no edge in Safari.
There also more artefacts visible on the inside in comparison to @bhouston screenshot on M1.

[zalo]

That artifact is very interesting; I think Radeon is the common element there. My wildest guess is that negative orthographic near clipping planes aren't supported on Radeon graphics cards on Mac...?

Safari was acting broken on my Mid-2015 Intel Iris Macbook, but Chrome works flawlessly. It turns out that is because renderer.extensions.has( 'EXT_frag_depth' ) returns true on Mac Safari, but it's not true! It's a lie! It also runs much more slowly than Chrome, so I'm additionally downgrading Safari to mobile settings.

Do any of those changes help on Radeon Chrome/Firefox, @gkjohnson and @carstenschwede ? If not, could you also hit the "Debug Lightmap" setting to see what that mysterious line looks like there? Perhaps also try moving the lighting/object around to see if the line moves? Updated Here: https://zalo.github.io/three.js/examples/#webgl_shadowmap_progressive

With regards to a new example, I'd really like to get the code cleaned up, add more configurability, add more bounces, and increase the complexity of the example scene/("unit test") to make this a suitable addition to three.js. The radiosity component currently triples the surface area of the lightmapping script to ~700 lines, so I'd like to par that down if possible...

[carstenschwede]

Looking more closely now, there are actually different edges in Safari and Chrome (Radeon 5600M, macOS).

Chrome:

Safari:

Edge does not move with lights/objects and is not present in early iterations.

[carstenschwede]

Overriding software rendering list and enabling draft extensions in Chrome/macOS/Radeon also has no effect on the edge.

[carstenschwede]

Here is a montage of how the edge appears in later iterations (Chrome/MacOS/Radeon)

[bhouston]

If this is in an example folder, rather than being a core feature, I wouldn't require it to be absolutely perfect on all devices in the first PR, because that could stall it from getting in.

[carstenschwede]

The light target is currently on top of the object. Would it make sense to add a corresponding helper to indicate the orientation of the DirectionalLight (or give the target its own control)? I assumed the control was for the origin of a PointLight and was irritated when the shadows were pointing towards it.

Also Safari appears to converge much slower which is why I think the edge is less apparent there. Increasing the LightMap resolution helps with the edge on the green plane, although they are still a few artefacts visible in the red one. Nevertheless a super impressive result - in terms of quality and convergence time!

[EliasHasle]

@EliasHasle is there any chance you could check the console for errors and report them to me? As far as I know, the most exotic things I’m doing are float buffers and writing depth in the fragment shader...

Beginning of log

MESSAGE: Safari: false

LEVEL: LOG TIMESTAMP: 14:28:37

LINE: 40 IN: https://zalo.github.io/three.js/examples/jsm/misc/ProgressiveLightMap.js

MESSAGE: WebGL2: true

LEVEL: LOG TIMESTAMP: 14:28:37

LINE: 41 IN: https://zalo.github.io/three.js/examples/jsm/misc/ProgressiveLightMap.js

MESSAGE: FragDepth: false

LEVEL: LOG TIMESTAMP: 14:28:37

LINE: 42 IN: https://zalo.github.io/three.js/examples/jsm/misc/ProgressiveLightMap.js

MESSAGE: THREE.WebGLRenderer: EXT_color_buffer_float extension not supported.

LEVEL: WARNING TIMESTAMP: 14:28:41

LINE: 14820 IN: https://zalo.github.io/three.js/build/three.module.js

MESSAGE: THREE.WebGLRenderer: EXT_color_buffer_float extension not supported.

LEVEL: WARNING TIMESTAMP: 14:28:44

LINE: 14820 IN: https://zalo.github.io/three.js/build/three.module.js

MESSAGE: THREE.WebGLRenderer: EXT_color_buffer_float extension not supported.

LEVEL: WARNING TIMESTAMP: 14:28:46

LINE: 14820 IN: https://zalo.github.io/three.js/build/three.module.js

MESSAGE: THREE.WebGLRenderer: EXT_color_buffer_float extension not supported.

LEVEL: WARNING TIMESTAMP: 14:28:47

LINE: 14820 IN: https://zalo.github.io/three.js/build/three.module.js

MESSAGE: [.WebGL-0x7e3ee3b400]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glClear: framebuffer incomplete

LEVEL: WARNING TIMESTAMP: 14:28:47

LINE: 0 IN: https://zalo.github.io/three.js/examples/webgl_shadowmap_progressive.html

MESSAGE: [.WebGL-0x7e3ee3b400]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawElements: framebuffer incomplete

LEVEL: WARNING TIMESTAMP: 14:28:47

LINE: 0 IN: https://zalo.github.io/three.js/examples/webgl_shadowmap_progressive.html

MESSAGE: [.WebGL-0x7e3ee3b400]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : GetShaderiv:

LEVEL: WARNING TIMESTAMP: 14:28:47

LINE: 0 IN: https://zalo.github.io/three.js/examples/webgl_shadowmap_progressive.html

MESSAGE: [.WebGL-0x7e3ee3b400]GL ERROR :GL_INVALID_FRAMEBUFFER_OPERATION : glDrawElements: framebuffer incomplete

So yeah, it looks like my phone, an old Sony Xperia, fails to support EXT_color_buffer_float. That's not to blame on you! 😄

[arturitu]

@zalo if I export a simple mesh from Blender with a UV map it doesn't work as expected. It is all black. Do you know if it is needed to do something special with the UVs before export it? (I tried with importing other gltf files included on the examples folder like 'Parrot.glb' or 'Horse.glb' and it happens too) (Surely I'm missing something)

[zalo]

@arturitu This script uses the uv2 channel (which is the idiomatic lightmapping channel?) by default.

It may make sense to copy the UVs from uv to uv2 if uv2 doesn't exist, but the uv2 channel has extra padding requirements in my naive implementation...

[arturitu]

It may make sense to copy the UVs from uv to uv2 if uv2 doesn't exist, but the uv2 channel has extra padding requirements in my naive implementation...

Yes, I saw that and I think it makes sense because three.js I think only support 2 sets of UVs, but If I export from Blender a simple Icosphere (or any other mesh) with this UVs ...

... it looks like this:

If I export the same mesh with this other UVs ...

... it looks better but (I think) not all faces are properly calculated:

Here the blend and the .glb files used with your code: test-UVs.zip

BUT, if you add directly on you webgl_shadowmap_progressive a Mesh with a THREE.Icosahedron geometry it works as expected. I replicated it using the threejs editor, exporting the glb and seeing in blender what UVs it generated:

... and now, it looks as expected:

And then I tried to unwrap like this on Blender using seams but it doesn't work as well. I suspect that it may be related to the order of the vertices of the geometry, that maybe your code expects them in a certain order and with other glbs the order in the UVs is more complex.

My goal is to find if is possible to use any kind of generated meshes with a properly UVs to be used with this global illumination method.

[mishagrin]

@arturitu
Any progress on this?

[arturitu]

@arturitu Any progress on this?

Nope. I did not go any further along this line because I did not get a workflow that I could control for the project.

[ToJans]

FYI this might also be a viable (intermediate? good enough?) route: https://github.com/0beqz/screen-space-reflections

[zalo]

Wow; that looks great!

And any screen-space technique is going to be a lot less invasive than a lightmapping-space technique :-)

I'm a big fan of what people have been able to accomplish with velocity and reprojection buffers.
It's even possible that the light+shadow jittering method could benefit from working in screenspace that way.

[PetterGs]

If anyone is curious how to use procedural geometry to make an atlas for progressive lightmaps. I used xatlas-three to unwrap and atlas-pack the uv2. The trick is to set fixWinding to true and rotateCharts to false in the xatlas options.

Here is a TypeScript example:

import { UVUnwrapper } from "xatlas-three"
const unwrapper = new UVUnwrapper({ BufferAttribute: BufferAttribute });

enum ProgressCategory {
    AddMesh,
    ComputeCharts,
    PackCharts,
    BuildOutputMeshes
};

const logProgress = true;

export const loadXAtlasThree = async () => {
    if (_loaded)
        return;

    const onProgress = (mode: number, progress: number) => {
        logProgress && console.log(`UV unwrap state: ${ProgressCategory[mode]} ${progress}%`);
    }
    await unwrapper.loadLibrary(
        onProgress,
        'https://cdn.jsdelivr.net/npm/xatlasjs@0.1.0/dist/xatlas.wasm',
        'https://cdn.jsdelivr.net/npm/xatlasjs@0.1.0/dist/xatlas.js',
    );
    _loaded = true;
};

export const unwrapUv2 = async (geometries: BufferGeometry[]) => {
    await unwrapper.packAtlas(geometries, 'uv2' as any);
    return geometries;
}

unwrapper.chartOptions = {
    fixWinding: true,
    maxBoundaryLength: 0,
    maxChartArea: 0,
    maxCost: 2,
    maxIterations: 1,
    normalDeviationWeight: 2,
    normalSeamWeight: 4,
    roundnessWeight: 0.009999999776482582,
    straightnessWeight: 6,
    textureSeamWeight: 0.5,
    useInputMeshUvs: false,
}

unwrapper.packOptions = {
    bilinear: true,
    blockAlign: false,
    bruteForce: false,
    createImage: false,
    maxChartSize: 0,
    padding: 0,
    resolution: 0,
    rotateCharts: false,
    rotateChartsToAxis: true,
    texelsPerUnit: 0
}