Implement LightmapGI shadowmasks

[BlueCube3310]

Salvages #77284
Closes: godotengine/godot-proposals#2354

Adds support for baked shadowmasks to LightmapGI. The shadowmasks are saved as compressed images and applied on top of the dynamic shadowmaps.

Progress:

• Support baking shadowmasks,
• Support Forward+ renderer,
• Support Mobile renderer,
• Support Compatibility renderer,
• (Bugfix) Allow lightmaps without shadowmasks,
• Dilate,
• Fix blending artifacts, Edit: Caused by DXT1 Compression,
• Document everything properly,
• Clean up and optimize the code,

To consider:

• Consider using specialization constants,
• Consider blending seams.

To do in the future (important):

• Compress grayscale textures as RGTC_R, Edit: done
• Implement and use bicubic sampling (See Implement bicubic sampling for lightmaps #89919) Edit: done,
• Further optimizations and cleanup,
• Add a shadowmask_resolution_ratio property to scale shadowmasks independently from lightmaps.
• Remind the user when lighting needs to be rebuilt,

[AThousandShips]

Since this is salvaged from a previous PR, unless it doesn't contain any of their code, please add them as co-author

[BlueCube3310]

Did I do it correctly? Sorry, I'm still new to github.

[BlueCube3310]

I think it's ready for review now. I'm still not certain whether the documentation is clear enough, or if the seams are noticeable enough to require blending them, though.

[Calinou]

Filter the shadowmasks in the Lightmapper,

Reimplementing support for bicubic sampling could resolve this. If this doesn't suffice, we could add an option that uses Image::resize() to halve the shadowmask resolution, then use Image::resize() on the small image to bring it back to the original size with cubic resampling. This will effectively blur the shadowmask without requiring an expensive filter.

Add a shadowmask_resolution_ratio property to scale shadowmasks independently from lightmaps.

This is a clever idea I hadn't thought of before. It makes a lot of sense if you need sharp distant shadows but don't need indirect lighting data to be very precise 🙂

That said, in practice, you don't want distant shadows to be too precise as you run the risk of having aliasing visible in the distance (lightmaps and shadowmasks aren't mipmapped).

[Calinou]

Tested locally, it mostly works as expected (I tested Forward+ and Mobile).

Testing project: test_lightmapgi_mobile.zip

The way shadowmask interacts with real-time shadows looks strange. I don't recall this happening with previous PRs, but maybe I started noticing this issue just now. This is most noticeable with Directional Soft Shadow Filter Quality set to Ultra, but it occurs with any setting:

I'd expect the final shadow color to be min(shadowmask, real_time_shadowmap). Right now, it seems to be real_time_shadowmap > 0.0 ? real_time_shadowmap : shadowmask. This issue affects both Forward+ and Mobile.

If you set Directional Soft Shadow Filter Quality to Hard, notice how shadowed area transitions look much sharper than intended:

Regarding fixing seams, I'd say it's worth the effort as right now, seams can be quite noticeable in the distance:

Real-time shadow	Shadowmask

[BlueCube3310]

After merge LightmapGI Blocking bug Lightmap Probes are completely broken on both Mobile and Compatibility renderers, making LightmapGI unusable on the devices that would benefit most from it.

I believe this bug should be prioritized.

#99367 should fix the mobile version

[clayjohn]

We discussed this in the Rendering meeting today and the consensus is that we should move forward and merge this as soon as possible. However, we are a bit concerned about the current ShadowmaskMode options. They are a bit confusing and they seem to cater to very niche use-cases that may not be widely used.

• Replace seems to be what most people expect from this feature. It allows for having baked shadows at a distance and high quality shadows when up close when baking indirect lighting
• Overlay seems interesting, but problematic. It will break if you have any animated objects. I suspect that there will be few cases where users want to have the low resolution lightmap fully overlaid on dynamic shadows. I suspect that it would be better to have an option to blend between the shadowmask and shadow map at the edges of the shadow range instead
• Only appears to be an inferior version of using the "Static" lightmapping mode on Light3Ds. It forces using the shadowmask (which is good for performance), but still uses real time direct lighting, so it is worse for performance than static lights. Ultimately, it presents a clunky workflow to do something that is already supported by the engine. I suspect that this option was more useful before we implemented bicubic sampling because users were unhappy with the quality of baked shadows before.

Accordingly, we agreed that two things are needed before merging:

1. Un-expose the SHADOWMASK_MODE_ONLY enum. Its fine to leave the code in internally for now, but we don't want to expose it to users unless we need to.
2. Mark the shadowmask_mode as experimental. (We also need to call it out in the release notes as something that will likely change)

Personally, I suspect that we can get away with making the shadowmask a binary selection "SHADOW_MASK_ENABLED", "SHADOW_MASK_DISABLED" with a boolean that pops up when enabled to turn on/off blending. But I would like to get some more user feedback about what they need before we go down the rabbit hole of trying to decide on users problems for them.

[SpockBauru]

I believe that the Shadowmask Only will be beneficial when the option to increase shadowmask resolution is implemented, so we can have high frequency shadowmask and low frequency lightmap. But this feature can wait.

For now I fully agree with @clayjohn: we should un-expose the shadowmask modes, keep only the binary selection and merge as soon as possible. Just keep the code for future improvements.

[BlueCube3310]

Done, this should now work as suggested

[BlueCube3310]

It looks like now a line appears where the shadow splits should end (when using blended shadows), but that also happens on the master branch and is unrelated to this PR

[SpockBauru]

Tested my MRP on Forward+, Mobile and Compatibility renderers. Shadowmasks is working as intended without any visual issues. Baked the LightmapGI on each renderer and everything is working.

Also made a quick test on my (halted) personal project. Out of the box, it works as expected without any issues. Far shadows are now present:

Shadowmaps None	Shadowmaps Replace

With this developers can make an option to reduce shadowmaps resolution and distance, improving the performance on weak systems.

I didn't find any issue related to this PR and believe that is ready to merge.

[clayjohn]

Let's go ahead and merge this!

It looks great and while I think there is room for iteration, I think it's time to get this in the hands of users so we can get some feedback

[akien-mga]

Thanks!

[Calinou]

@BlueCube3310 I noticed shadowmasks are saved as PNGs, but saving them as lossless WebPs can make them much smaller on disk. Example with a 4096×4096 shadowmask texture:

321k test_shadow.png
151k test_shadow.webp

This does not affect memory usage, but it'll reduce file sizes in version control, which is helpful as lightmap data is often iterated upon. In the codebase, it should be as simple as replacing save_png() to save_webp() (which is lossless by default).

The downside is that we can only do that for images that are 16383×16383 or smaller, so shadowmasks that are larger would have to be resized down. In practice, only 16384×16384 can be an issue here¹, so we could downsize it to 16383×16383 upon saving (Godot's VRAM compression will pad the last pixel automatically).

Footnotes

1. The maximum lightmap image size that can be saved by Godot is 16384×16384. ↩

[michaelharmonart]

Not sure if this is the place to mention it, but after some testing it seems that this doesn't take into account whether shadow casting is on per mesh. Would be nice to have that supported as well to be able to disable shadow casters for baking as well.

[clayjohn]

Not sure if this is the place to mention it, but after some testing it seems that this doesn't take into account whether shadow casting is on per mesh. Would be nice to have that supported as well to be able to disable shadow casters for baking as well.

That's an existing issue in LightmapGI. #98684

[michaelharmonart]

ahh ok. l never noticed since there was never a way to blend between real-time and baked lighting like this before now

I noticed now only since there was a clear transition line between realtime and baked shadows only since there was a mismatch in which objects had shadows.

[IsaacMarovitz]

In 4.4.dev7 despite having shadowmask mode set to Replace, baking does not seem to build the shadowmasks.

I can't tell if I'm using it wrong, or if this is a bug.

[SpockBauru]

baking does not seem to build the shadowmasks.

@IsaacMarovitz It's working here, did you remembered to set shadowmasks to replace before baking?

[IsaacMarovitz]

Nvm, I was using it wrong, my directional light was set to static and I missed the warn in the console. The DirectionalLight Bake Mode documentation needs to be updated to reflect this change.

[unfa]

Hey, where can I find a good explanation of this feature? I was looking around and couldn't really find anything.

[SpockBauru]

Officially there's an explanation on the "latest" version of the docs, not the "stable" one since 4.4 stable was not released yet. Be aware that it's an experimental feature so there's no much explanation yet:

• https://docs.godotengine.org/en/latest/classes/class_lightmapgi.html#class-lightmapgi-property-shadowmask-mode
• https://docs.godotengine.org/en/latest/classes/class_lightmapgidata.html#enum-lightmapgidata-shadowmaskmode

I'm also answering people on this Reddit thread: https://www.reddit.com/r/godot/comments/1hcr73g