BakedLightmap CPU lightmapper crashes on large scenes

[NHodgesVFX]

Godot version:

Godot 3.2.4 Beta 6 Compiled with MSVC

OS/device including version:

GLES3 RTX2080s Windows 10 64bit

Issue description:

While attempting to bake the Godot tps demo, It often crashes at the fixing seems and denoising or the capture step. I can stop the first crash by setting the quality level to low or disabling denoising. The 2nd crash is fixed by turning off capture or setting the cell size to a large value like 25.

I remember reading the godot blog about how the light octree used in the capture setting is inefficient and that light probes would be better for larger levels and reduz confirmed this. The primary issue is the first crash though.

Steps to reproduce:

1. Download the tps demo from master branch https://github.com/godotengine/tps-demo
2. Open it in godot and go into the level->geometry->models folder and find all the gltf files. Reimport them with lightmap baking enabled.
3. open level.tscn, delete gi probe and setup a bakedlightmap node so it fits the scene. Enable environment set to scene. set bounces to 5.
4. try baking it will most likely crash on the fixing seems and denoising.
5. turn off denoising and set the bake quality to low, you should now be able to get past the first crash.
6. Observe it now crashes at the capture step, raising the cell size to 25 fixes it.

Minimal reproduction project:

This bug only shows it self on large projects like the tps demo. Most likely due to the scale and amount of meshes to bake. I managed to get it on a test scene with kitbash3d assets as well but cant share due to licensing.

I can link to a zip with the tps demo setup but its large and the changes which I made to the scene are listed above.

[qarmin]

Backtrace:

ERROR: operator[]: FATAL: Index p_index = 4 is out of bounds (count = 4).
   At: ./core/local_vector.h:163.
handle_crash: Program crashed with signal 4
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x46210) [0x7fedab9c4210] (??:0)
[2] LocalVector<int, unsigned int, false>::operator[](unsigned int) (/mnt/Miecz/godot3.2/./core/local_vector.h:163 (discriminator 1))
[3] LightmapperCPU::_compute_indirect_light(unsigned int, void*) (/mnt/Miecz/godot3.2/modules/lightmapper_cpu/lightmapper_cpu.cpp:859)
[4] LightmapperCPU::_thread_func_wrapper(unsigned int, LightmapperCPU::ThreadData*) (/mnt/Miecz/godot3.2/modules/lightmapper_cpu/lightmapper_cpu.cpp:222 (discriminator 4))
[5] ThreadArrayProcessData<LightmapperCPU, LightmapperCPU::ThreadData*>::process(unsigned int) (/mnt/Miecz/godot3.2/./core/os/threaded_array_processor.h:50 (discriminator 4))
[6] void thread_process_array<LightmapperCPU, void (LightmapperCPU::*)(unsigned int, LightmapperCPU::ThreadData*), LightmapperCPU::ThreadData*>(unsigned int, LightmapperCPU*, void (LightmapperCPU::*)(unsigned int, LightmapperCPU::ThreadData*), LightmapperCPU::ThreadData*) (/mnt/Miecz/godot3.2/./core/os/threaded_array_processor.h:78)
[7] LightmapperCPU::_thread_func_callback(void*) (/mnt/Miecz/godot3.2/modules/lightmapper_cpu/lightmapper_cpu.cpp:212)
[8] ThreadPosix::thread_callback(void*) (/mnt/Miecz/godot3.2/drivers/unix/thread_posix.cpp:76)
[9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fedabecc609] (??:0)
[10] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fedabaa0293] (??:0)

To speed up importing this project https://crates.io/crates/godot_image_replacer can be used which resize all images to 1x1 and now project imports in 1 instead 30 minutes.

[JFonS]

I think the OP issue and @qarmin's stack trace are separate issues. I have found a fix for @qarmin's crash, but I didn't manage to reproduce OP's crash.

I do remember having some high RAM usage in some scenes, resulting in the OS killing the execution.
@NHodgesVFX Do you think that could be the case? Do you have any errors or stack trace in the output right before the crash?

[NHodgesVFX]

I think the OP issue and @qarmin's stack trace are separate issues. I have found a fix for @qarmin's crash, but I didn't manage to reproduce OP's crash.

I do remember having some high RAM usage in some scenes, resulting in the OS killing the execution.
@NHodgesVFX Do you think that could be the case? Do you have any errors or stack trace in the output right before the crash?

It spams LightmapDenoiser: Out Of memory and printed ERROR: Memory :: alloc_static condition !mem is true. Returned: 0 at core/os/memory.cpp:84 according to task manager my pc still has like 10gb of memory, not sure how accurate that is though.

[JFonS]

Something weird is going on in OpenImageDenoise then. I will see if upgrading it to the latest version gets rid of the issue.

[NHodgesVFX]

Thought I would mention this also shows in the build compiled by gcc by akien.

[akien-mga]

Is this still reproducible in Godot 3.4 RC 1 or later?

[Qrt42]

I encountered this problem in version 3.4 rc 2. Changing capture_cell_size from 0.5 to 1.0 fixed it

[Calinou]

Related to #54679.

@NHodgesVFX Can you reproduce the denoising crash on 3.5beta2, which includes #58102?

[NHodgesVFX]

@Calinou I tested it again using 3.5 beta 2 and light map generation with capture off did not crash but the capture generation at 0.5 still crashes.

[gdlq]

@NHodgesVFX did you get multiple baked atlases? No matter what I do it never works as intended, always single atlas with insaine resolution.

[NHodgesVFX]

Nope for me it creates a single atlas with the size of 256 x 1024, which is less than the max atlas size of 4096, so that seems to be working for me. Although I am surprised the file has such a small resolution for the TPS demo.