Fix GLES3 changing 2D shadow atlas size is broken

[WhalesState]

Fixes #80148
Fixes #80186
Fixes #80396

[WhalesState]

it's already clamping the texture size to the max_texture_size allowed so this is just a friendly warning when you set a larger size

[WhalesState]

there are still 3 issues,
1- you are allowed to set a negative size in project settings which will spam errors to the compiler.
2- we need to apply a smooth texture filter to the texture if it has a small size
3- the editor is not updating the changes on running the scene and it doesn't respect the size in settings

[lawnjelly]

it's already clamping the texture size to the max_texture_size allowed so this is just a friendly warning when you set a larger size

Ah, I missed this I assumed it was doing a reject of the command. Clamping makes sense. However it does look like the check for no change is made above this, so could be moved below the clamping, maybe we could fix this at same time.

[WhalesState]

you are right, it's the same in vulkan too, let me edit both

Edit: there's no clamping in vulkan

[WhalesState]

please don't approve since this is not finished, i will try to track why it doesn't respect the new size on running the scene.

[lawnjelly]

please don't approve since this is not finished, i will try to track why it doesn't respect the new size on running the scene.

If you are still working on it, just convert to draft, there is a (hard to see 😀 ) button on github on the top right of the page in the reviewers section. Then reviewers know to hold off, and it also won't be merged by accident. Then click ready for review to move out of draft when finished.

[WhalesState]

this was helpful, now we need to put this line somewhere when the game starts to load settings from project settings.

RS::get_singleton()->canvas_set_shadow_texture_size(GLOBAL_GET("rendering/2d/shadow_atlas/size"));

it already exists in the editor _load_from_settings() so where to add it ? in the ready() of the game in main.cpp

[akien-mga]

Is this meant to fix #80148? If so, please mention it in the PR description, ideally with "Fixes #80148" which is interpreted by GitHub so it would close the issue if/when the PR merged.

[WhalesState]

@lawnjelly have a look on this

it seems like the shadow_atlas gets resized on gles3 then the shadow polygons are drawn over the small texture and rendered in high resolution on screen

this supposed to be a shadow atlas of size [32]

this is [128]

and this is [2048]

[WhalesState]

while in vulkan it looks like this

while both seems to have issues in resize but the GLES3 is much cleaner in drawing polygons over a small texture

[32]

[128]

[2048]

zooming in [2048]

using Camera2D to show actual pixel size on screen

[clayjohn]

Great work! I just left one suggestion, after fixing that one thing this should be ready to merge

[WhalesState]

@clayjohn Thank you, please feel free to review and edit the code since I have no experience with opengl, I also have found another issue on this file.

the default polygon color in opengl is black, and here it doesn't update the polygon color if it's white, means drawing with white color will not update the polygon color in opengl but any other color will do, this issue only appears when building without Vulkan and trying to draw any polygon with white color, like a line or a rect inside the draw function will appear black. can i fix this one too ?

it happens like, the first draw call is black when the color is white, the second draw call will be red when the color is red and will fix the first call because it will update the polygon color after this, also another draw call in white color may fix it as i remember since the color array will have more than one color and will not use those fixed lines as i understand, and also will fix the polygon color, so using only one draw_rect with white color will appear black if you didn't draw any other polygons in the same frame. is there's a better way to set the default polygon color to white when the gles3 renderer starts ?

like running this line once when it starts glVertexAttrib4f(RS::ARRAY_COLOR, 1.0, 1.0, 1.0, 1.0);
another option is to give the polygon buffer color a default white value in the .h file, then using glVertexAttrib4f(RS::ARRAY_COLOR, pb->color.r, pb->color.g, pb->color.b, pb->color.a)

[WhalesState]

edited the wrong line. lol!

sorry, got confused. it's the correct line.

thought it's inside free_polygon!

also the change in the .h file is not needed, i will test without it to see if the issue is fixed then will remove the default polygon buffer's color since it will be updated in every draw call.

Fixed, will remove the changes from the .h file when the checks are finished.

[clayjohn]

Looks good now! Thanks for working on this.

As a note for the future, we try really hard to keep separate changes in their own PRs. So, in the future, please open a new PR when you want to fix a separate bug. In this case I won't make you open a new PR because it's literally a two line change. But even for small changes like this it really is better to open a separate PR.

[WhalesState]

Got it, Thank you ^^
Should I keep the changes in the header file or remove them after the checks? since they make no difference to those fixes.

Anyway, it's ok to merge at this point since it can tell that we are using white as the default polygon color.

[clayjohn]

Should I keep the changes in the header file or remove them after the checks? since they make no difference to those fixes.

You can keep them. I prefer being explicit about initialization anyway.

[akien-mga]

Thanks!