Add some editor startup benchmarks

[OverloadedOrama]

My attempt to add the following benchmarks from #36:

• 🟦Startup🟦 [Editor startup]: with no shader cache
• 🟦Startup🟦 [Editor startup]: with shaders cached

I've only added these two benchmarks for starters so we can decide how we want to handle editor startup benchmarking.

The way I have implemented it, is that the benchmarks first create a project.godot file in user://tmp, and then call the Godot editor to open the project with OS.execute() with the --quit parameter, and the benchmark simply measures the time it takes for the editor to open and then close. For the shaders cached benchmark, the benchmark is copying all of the cached shader files from the godot-benchmark's .godot directory itself into the temp project folder, and then deletes all the files and directories inside user://tmp. An issue with this is approach is that the time that it takes to copy and then delete the files is included in the benchmark, and I am worried this may be worse with benchmarks that have to do with multiple files, as we have to copy (or generate) them on the fly, and then delete them.

Another approach would be to have smaller projects inside supplemental/projects where everything is already set up and we don't need to do any copying and deleting, but this would raise the size of the entire benchmarks project as we'd have multiple .godot folders for each small project, and I'm not sure if we can make it work with git, as it ignores .godot folders.

I also assume that this wouldn't work on an exported project, because it's using OS.get_executable_path() to get the Godot path. Is this an issue? If it is, we'd need a way for the user to input the Godot path themselves.

Results on my PC

{
"benchmarks": [
{
"category": "Startup > Editor",
"name": "Start Editor No Shader Cache",
"results": {
"time": 7968
}
},
{
"category": "Startup > Editor",
"name": "Start Editor Shader Cache",
"results": {
"time": 6357
}
}
],
"engine": {
"version": "v4.3.beta2.official",
"version_hash": "b75f0485ba15951b87f1d9a2d8dd0fcd55e178e4"
},
"system": {
"cpu_architecture": "x86_64",
"cpu_count": 12,
"cpu_name": "AMD Ryzen 5 1600 Six-Core Processor",
"os": "Linux"
}
}

[Calinou]

I'm not sure if this should be added a standard benchmark. It may be better done as a special benchmark that is implemented in run-benchmarks.sh, especially since it has special run conditions:

• It should only be done in debug mode (editor build), not release mode which uses a release export template (and therefore lacks editor functionality).
• It should be run 10 times in a row to reduce variance. The average value is retained.

I suggest opening the same empty project that is created for benchmarking project startup/shutdown time, so that the number of files in the editor does not affect performance. If we use the main benchmarks project for that, performance would slow down as the project size increases, which could be misleading. Tracking performance in a project with 10,000+ files could be worth tracking separately, but it needs to remain constant over time.

We can do both cold and warm shader cache tests this way, but I'd focus on warm shader cache tests first as they're the scenario most commonly encountered by users.

It would then become an extra line on the results page's header:

It could also be graphed independently on the homepage like other statistics (such as build time and binary size), although this hasn't been implemented yet.

[OverloadedOrama]

In that case, if I'm understanding run-benchmarks.sh correctly, the warm shader cache benchmark would be identical to the debug startup + shutdown time, right? And the cold shader cache would also be similar, but it would also remove the .godot/shader_cache directory before running the benchmark?

To be honest, I am unsure how to test benchmarks that are in run-benchmarks.sh. It seems hardcoded to be used for the benchmarking server. This makes it hard both for me to create and test them, but also for users who may want to run them in their local devices and compare results. Perhaps we could have another bash script that only contains benchmarks and is device-agnostic and thus can be easily used by everyone, and run-benchmarks.sh could also call?

Alternatively, I also feel like the special run conditions can also be mitigated when running them inside the Godot Editor:

• We could use OS.has_feature("debug") and let these benchmarks run only if that's true.
• We could refactor the benchmark running system to allow for individual benchmarks to return their own times, as integers. That way, we could run the benchmark 10 times using a for loop, calculate the time it takes inside the benchmark method itself, and return the average. Benchmarks returning void or a Node type could have their time calculated on manager.gd as it happens now, and benchmarks returning int could calculate their own time and just return that to manager.gd. That way we could also exclude the time the benchmark takes to move/copy/delete/etc the files, and just calculate the startup time on its own.

[Calinou]

Perhaps we could have another bash script that only contains benchmarks and is device-agnostic and thus can be easily used by everyone, and run-benchmarks.sh could also call?

This can be done, but it's a lot of added complexity for a few benchmarks that I think not a lot of people will run locally anyway. The compilation benchmarks in particular can be difficult to set up as they require following distribution-specific instructions for installing dependencies to compile Godot.

[OverloadedOrama]

All right, I made an attempt to add the benchmarks to run-benchmarks.sh instead. I made a temporary script so I could test that the benchmarks work and they do indeed work on my end, hopefully they work on run-benchmarks.sh too. Apologies it took so long, it has been a very busy month for me. 😅

Results on my PC:

"empty_project_editor_startup_shader_cache": {
	"debug": 6390,
},
"empty_project_editor_startup_no_shader_cache": {
	"debug": 8000,
}

[Calinou]

Thanks!