Consolidate benchmarks and use functions rather than scenes as the base unit #22

myaaaaaaaaa · 2022-12-07T22:12:21Z

GDScripts are now used as categories rather than directories, and functions are now the basic benchmarking unit rather than scenes. This allows nearly 40 files to be consolidated into 3 GDScripts, and makes it much easier to write new benchmarks since contributors only need to add a single script.

On startup, the loader now scans benchmarks/ for GDScripts, opens them up, and scans those for functions that begin with benchmark_ for automatic registration.

Instead of the time_limit mechanism, the benchmark_() functions return a Node to be added to the scene, which if null will signal to the manager that there is no time limit. This is admittedly a bit hacky, but it was the easiest way to do this consolidation while minimizing the amount of refactoring needed.

myaaaaaaaaa · 2022-12-19T17:36:55Z

Hello, are there any questions or concerns with this proposed format that I should address before it can be merged?

Calinou · 2022-12-20T19:36:53Z

Hello, are there any questions or concerns with this proposed format that I should address before it can be merged?

I've been busy lately and haven't had time to review pull requests. Holiday season is also starting, so it will take a few weeks until I can get to this and give it enough time for a proper review.

Rest assured, your work is very appreciated 🙂

myaaaaaaaaa · 2022-12-20T23:40:48Z

I've been busy lately and haven't had time to review pull requests. Holiday season is also starting, so it will take a few weeks until I can get to this and give it enough time for a proper review.

Okay, thank you for the update. Happy holidays!

…unit

Calinou · 2023-03-24T20:57:29Z

Benchmark results:

`main`

{
	"benchmarks": [
		{
			"category": "Gdscript",
			"name": "Packed String Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 1653
			}
		},
		{
			"category": "Gdscript",
			"name": "Typed Int Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 2538
			}
		},
		{
			"category": "Gdscript",
			"name": "Typed String Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 3652
			}
		},
		{
			"category": "Gdscript",
			"name": "Untyped Int Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 2565
			}
		},
		{
			"category": "Gdscript",
			"name": "Untyped String Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 3648
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Basic Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.56,
				"render_gpu": 0.57,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Directional Light Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.87,
				"render_gpu": 0.75,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Dynamic Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 3.12,
				"render_gpu": 0.55,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Dynamic Light Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.73,
				"render_gpu": 0.61,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Dynamic Light Cull With Shadows",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 1.59,
				"render_gpu": 0.64,
				"time": 5007
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Static Light Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.58,
				"render_gpu": 0.59,
				"time": 5003
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "1000 Polygon 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.18,
				"render_gpu": 0.03,
				"time": 5006
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "100 Polygon 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.04,
				"render_gpu": 0.02,
				"time": 5003
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "10 Polygon 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.02,
				"render_gpu": 0.02,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Sprite 2d",
			"name": "5000 Sprite 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.36,
				"render_gpu": 0.31,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Sprite 2d",
			"name": "500 Sprite 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.06,
				"render_gpu": 0.05,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Sprite 2d",
			"name": "50 Sprite 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.03,
				"render_gpu": 0.03,
				"time": 5006
			}
		}
	],
	"engine": {
		"binary_size": 115410976,
		"build_type": "editor",
		"version": "v4.0.stable.official",
		"version_hash": "92bee43adba8d2401ef40e2480e53087bcb1eaf1"
	},
	"system": {
		"cpu_architecture": "x86_64",
		"cpu_count": 32,
		"cpu_name": "13th Gen Intel(R) Core(TM) i9-13900K",
		"gpu_name": "NVIDIA GeForce RTX 4090",
		"gpu_vendor": "NVIDIA",
		"os": "Linux"
	}
}

This PR

{
	"benchmarks": [
		{
			"category": "Gdscript > Array",
			"name": "Packed String Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 1652
			}
		},
		{
			"category": "Gdscript > Array",
			"name": "Typed Int Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 2705
			}
		},
		{
			"category": "Gdscript > Array",
			"name": "Typed String Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 3668
			}
		},
		{
			"category": "Gdscript > Array",
			"name": "Untyped Int Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 2485
			}
		},
		{
			"category": "Gdscript > Array",
			"name": "Untyped String Array",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0,
				"render_gpu": 0,
				"time": 3651
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Basic Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.84,
				"render_gpu": 0.57,
				"time": 5004
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Directional Light Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 2.22,
				"render_gpu": 0.79,
				"time": 5007
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Dynamic Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 4.44,
				"render_gpu": 0.58,
				"time": 5002
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Dynamic Light Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 2.08,
				"render_gpu": 0.61,
				"time": 5002
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Dynamic Light Cull With Shadows",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 2.9,
				"render_gpu": 0.63,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Culling",
			"name": "Static Light Cull",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 1.89,
				"render_gpu": 0.6,
				"time": 5006
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "1000 Polygon 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.16,
				"render_gpu": 0.03,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "100 Polygon 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.03,
				"render_gpu": 0.02,
				"time": 5007
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "10 Polygon 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.02,
				"render_gpu": 0.02,
				"time": 5005
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "5000 Sprite 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.35,
				"render_gpu": 0.3,
				"time": 5000
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "500 Sprite 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.05,
				"render_gpu": 0.05,
				"time": 5008
			}
		},
		{
			"category": "Rendering > Polygon 2d",
			"name": "50 Sprite 2d",
			"results": {
				"idle": 0,
				"physics": 0,
				"render_cpu": 0.02,
				"render_gpu": 0.03,
				"time": 5008
			}
		}
	],
	"engine": {
		"binary_size": 115410976,
		"build_type": "editor",
		"version": "v4.0.stable.official",
		"version_hash": "92bee43adba8d2401ef40e2480e53087bcb1eaf1"
	},
	"system": {
		"cpu_architecture": "x86_64",
		"cpu_count": 32,
		"cpu_name": "13th Gen Intel(R) Core(TM) i9-13900K",
		"gpu_name": "NVIDIA GeForce RTX 4090",
		"gpu_vendor": "NVIDIA",
		"os": "Linux"
	}
}

Results are mostly similar, but render_cpu is much higher with this PR for 3D benchmarks (both runs were made with 4.0.stable official binaries). Can you reproduce this locally?

myaaaaaaaaa · 2023-03-24T23:22:41Z

This may be due to shader compilation, I'm observing a similar render_cpu difference from running the main branch twice. Can you still reproduce this even after doing a warm-up run?

manager.gd currently skips a few frames before each benchmark in an attempt to avoid this, but apparently this isn't enough.

FYI, you can use diff --color -u5 <(cat a.json | jq .) <(cat b.json | jq .) (requires jq) for an easier time comparing results 🙂

Calinou · 2023-03-25T02:07:49Z

This may be due to shader compilation, I'm observing a similar render_cpu difference from running the main branch twice. Can you still reproduce this even after doing a warm-up run?

CPU time is much lower on a second run in 3D benchmarks, which indicates an issue with shader compilation times being counted in CPU time.

Edit: Issue opened: #29

Calinou

Thanks!

Calinou added the enhancement label Dec 8, 2022

myaaaaaaaaa mentioned this pull request Dec 10, 2022

Add HLOD benchmarks #23

Merged

This was referenced Jan 28, 2023

Add node hierarchy benchmarks #24

Merged

Add lights and meshes benchmark #25

Merged

myaaaaaaaaa added 3 commits February 4, 2023 22:15

Rework benchmark API to use functions rather than scenes as the base …

0fedc97

…unit

Consolidate benchmarks and port to new API

640199c

Update CONTRIBUTING.md for new benchmark format

f8d528d

myaaaaaaaaa mentioned this pull request Feb 8, 2023

Miscellaneous cleanup #26

Merged

Calinou approved these changes Mar 27, 2023

View reviewed changes

Calinou merged commit 0c56e6b into godotengine:main Mar 27, 2023

Calinou mentioned this pull request Mar 27, 2023

Shader compilation time is included in render CPU time measurements #29

Open

myaaaaaaaaa deleted the sub-benchmarks branch March 27, 2023 21:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate benchmarks and use functions rather than scenes as the base unit #22

Consolidate benchmarks and use functions rather than scenes as the base unit #22

myaaaaaaaaa commented Dec 7, 2022 •

edited

Loading

myaaaaaaaaa commented Dec 19, 2022

Calinou commented Dec 20, 2022 •

edited

Loading

myaaaaaaaaa commented Dec 20, 2022

Calinou commented Mar 24, 2023 •

edited

Loading

myaaaaaaaaa commented Mar 24, 2023

Calinou commented Mar 25, 2023 •

edited

Loading

Calinou left a comment

Consolidate benchmarks and use functions rather than scenes as the base unit #22

Consolidate benchmarks and use functions rather than scenes as the base unit #22

Conversation

myaaaaaaaaa commented Dec 7, 2022 • edited Loading

myaaaaaaaaa commented Dec 19, 2022

Calinou commented Dec 20, 2022 • edited Loading

myaaaaaaaaa commented Dec 20, 2022

Calinou commented Mar 24, 2023 • edited Loading

main

This PR

myaaaaaaaaa commented Mar 24, 2023

Calinou commented Mar 25, 2023 • edited Loading

Calinou left a comment

Choose a reason for hiding this comment

myaaaaaaaaa commented Dec 7, 2022 •

edited

Loading

Calinou commented Dec 20, 2022 •

edited

Loading

Calinou commented Mar 24, 2023 •

edited

Loading

`main`

Calinou commented Mar 25, 2023 •

edited

Loading