Pack multiple meshes into vertex and index buffers. #13218

pcwalton · 2024-05-03T20:39:23Z

The underlying allocation algorithm is offset-allocator, which is a port of Sebastian Aaltonen's OffsetAllocator. It's a fast, simple hard real time allocator in the two-level segregated fit family.

Allocations are divided into two categories: regular and large. Regular allocations go into one of the shared slabs managed by an allocator. Large allocations get their own individual slabs. Due to platform limitations, on WebGL 2 all vertex buffers are considered large allocations that get their own slabs; however, index buffers can still be packed together. The slab size is 32 MB by default, but the developer can adjust it manually.

The mesh bin key and compare data have been reworked so that the slab IDs are compared first. That way, meshes in the same vertex and index buffers tend to be drawn together. Note that this only works well for opaque meshes; transparent meshes must be sorted into draw order, so there's less opportunity for grouping.

The purpose of packing meshes together is to reduce the number of times vertex and index buffers have to be re-bound, which is expensive. In the future, we'd like to use multi-draw, which allows us to draw multiple meshes with a single drawcall, as long as they're in the same buffers. Thus, this patch paves the way toward multi-draw, and with it a GPU-driven pipeline.

Even without multi-draw, this patch results in significant performance improvements. For me, the command submission time (i.e. GPU time plus driver and wgpu overhead) for Bistro goes from 4.07ms to 1.42ms without shadows (2.8x speedup); with shadows it goes from 6.91ms to 2.62ms (2.45x speedup). The number of vertex and index buffer switches in Bistro is reduced from approximately 3,600 to 927, with the vast majority of the remaining switches due to the transparent pass.

Bistro, without shadows. Yellow is this PR; red is main.

Bistro, with shadows. Yellow is this PR; red is main.

Changelog

Added

Multiple meshes can now be packed together into vertex and index buffers, which reduces state changes and provides performance improvements.

Migration Guide

The vertex and index data in GpuMesh are now GpuAllocations instead of Buffers, to facilitate packing multiple meshes in the same buffer. To fetch the buffer corresponding to a GpuAllocation, use the buffer() method in the new GpuAllocator resource. Note that the allocation may be located anywhere in the buffer; use the offset() method to determine its location.

The underlying allocation algorithm is [`offset-allocator`], which is a port of [Sebastian Aaltonen's `OffsetAllocator`]. It's a fast, simple hard real time allocator in the two-level segregated fit family. Allocations are divided into two categories: *regular* and *large*. Regular allocations go into one of the shared slabs managed by an allocator. Large allocations get their own individual slabs. Due to platform limitations, on WebGL 2 all vertex buffers are considered large allocations that get their own slabs; however, index buffers can still be packed together. The slab size is 32 MB by default, but the developer can adjust it manually. The mesh bin key and compare data have been reworked so that the slab IDs are compared first. That way, meshes that the same vertex and index buffers tend to be drawn together. Note that this only works well for opaque meshes; transparent meshes must be sorted into draw order, so there's less opportunity for grouping. The purpose of packing meshes together is to reduce the number of times vertex and index buffers have to be re-bound, which is expensive. In the future, we'd like to use *multi-draw*, which allows us to draw multiple meshes with a single drawcall, as long as they're in the same buffers. Thus, this patch paves the way toward multi-draw, and with it a GPU-driven pipeline Even without multi-draw, this patch results in significant performance improvements. For me, the command submission time (i.e. GPU time plus driver and `wgpu` overhead) for Bistro goes from 4.07ms to 1.42ms without shadows (2.8x speedup); with shadows it goes from 6.91ms to 2.62ms (2.45x speedup). The number of vertex and index buffer switches in Bistro is reduced from approximately 3,600 to 927, with the vast majority of the remaining switches due to the transparent pass. [`offset-allocator`]: https://github.com/pcwalton/offset-allocator/ [Sebastian Aaltonen's `OffsetAllocator`]: https://github.com/sebbbi/OffsetAllocator/

NthTensor · 2024-05-04T13:34:16Z

Awesome! I will try to read through and test this over the weekend.

Will this also give us a second perf improvement when order-independent-transparency lands? That should mean we can basically stop sorting, right?

pcwalton · 2024-05-04T22:28:47Z

@NthTensor Yes, I would think so.

Swoorup · 2024-05-06T08:15:13Z

crates/bevy_render/src/allocator.rs

+/// position*, we must only group meshes with identical vertex buffer layouts
+/// into the same buffer.
+#[derive(Clone, PartialEq, Eq, Hash, Debug)]
+pub enum GpuAllocationClass {


Curious if it makes sense to add other GpuAllocationClass textures, samplers, pipeline layout, etc? Mostly looking at rerun's code as an example

I think we should add those as we come to them. YAGNI :)

NthTensor · 2024-05-06T22:54:12Z

crates/bevy_core_pipeline/src/core_3d/mod.rs

+    /// The mesh.
+    ///
+    /// Although we don't have multidraw capability yet, we place this at the
+    /// end to maximize multidraw opportunities in the future.


Why does order matter here? It's not repr(c) or anything.

PartialOrd compares from first struct field to last struct field. So fields at the top of the struct will generally be placed together more often.

TIL, Thanks.

NthTensor

Some of the code involving the allocator itself is a bit beyond me, but I don't see any issues reading it over. Very pleased with the performance trying this out on my fairly low quality hardware.

re0312

Outstanding work ! , but looks like something was broken in 2d_shapes

re0312 · 2024-05-07T09:19:04Z

crates/bevy_render/src/allocator.rs

+impl GpuAllocator {
+    /// Returns the slab that the given allocation is stored in.
+    pub fn buffer(&self, allocation: &GpuAllocation) -> &Buffer {
+        &self.slabs[&allocation.slab_id]


Instead of panicking when encountering an invalid allocation, we could return an Option<&Buffer> here

In general I prefer to not panic, but I'm not sure how we would handle the option here in the render system. It looks like this only panics if there's a (unrecoverable) memory error in this or the allocator. Perhaps we could unwrap for a more useful error message?

If it fails it always indicates a bug, because we should be spilling into large allocations if a small allocation can't handle it. So I'm not sure returning None would be helpful.

crates/bevy_render/src/allocator.rs

NthTensor · 2024-05-07T14:45:15Z

I can replicate the issues with 2d shapes. That's weird.

pcwalton · 2024-05-17T05:48:36Z

Updated to main and fixed the 2D meshes problem. It was a simple mistake when porting the logic from 3D over: in the indexed path, for the draw_indexed method call I forgot to switch the base_vertex parameter from 0 to the actual location in the buffer.

crates/bevy_render/src/allocator.rs

pcwalton · 2024-05-21T02:19:11Z

Comments addressed

alice-i-cecile · 2024-05-21T18:51:02Z

[2:32 PM]pcwalton: I think 0.15 would be best for that.
[2:32 PM]pcwalton: Specifically my concern is the size of the slabs: the heuristic hasn't been well tuned and might balloon memory usage in some cases. We won't know without testing.
[2:32 PM]pcwalton: It's the kind of thing we'll only know through a cycle of testing on main so I'd be uncomfortable with 0.14. Besides, @griffin found that the savings are very situational

Blocking until 0.14 is shipped.

pcwalton · 2024-05-21T18:55:12Z

This is ready to go, but I think it would be best to wait until 0.15 and not merge for 0.14. The reason is that the memory usage heuristics haven't been well tuned yet. We'll only know what the best heuristics are through a cycle of testing.

BD103 · 2024-05-22T21:31:42Z

I would like to nominate this for the release notes. I think the performance gains are significant enough that users would enjoy reading about it. (Not to mention I've been watching this in the background every since offset-allocator hit the front page of HN!)

alice-i-cecile · 2024-05-22T21:33:27Z

Agreed :) In the future, feel free to just add the label yourselves: it's easy to make the editorial call to split or lump things during the final release notes process.

Elabajaba · 2024-07-07T06:25:47Z

crates/bevy_render/src/allocator.rs

+const SWEEP_INTERVAL: Duration = Duration::from_secs(10);
+
+/// The default size of a slab, in bytes.
+const DEFAULT_SLAB_SIZE: u64 = 32 * 1024 * 1024;


Should this (in the future) be synced up with wgpu's allocation sizes as well? (gfx-rs/wgpu#5875)

Elabajaba · 2024-07-07T06:36:56Z

crates/bevy_render/src/allocator.rs

+        class: &GpuAllocationClass,
+        contents: &[u8],
+    ) -> GpuAllocation {
+        // If this is going to overflow a slab, give it


Suggested change

// If this is going to overflow a slab, give it

// If this is going to overflow a slab, give it its own large slab.

pcwalton · 2024-07-08T17:14:23Z

I'm closing this because I've written it.

This commit uses the [`offset-allocator`] crate to combine vertex and index arrays from different meshes into single buffers. Since the primary source of `wgpu` overhead is from validation and synchronization when switching buffers, this significantly improves Bevy's rendering performance on many scenes. This patch is a more flexible version of bevyengine#13218, which also used slabs. Unlike bevyengine#13218, which used slabs of a fixed size, this commit implements slabs that start small and can grow. In addition to reducing memory usage, supporting slab growth reduces the number of vertex and index buffer switches that need to happen during rendering, leading to improved performance. To prevent pathological fragmentation behavior, slabs are capped to a maximum size, and mesh arrays that are too large get their own dedicated slabs. As an additional improvement over bevyengine#13218, this commit allows the application to customize all allocator heuristics. The `MeshAllocatorSettings` resource contains values that adjust the minimum and maximum slab sizes, the cutoff point at which meshes get their own dedicated slabs, and the rate at which slabs grow. Hopefully-sensible defaults have been chosen for each value. Unfortunately, WebGL 2 doesn't support the *base vertex* feature, which is necessary to pack vertex arrays from different meshes into the same buffer. `wgpu` represents this restriction as the downlevel flag `BASE_VERTEX`. This patch detects that bit and ensures that all vertex buffers get dedicated slabs on that platform. Even on WebGL 2, though, we can combine all *index* arrays into single buffers to reduce buffer changes, and we do so.

…#14257) This commit uses the [`offset-allocator`] crate to combine vertex and index arrays from different meshes into single buffers. Since the primary source of `wgpu` overhead is from validation and synchronization when switching buffers, this significantly improves Bevy's rendering performance on many scenes. This patch is a more flexible version of #13218, which also used slabs. Unlike #13218, which used slabs of a fixed size, this commit implements slabs that start small and can grow. In addition to reducing memory usage, supporting slab growth reduces the number of vertex and index buffer switches that need to happen during rendering, leading to improved performance. To prevent pathological fragmentation behavior, slabs are capped to a maximum size, and mesh arrays that are too large get their own dedicated slabs. As an additional improvement over #13218, this commit allows the application to customize all allocator heuristics. The `MeshAllocatorSettings` resource contains values that adjust the minimum and maximum slab sizes, the cutoff point at which meshes get their own dedicated slabs, and the rate at which slabs grow. Hopefully-sensible defaults have been chosen for each value. Unfortunately, WebGL 2 doesn't support the *base vertex* feature, which is necessary to pack vertex arrays from different meshes into the same buffer. `wgpu` represents this restriction as the downlevel flag `BASE_VERTEX`. This patch detects that bit and ensures that all vertex buffers get dedicated slabs on that platform. Even on WebGL 2, though, we can combine all *index* arrays into single buffers to reduce buffer changes, and we do so. The following measurements are on Bistro: Overall frame time improves from 8.74 ms to 5.53 ms (1.58x speedup): ![Screenshot 2024-07-09 163521](https://github.com/bevyengine/bevy/assets/157897/5d83c824-c0ee-434c-bbaf-218ff7212c48) Render system time improves from 6.57 ms to 3.54 ms (1.86x speedup): ![Screenshot 2024-07-09 163559](https://github.com/bevyengine/bevy/assets/157897/d94e2273-c3a0-496a-9f88-20d394129610) Opaque pass time improves from 4.64 ms to 2.33 ms (1.99x speedup): ![Screenshot 2024-07-09 163536](https://github.com/bevyengine/bevy/assets/157897/e4ef6e48-d60e-44ae-9a71-b9a731c99d9a) ## Migration Guide ### Changed * Vertex and index buffers for meshes may now be packed alongside other buffers, for performance. * `GpuMesh` has been renamed to `RenderMesh`, to reflect the fact that it no longer directly stores handles to GPU objects. * Because meshes no longer have their own vertex and index buffers, the responsibility for the buffers has moved from `GpuMesh` (now called `RenderMesh`) to the `MeshAllocator` resource. To access the vertex data for a mesh, use `MeshAllocator::mesh_vertex_slice`. To access the index data for a mesh, use `MeshAllocator::mesh_index_slice`. [`offset-allocator`]: https://github.com/pcwalton/offset-allocator

superdump added A-Rendering Drawing game state to the screen C-Performance A change motivated by improving speed, memory usage or compile times X-Uncontroversial This work is generally agreed upon labels May 3, 2024

Elabajaba self-requested a review May 4, 2024 05:07

Swoorup reviewed May 6, 2024

View reviewed changes

NthTensor reviewed May 6, 2024

View reviewed changes

NthTensor approved these changes May 6, 2024

View reviewed changes

re0312 reviewed May 7, 2024

View reviewed changes

pcwalton added 3 commits May 7, 2024 13:58

Merge remote-tracking branch 'origin/main' into uberbuffers

9369723

Merge remote-tracking branch 'origin/main' into uberbuffers

b2dca4b

Take vertex offset into account for 2D meshes

3a20a1e

Address review comment

9b81802

pcwalton requested review from re0312 and superdump and removed request for re0312 May 17, 2024 05:51

pcwalton added this to the 0.15 milestone May 20, 2024

pcwalton added the S-Needs-Review Needs reviewer attention (from anyone!) to move forward label May 20, 2024

ricky26 reviewed May 21, 2024

View reviewed changes

crates/bevy_render/src/allocator.rs Outdated Show resolved Hide resolved

crates/bevy_render/src/allocator.rs Outdated Show resolved Hide resolved

crates/bevy_render/src/allocator.rs Outdated Show resolved Hide resolved

pcwalton added 2 commits May 20, 2024 19:04

Merge remote-tracking branch 'origin/main' into uberbuffers

5b199ac

Address review comments

9742520

pcwalton requested a review from ricky26 May 21, 2024 02:19

ricky26 approved these changes May 21, 2024

View reviewed changes

NthTensor added S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels May 21, 2024

alice-i-cecile added the D-Complex Quite challenging from either a design or technical perspective. Ask for help! label May 21, 2024

alice-i-cecile modified the milestones: 0.15, 0.14 May 21, 2024

alice-i-cecile added the S-Blocked This cannot move forward until something else changes label May 21, 2024

alice-i-cecile added the M-Needs-Release-Note Work that should be called out in the blog due to impact label May 22, 2024

Merge remote-tracking branch 'origin/main' into uberbuffers

26d49f8

Elabajaba approved these changes Jul 7, 2024

View reviewed changes

pcwalton closed this Jul 8, 2024

pcwalton mentioned this pull request Jul 9, 2024

Pack multiple vertex and index arrays together into growable buffers. #14257

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pack multiple meshes into vertex and index buffers. #13218

Pack multiple meshes into vertex and index buffers. #13218

pcwalton commented May 3, 2024 •

edited

Loading

NthTensor commented May 4, 2024

pcwalton commented May 4, 2024

Swoorup May 6, 2024 •

edited

Loading

pcwalton May 7, 2024

NthTensor May 6, 2024

pcwalton May 7, 2024

NthTensor May 7, 2024

NthTensor left a comment

re0312 left a comment

re0312 May 7, 2024

NthTensor May 7, 2024

pcwalton May 7, 2024 •

edited

Loading

NthTensor commented May 7, 2024

pcwalton commented May 17, 2024

pcwalton commented May 21, 2024

alice-i-cecile commented May 21, 2024

pcwalton commented May 21, 2024

BD103 commented May 22, 2024

alice-i-cecile commented May 22, 2024

Elabajaba Jul 7, 2024

Elabajaba Jul 7, 2024

pcwalton commented Jul 8, 2024

	// If this is going to overflow a slab, give it
	// If this is going to overflow a slab, give it its own large slab.

Pack multiple meshes into vertex and index buffers. #13218

Pack multiple meshes into vertex and index buffers. #13218

Conversation

pcwalton commented May 3, 2024 • edited Loading

Changelog

Added

Migration Guide

NthTensor commented May 4, 2024

pcwalton commented May 4, 2024

Swoorup May 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NthTensor left a comment

Choose a reason for hiding this comment

re0312 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcwalton May 7, 2024 • edited Loading

Choose a reason for hiding this comment

NthTensor commented May 7, 2024

pcwalton commented May 17, 2024

pcwalton commented May 21, 2024

alice-i-cecile commented May 21, 2024

pcwalton commented May 21, 2024

BD103 commented May 22, 2024

alice-i-cecile commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcwalton commented Jul 8, 2024

pcwalton commented May 3, 2024 •

edited

Loading

Swoorup May 6, 2024 •

edited

Loading

pcwalton May 7, 2024 •

edited

Loading