Add capability to render to cpu buffer or double buffer #4572

DGriffin91 · 2022-04-24T02:24:14Z

Objective

Provides a portion of the functionally needed for #22

The goal of the PR is to provide a straightforward way to:

Render to a cpu buffer that can be used to output images files.
Render to a double buffer, allowing the image to be easily used as a texture in the same render layer.
Simplify rendering to a texture. Doing so without having to directly manipulate the render graph.

There are three examples showing each of these respective methods:
examples/3d/frame_capture_cpu.rs
examples/3d/frame_capture_double_buffer.rs
examples/3d/frame_capture_gpu.rs

The FrameCapturePlugin is also setup to allow use of multiple, simultaneous, renders using a variety of targets.

The location of the FrameCapturePlugin was lightly discussed on discord. There isn't currently a great place for utility plugins like this one.

alice-i-cecile · 2022-04-24T03:48:23Z

examples/3d/frame_capture_cpu.rs

+use bevy::render::renderer::RenderDevice;
+
+#[derive(Component, Default)]
+pub struct CaptureCamera1;


Is there a reason why this needs to be numbered?

Just a left over from when I was testing using multiple cameras. I'll ditch the number.

alice-i-cecile · 2022-04-24T03:52:06Z

Super cool! A few comments on the examples:

Please add quick module level doc comments at the top of each explaining what their trying to do.
It feels very awkward that these are in the 3D folder. There's fundamentally nothing in them that wouldn't work for 2D. @mockersf, are you on board with creating a rendering examples folder? I know we've discussed this before, but this is a clear case of the need.
Can we save the rendered textures to file? If so, that would be incredibly valuable to demonstrate in an example in the root level tests folder to enable the creation and validation of "golden images", which should be stable across changes.

DGriffin91 · 2022-04-24T04:37:22Z

@alice-i-cecile

Please add quick module level doc comments at the top of each explaining what their trying to do.

Just added some doc comments. Is what I added the sort of thing you are looking for?

It feels very awkward that these are in the 3D folder.

Agreed. I'll have to do some testing to make sure these can work in 2D as well.

Can we save the rendered textures to file?

Yep! That's what the frame_capture_cpu example currently does, and is part of the motivation for adding this feature.

alice-i-cecile · 2022-04-24T05:58:34Z

WRT the comments, see #4438 :)

mockersf · 2022-04-24T07:44:06Z

crates/bevy_camera/Cargo.toml

+bevy_app = { path = "../bevy_app", version = "0.8.0-dev" }
+bevy_asset = { path = "../bevy_asset", version = "0.8.0-dev" }
+bevy_ecs = { path = "../bevy_ecs", version = "0.8.0-dev" }
+bevy_utils = { path = "../bevy_utils", version = "0.8.0-dev" }
+bevy_render = { path = "../bevy_render", version = "0.8.0-dev" }
+bevy_math = { path = "../bevy_math", version = "0.8.0-dev" }
+bevy_core = { path = "../bevy_core", version = "0.8.0-dev" }
+bevy_core_pipeline = { path = "../bevy_core_pipeline", version = "0.8.0-dev" }


could you reorder them alphabetically and check that they are all needed? I'm not sure for bevy_utils, bevy_math or bevy_core

mockersf · 2022-04-24T22:24:17Z

crates/bevy_camera/src/frame_capture.rs

+
+        let gpu_image = images.add(image);
+
+        let padded_bytes_per_row = RenderDevice::align_copy_bytes_per_row(width as usize) * 4;


I'm not happy about the * 4 there (and the other time there's a 4). Some texture formats have less than 4 components, and some uses more than one byte per component.
Should this be * format.describe().components as usize * format.describe().block_size as usize instead?

Good catch.

Based on this: ImageDataLayout I think it may be:

let padded_bytes_per_row = RenderDevice::align_copy_bytes_per_row( (gpu_image.size.width as usize / format.block_dimensions.0 as usize) * format.block_size as usize, );

It may be still more complex. I'm not dealing with rows_per_image yet. But I don't know if that matters here since I would think the texture would always have only one layer in this context.

This is what Bevy does for prepare_asset in image.rs

And it looks like pixel_size() just does info.type_size * info.num_components. I'm not sure if this ends up being always correct if the texture is compressed.

There's also the issue of the buffer being larger than the user might expect since it's aligned to 256 bytes

This is fine for both GPU examples since it's what the gpu expects on both ends. But the one that copies to the CPU results in row padded data.

Do we crop the buffer for the user? If so, I'm not sure exactly what the most efficient way of doing that is. Each row is potentially too long, so we can't just slice off the end of the flat buffer.

mockersf · 2022-04-24T22:32:49Z

crates/bevy_camera/src/frame_capture.rs

+                    RenderPhase::<Opaque3d>::default(),
+                    RenderPhase::<AlphaMask3d>::default(),
+                    RenderPhase::<Transparent3d>::default(),


would it be possible to have it work also for 2d / ui?

mockersf · 2022-04-24T22:35:55Z

examples/3d/frame_capture_cpu.rs

+                bevy::camera::TargetBuffer::CPUBuffer(target_buffer) => {
+                    target_buffer.get(&render_device, |buf| {
+                        image::save_buffer(
+                            format!("../test{i}.png"),


this will write every frame to the same file, with a good chance of leaving it corrupted (with 120 writes per second on my laptop)

could this system be put on a tilmestep, maybe every second, and changing the filename every time?

alice-i-cecile · 2022-04-26T23:14:30Z

@DGriffin91 does this fix #22? If so, please add it to the PR description.

DGriffin91 · 2022-04-27T02:04:54Z

@alice-i-cecile It should at least help with the "copy texture data from the gpu to an image file" portion. I just tried making a headless example and haven't been able to.

If I manually add plugins:

.add_plugin(CorePlugin)
.add_plugin(TransformPlugin)
.add_plugin(HierarchyPlugin)
.add_plugin(AssetPlugin)
.add_plugin(ScenePlugin)
.add_plugin(RenderPlugin)
.add_plugin(CorePipelinePlugin)
.add_plugin(GltfPlugin)
.add_plugin(PbrPlugin)

I get an error on this line:

bevy/crates/bevy_render/src/lib.rs

Line 128 in 328c26d

let windows = app.world.resource_mut::<bevy_window::Windows>();

thread 'main' panicked at 'Requested resource bevy_window::windows::Windows does not exist in the `World`. 
                Did you forget to add it using `app.add_resource` / `app.init_resource`?
                Resources are also implicitly added via `app.add_event`,
                and can be added by plugins.', crates\bevy_render\src\lib.rs:128:41

And if I try the wgpu backend: None from the headless_defaults.rs example:

.insert_resource(WgpuSettings {
    backends: None,
    ..default()
})
.add_plugins(DefaultPlugins)

I get the error:

thread 'main' panicked at 'Sub-App with label 'RenderApp' does not exist', 
[...]\bevy_app\src\app.rs:884:27

So I'm not sure if headless rendering in generally possible in bevy yet. And I also don't know if making headless rendering possible is in the scope of this PR.

alice-i-cecile · 2022-04-27T03:06:04Z

Sounds good. Headless rendering has been a bit of a headache for a long time, so that doesn't surprise me. Fixing that is definitely out of scope.

mockersf · 2022-05-02T19:50:51Z

waiting on the "high level camera driven render targets" PR

YoshieraHuang · 2022-05-25T07:49:58Z

@alice-i-cecile It should at least help with the "copy texture data from the gpu to an image file" portion. I just tried making a headless example and haven't been able to.

If I manually add plugins:
.add_plugin(CorePlugin)
.add_plugin(TransformPlugin)
.add_plugin(HierarchyPlugin)
.add_plugin(AssetPlugin)
.add_plugin(ScenePlugin)
.add_plugin(RenderPlugin)
.add_plugin(CorePipelinePlugin)
.add_plugin(GltfPlugin)
.add_plugin(PbrPlugin)
I get an error on this line:

bevy/crates/bevy_render/src/lib.rs

Line 128 in 328c26d

let windows = app.world.resource_mut::<bevy_window::Windows>();
thread 'main' panicked at 'Requested resource bevy_window::windows::Windows does not exist in the `World`. 
                Did you forget to add it using `app.add_resource` / `app.init_resource`?
                Resources are also implicitly added via `app.add_event`,
                and can be added by plugins.', crates\bevy_render\src\lib.rs:128:41
And if I try the wgpu backend: None from the headless_defaults.rs example:
.insert_resource(WgpuSettings {
    backends: None,
    ..default()
})
.add_plugins(DefaultPlugins)
I get the error:
thread 'main' panicked at 'Sub-App with label 'RenderApp' does not exist', 
[...]\bevy_app\src\app.rs:884:27
So I'm not sure if headless rendering in generally possible in bevy yet. And I also don't know if making headless rendering possible is in the scope of this PR.

Only the WinitPlugin should be disabled to prevent the windowing. This script works with the frame_capture_cpu example:

App::new()
...
        .add_plugins_with(DefaultPlugins, |group| group.disable::<WinitPlugin>() )
...

However, no frame is rendered and then the app exits, since WinitPlugin provides built-in eventloop to trigger app to run forever. An explicit runner should be provided.

Update:
ScheduleRunnerPlugin provides a standalone runner. Add ScheduleRunnerPlugin to the app and the app starts to output png files. Here is the current main function:

    App::new()
        .insert_resource(Msaa { samples: 4 }) // Use 4x MSAA
        .insert_resource(AmbientLight {
            color: Color::WHITE,
            brightness: 1.0 / 5.0f32,
        })
        .add_plugins_with(DefaultPlugins, |group| group.disable::<WinitPlugin>())
        .add_plugin(CameraTypePlugin::<CaptureCamera>::default())
        .add_plugin(FrameCapturePlugin)
        .insert_resource(ScheduleRunnerSettings{
            run_mode: RunMode::Loop { wait: Some(Duration::from_millis(100)) }
        })
        .add_plugin(ScheduleRunnerPlugin)
        .add_startup_system(setup)
        .add_system(animate_light_direction)
        .add_system(save_img)
        .run();

The ScheduleRunnerSettings is used to determine how fast this app ticks.

However, the output file may be empty or corrputed if the app ends using Ctrl+C. If you monitor the output file when the app runs, you can see the size of file jumping from zero to about 100 kb repeatedly.

mockersf · 2022-06-15T23:12:43Z

@DGriffin91 #4898 has been merged, could you update this PR? I would love to see it move forward.

zimond · 2022-06-16T01:39:27Z

I tested this PR on 2d camera and it just works (by changing 3d subgraph rendering to 2d, etc). I think maybe the whole extract_camera_phases thing is creating redundant code, as it is identical to what is done for 2d/3d camera

DGriffin91 · 2022-06-16T02:01:18Z

I tested this PR on 2d camera and it just works (by changing 3d subgraph rendering to 2d, etc). I think maybe the whole extract_camera_phases thing is creating redundant code, as it is identical to what is done for 2d/3d camera

If I remove this (below) from extract_camera_phases with the frame_capture_cpu example, it doesn't output the 3D model and I just get a blank grey png.

.insert_bundle((
    RenderPhase::<Opaque3d>::default(),
    RenderPhase::<AlphaMask3d>::default(),
    RenderPhase::<Transparent3d>::default(),
))

zimond · 2022-06-16T03:02:05Z

@DGriffin91 Yes, I didn't mean it's unnecessary, it's just a duplication of what is done in camera3d pipeline.

Anyway, I think we could extract the logic or at least generalize this method to support 2d/ui cameras, e.g. for 2d camera:

.insert_bundle((
    RenderPhase::<Transparent2d>::default()
))

By the way this is the feature I really want & thanks for your hard work.

DGriffin91 · 2022-06-16T04:59:38Z

@zimond Gotcha, that makes sense. And thank you! We are discussing how to get this setup with the new camera driven stuff on discord here in case you are interested: https://discord.com/channels/691052431525675048/986788850535915540/

superdump · 2022-07-13T10:09:20Z

crates/bevy_core_pipeline/src/core_3d/main_pass_3d_node.rs

@@ -194,6 +199,31 @@ impl Node for MainPass3dNode {
            }
        }

+        // TODO: should this live here or in a separate node?


Separate node imo.

superdump · 2022-07-13T10:12:03Z

crates/bevy_render/src/camera/camera.rs

@@ -247,6 +247,8 @@ pub enum RenderTarget {
    Window(WindowId),
    /// Image to which the camera's view is rendered.
    Image(Handle<Image>),
+    /// Buffered Image to which the camera's view is rendered.


I feel like this description could be clearer. The first image is used for rendering into and is then copied into the second image, right?

superdump · 2022-07-13T10:12:47Z

crates/bevy_core_pipeline/src/core_3d/main_pass_3d_node.rs

+        // TODO: don't just have duplicated code between MainPass3dNode/MainPass2dNode
+        if let RenderTarget::BufferedImage(image_handle, buffer_image_handle) = &camera.target {
+            let gpu_images = world.get_resource::<RenderAssets<Image>>().unwrap();
+            let gpu_image = gpu_images.get(&image_handle).unwrap();


Maybe source and destination images? i.e. src_image, dst_image

superdump · 2022-07-13T10:13:33Z

examples/3d/render_to_double_buffer.rs

+            usage: TextureUsages::TEXTURE_BINDING
+                | TextureUsages::COPY_DST
+                | TextureUsages::COPY_SRC
+                | TextureUsages::RENDER_ATTACHMENT,


I think this one only needs RENDER_ATTACHMENT | COPY_SRC.

superdump · 2022-07-13T10:14:57Z

examples/3d/render_to_double_buffer.rs

+            usage: TextureUsages::TEXTURE_BINDING
+                | TextureUsages::COPY_DST
+                | TextureUsages::COPY_SRC
+                | TextureUsages::RENDER_ATTACHMENT,


I think this one only needs COPY_DST | TEXTURE_BINDING.

paulkre · 2022-09-05T17:41:20Z

I'm looking forward to using this feature. I hope it gets merged soon!

paulkre · 2022-09-06T16:05:33Z

Is it possible to save a PNG image with the wgpu::Texture struct? I only found a way to do this with the wgpu::Buffer which requires the use of wgpu::CommandEncoder::copy_texture_to_buffer method.

DGriffin91 · 2022-09-06T19:16:45Z

@paulkre Check out this newer PR. It outputs PNG images: #5550

You should be able to use image_copy.rs in bevy 0.8 as a plugin iirc.

Yes I am using copy_texture_to_buffer

alice-i-cecile · 2022-09-06T20:03:54Z

@DGriffin91 shall I close this out in favor of #5550?

DGriffin91 · 2022-09-06T20:11:05Z

@alice-i-cecile Yep, I think that makes sense.

github-actions bot added the S-Needs-Triage This issue needs to be labelled label Apr 24, 2022

DGriffin91 changed the title ~~Add capability to render to cpu image buffer or double buffer~~ Add capability to render to cpu buffer or double buffer Apr 24, 2022

alice-i-cecile added C-Feature A new feature, making something new possible A-Rendering Drawing game state to the screen and removed S-Needs-Triage This issue needs to be labelled labels Apr 24, 2022

alice-i-cecile reviewed Apr 24, 2022

View reviewed changes

mockersf reviewed Apr 24, 2022

View reviewed changes

DGriffin91 mentioned this pull request May 17, 2022

Moving camera controller into separate crate in examples #4458

Closed

DGriffin91 force-pushed the frame_capture branch 2 times, most recently from f74b4c4 to 379515d Compare June 17, 2022 00:16

DGriffin91 added 4 commits June 21, 2022 13:32

bevy_camera: FrameCapturePlugin

0cd2e89

get_buffer method for getting BufferView

ad1e2f1

remove second camera from example

71cab6c

add modes for gpu, double buffer

e5dd62b

DGriffin91 added 7 commits June 21, 2022 13:32

add doc comments, remove camera number

29445c2

remove unnecessary dependencies

21957de

make proper doc comments

1fe10a6

rebase & update to use render RenderTarget

e65cdba

clean up example a bit

041aaef

remove image from Cargo.toml for now

d07edbb

remove extraneous line break

fdfdb78

DGriffin91 force-pushed the frame_capture branch from 1753a34 to fdfdb78 Compare June 21, 2022 20:35

superdump requested changes Jul 13, 2022

View reviewed changes

DGriffin91 mentioned this pull request Aug 2, 2022

Frame capture testing tool #5550

Closed

alice-i-cecile closed this Sep 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add capability to render to cpu buffer or double buffer #4572

Add capability to render to cpu buffer or double buffer #4572

DGriffin91 commented Apr 24, 2022 •

edited

Loading

alice-i-cecile Apr 24, 2022

DGriffin91 Apr 24, 2022

alice-i-cecile commented Apr 24, 2022 •

edited

Loading

DGriffin91 commented Apr 24, 2022

alice-i-cecile commented Apr 24, 2022

mockersf Apr 24, 2022

mockersf Apr 24, 2022

DGriffin91 Apr 26, 2022 •

edited

Loading

mockersf Apr 24, 2022

mockersf Apr 24, 2022

alice-i-cecile commented Apr 26, 2022

DGriffin91 commented Apr 27, 2022 •

edited

Loading

alice-i-cecile commented Apr 27, 2022

mockersf commented May 2, 2022

YoshieraHuang commented May 25, 2022 •

edited

Loading

mockersf commented Jun 15, 2022

zimond commented Jun 16, 2022 •

edited

Loading

DGriffin91 commented Jun 16, 2022

zimond commented Jun 16, 2022 •

edited

Loading

DGriffin91 commented Jun 16, 2022

superdump Jul 13, 2022

superdump Jul 13, 2022

superdump Jul 13, 2022

superdump Jul 13, 2022

superdump Jul 13, 2022

paulkre commented Sep 5, 2022

paulkre commented Sep 6, 2022

DGriffin91 commented Sep 6, 2022 •

edited

Loading

alice-i-cecile commented Sep 6, 2022

DGriffin91 commented Sep 6, 2022


		let gpu_image = images.add(image);

		let padded_bytes_per_row = RenderDevice::align_copy_bytes_per_row(width as usize) * 4;

Add capability to render to cpu buffer or double buffer #4572

Add capability to render to cpu buffer or double buffer #4572

Conversation

DGriffin91 commented Apr 24, 2022 • edited Loading

Objective

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alice-i-cecile commented Apr 24, 2022 • edited Loading

DGriffin91 commented Apr 24, 2022

alice-i-cecile commented Apr 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DGriffin91 Apr 26, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alice-i-cecile commented Apr 26, 2022

DGriffin91 commented Apr 27, 2022 • edited Loading

alice-i-cecile commented Apr 27, 2022

mockersf commented May 2, 2022

YoshieraHuang commented May 25, 2022 • edited Loading

mockersf commented Jun 15, 2022

zimond commented Jun 16, 2022 • edited Loading

DGriffin91 commented Jun 16, 2022

zimond commented Jun 16, 2022 • edited Loading

DGriffin91 commented Jun 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paulkre commented Sep 5, 2022

paulkre commented Sep 6, 2022

DGriffin91 commented Sep 6, 2022 • edited Loading

alice-i-cecile commented Sep 6, 2022

DGriffin91 commented Sep 6, 2022

DGriffin91 commented Apr 24, 2022 •

edited

Loading

alice-i-cecile commented Apr 24, 2022 •

edited

Loading

DGriffin91 Apr 26, 2022 •

edited

Loading

DGriffin91 commented Apr 27, 2022 •

edited

Loading

YoshieraHuang commented May 25, 2022 •

edited

Loading

zimond commented Jun 16, 2022 •

edited

Loading

zimond commented Jun 16, 2022 •

edited

Loading

DGriffin91 commented Sep 6, 2022 •

edited

Loading