Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement minimal reflection probes (fixed macOS, iOS, and Android). #11366

Merged
merged 64 commits into from
Jan 19, 2024

Conversation

pcwalton
Copy link
Contributor

This pull request re-submits #10057, which was backed out for breaking macOS, iOS, and Android. I've tested this version on macOS and Android and on the iOS simulator.

Objective

This pull request implements reflection probes, which generalize environment maps to allow for multiple environment maps in the same scene, each of which has an axis-aligned bounding box. This is a standard feature of physically-based renderers and was inspired by the corresponding feature in Blender's Eevee renderer.

Solution

This is a minimal implementation of reflection probes that allows artists to define cuboid bounding regions associated with environment maps. For every view, on every frame, a system builds up a list of the nearest 4 reflection probes that are within the view's frustum and supplies that list to the shader. The PBR fragment shader searches through the list, finds the first containing reflection probe, and uses it for indirect lighting, falling back to the view's environment map if none is found. Both forward and deferred renderers are fully supported.

A reflection probe is an entity with a pair of components, LightProbe and EnvironmentMapLight (as well as the standard SpatialBundle, to position it in the world). The LightProbe component (along with the Transform) defines the bounding region, while the EnvironmentMapLight component specifies the associated diffuse and specular cubemaps.

A frequent question is "why two components instead of just one?" The advantages of this setup are:

  1. It's readily extensible to other types of light probes, in particular irradiance volumes (also known as ambient cubes or voxel global illumination), which use the same approach of bounding cuboids. With a single component that applies to both reflection probes and irradiance volumes, we can share the logic that implements falloff and blending between multiple light probes between both of those features.

  2. It reduces duplication between the existing EnvironmentMapLight and these new reflection probes. Systems can treat environment maps attached to cameras the same way they treat environment maps applied to reflection probes if they wish.

Internally, we gather up all environment maps in the scene and place them in a cubemap array. At present, this means that all environment maps must have the same size, mipmap count, and texture format. A warning is emitted if this restriction is violated. We could potentially relax this in the future as part of the automatic mipmap generation work, which could easily do texture format conversion as part of its preprocessing.

An easy way to generate reflection probe cubemaps is to bake them in Blender and use the export-blender-gi tool that's part of the bevy-baked-gi project. This tool takes a .blend file containing baked cubemaps as input and exports cubemap images, pre-filtered with an embedded fork of the glTF IBL Sampler, alongside a corresponding .scn.ron file that the scene spawner can use to recreate the reflection probes.

Note that this is intentionally a minimal implementation, to aid reviewability. Known issues are:

  • Reflection probes are basically unsupported on WebGL 2, because WebGL 2 has no cubemap arrays. (Strictly speaking, you can have precisely one reflection probe in the scene if you have no other cubemaps anywhere, but this isn't very useful.)

  • Reflection probes have no falloff, so reflections will abruptly change when objects move from one bounding region to another.

  • As mentioned before, all cubemaps in the world of a given type (diffuse or specular) must have the same size, format, and mipmap count.

Future work includes:

  • Blending between multiple reflection probes.

  • A falloff/fade-out region so that reflected objects disappear gradually instead of vanishing all at once.

  • Irradiance volumes for voxel-based global illumination. This should reuse much of the reflection probe logic, as they're both GI techniques based on cuboid bounding regions.

  • Support for WebGL 2, by breaking batches when reflection probes are used.

These issues notwithstanding, I think it's best to land this with roughly the current set of functionality, because this patch is useful as is and adding everything above would make the pull request significantly larger and harder to review.


Changelog

Added

  • A new LightProbe component is available that specifies a bounding region that an EnvironmentMapLight applies to. The combination of a LightProbe and an EnvironmentMapLight offers reflection probe functionality similar to that available in other engines.

Current problems:

* There's no logic yet to determine which reflection probe to apply to
  each mesh; it always takes the first one.

* Reflection probes are regenerated on every frame.
@mockersf
Copy link
Member

Support for WebGL 2, by breaking batches when reflection probes are used.

Should WebGPU be supported now, or is that for later?

@pcwalton
Copy link
Contributor Author

pcwalton commented Jan 18, 2024

@mockersf How are you testing this on Android? I tested cd examples/mobile && cargo apk run and it works fine on my Pixel 7 Pro. Perhaps this is Android-GPU-specific?

As for your other question, WebGPU has no support for binding arrays and so reflection probes are presently unsupported there.

@mockersf
Copy link
Member

mockersf commented Jan 18, 2024

I'm testing on the simulator on an m1 Mac, I don't have an Android device at the moment

event crates/bevy_diagnostic/src/system_information_diagnostics_plugin.rs:130:  SystemInfo { os: "Android 13 sdk_gphone64_arm64", kernel: "5.15.41-android13-8-00055-g4f5025129fe8-ab8949913", cpu: "", core_count: "4", memory: "1.9 GiB" }
event crates/bevy_render/src/renderer/mod.rs:144:  AdapterInfo { name: "SwiftShader Device (LLVM 10.0.0)", vendor: 6880, device: 49374, device_type: VirtualGpu, driver: "?", driver_info: "?", backend: Vulkan }

With the Pixel 7 pro, it may be possible you have a more powerful phone with a better GPU, or it may be that the simulator is just not good. But I probably expect the latest google flagship to support more things than most Android phones

@pcwalton
Copy link
Contributor Author

Gotcha, I'll try on the simulator next. I suspect that SwiftShader has a relatively low limit while modern Mali has a higher limit. I can try to detect the maximum number of texture bindings and disable reflection probes if it isn't at least 24 (or whatever).

@mockersf
Copy link
Member

mockersf commented Jan 18, 2024

it doesn't crash on the Android simulator if I check on render_device.limits().max_storage_textures_per_shader_stage

@pcwalton
Copy link
Contributor Author

it doesn't crash on the Android simulator if I check on render_device.limits().max_storage_textures_per_shader_stage

OK, I check that now. Seems to work on the default emulator that comes with Android Studio.

Copy link
Contributor

@atlv24 atlv24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

Copy link
Contributor

@JMS55 JMS55 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't do a very thorough check, but approving considering it fixes the crashes, correctly detects feature support at runtime now, and the code generally looks fine and was fine last time I reviewed it in the previous PR. If there are bugs we missed, let's just iterate on main.

@mockersf mockersf added the S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it label Jan 19, 2024
@mockersf mockersf added this pull request to the merge queue Jan 19, 2024
Merged via the queue into bevyengine:main with commit 83d6600 Jan 19, 2024
27 checks passed
github-merge-queue bot pushed a commit that referenced this pull request Jan 20, 2024
# Objective

- Since #11366, feature `glsl` of
`naga_oil` is not enabled by default
- It is needed for example `shader_material_glsl`

```
thread 'Compute Task Pool (0)' panicked at crates\bevy_render\src\render_resource\shader.rs:238:35:
GLSL is not supported in this configuration; use the feature `shader_format_glsl`
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_render::render_resource::pipeline_cache::PipelineCache::process_pipeline_queue_system`!
thread 'main' panicked at crates\bevy_render\src\pipelined_rendering.rs:145:45:
called `Result::unwrap()` on an `Err` value: RecvError
```

## Solution

- Add feature `shader_format_glsl` as a required feature for example
`shader_material_glsl`
github-merge-queue bot pushed a commit that referenced this pull request Jan 27, 2024
… `EnvironmentMapLight` (#11487)

# Objective

DXC+DX12 debug builds with an environment map have been broken since
#11366 merged due to an internal
compiler error in DXC. I tracked it down to a single `break` statement
and reported it upstream
(microsoft/DirectXShaderCompiler#6183)

## Solution

Workaround the ICE by setting the for loop index variable to the max
value of the loop to avoid the `break` that's causing the ICE.

This works because it's the last thing in the for loop.

The `reflection_probes` and `pbr` examples both appear to still work
correctly.
tjamaan pushed a commit to tjamaan/bevy that referenced this pull request Feb 6, 2024
… `EnvironmentMapLight` (bevyengine#11487)

# Objective

DXC+DX12 debug builds with an environment map have been broken since
bevyengine#11366 merged due to an internal
compiler error in DXC. I tracked it down to a single `break` statement
and reported it upstream
(microsoft/DirectXShaderCompiler#6183)

## Solution

Workaround the ICE by setting the for loop index variable to the max
value of the loop to avoid the `break` that's causing the ICE.

This works because it's the last thing in the for loop.

The `reflection_probes` and `pbr` examples both appear to still work
correctly.
github-merge-queue bot pushed a commit that referenced this pull request Feb 6, 2024
# Objective

Bevy could benefit from *irradiance volumes*, also known as *voxel
global illumination* or simply as light probes (though this term is not
preferred, as multiple techniques can be called light probes).
Irradiance volumes are a form of baked global illumination; they work by
sampling the light at the centers of each voxel within a cuboid. At
runtime, the voxels surrounding the fragment center are sampled and
interpolated to produce indirect diffuse illumination.

## Solution

This is divided into two sections. The first is copied and pasted from
the irradiance volume module documentation and describes the technique.
The second part consists of notes on the implementation.

### Overview

An *irradiance volume* is a cuboid voxel region consisting of
regularly-spaced precomputed samples of diffuse indirect light. They're
ideal if you have a dynamic object such as a character that can move
about
static non-moving geometry such as a level in a game, and you want that
dynamic object to be affected by the light bouncing off that static
geometry.

To use irradiance volumes, you need to precompute, or *bake*, the
indirect
light in your scene. Bevy doesn't currently come with a way to do this.
Fortunately, [Blender] provides a [baking tool] as part of the Eevee
renderer, and its irradiance volumes are compatible with those used by
Bevy.
The [`bevy-baked-gi`] project provides a tool, `export-blender-gi`, that
can
extract the baked irradiance volumes from the Blender `.blend` file and
package them up into a `.ktx2` texture for use by the engine. See the
documentation in the `bevy-baked-gi` project for more details as to this
workflow.

Like all light probes in Bevy, irradiance volumes are 1×1×1 cubes that
can
be arbitrarily scaled, rotated, and positioned in a scene with the
[`bevy_transform::components::Transform`] component. The 3D voxel grid
will
be stretched to fill the interior of the cube, and the illumination from
the
irradiance volume will apply to all fragments within that bounding
region.

Bevy's irradiance volumes are based on Valve's [*ambient cubes*] as used
in
*Half-Life 2* ([Mitchell 2006], slide 27). These encode a single color
of
light from the six 3D cardinal directions and blend the sides together
according to the surface normal.

The primary reason for choosing ambient cubes is to match Blender, so
that
its Eevee renderer can be used for baking. However, they also have some
advantages over the common second-order spherical harmonics approach:
ambient cubes don't suffer from ringing artifacts, they are smaller (6
colors for ambient cubes as opposed to 9 for spherical harmonics), and
evaluation is faster. A smaller basis allows for a denser grid of voxels
with the same storage requirements.

If you wish to use a tool other than `export-blender-gi` to produce the
irradiance volumes, you'll need to pack the irradiance volumes in the
following format. The irradiance volume of resolution *(Rx, Ry, Rz)* is
expected to be a 3D texture of dimensions *(Rx, 2Ry, 3Rz)*. The
unnormalized
texture coordinate *(s, t, p)* of the voxel at coordinate *(x, y, z)*
with
side *S* ∈ *{-X, +X, -Y, +Y, -Z, +Z}* is as follows:

```text
s = x

t = y + ⎰  0 if S ∈ {-X, -Y, -Z}
        ⎱ Ry if S ∈ {+X, +Y, +Z}

        ⎧   0 if S ∈ {-X, +X}
p = z + ⎨  Rz if S ∈ {-Y, +Y}
        ⎩ 2Rz if S ∈ {-Z, +Z}
```

Visually, in a left-handed coordinate system with Y up, viewed from the
right, the 3D texture looks like a stacked series of voxel grids, one
for
each cube side, in this order:

| **+X** | **+Y** | **+Z** |
| ------ | ------ | ------ |
| **-X** | **-Y** | **-Z** |

A terminology note: Other engines may refer to irradiance volumes as
*voxel
global illumination*, *VXGI*, or simply as *light probes*. Sometimes
*light
probe* refers to what Bevy calls a reflection probe. In Bevy, *light
probe*
is a generic term that encompasses all cuboid bounding regions that
capture
indirect illumination, whether based on voxels or not.

Note that, if binding arrays aren't supported (e.g. on WebGPU or WebGL
2),
then only the closest irradiance volume to the view will be taken into
account during rendering.

[*ambient cubes*]:
https://advances.realtimerendering.com/s2006/Mitchell-ShadingInValvesSourceEngine.pdf

[Mitchell 2006]:
https://advances.realtimerendering.com/s2006/Mitchell-ShadingInValvesSourceEngine.pdf

[Blender]: http://blender.org/

[baking tool]:
https://docs.blender.org/manual/en/latest/render/eevee/render_settings/indirect_lighting.html

[`bevy-baked-gi`]: https://github.com/pcwalton/bevy-baked-gi

### Implementation notes

This patch generalizes light probes so as to reuse as much code as
possible between irradiance volumes and the existing reflection probes.
This approach was chosen because both techniques share numerous
similarities:

1. Both irradiance volumes and reflection probes are cuboid bounding
regions.
2. Both are responsible for providing baked indirect light.
3. Both techniques involve presenting a variable number of textures to
the shader from which indirect light is sampled. (In the current
implementation, this uses binding arrays.)
4. Both irradiance volumes and reflection probes require gathering and
sorting probes by distance on CPU.
5. Both techniques require the GPU to search through a list of bounding
regions.
6. Both will eventually want to have falloff so that we can smoothly
blend as objects enter and exit the probes' influence ranges. (This is
not implemented yet to keep this patch relatively small and reviewable.)

To do this, we generalize most of the methods in the reflection probes
patch #11366 to be generic over a trait, `LightProbeComponent`. This
trait is implemented by both `EnvironmentMapLight` (for reflection
probes) and `IrradianceVolume` (for irradiance volumes). Using a trait
will allow us to add more types of light probes in the future. In
particular, I highly suspect we will want real-time reflection planes
for mirrors in the future, which can be easily slotted into this
framework.

## Changelog

> This section is optional. If this was a trivial fix, or has no
externally-visible impact, you can delete this section.

### Added
* A new `IrradianceVolume` asset type is available for baked voxelized
light probes. You can bake the global illumination using Blender or
another tool of your choice and use it in Bevy to apply indirect
illumination to dynamic objects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants