WIP: OpenXR integration #2166

blaind · 2021-05-14T16:31:44Z

Here's a proof-of-concept of OpenXR rendering, using wgpu & underlying stack. See https://github.com/blaind/xrbevy

Relates to #115

There's quite a bit to discuss here, as the integration would touch quite a bit core parts of bevy.

Note that the current pull request is based on Feb version of bevy.

A few initial notes:

Rgba8Unorm patched temporarily to many places, mostly because it was one that worked with Quest
Not sure if the mat4 ViewProj2; is required in all shaders
Some hardcoded constants (e.g. resolutions) in code currently...
etc. etc.

For architecture of the pull request, see https://github.com/blaind/xrbevy/blob/main/docs/architecture.md (evolving document)

NathanSWard · 2021-05-14T17:06:41Z

Hmm, due to this size of this change, this might be something worth first creating an RFC for.
However, @alice-i-cecile would know if that is appropriate for this :)

NathanSWard · 2021-05-14T17:08:35Z

And secondly, very large changes such as this one are difficult to review all at once.
While this is WIP, it would be nice if eventually this could be split apart into a couple individual PRs each adding some specific functionality.

blaind · 2021-05-14T17:25:53Z

In addition to possible bevy RFC, this will most probably require coordination between multiple ecosystem crates too.

See:

alice-i-cecile · 2021-05-14T19:03:12Z

Agreed; this is a great candidate for an RFC explaining what's going on and what the contentious choices are.

I'll give this a scan and try and highlight any changes that I think can be broken apart.

mockersf · 2021-05-14T19:47:20Z

From a (very) quick review, it seems there wasn't a lot of big changes to Bevy itself, and those can mostly be feature gated 🎉

I think once all the dependencies are updated and the Bevy render update is done, some of those changes makes sense to include in Bevy planning for XR (for example a second camera) without a RFC.
Then there will be the big chunk of bevy_openxr...

Even though I don't have an headset, I played at work with one and Godot and it was very fun to be able to move in my 3D scenes, I hope Bevy can do that soon 👍 thank you for the groundwork!

alice-i-cecile · 2021-05-14T21:37:23Z

crates/bevy_app/src/plugin_group.rs

@@ -96,6 +96,21 @@ impl PluginGroupBuilder {
        self
    }

+    pub fn remove<T: Plugin>(&mut self) -> &mut Self {


I like this change; can you spin it out into a seperate tiny PR?

Here it is: #2171

Based on discussion at pull request, this will be migrated after #2039 has been merged

alice-i-cecile · 2021-05-14T21:42:49Z

crates/bevy_gltf/src/loader.rs

@@ -365,6 +368,7 @@ fn load_node(
                    projection_matrix: perspective_projection.get_projection_matrix(),
                    ..Default::default()
                });
+                // FIXME how to differentiate between CAMERA_XR and CAMERA_3D?


I'm not entirely sure what is best here, but I believe that the UI camera differentiation just operates off of names right now.

alice-i-cecile · 2021-05-14T21:47:00Z

Huh cool, this actually isn't too bad at all. There's one change that I think belongs as its own PR (plugin removal), but otherwise this seems quite minimal and clear.

I think with some more documentation (and testing from folks who have the devices) I'd be quite happy to see this get merged. For documentation:

README.md for the new crate, explaining what OpenXR is and why we want to use it.
Examples + getting started guide.
Some more doc strings on the new code added.

jkelleyrtp · 2021-05-20T22:58:16Z

Not to derail this PR, but I'm curious if there's been any thought on trying to get iOS / ARKit enabled. I've done AR in swift-land build would like to stick with Rust-land if I can. Unfortunately, Apple doesn't support OpenXR so I'm curious how much work would be required to get ARKit/Metal mocked behind the OpenXR API. The OpenXR API surface seems huge, but I'm sure there might be a limited MVP that could enable AR with Bevy?

As a side note, I've got a PoC of iOS Bevy AR working by sending the ARKit camera projection matrix through the FFI layer into the Bevy App, which is pretty neato. I imagine OpenXR would open up a lot more functionality. I think ARKit/iOS is of a high level of importance because it's currently the most accessible AR device to most developers, so supporting it would open up the community of contributors (ie, me!).

cart · 2021-05-20T23:24:07Z

@jkelleyrtp

there's been any thought on trying to get iOS / ARKit enabled

I don't recall any existing efforts. But it would definitely be welcome.

As a side note, I've got a PoC of iOS Bevy AR working by sending the ARKit camera projection matrix through the FFI layer into the Bevy App, which is pretty neato.

Definitely agreed. This is very neato 😄

I think ARKit/iOS is of a high level of importance because it's currently the most accessible AR device to most developers, so supporting it would open up the community of contributors (ie, me!).

Agreed. The only thing missing is someone to drive the effort. If you could open a new issue and share what you've learned so far, that would be very helpful. It would also provide a centralized place for interested people to coordinate.

blaind · 2021-05-21T05:48:31Z

This PR is more or less of a demonstration that it can be done, so definitely not derailing anything here :) The implementation approach has already changed quite a bit into more abstracted way from the code what's present here currently.

I guess it would also eventually be good to check Vuforia, ARCore, etc. API's. Does ARKit require access to graphics device, or is it more or less at the input layer?

OpenXR also supports handheld devices form factors, its API could be a good abstraction point.

Here's a rough draft of what the abstraction(s) could look:

bevy_openxr_core would be quite low-level conversion layer from openxr events into bevy event system, while also exposing an xr-initialized wgpu objects for bevy_wgpu to use. bevy_openxr would contain higher-level functionality related e.g. to camera, maybe some basic hands / controller objects, etc.

Zarik5 did an excellent work with ideas he initially presented on blaind/xrbevy#1, having gfx-rs/gfx#3762 merged into gfx. Basically that will expose vulkan raw handles from gfx, and next steps would be to pull the handles through wgpu and wgpu-rs API layers.

Eventually some upper abstraction can take those raw handles and do required initializations for xr-related work. For openxr, gfx-rs/gfx#3219 (comment) contains sequence diagram of what happens with Vulkan (using gfx - objects there).

Before your post @jkelleyrtp, I thought this initialization could be in bevy_openxr_core, but there should be some ecosystem crate which would function somewhat like what gfx is for graphics (gfx big picture). It would wrap different XR-implementations, and expose OpenXR-ish API on top of them? At the same time doing initializations with graphics API's through wgpu, if needed. That crate could be also usable by other rendering engines as well.

jkelleyrtp · 2021-05-21T15:33:39Z

I guess it would also eventually be good to check Vuforia, ARCore, etc. API's. Does ARKit require access to graphics device, or is it more or less at the input layer?

ARKit can be used without access to the graphics APIs - the 3d rendering is handled by SceneKit, RealityKit, and SpriteKit. So, ARKit just provides the camera projection matrix, the pointclouds, feature tracking, and information about lighting. iOS doesn't provide any hand tracking APIs, so currently the only interaction is touch (something I don't believe works in Bevy-iOS atm).

However, to pass through the "AR Frame" as ARKit calls it, you need a way to pull the camera frame from metal into a texture.

ARKit's AR basically just renders a 3D scene on top of a video feed and adjusts the camera view to that of the device's orientation.

Before your post @jkelleyrtp, I thought this initialization could be in bevy_openxr_core, but there should be some ecosystem crate which would function somewhat like what gfx is for graphics (gfx big picture). It would wrap different XR-implementations, and expose OpenXR-ish API on top of them? At the same time doing initializations with graphics API's through wgpu, if needed. That crate could be also usable by other rendering engines as well.

OpenXR provides a good entry point for an API, so anything "high level" like ARCore/ARKit would probably be best served by providing a shim at the openxr-sys level. I personally haven't dove too deep into OpenXR.rs but the API definitely has that bindgen vibe to it. Perhaps it makes sense to build a crate on top of OpenXR that integrates ARCore/ARKit/Vuforia.

I'm definitely bullish on WGPU being the defacto renderer for the Rust ecosystem, so finding a way to reuse it and the engines that leverage it (Bevy) would be amazing for cross-platform AR development.

blaind · 2021-05-21T16:54:33Z

ARKit can be used without access to the graphics APIs - the 3d rendering is handled by SceneKit, RealityKit, and SpriteKit. So, ARKit just provides the camera projection matrix, the pointclouds, feature tracking, and information about lighting. iOS doesn't provide any hand tracking APIs, so currently the only interaction is touch (something I don't believe works in Bevy-iOS atm).

However, to pass through the "AR Frame" as ARKit calls it, you need a way to pull the camera frame from metal into a texture.

ARKit's AR basically just renders a 3D scene on top of a video feed and adjusts the camera view to that of the device's orientation.

Sounds great! I think you gave ideas for a lot of abstractions for this pull request too. Bevy could also be abstracted to be able to handle this completely through plugin(s), if:

A plugin can set camera projection matrices (per view):
- (already possible just by getting from resource graph?)
- The code in this pull request just need to be modified so that a plugin will set the matrices in pre-render stage. Will also become a lot cleaner since no need to access XR object in camera node
A plugin can configure the bevy render graph:
- Should make it possible to configure at least graph items MAIN_DEPTH_TEXTURE, PRIMARY_SWAP_CHAIN, MAIN_SAMPLED_COLOR_ATTACHMENT
- Those graph items may contain texture(views), which must be addable to a render graph (by a plugin)
- For AR, plugins should be able to add a "base" texture layer to the graph. 3D rendered content would be rendered on top of this
- The code in this pull request can probably be adapted, the CAMERA_XR should be scrapped away and just the already existing CAMERA_3D could do?
A plugin can disable/enable multiview rendering
- Multiview rendering should support a projection matrix per view
- Could go through GL_EXT_multiview extension
- (needed for XR, not for AR)
- The code in this pull request somewhat contains a base --> maybe a good idea to extract into a separate pull req
Allow plugin to configure pre- and post hooks just before render and after render
- Already possible through stages?
[Really not sure yet about this]: Allow a plugin to construct a WgpuRenderer

Point clouds and other data could be handled through events? Input system integration (actions in openxr) require more thought.

OpenXR provides a good entry point for an API, so anything "high level" like ARCore/ARKit would probably be best served by providing a shim at the openxr-sys level. I personally haven't dove too deep into OpenXR.rs but the API definitely has that bindgen vibe to it. Perhaps it makes sense to build a crate on top of OpenXR that integrates ARCore/ARKit/Vuforia.

Sounds doable. I tried to look into handheld device use cases of OpenXR, also checked the Khronos API description for examples (e.g. about point cloud, etc.) but did not find any yet.

I'm definitely bullish on WGPU being the defacto renderer for the Rust ecosystem, so finding a way to reuse it and the engines that leverage it (Bevy) would be amazing for cross-platform AR development.

I hope that WGPU will handle all cases, but if it turned out not to, probably would be easy(ish) to provide both WGPU- and GFX abstractions to select from.

blaind · 2021-05-30T10:48:55Z

Status update: this pull request currently contains the today's understanding of mandatory minimal changes required to bevy for getting XR support. I've extracted the openxr-related code to separate crate (see https://github.com/blaind/bevy_openxr/). There is a functional PoC that works on Windows and on Monado /Linux (local emulation) at https://github.com/blaind/xrbevy/

Currently needed changes to bevy (but still iterating):

Multiview support

mat4 ViewProj2; in shaders. Could be doable without, but a large performance hit
Not sure if all the shaders require this change - at least pbr_pipeline is needed but what about bevy_ui and bevy_sprite - will they even relevant in 3d/xr?
gl multiview extension #extension GL_EXT_multiview : enable in main vertex shader. Not sure if all runtimes support this (most probably not)
providing N (currently 2, but maybe more) view projection matrices in camera

Camera projection and position matrices calculation

see https://github.com/bevyengine/bevy/pull/2166/files#diff-d90a7b78c3535c3c4cd112f4d3f99aba3062ab0022debc7891d9a6fc31eeddaa
this is somewhat a hack still, but I added get_view_projection method to camera along with position_matrices (instead of GlobalTransform) which is updated from XR code

Possibility to edit render graph from plugins

I'm replacing the graph nodes with XR-specific textures. Added replace_node method. See https://github.com/blaind/bevy_openxr/blob/main/crates/bevy_openxr/src/render_graph/xr_render_graph.rs for how its being used

Wgpu

This part is still very much in flux, e.g. the solution might differ a lot. Especially since the underlying crates (gfx, wgpu, wgpu-rs) are heavily patched for now. For current iteration, state sharing is needed at least for:

adding (xr swapchain) textures into wgpu render graph
detecting whether the render loop should happen (XR can be unfocused / at background) - should the whole loop be stopped instead of just wgpu part?
exposing wgpu::Device

blaind · 2021-07-23T07:23:41Z

Work continues in #2319, I'll close this

blaind added 2 commits May 14, 2021 16:52

Initial commit of bevy_openxr

c625b14

Add bevy_openxr*

c6994f8

blaind marked this pull request as draft May 14, 2021 16:36

alice-i-cecile added C-Feature A new feature, making something new possible S-Needs-Design-Doc This issue or PR is particularly complex, and needs an approved design doc before it can be merged labels May 14, 2021

alice-i-cecile reviewed May 14, 2021

View reviewed changes

Merge remote-tracking branch 'bevy/main' into bevy_openxr

0ba8d6e

blaind mentioned this pull request May 15, 2021

Added new method remove to PluginGroup, also adds unit tests for PluginGroup #2171

Closed

blaind added 6 commits May 15, 2021 15:45

Code simplification

5327e2c

Simplify code

23ce4d3

Events draft

0f3c8e6

Debug

6226677

Initial integration test, move to events, etc.

d634cdb

Set Wgpu Backend type in settings

5b44ed4

blaind added 2 commits May 23, 2021 16:01

Dev

1ccd2d6

Simplify code

e83c13d

blaind force-pushed the bevy_openxr branch from deb3ba8 to e83c13d Compare May 23, 2021 16:49

Extract bevy_openxr*

dacaa89

blaind added 7 commits May 23, 2021 20:34

Remove unnecessary code

36a6f83

Simplify

a12b6cc

Remove unnecessary diff's from main

29a7a65

Allow to add textures into render graph after startup

72f8259

Revert texture changes, and find more abstract way if needed

dc57a63

Fix texture

77909e2

Plugin group removal not needed anymore

eae0f06

bjorn3 mentioned this pull request Jun 8, 2021

XR support with OpenXR backend #2319

Closed

blaind closed this Jul 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: OpenXR integration #2166

WIP: OpenXR integration #2166

blaind commented May 14, 2021 •

edited

Loading

NathanSWard commented May 14, 2021

NathanSWard commented May 14, 2021

blaind commented May 14, 2021

alice-i-cecile commented May 14, 2021

mockersf commented May 14, 2021 •

edited

Loading

alice-i-cecile May 14, 2021

blaind May 15, 2021

blaind May 16, 2021

alice-i-cecile May 14, 2021

alice-i-cecile commented May 14, 2021

jkelleyrtp commented May 20, 2021

cart commented May 20, 2021

blaind commented May 21, 2021 •

edited

Loading

jkelleyrtp commented May 21, 2021

blaind commented May 21, 2021

blaind commented May 30, 2021 •

edited

Loading

blaind commented Jul 23, 2021

WIP: OpenXR integration #2166

WIP: OpenXR integration #2166

Conversation

blaind commented May 14, 2021 • edited Loading

NathanSWard commented May 14, 2021

NathanSWard commented May 14, 2021

blaind commented May 14, 2021

alice-i-cecile commented May 14, 2021

mockersf commented May 14, 2021 • edited Loading

alice-i-cecile May 14, 2021

Choose a reason for hiding this comment

blaind May 15, 2021

Choose a reason for hiding this comment

blaind May 16, 2021

Choose a reason for hiding this comment

alice-i-cecile May 14, 2021

Choose a reason for hiding this comment

alice-i-cecile commented May 14, 2021

jkelleyrtp commented May 20, 2021

cart commented May 20, 2021

blaind commented May 21, 2021 • edited Loading

jkelleyrtp commented May 21, 2021

blaind commented May 21, 2021

blaind commented May 30, 2021 • edited Loading

Multiview support

Camera projection and position matrices calculation

Possibility to edit render graph from plugins

Wgpu

blaind commented Jul 23, 2021

blaind commented May 14, 2021 •

edited

Loading

mockersf commented May 14, 2021 •

edited

Loading

blaind commented May 21, 2021 •

edited

Loading

blaind commented May 30, 2021 •

edited

Loading