Bevy Asset System Evolution requirement gathering

[cart]

In the very near future I will be shifting my focus to evolving Bevy's asset system to better accommodate our needs. This will either involve iterating on bevy_asset or replacing it with distill.

This is a place to lazily collect and discuss requirements and scenarios over time, not a formal design document. This is also not a place to dictate particular implementations. High level requirements and scenarios only at this point please!

Some initial high level requirements to get the ball rolling:

• Asset Preprocessing: it should be possible to "preprocess" / "compile" / "crunch" assets at "development time" rather than when the game starts up. This enables offloading expensive work from deployed apps, faster asset loading, less runtime memory usage, etc.
• Asset .meta files: asset configuration files stored adjacent to the asset in question, which both assigns them a unique and stable asset id and allows the user to configure asset-type-specific settings. These settings should be accessible during the pre-processing phase. Modifying a .meta file should trigger a re-processing / re-load of the asset.
• Asset Dependencies: assets should be able to define dependencies on other assets. this might be split out into "hard" and "soft" dependencies, where "hard" dependencies must be available during pre-processing and invalidate parent assets when they change. And "soft" dependencies are just loaded when the parent is loaded (at runtime).
• Runtime Asset Loading: it should be (optionally) possible to load arbitrary assets dynamically at runtime. This necessitates being able to deploy and run the asset server alongside Bevy Apps on all platforms. For example, we should be able to invoke the shader compiler at runtime, stream scenes from sources like the internet, etc. To keep deployed binaries (and startup times) small, the runtime asset server configuration should be configurable with different settings compared to the "pre processor asset server".

Some history:

• My initial "bevy asset rework" framing from over a year ago. A lot has changed since then, both in terms of bevy_asset's capabilities and atelier (which was renamed to distill).
• My prototype rework of bevy_asset to add support for nested asset loading, imports, .meta config files, and asset redirects (a precursor to asset preprocessing). Ultimately I ported aspects of this to upstream bevy_asset, such as nested asset loading, which enabled things like "gltf scene loading".
• The migrate to distill issue tracking the various things needed for us to migrate to distill. As stated above, migrating to distill is no longer a set-in-stone plan. As part of the upcoming bevy asset design process, we will re-evaluate the best path forward.

[cart]

Asset Preprocessing: it should be possible to "preprocess" / "compile" / "crunch" assets at "development time" rather than when the game starts up. This enables offloading expensive work from deployed apps, faster asset loading, less runtime memory usage, etc.

[NHodgesVFX]

I think as apart of this there should be a built in way to encrypt assets as well. As most licenses for assets on asset stores like CGTrader require a best effort attempt to protect assets when distributed in games.

[alice-i-cecile]

The requirements from the asset stores is a fair point. I don't think "best effort" is going to be very good in the end (storing encryption keys and code on the client is doomed to fail), but I'm fine with including some easy functionality for this, either in the main repo or in a lighweight well-integrated crate.

[mousetail]

Very late, but I feel this would be better as a (maybe officially supported) external tool. Projects shouldn't require a build system if they don't need to. All bevy needs is the ability to load compressed/optimized assets from archives. This would make it significantly easier to do weird custom processing when needed, and means other systems don't need to be aware of the dependency graph.

[soqb]

but bevy should therefore have the capacity for it. e.g. not relying on a single texture format

[mousetail]

Yea I just said it should support compressed texture formats, as well as normal texture formats. In fact bevy already does this.

[cart]

Asset .meta files: asset configuration files stored adjacent to the asset in question, which both assigns them a unique and stable asset id and allows the user to configure asset-type-specific settings. These settings should be accessible during the pre-processing phase. Modifying a .meta file should trigger a re-processing / re-load of the asset.

[soqb]

i am strongly against all asset meta files being stored directly next to their main asset files at compile time. since every asset needs a meta file, for most users, most files will be shadowed by an arbitrary and seemingly useless file that they are not allowed to remove. we can do some things to mitigate these issues:

• if we have some "asset view" in the editor hide asset meta files there and allow configuration somewhere else, this will not aid users who edit a majority of their asset files outside of the bevy editor
• rename the meta files from asset.meta to .asset.meta or similar so that users sorting alphabetically will have their meta files grouped and out of the way
• allow moving the meta files away from the place their main asset files are stored/configurable meta file resolution (e.g.) assets/path/to/asset.meta then asset_meta/path/to/asset.meta and maybe even /system/wide/configuration/asset.meta

i would like to see some explanation of why meta files are necessary for all assets:

• why do we need all assets to have stable ids that persist over rename? it seems it would be done to save the user the trouble of changing all their references to the file in their configuration and codebase, at the cost of renaming the meta file too. ids are also difficult to decipher and match up to assets when manually editing. the addition of an id-driven asset pipeline feels like a step backward for non-editor/manual workflows.
• what kind of additional configuration would be stored in the meta files that wouldn't be better stored either in the file itself, where the assets are referenced (e.g. in scene/prefab files) or in some runtime resource for asset configuration?
• why even have a standardized system of meta files rather than entirely new assets like MyMetaShader that points to assets/my_shader.meta.ron which depends on assets/my_shader.wgsl? i don't think just the stable ids alone is a compelling enough reason to have meta files

the addition of asset meta files strikes me as a change too opinionated and rife with drawbacks and tradeoffs as to not fit in with the rest of bevy's design.
an asset driven workflow should be available, but i believe as a third-party crate or at the very least a completely optional and feature identical supplement to the proposed asset pipeline and workflow.

[bjorn3]

allow moving the meta files away from the place their main asset files are stored/configurable meta file resolution (e.g.) assets/path/to/asset.meta then asset_meta/path/to/asset.meta and maybe even /system/wide/configuration/asset.meta

How would you then know where to find the asset itself? If it is next to the asset file you can simply strip the .meta extension, but if it isn't next to the asset file you did have to embed the path itself, which would make it impossible to simply move the asset without breaking the link.

[bjorn3]

what kind of additional configuration would be stored in the meta files that wouldn't be better stored either in the file itself, where the assets are referenced (e.g. in scene/prefab files) or in some runtime resource for asset configuration?

For textures the desired filtering mode, the desired mipmap levels, ... For 3d models the desired lod levels and method to generate said lod levels. All those things can't be stored directly in the asset itself as it doesn't have anywhere to store it. Especially not in a way that external editors will not destroy it.

[bjorn3]

why do we need all assets to have stable ids that persist over rename?

What about save files? You wouldn't want those to break if the name of assets changes when using scenes as save files.

[soqb]

What about save files? You wouldn't want those to break if the name of assets changes when using scenes as save files.

yes. renaming assets would invalidate saves, but so would renaming or adding a field; so would adding literally any non-optional component or non-runtime resource anywhere. devs will need to implement save-porting and backwards-compatible, defensive systems. bevy (or a 3rd party crate) should be capable of handling most of the work of converting between versions with minimal configuration. i fail to see why asset renaming shouldn't be part of it.

besides, workflows that favour scene-based saves (likely using a standalone editor) typically also don't suffer from as many of the problems i've outlined, but bevy users shouldn't be forced into them.

to reiterate i don't believe asset meta files do not have value, i just don't believe either that they're necessary for all assets in all workflows or worth forcing users into the overhead of an editor if they wouldn't otherwise benefit. users should be able to leverage every other bit of the asset pipeline proposed at the top of this discussion without being bound to asset ids and asset meta files.

[cart]

Asset Dependencies: assets should be able to define dependencies on other assets. this might be split out into "hard" and "soft" dependencies, where "hard" dependencies must be available during pre-processing and invalidate parent assets when they change. And "soft" dependencies are just loaded when the parent is loaded (at runtime).

[aclysma]

FYI it has come up in using distill that it would be nice to be able to load an asset that references another asset at runtime, but doesn’t implicitly load that asset. For example, a data file that lists all levels in a game might reference the levels, which references objects, which reference meshes/textures. Obviously one would want to read the metadata for the level files and trigger loading them explicitly, without implicitly triggering a load of all assets ref’ed by all levels. :)

Also I would just call out in general, it is easy to accidentally design a chain of dependencies where tweaking one vertex in a mesh can cascade a huge amount of rebuilding. (All cached AABBs, visibility structures, navmesh, lighting, across many levels. The rebuilds in those levels may walk a lot of unrelated assets to do the rebuild. So you could imagine a tiny tweak that you just want to preview triggers nearly a full rebuild of all assets. Finding ways to limit/break dependency chains like this will be important imo.

[nicopap]

Exploring how to load FBX files, I've hit all the limitations of bevy's asset system. I think composable asset loading is a requirement.

The issue I'm currently dealing with is that FBX is very versatile and can define all sorts of materials in all sorts of way (basically every software exporting to FBX have their own custom material definition). So it's necessary to have more than a single material loader for it, and ideally custom user-provided ones.

The dream would be to just pass a handle to the FBX definition to an (or many) asset loader and get back a material.

My current workaround is to ask the user for a list of loaders (a struct with a function pointer field that takes my FBX material handle and returns a StandardMaterial)

[JosephCatrambone]

Idle consideration: I wonder how much can be accomplished with things like Bevy-supported placeholders if a hard/soft dependency is unloaded. Would having empty sounds which "play" but have no audio and empty textures which "render" but are invisible be a nice simplifying assumption (inasfar as they mean we can always reference textures and sounds), or would the headache of temporaries be too problematic?

[Longor1996]

Would that not depend on the usage domain (color, mask, normal, pbr-map, density, etc. etc.) of the texture?

[djeedai]

Linking #7479 because this thread seems to be lacking on more complex use cases for asset dependencies.

[cart]

Runtime Asset Loading: it should be (optionally) possible to load arbitrary assets dynamically at runtime. This necessitates being able to deploy and run the asset server alongside Bevy Apps on all platforms. For example, we should be able to invoke the shader compiler at runtime, stream scenes from sources like the internet, etc. To keep deployed binaries (and startup times) small, the runtime asset server configuration should be configurable with different settings compared to the "pre processor asset server".

[igorhoogerwoord]

This was a reason for switching from Godot to Bevy -> our game includes creative tools which allow any assets (audio, images, meshes) to be created and uploaded/downloaded from a web server at any time. In our case 'loading/processing' times are acceptable because of the nature of our game. This wasn't possible in Godot because asset processing and loading is restricted to the Godot Editor and not in exported games.

[cart]

Multiple Backends: It would be nice if we could load assets from a variety of sources (filesystems, the internet, remote asset servers, etc).

[james7132]

Until we have an editor, we should definitely have a baked in local backend for local testing without a dedicated asset server, which I think distill unfortunately requires in all contexts right now. Might have changed since I last checked.

+1 to the remote asset servers for loading runtime assets. This is basically mandatory for any non-trivial mobile game nowadays. See Unity's Asset Bundle system and their built in support for loading it from arbitrary HTTP file servers.

[bjorn3]

The daemon is a library, so I think you could just start it from within the game itself. It will still use a port on localhost for rpc though it seems. Distill has websocket support for rpc, so the daemon should be able to work over the internet. Not sure about the security of doing so though.

[kabergstrom]

It'd be trivial to write an in-process transport for the RPC if one would want it. And no, the RPC does not use authentication of any sort/

[james7132]

Not included in the main requirements, but one key thing to think of is developer UX. Asset loading in bevy right now is pretty involved and error prone, even for simple dependent asset loads. We can obviously build more user friendly asset APIs on top of a low level core, but building the right user-facing abstractions here can go a very long way.

[alice-i-cecile]

IMO bevy_asset_loader gets this right, at least as a first approximation.

Users need to be able to know if all of the relevant things that they've requested to load are loaded, and to block the app from advancing until they are.

States are a natural tool for this, especially once we can add third-party plugins to states.

The way that asset collections work, allowing you to define a central resource with paths and saved handles is also very appealing.

[mockersf]

Writing an asset loader for a new format is not obvious when the format gets complicated... how to deal with dependents asset loads is explained here for example.
The gltf loader doesn't do it that way and instead uses direct file access because its dependent assets can be either in their own files or packed in a binary buffer, and loading an image from its own file or from the binary buffer needed to take the same path in code, instead of relying on the existing image loader only for files.

All this to say: dependent assets are hard, a good UX for them is harder

[james7132]

bevy_asset_loader is a step in the right direction, but IMO it moves what should be in the domain of an editor into hard Rust types. It's a good substitute until we have an actual editor with a fully functioning asset pipeline constantly watching asset files and injecting handles where they need to go.

[kabergstrom]

I think one oversight in bevy_assets is that it's impossible to load/process a source file and an asset (or a graph of assets) without having it become visible to the running app state.

[Neo-Zhixing]

Efficient handling of CPU -> GPU dataflow
Right now for each asset to be loaded, the asset loader gets passed a &[u8] which needs to be copied around multiple times in memory. For example, to load a texture, it goes through the following steps:

• From the original &[u8] slice to the main world
• From the main world to the render world
• From the render world to host visible (staging) buffer
• In the case of discrete graphics card, it gets copied to device local buffer

The new asset system needs to support at least the following types of asset dataflows:

• CPU-only assets: the scene file
• GPU-only assets: textures, mesh, shaders
• GPU and CPU assets

For GPU-only assets, the ownership of the data should be inserted into the render world directly without requiring an extra copy step on the CPU through the main world.

We also need to be forward-looking here and consider RDMA / DirectStorage where the asset data gets streamed and decompressed directly from SSD to VRAM with minimal intervention from the host.

[sokorototo]

Moving around a Box<[u8]> would mostly alleviate the excess copying without excessive complexity. Do note that one must move the box, not clone or copy otherwise no extra efficiency is gained.

[Evrey]

Make that box a Cow and i’m in. I use an asset format that is mmap-friendly. Any copy, including to heap, is a copy too much.

[sokorototo]

Rust Cows are legit under utilised

[Neo-Zhixing]

Asset Unloading: It should be able to unload certain assets from memory temporarily but still retain a handle to it. The user can then use that handle to load it back up from disk on demand.

[superdump]

Asset Unloading: It should be able to unload certain assets from memory temporarily but still retain a handle to it. The user can then use that handle to load it back up from disk on demand.

We might want to implement a strategy for that for web to avoid redownloading large assets, or just trust the browser cache perhaps.

[minecrawler]

I'm all for the browser cache, since that's the default way to handle caching and it can be influenced by the server (via HTTP headers). Especially using max-age and ETag (did the online resource change at all after max-age?) is something which already works great, is web specific and does not need to be re-invented :)

[superdump]

I’ve not had to fiddle with caching things before so I’ve only heard of etag, but I did figure as I was writing that we should probably rely on browser cache.

[MiniaczQ]

I didn't see anyone mention hot-reloading, which is pretty important considering almost every suggestion here would complicate it in some way.

[SpecificProtagonist]

Additionally, unlike the current system the last successfully loaded version of an asset should stay available, even if the asset source gets updated (and loading it fails). When live-editing assets, they're not always in an valid state, especially if you have additional requirements on them beyond the file format.

[heavyrain266]

I see that you mentioned replacing current asset handler with distil; From my experience of porting rust, bevy and general gamedev ecosystem to PlayStation consoles (not homebrew), distil causes some problems which prevents you from loading assets on console(s). Distil uses daemon as IPC system for managing assets from external process, this is main problem on consoles, you are not allowed to use any kinds of daemons which are living next to your game. Each process is sandboxed for security reasons and you will have to exploit runtime to send assets between daemon and the game, of course no one wants to do that.

[Nilirad]

From the readme it looks like that by default the daemon is only used in development, not in the shipped game. Do console SDKs have the same requirement?

[heavyrain266]

Yeah, I meant debug builds which requires daemon.

[kabergstrom]

To clarify a bit, the daemon is not required to live in a separate process. It can live in the same process. The transport for communication between daemon and loader is also pluggable, and it would be easy enough to create a purely userspace transport even if none exists right now.

In the case of a console though, the intended use case is that the daemon should run on the host computer so that changes can be hot reloaded. The console connects to the host daemon over TCP.

Here's a hot reloading demo on a phone: https://youtu.be/Ks_HQbejHE4

[heavyrain266]

In the case of a console though, the intended use case is that the daemon should run on the host computer so that changes can be hot reloaded. The console connects to the host daemon over TCP.

Ohh, that helps a lot. Honestly, didn't know that it's possible, thanks for clarification and will try transporting assets from PC to devkit this way today.

[BGR360]

Feel free to remove this if this isn't high level enough (and/or outdated; I last used bevy in like February).

I think it's worth considering that an asset's file extension doesn't necessarily determine the asset type.

Motivating example: I was trying to write a Minecraft client, and the data pack format uses .json files for multiple different types of asset (e.g., block model and animation).

As far as I could tell, the AssetServer dispatches load requests based on the file extension. And the information in each .json isn't enough to (easily) determine which type of asset it is. So that precluded me from using the AssetServer to load data pack files.

[CobaltCause]

I agree, I also think this is a little strange. I would have expected some trait impl on T in bevy::asset::Handle<T> to determine how the data should be loaded.

[komadori]

You might also want to determine the asset type by MIME type. glTF specifies image file formats for textures this way.

[CobaltCause]

MIME type is better than file extension IMO but still much more prescriptive than "some trait impl on T in bevy::asset::Handle<T>" in that, back to the example in the main comment, JSON and other reusable/self-describing/multipurpose/adjective serialization formats exist, and MIME types would not cover those cases.

[sullyj3]

Just encountered this while thinking about how to implement level loading. For now I'm creating levels as .png files, the pixels of which determine the locations of entities. I briefly considered adding a custom AssetLoader for my level struct, before realising it might clash with ImageTextureLoader

I'm not a rust expert, so I don't know if this is possible, but it seems like the API you'd want would be something like

let texture_handle: Handle<Image> = asset_server.load::<Image>("a_texture.png");
let level_handle: Handle<Level> = asset_server.load::<Level>("level1.png");

[mousetail]

The type Image or Level is already required by the Handle<Image> and Handle<Level> annotations.

[Demiu]

More asynchronous/ability to control loading files into memory on a per asset type basis. Right now the AssetLoader::load gets passed the entire file so getting a small part of a very large archive would waste a lot of memory. A custom AssetIO can be used, but as I understand it's a global resource. It would also break the separation of AssetIO handling loading from any path, unaware of the type.

I would highlight 3 layers to describe an asset: source (local file, network, another asset), format (json, png), actual game asset type (as mentioned: a model json vs an animation json). It would be nice if it was possible to change a layer to something else easily. An extension could just be used to set a "default" format and game asset type.

[bjorn3]

If you have a large archive file containibg multiple assets, you would add an asset loader which loads assets from said archive file, right? In that case the asset loader is free to only load parts of the asset.

[Demiu]

Well, no. AssetLoader::load has a bytes arguments which is just a regular slice reference to all the bytes of a file, already in memory. AssetLoader doesn't really load anything, it's really more of an AssetParser that turns bytes of a file into an Asset.
AssetLoader::load is called from AssetServer::load_async, which in turns takes the bytes from a AssetIo::load_path. The default AssetIo is FileAssetIo which just uses std::fs::File::read_to_end

[bjorn3]

I meant using an AssetIo for the archive, not AssetLoader. My bad.

[Demiu]

Yes, but you can only have one AssetIo as it is a owned by the AssetServer which is a resource. The implementation for a custom AssetIo could get quite big as you add more ways to load an asset. Plus from code requesting to load an asset you have no way to signal anything to the AssetIo, other than the Path (which maybe should be a URI or an URL instead).
Actually, no you can't because the AssetIo doesn't even get the label, so it has no idea which part of an archive is wanted.
Edit: And obviously this breaks separation of concerns heavily, if AssetIo could do it, now it has to parse a format, like an AssetLoader and match extension to code responsible for loading it, like the AssetServer.

[Neo-Zhixing]

Integration with a version control system.
Bevy asset should make it easy for the user to manage their large asset files.

Unity has integration with Perforce and Plastic SCM
https://docs.unity3d.com/Manual/Versioncontrolintegration.html

Godot has a Git Plugin to enable version control in the editor:
https://docs.godotengine.org/en/stable/tutorials/best_practices/version_control_systems.html

Bevy does not have an editor yet, but integration with a version control system would still enable features such as:

• Checking out the asset file from GitLFS on demand during development (when they're actually loaded)
• Generating incremental upgrade packages containing only changed assets
• Cache management for web-based games (Call CDN APIs to flush stale assets)

[kurtkuehnert]

Loading partial asset data.
I think that loading only a part of an asset and updating it over time, would be a useful feature of the asset system.

The main use case that comes to my mind is loading array textures, from multiple files, which may happen distributed over the lifecycle of the application.
Currently this is not supported. To work around this issue, you currently have to create a manual mapping of the individual layers and the actual texture and then after loading the layers copy the data over.

[BeastLe9enD]

Or for example for loading a model lod it would also be useful, otherwise you have to pack them in different files

[ameaninglessname]

I was expecting it could also do something like virtual asset

[TimJentzsch]

WASM support: As the requirements for serving the game on the web are quite different, I think it's worth to list this as separate requirement.
While it's often best to bundle the assets with the game files for desktop games, binary size matters a lot for the web (due to the long loading times).

Ideally, it would be very easy to apply different asset loading/preprocessing techniques on WASM builds compared to "normal" builds.
I assume it will be best to serve assets for WASM dynamically over the network instead of loading everything upfront.
Some of the asset processing can probably also be outsourced to the browser in the web context.

[viridia]

Better APIs for loading custom assets

Currently loading assets for built-in types such as images or meshes is pretty simple. You just load the asset, get a handle, and then you can plug that handle directly into the various systems that consume them. Those systems know how to check when the asset is loaded, and can utilize the data once it is available.

However, for custom assets - things like game maps, quests, dialogue trees, inventory items, skill and spell definitions, the process is much more complicated:

• You need a system to begin loading the asset. For assets that are loaded once at Start, this will run once; for assets which are loaded dynamically during play this will run on Update.
• You need a resource or some place to store the handle until the asset is loaded.
• You need another system to poll the handle until the asset is loaded.
• You need another resource or component to store the loaded data.
• You need some kind of state flag to let you know that loading has finished, so that you don't keep loading / getting the same asset over and over again.
• Finally, you need to write the code that actually uses the asset.

This is a lot of moving parts, but the situation is even more complicated if you're trying to use the asset inside of an async task - now you have to clone handles and possibly use some combination of Mutex/Rc to actually use the data within the task closure.

One thing that would simplify this is the ability to await on an asset load. That is, you could have a single system which runs at startup that spawns an async task. The task loads whatever assets it needs and waits on them - just like Promises in JavaScript. This eliminates the need for a second system to poll for the asset completion. Overall, the code is much simpler. Ultimately, you're going to want the task to create resources or components, but that's outside of the scope of this proposal.

Another improvement would be to have some easier way to insert assets into the world once loaded. In JavaScript, you'd do something like this:

assets.get(path).then(data => {
  commands.spawn().insert(data);
})

In other words, rather than having to build a separate system which polls the loading state, have some async closure which triggers when the load is done and which can issue commands.