-
Notifications
You must be signed in to change notification settings - Fork 69
Asset Resolution and Version Selection at Animal Logic
In Animal Logic Terminology Version Selection refers to choosing a specific version of an asset from a set of available versions. Asset Version Packaging (or usually just Packaging, but it's nice to have a name that disambiguates) refers to the aggregation of a number of version selections in one place in a way that can be used from our tools. Usually this aggregation/grouping has some kind of functional meaning (e.g "the latest approved versions of all of the Lighting assets in Shot X", or "The latest delivery of assets for the Gotham Downtown Set from the Lego Batman Environment Team" ). The intent of Packaging is (to the best of my knowledge) similar to the process known as "Pinning", "Bundling" or "Freezing" in other studios, the implementation/mechanics of which may vary - it may involve (for example at Pixar) writing a folder containing a set of symlinks to particular asset versions.
Approaches to Version Selection include:
1. Selection based off asset metadata such as:
- creation time ("give me the latest assets at date/time x")
- status (e.g "delivered" , "approved")
- dependencies (we may want to e.g. package the dependencies of a render which has been approved as a "delivery package")
2. Selection based on dependency analysis
- e.g "get me the latest dependent assets that are compatible with Asset X" (analogous to software package management with rez, RPM, yum etc)
3. Explicit Version Selection:
- Often done manually through a UI of some kind
Other than manual selection which has to have it's results recorded directly, version selection can either be
- dynamic (occurs at asset resolution time)
- static (or cached) calculated once and stored as an artefact which can be retrieved later
In the Animal AMS (ARK), many versions of assets are available at the same time on the filesystem, for example, here are 3 versions of a single asset:
| URI/Asset Identifier | Resolved Location |
|---|---|
| ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset)?version=1?ext=.abc | /jobs/sydney/this/is/an/asset/theAsset_v001.abc |
| ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset)?version=2?ext=.abc | /jobs/sydney/this/is/an/asset/theAsset_v002.abc |
| ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset)?version=3?ext=.abc | /jobs/sydney/this/is/an/asset/theAsset_v003.abc |
In our Scene Description files (Glimpse, USD, others), there are a few ways of referencing these assets
| Type | Example |
|---|---|
| Non-versioned URI | ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset) |
| Versioned URI | ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset)?version=3 |
| Filepath | /jobs/sydney/this/is/an/asset/theAsset_v003.abc |
In the case of Non-Versioned, version selection is required - if there is no explicit configuration provided, our Asset Resolver system will choose the latest version available (we can also block this "defaulting to latest" behaviour if we like)
When we set up asset resolution, it's configuration can include references to packages (and other sources of version information) Their content is a set (any asset can only appear once) of asset Identifiers and versions as mentioned above. A simple example looks something like this (where the first column is the AMS Identifier, and the second column the version to use)
/TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset) : 3
/TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(another)/LEAF(asset) : 7
For any AMS identifer, we force a specific asset version to use (version 3 and version 7 respectively in the example above). Note that this AMS identifier is not a full URI as described, it's only the part that identifies the "Asset" in our AMS DB rather than a full URI which references a specific filepath and/or versions (full spec/discussion of our URI schemes, similar to Bluesky is outside the scope of this discussion)
This package can be stored in a file or in a Database - the important thing is that we can pass a reference to it to our AssetResolver before a Scene Description file is opened, and for every input Asset Identifier, the package is checked, and the corresponding version is used if found.
It would be reasonable to imagine that this fairly hardcoded mapping between the AMS Identifier part of the URI and a version, which modifies a URI from: "ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset)" to "ark://TYPE(this)/SUBTYPE(is)/SUBSUBTYPE(an)/LEAF(asset)?version=X" could easily be generalised into a more general purpose expression system to match and replace patterns, and we've gone part of the way along this path
Packages are generated in 2 ways:
- A fairly rudimentary UI in the AMS where templated Asset Identifier inputs can be versioned, usually used by Department Coordinators
- via an API which allows arbitrary packages to be authored
There are a few limitations with the current packaging system
- Although Packages themselves can be recursive, the APIs for accessing them return a single flat list of Asset Identifier, version pairs. This makes it impossibly to have multiple versions of the same asset in a given context (e.g a shot or set). One way it's been worked around in the past is to have "bake" stages in the Scene Building pipeline where scenes are written out with filepaths or versioned URIs
- Traditionally, the list of input AMS Identifiers to a Package is not generated by introspecting the Scene Description data or through dependency analysis, but has come from a predefined, simplified approximation which predates the use of deeply nested/recursive references in our assets
We have built an AssetResolver library which is used directly by many of our proprietary apps and tools such as Glimpse, ALF etc. We haven't so far found it necessary to implement Resolver plugins in DCC apps like Houdini and Maya.
The AL Asset Resolver system is responsible for taking URIs such as the ones above and translating them to paths on a file system. It allows us to:
- keep our asset references multi-site
- Control versioning policy by setting rules dynamically
When setting up a render or other Asset Resolution heavy process, we:
- Resolve all of the assets once in a prepass which caches the URI->Resolved Location lookup in memory
- Write that cache to disk
- Use that cache for all frames and passes of the render.
We do this to:
- improve performance and eliminate load on our Asset Management DB
- ensure that asset versions don't change part way through a multi-part process where temporal consistency of assets is important (e.g a render) when using dynamic version selection rules (e.g for "use latest" - caching this "freezes" the assets to the latest version available at the time the cache was generated - this is similar in intent AFAIK to what Pixar is doing with symlinks)
We can manipulate this cache to copy the contents to a new location (like a primitive version of usdz) and update the serialised cache to point to the new location. This has a couple of nice properties:
- It doesn't modify the Scene Description files themselves - when they go through the Asset Resolver system the original URI lookups are redirected to another location on disk
- We can record all asset references without a complex and error-prone introspection/dependency analysis approach (although we do have to open all of the references, which in the case of USD would mean expanding all payloads, or in the case of a render scene triggering shader compilation etc)
The system can be configured to look up specific packages and other sources of version information via it's API. This configuration can be represented as a blob of JSON for serialisation. The configuration is stored in different places depending on the use case - Glimpse stores it's configuration in it's Render Scene Description files, in Maya it is stored in a specific maya node, and shared between multiple tools/libraries
Some relevant explanatory posts from Pixar on USD-Interest:
-
"We find it invaluable to bake a shot's asset asset (sic) paths before sending the shot off to the farm to render.
We refer to it as "freezing" rather than baking because it is freezing the floating (searchpath) asset references to a particular version. This is done not so much for performance as it is for change isolation, though it may result in a modest performance boost also. The way we do it leverages "dependency analysis", which we provide the basic building block for in UsdUtilsExtractExternalReferences() (which you can use in an iterative fashion), and basic searchpath resolution behavior. We always have an early entry in our resolver searchpath be "./pins", and we use the results of dependency analysis to deposit symlinks in that dir that will "pin" all the assets to the versions resolved at the time the job launches. One nice thing about this approach is that it requires no modifcations to the resolver plugin (assuming your resolver isn't one that manufactures files on the fly)" See here for full post -
"You've addressed that part of the problem by relying on a db (presumably one you're already touching as part of publishing a new version, so not much more awkwardness). Although we haven't re-profiled in several years, we invoked the ire of our system facilities group back in Presto days when asset-resolution involved talking to databases, because at renderfarm-scale, we wound up overtaxing the database. So we have a firm policy of no database lookups for asset resolution" see here for full post
-
"In our own custom resolver, we rely on external state to resolve versions of assets. Our AssetResolver, which can be configured by a program, or falling back to a default configuration based on an env-var, will search a set of relevant directories on the filesystem for "asset pins" (which are really just symlinks managed by an API that consults a database). When resolving the asset-path @Buzz/usd/Buzz.usd@, the resolver will examine the pins to see if there is a pin for "Buzz", and if so, redirect through that symlink; otherwise, it finds the "stable" version of Buzz on our "model farm"". See here for full post
To summarise my understanding, Pixar have a system that relies on an external process querying their AMS DB, writing out symlinks to snapshot the "right" versions at particular times (e.g every night), and then configuring their Resolver via search paths to look at the file locations where those symlinks live. Their asset references are mostly relative, and contain no version information. They do use USD's ARAssetInfo to store debug information about what version of an asset was in the symlink at the time a USD file was generated
In comparison:
- AL doesn't use symlinks at all in any Asset Resolution/Packaging related workflows
- We do database lookups at Asset Resolution time - but try not to do them more than necessary (once)
- We don't use search paths in our resolver - all assets are referenced explicitly by URI. This can make our USD files more dependent on having our AMS database available, and make the files less portable - unless you use the "archiving/caching" solution as described above. We still don't have a very elegant solution to allowing URI based assets to be redirected to a users work area for example (allowing local versions of assets to override AMS managed ones), which might be referred to as "Context Specific Asset Resolution"
We've implemented a USD AssetResolver on top of our Asset Resolver library
Some feedback on this:
-
The USD Asset Resolver API contains a large number of pure virtual methods that we don't use but have had to implement
-
The USD API is called many times and relies on implementer caching to reduce server load/repeated calls etc (Possibly ARResolverScopedCache already provides this? In any case this makes it a bit tricky to debug
-
In environments where our proprietary tools "own" any USDStages created, we can control Asset Resolver configuration by creating ResolverContext objects and adding whatever configuration we like.
-
When using OSS tools such as usdview, the API to ResolverContext does not contain any standardised way of getting or setting configuration, so we need to rely on "hacks" like reading environment variables. A suggested improvement is covered here, but the general idea would be to just have some methods on the ResolverContext API like:
- setConfiguration(std::string&)
- std::string getConfiguration()
- etc
-
in AL_USDMaya, where there may be many ProxyShape/Stage combinations with different Asset Resolver configuration, we rely on a maya string attribute called "assetResolverConfig" - see code. We abuse the [ConfigureResolverForAsset] (https://github.com/PixarAnimationStudios/USD/blob/master/pxr/usd/lib/ar/resolver.h#L70) method to "inject"configuration (in our USD Asset Resolver code, if we detect the input string is a blob of JSON rather than a filepath, we use it to configure the resolver). This abuse of the available API makes us feel uneasy!
-
Unlike in Glimpse, where a Render Scene Description is read in a specific order, and we can place Render Globals and AssetResolver Configuration early on in that file and be confident they will be read before and files are resolved or pixels rendered, we can't store our Asset Resolver config easily in a USD file as it will suffer from the classic "bootstrapping" problem and have to be read twice - once to configure the Asset Resolver , and second to resolve references using that Asset Resolver configuration
-
We've wanted to wrap our USD AssetResolver in a Maya Asset Resolver so that references that survive translation to Maya (e.g texture paths on lights) could be handled uniformly.. see https://groups.google.com/d/msg/usd-interest/_WVE4tHojdc/nUTgG2aLCQAJ. I've promised Spiff I would log an issue for this..
-
Documentation on the USD Asset Resolver system is fairly limited - Spiff has done a great job of explaining it in many posts on the USD Google group, this probably being the most comprehensive, but it's still difficult to see it all in one place