Named Variants

[dgovil]

This proposal describes the addition of Named Variants to Rez, by changing the Variants list into a dictionary.

Motivation

Variants are very useful in Rez to provide mutually exclusive packages that correspond to a single version of the primary package.

However the current design has a few issues and limitations that come up in production use.  These issues often manifest in resolves that bring in the incorrect variant or require complex package.py setups to configure a package based on the variant chosen.

Named variants aim to provide simplicity to packages, and provide more intuitive behaviour in Rez.

Description

Named variants would change the package.py key for variants to a dictionary, instead of the current list.

For example, today we have

variants = [
	["foo", "spam", "maya"],
	["foo", "houdini"],
	["foo", "python-3"]
]

Changing this to a dictionary would give you the following

variants = {
	"maya" : {"requires":["foo", "spam", "maya"], "private_build_requires":["maya_devkit]"]},
	"houdini" : {"requires":["foo", "houdini"]},
	"standalone" : {"requires":["foo", "python-3"]}
}

This has several benefits over the current system of variants

1. Variants can specify variant specific metadata, which would remove a lot of complexity in package.py setups that make use of @late to handle differences dynamically to compensate. Here, the Maya variant can specify it needs the maya devkit without affecting the other variants.
2. If a new variant is added, there's no risk of the index changing which could in turn affect build scripts and dynamic commands. This makes it more resilient.
3. This could allow dependencies and user requests to optionally specify the variant by name like so eggs-1.0?standalone. This would prevent the issue with the current system where the Maya variant may be expected, but a Houdini variant is picked. This problem is described in the Wiki..
4. Different variants could still have the same package lists, but differ in terms of functionality, using the name as a differentiator.
5. Variant paths could use the name of the variant, making them shorter and more intuitive. Hashed variants solve the issue where variant paths are too long, or where asymmetry caused directory conflicts. Named variants have the same benefits but while providing more intuitive structuring.
6. Dictionaries do not imply order, which would remove one source of confusion with variant lists today, where the order is implied by index selection but not actually indicative of the order used in resolves. That said, dictionaries can be assumed to be ordered if we can assume Python 3.7

I believe this would solve many of the friction points people encounter with Rez, while having very minimal changes to the underlying system.

Backwards Compatibility

This proposal would allow for backwards compatibility with existing list based variants.
Rez could treat the list indices as dictionary keys.

Taking the example above,

variants = [
	["foo", "spam", "maya"],
	["foo", "houdini"],
	["foo", "python-3"]
]

It could stay unchanged. Rez would implicitly convert this to the following structure under the hood, thereby providing compatibility with the new system.

variants = {
	"0":{"requires":["foo", "spam", "maya"]},
	"1":{"requires":["foo", "houdini"]},
	"2":{"requires":["foo", "python-3"]}
}

From the standpoint of users of Rez, nothing would have changed for them for older packages, while newer packages could adopt the new functionality as they see fit.

Practical Use

This section describes how named variants may be used in practice

Build Time Variant Selection and Detection

Today variants are selected or detected by their index when building.

rez-build --variant 0 to build the first variant.
REZ_BUILD_VARIANT_INDEX returns 0 for the first variant.

With the proposed changes, these behaviours would still work due to the backwards compatibility behaviour above.

However named variants would also additionally allow for

rez-build --variant maya to build the maya variant.
REZ_BUILD_VARIANT_INDEX could return maya when building the Maya variant.

This would mean that adding new variants wouldn't risk breaking existing build scripts by accidentally changing the index.

It would also make it clearer when reading build scripts, what the variant is.

Requests

Having named variants would allow requests and packages to optionally specify a specific variant, removing ambiguity of which variant will resolve.

This could be allowed with a new delimiter such as ? in the form of <package_name>-<version>?<variant>.

This would also allow users to specify the functionality they want without specifying the version. For example, many of our artists don't care about the USD version they get, but they want to make sure they get the standalone version instead of the Maya one.

Now they could specify usd?standalone, whereas our scripts for Maya tools can more safely say usd?maya.

Alternate Versions

Versions allow alternates like python-3.7|3.9 which might also be future desired behaviour for named variants.

For the sake of keeping this proposal small, we suggest that the first iteration does not make use of alternates, till we can learn more about what combinatorial behaviour users expect.

Therefore , the first version would allow either specifying a single variant or not, but wouldn't allow specifying multiple choices.

Provides

Another common Rez request is giving packages the ability say they provide another package. e.g Maya could say it provides Python itself.

While Named Variants would not replace the need for provides, they do potentially allow for the ability to create a temporary workaround in the meantime.

The Python package could add a provided variant that does no build, and is basically empty. The commands function could detect the variant is provided and do no setup.

Maya could then add ~python-3.10?provided as the requires for Python. If a rez-request then has packages that need python, the variant here would prevent lib clashes.

I would reiterate that this is NOT a proposal to create provides , just to mention that it would allow for new workarounds in the meantime. When provides becomes an official Rez feature, it would allow skipping this altogether.

Variant Metadata

For the first version of this proposal, we suggest that the following keys be allowed for variants. Future keys could be added in future proposals, but I think simplicity would be good to start.

The keys we suggest are:

• requires
• build_requires
• private_build_requires
• arch
• platform

The keys we suggest omitting are:

• commands
• build_command
• name
• version

Since both commands and build_command can lookup the variant name while setting themselves up, there's no current benefit in setting them as per variant metadata.

Name and version would of course not make sense to vary per variant.

Hashed Variants

Hashed variants solve some of the issues that Named Variants would also aim to solve, namely path lengths and path conflicts.

The addition of Named variants should play well with hashed variants. The entire named variant could be hashed, much the same way as variants are today, preserving the same behaviour when hashing is enabled.

[herronelou]

I like this, I've been using hashed variants frequently because a lot of my variants require version ranges, but paths aren't happy with some of the special characters used to specify version ranges. A named variant would also solve this, while allowing for more human friendly names.

[bfloch]

Thanks Dhruv! Looks like you are taking a shot at some of our most wanted features in one go 🎉
My thoughts:

• Named Variants
  • No brainer
• Variants on metadata
  • Better syntax and adds access to more primitives to variants without "code". Nice.
• Explicit Variant selection
  • Drawback I see is that packages will now have knowledge about the internal named variants of another package, but at least studios can come up with their own naming conventions to keep it clean
  • Could it handle cases like Unexpected behaviour with variants #1458 or other problems where ambiguous resolution would lead to unexpected results? Or will it suffer from similar issues where lack of mutual exclusivity will creep in. I always felt we need both: Explicit variant selection and easier way to express mutual exclusivity
  • Maybe outside of this proposal, but I wish we had a primitive that would express: This variant is mutually exclusive to all the definitions in other variants, for packages that are not intersecting. This would mean that I don't need to re-release variants when introducing a new one and while not perfect work for many of the simple cases. Nice that you provide a framework that could be extended with the new syntax. E.g. without spending much thought, something like:

variants = {
	"maya":{"requires":["foo", "spam", "maya"], "exclusive_to": ["houdini/*"] },
	"houdini":{"requires":["foo", "houdini"], "exclusive_to: ["maya/spam", "maya/maya", "python/*"]"},
	"python":{"requires":["foo", "python-3"], "exclusive_to" ["*/*"]}
}
# Expect "foo" to be valid, but not the other variant packages.

• Bonus: Nicer "Provides" workaround
  • Neat. We had a similar concept called "dodging" but expressed it as a separate package with tons of mutual exclusive variants. So much cleaner

Wondering if it made sense to split it into separate proposals once we had an initial discussion on the package. So much good stuff in here.

[dgovil]

Thank, Blazej, for the kind words. For the couple questions you brought up… With regards to your point on mutual exclusivity, i don’t think it inherently solves the problem so much as gives a new knob to prevent it. E.g you could still get silent conflicts like you mentioned but you could now at least specify the exact variant either as a package dependency or an environment request in cases where ambiguity crops up. That would let you side step the issue. And I agree that it could potentially be an issue where variants become a leaky implementation, but unfortunately I’m already hitting that today where I need to add packages to variants that force a conflict so I know they’re mutually exclusive, and then adding those sentinel packages to dependencies requires to keep things behaving. So having the ability to request a specific variant would let me have a more foolproof solution. Hopefully that answers those questions, but let me know if not! Thanks again for the feedback

…

On Wed, Jun 28, 2023 at 1:21 PM Blazej Floch ***@***.***> wrote: Thanks Druhv! Looks like you are taking a shot at some of our most wanted features in one go 🎉 My thoughts: - Named Variants - No brainer - Variants on metadata - Better syntax and adds access to more primitives to variants without "code". Nice. - Explicit Variant selection - Drawback I see is that packages will now have knowledge about the internal named variants of another package, but at least studios can come up with their own naming conventions to keep it clean - Could it handle cases like #1458 <#1458> or other problems where ambiguous resolution would lead to unexpected results? Or will it suffer from similar issues where lack of mutual exclusivity will creep in. I always felt we need both: Explicit variant selection and easier way to express mutual exclusivity - Maybe outside of this proposal, but I wish we had a primitive that would express: This variant is mutually exclusive to all the definitions in other variants, for packages that are not intersecting. This would mean that I don't need to re-release variants when introducing a new one and while not perfect work for many of the simple cases. Nice that you provide a framework that could be extended with the new syntax. E.g. without spending much thought, something like: variants = { "maya":{"requires":["foo", "spam", "maya"], "exclusive_to": ["houdini/*"] }, "houdini":{"requires":["foo", "houdini"], "exclusive_to: ["maya/spam", "maya/maya", "python/*"]"}, "python":{"requires":["foo", "python-3"], "exclusive_to" ["*/*"]} } # Expect "foo" to be valid, but not the other variant packages. - Bonus: Nicer "Provides" workaround - Neat. We had a similar concept called "dodging" but expressed it as a separate package with tons of mutual exclusive variants. So much cleaner Wondering if it made sense to split it into separate proposals once we had an initial discussion on the package. So much good stuff in here. — Reply to this email directly, view it on GitHub <#1503 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB4XUR5CTU2FGRUHUM4MNLXNSG33ANCNFSM6AAAAAAZV6XY4U> . You are receiving this because you authored the thread.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1503/comments/6308675@ github.com>

[JeanChristopheMorinPerso]

Thanks @dgovil! I think this is a useful feature and is something I wanted to introduce a long time ago too. I think we have some GitHub issues where this was mentioned and our old mailing list probably has some references to dict-based variants.

Dictionaries do not imply order, which would remove one source of confusion with variant lists today, where the order is implied by index selection but not actually indicative of the order used in resolves.

This could make the builds less reproducible I think. At anaconda, we have this issue where conda-build will build variants in an unspecified order that changes all the time. It's quite annoying. I would prefer if the order was kept as is.

REZ_BUILD_VARIANT_INDEX could return maya when building the Maya variant.

It would be safer to introduce a new variable, REZ_BUILD_VARIANT_NAME or something like that. We don't know how REZ_BUILD_VARIANT_INDEX is being used in the wild and we might have users that convert the value into an int.

This would mean that adding new variants wouldn't risk breaking existing build scripts by accidentally changing the index.

I'm not sure to follow what you mean here. Can you clarify what you mean please?

#Variant Metadata
For the first version of this proposal, we suggest that the following keys be allowed for variants. Future keys could be added in future proposals, but I think simplicity would be good to start.

The keys we suggest are:

• requires
• build_requires
• private_build_requires
• arch
• platform

What is arch and platform here and how would they work/what would they do?

General questions/comments:

• What happens if both dict variants and requires, build_requires, etc are present?

variants = {
    'var1': {'requires': ['packageA']}
}

requires = ['asd']

private_build_requires = ['other_package]

• Should we limit what characters can be used in variant names?

Lastly, regarding mutual exclusivity, I would put that aside for now. I'd prefer if we could make a separate proposal to rework how variants work so that they behave more like what our users expect compared to how they behave today. But that'll be a huge project. @dgovil's proposal lays the ground for this, or at least it opens one door. I think we'd still want to suppost list based variants.

Thanks!

[dgovil]

@JeanChristopheMorinPerso Hmm, maybe I mis-rememberred or its something custom someone did internally. I'll remove the mention of arch and platform from the proposal and add mention of dictionary ordering as well

[JeanChristopheMorinPerso]

One way to keep the variants order would be to specify the variants like this:

variants = [
    {"name": "maya" : "requires":["foo", "spam", "maya"], "private_build_requires":["maya_devkit]"]},
    {"name": "houdini" : "requires":["foo", "houdini"]},
    {"name": "standalone" : "requires":["foo", "python-3"]}
]

Another alternative would be to define a Variants class and use it like this:

variants = Variants(
    maya={"requires": ["foo", "spam", "maya"], "private_build_requires": ["maya_devkit]"]},
    houdini={"requires": ["foo", "houdini"]},
    standalone={"requires": ["foo", "python-3"]}
)

Using a class could open the door to some cool features.

Alternatively, we control everything, so we could always monkeypatch the dict class on Python <3.7 to replace it with an collections.OrderedDict when loading and evaluating the package definitions.

[dgovil]

You could do it like so too

variants = [
     Variant("maya", requires=["foo", "spam", "maya"], private_build_requires=["maya_devkit"]),
     Variant("houdini", requires=["foo", "houdini"],
     ...
]

That has a couple advantages:

1. Ordering is maintained even on Python<3.7 since its just a list
2. The keys could be predefined so that typos would be mitigated. e.g You couldn't do reqqires instead of requires.
3. The class could have methods to access data like the variant path, which would give a nice way for studios to customize behaviour if needed. So {root} on a variant could be queried by doing VariantInstance.GetRoot(). The default for GetRoot would be the

But also, this would allow for the current list based variants to be transformed from ["foo", "spam", "maya"] to AnonymousVariant(["foo", "spam", "maya"]) implicitly, where AnonymousVariant would have GetRoot() defined as os.path.sep.join(**self.requires), preserving the current behaviour.

Anyway, just a thought riffing on your idea @JeanChristopheMorinPerso

[JeanChristopheMorinPerso]

I see where you are going! It would make sense. I didn't go with this option because on the code side of things, getting the variants would be slightly more awkward if the structure is kept as a list. Though, nothing stops us from converting the list into another structure type, either a dict or a class or something like that.

[dgovil]

Yeah. Personally I prefer the dictionary option for the simplicity for both the repo and users, but if we wanted to keep Python 2 support AND ordering, I think a list would make sense, or ask Python 2 users to use OrderedDict for their packages.

Though it would be nice to take a solution that doesn't need to consider Python 2 so we're not designing around "legacy" setups (legacy in quotes because I know its still in use). Being able to use the reference platform minimum of Python 3.7 would be nice.

[maxnbk]

AFAIK order is part of the determinism of the resolve, but as @nerdvegas was fond of saying, deterministic doesn't mean intuitive, which is really the issue for most people for whom this feature would be useful.

Though frankly the entire idea is compelling enough that it almost strikes me as a solid enough reason to drop py<3.7 officially, and keep the indexing for whatever the solver needs it for.

[dgovil]

I would very much be a fan of dropping Py<3.7 if that's something the maintainers are okay with. There's so many quality of life improvements that it would make it easier in general to do other contributions to Rez.

[herronelou]

Just chiming in to say that I'd rather not drop python 2 support for this, while we're working he'd to eliminate python 2 from our tech stack it's not going to be done for a while, and I've been eagerly awaiting certain new Rez features like "provides", so if that came out without py2 support that would be a problem.

…

On Fri, Jun 30, 2023, 09:08 Dhruv Govil ***@***.***> wrote: @JeanChristopheMorinPerso <https://github.com/JeanChristopheMorinPerso> Hmm, maybe I mis-rememberred or its something custom someone did internally. I'll remove the mention of arch and platform from the proposal and add mention of dictionary ordering as well — Reply to this email directly, view it on GitHub <#1503 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAV5BB6QII4AOZ4VCNXC3XN32ZDANCNFSM6AAAAAAZV6XY4U> . You are receiving this because you commented.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1503/comments/6327446@ github.com>

[JeanChristopheMorinPerso]

There is a lot of chances that provides will come after we drop support for Python 2. We already started talking about dropping support for Python 2.7 and the result from our last discussion is that the days are counted. Anyway, for now we are keeping support for it, but you can't expect us to keep maintaining and support rez on Python 2.7 forever.

[herronelou]

I'm one of those who care to somewhat support python 2, but currently, all of our variants ARE lists, if we want to switch to dicts and we care about python 2 and we care about ordering, we can use an ordered dict. It's less pretty but it should work.

…

On Wed, Jul 19, 2023 at 9:19 AM Dhruv Govil ***@***.***> wrote: Yeah. Personally I prefer the dictionary option for the simplicity for both the repo and users, but if we wanted to keep Python 2 support AND ordering, I think a list would make sense. Though it would be nice to take a solution that doesn't need to consider Python 2 so we're not designing around "legacy" setups (legacy in quotes because I know its still in use). Being able to use the reference platform minimum of Python 3.7 would be nice. — Reply to this email directly, view it on GitHub <#1503 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAV5E2BZSHBFT25RJ5RTTXRACHTANCNFSM6AAAAAAZV6XY4U> . You are receiving this because you commented.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1503/comments/6490771@ github.com>