Improve docs

[nerdvegas]

Please update this discussion with cases you've come across where you had to dig into code / ask around to figure out how something works, and where that info is not in the wiki (or, where the info is there but is misleading or difficult to understand). With enough participation, we should be able to identify common themes and thus be able to target our efforts on doc upgrades in a way that sees the most benefit.

When adding to the discussion, please provide links to existing docs where appropriate, or suggest where the docs you ned should be, if they're currently missing.

[nerdvegas]

I'll start simply with a dump of the current doc issues we have open:

• Document various strategies to release alpha/dev/beta packages with rex #1218
• Document package filters and orderers #1217
• improve docs wrt commands() not run on built package #1198
• Document how packages can be tested using rez-test #996
• .ignore pkg repo functionality not documented in wiki #856
• Document all variables available in package definition functions #805

[ColinKennedy]

Hey Allan, thank you for making this discussion to bring visibility to this. IMO the documentation in Rez currently is actually really good, but all things can be improved.

Here's a list off the top of my head of things which might be worthwhile to spin off as separate issues:

New Documentation Ideas: Basics / Workflow

• Rez extensions, how-to
  • Recommended: A new wiki entry
• The differences between "developer / source" and "installed / built / released" packages
  • Recommended: A new wiki entry. Maybe just a page called "Common Rez Terminology". Or something
• A workflow for getting test / feedback for packages
  • Relevant Rez tutorial: https://www.youtube.com/watch?v=rPe6JpzE_Nc
  • Relevant question post: Another way to test packages in dev stage #1263
  • Relevant thread: https://academysoftwarefdn.slack.com/archives/C0321B828FM/p1648615929395209
  • This comes up fairly often in forums. While Rez probably shouldn't claim "this is the way to do it", having a go-to answer to link to when people ask would be a great thing
• Rez and common build systems
  • CMake - How to approach this from scratch
    • Relevant video: https://www.youtube.com/watch?v=ZMRTFIkOJdU

New Documentation Ideas, Continued: Technical Literacy

(Making users more comfortable with tools that already exist)

• Best practices for contributing to Rez
  • Reference: https://academysoftwarefdn.slack.com/archives/C0321B828FM/p1649259168023869
• rez-pip common issues
  • e.g. how to handle relative files that live outside of the wheel / dist using pip_install_remaps
  • Common ways of determining if a failed package is due to user error or is a rez-pip bug (probably user error)

Existing Documentation Improvements

• Rez's Sphinx API documentation is out of date
  • Auto-update the Sphinx API documentation for Rez #1254
• Clarify that a package preprocess and package_preprocess_function run not just during build (docs imply it, in a couple places)

[Jawabiscuit]

My $0.02 would be to begin writing some Windows friendly docs starting with some workflows that don't apply directly to Windows from Linux. For example, I had to create substantial msvc/vs packages so that I could build with the MS compiler, Windows sdks, debug in VS, and use their built-in CMake. This process really highlighted how tedious it can get to rez-ify "all the things" on Windows. I wouldn't mind contributing, and I know there are some more experienced Windows users out there.

[ColinKennedy]

@Jawabiscuit Could you please point out which topics you'd like covered on Windows / think needs special consideration for Windows? I can guess but it'd be good to get a running list so we can focus our efforts there.

[Jawabiscuit]

Well, for starters, the Getting Started section was less then pleasant to work through exactly as it is on Windows since I needed to create the msvc package I mentioned. Only having cmake wasn't enough to build a hello world with rez. I needed the compiler and linker too which is probably the case on Linux but I never had to go through the "Getting Started" phase on Linux at work. Some people may encounter that and pivot to using python build scripts and that turns into a bit of a maze once you start looking at obsolete rezbuild.py examples and other utilities out there. Someone might go down that path only to wind up using a full featured build system like cmake again, never getting rid of the buildup of cruft.

Ideally, the docs should ensure the best information is the most convenient to access to avoid users going down many google rabbit holes. Maybe that means maintaining links to credible external sources.

I thought about creating a YT video actually before you released your videos on Linux. Now I'm a bit more inspired :) but I just need to find the time. I also wanted to gain more experience and not spread inaccurate information since there isn't much info out there for Windows users. I feel I'm much farther along that path these days.

What I learned outside of the wiki I got from monitoring and searching the Slack channel. If I were to make a list, which I could do, I'd start with the recent influx of questions there. Some are very reminiscent of what I went through. That sounds like it would go in the FAQ docs section and hopefully it wouldn't stray too far into the "hard to use" territory but rez is currently "hard to use" on Windows and the uptick in development will only see the experience getting more volitile unless thoughtful countermeasures are taken. I see discussions where that is the focus so I'm not overly concerned.

[ColinKennedy]

that turns into a bit of a maze once you start looking at obsolete rezbuild.py examples

That's true, I had forgotten about that. I was using Rez at the time when Bez was deprecated in favor of build_command but in retrospect, I'm not sure how well that new workflow is documented.

The overall impression I'm hearing from you, and I hope this is right, is the the "Getting Started" page is a step in the right direction but A. needs a bit of updating B. needs to cover a wider range of topics and special notes for different OSes, where applicable.

[Jawabiscuit]

Yeah, sorry I was formulating as I was writing there, so it's uncondensed :) Like many other gihub repos, a bit of cross platform documentation is usually present in Getting Started, FAQ, or other documents.

In summation:

1. Better cross-platform getting started with CMake
2. Maybe a Python getting started if it's determined CMake's scope is too big, or in addition to CMake
3. Maintainable, easy to access, links (submittable via PR? located in REAME?) to other "unofficial" but credible resources
4. Gather recent recurring, many are Windows, issues being brought up in Slack and document them in FAQ

[instinct-vfx]

There are many special cases on Windows and i agree that the docs deserve a lot more love in that area. I was wondering if this might grow into a hard "to use" discussion quickly. I think this should become a page in the Wiki or an issue where the description contains all the wishes merged with checkboxes so we can work through them easily.
Thoughts?

[ColinKennedy]

This next point I'm about to bring up may be out of scope for this discussion but, personally, I find updating Wiki pages for GitHub fairly unintuitive because I'm not sure how to preview the changes that I want to make, rendered in GitHub, prior to submitting PRs back to Rez.

If the pages were .rst files which could be built / verified with Sphinx locally, personally I'd really like that. Being in Sphinx also has several advantages, such as cross :ref:-ing to and from code. We build Sphinx docs for Rez already so it's not like there's no precedent either - we would just need to lean on those processes a bit more.

[JeanChristopheMorinPerso]

@nerdvegas readthedocs already supports versions and PR previews.

[ColinKennedy]

Versioned documentation of Sphinx-generated markdown pages using m2r? Now there's a project!

I used to like GitHub wiki until I realized you can cross reference to and from API pages using :ref:, :doc:, :mod:, :func:`, etc. Being able to reference hand-written documentation in Python code and refer to Python functions in hand-written docs is so great for readability as a user and is an advantage that I think GitHub wiki cannot easily provide.

However I'm seeing here / there you can get all of that while using markdown. As long as it "plays nice" with the Google-style Python API documentation which exists so far and link between them, I see that as a marrying those 2 technologies

[JeanChristopheMorinPerso]

Nowadays, the "official" way to write markdown in Sphinx is to use MySt. Anyway, it's details and these things shouldn't really be the focus of this discussion I think.

[ColinKennedy]

When you say "official", can you clarify what you mean? Sphinx's docs describe it as an optional extension that you have to separately pip install. I'm not on the pulse of Sphinx development so I might just be unaware. It's the first time I'd heard of MySt.

Anyway as you say, it probably isn't meant to be a focus, I was just curious. Vanilla Sphinx or a markdown extension to support what we've made so far IMO are strong routes for improving Rez.

[JeanChristopheMorinPerso]

The fact that it's even documented in the official Sphinx docs is a pretty good statement of the fact that it's a recommended extension. I should have said recommended though, as it's a better/more nuanced word. MySt is also actively developed and maintained.

[herronelou]

I come from many years working at studios with rez-like package managers but had never used rez until a few weeks ago.
I have recently started at a studio where there was no management being done other than having git repos, and felt a strong need to implement something, which got me looking into rez as the task of developing something similar was daunting.

I have read the wiki multiple times, yet I still have to try a few different pages every time I try to get back to some info I read before, I can never seem to find it right away, having the docs in a sphinx-like documentation which provides search would be nice.
The studio is on Windows, and I definitely ran into a few issues mentioned above that made rez a bit more challenging to implement. The getting started rez-bind commands mostly failed, and those that succeeded put the exact windows version as a variant, which turned into a bit of a headache later on as windows likes to update all the time.
I didn't even try setting up cmake on windows, and just went with a build.py.

As my needs started to get past what was provided by the command line tools, I saw myself wanting to use the API, but found no documentation for it besides the autodoc at https://nerdvegas.github.io/rez/index.html, having some pointers giving a few recipes on how to do common operations in API would be great (how do you list existing package families? how do you find the versions within these? how do you do a resolve? What kind of callbacks can perform?)

Reading the documentation at https://github.com/nerdvegas/rez/wiki/Variants#disk-structure led me to believe that there was something called 'base'. I thought this was great, so that I could have variants, each containing their own payload, but I could also have some common files directly in the base, but turns out we can't use {base} in the package.py. To this day I still don't know if it's possible, or if every single file needs to be duplicated for each variant.

I ran into quite a few issues with pip-rez, and the hashed variants. Pip-rez often puts incorrect dependencies or variants in the package.py that I mentioned in another post (like always putting the python version it ran on in the variants, even if the pip package is actually compatible with a range of versions). It's easy enough to change the package.py, but the hashed variants make moving the payloads quite painful, as there is no visibility into what the new hash should be (sure you can load the package, and it will print a message saying the path it tried to find, which gives the hash, but that's really time-consuming). I had to dig into the code to find the hash function and write a small utility that will regenerate the hashed payloads for me.
Talking about rez-pip, I ran into an issue which I believe is due to pip itself, not rez, but on windows when it generates executable python files as .exe it encodes the path to the python executable in there, which makes it nearly impossible to run certain packages with a different python executable, I had to also write some tools that will go into these .exe and change the shebang pointing at an absolute path to just say "python", so that I can use these utilities with whichever python executable is in my rez env.

There are also certain topics that I'm still confused about to this day, for example how to package up a dcc or some other external tools.
Right now I create the package.py manually, generate a timestamp for it, and place the files in the correct path. Am I supposed to have some setup where I can do a rez-release for these? I'm still unsure how to manage these packages.
I'm still not 100% certain what is the best way to make test releases so that some users can test certain fixes. If I release 1.2.beta, later it won't allow me to release 1.2 as that's a lower version, so I have to put an extra argument in the release command to allow that, but still 1.2.beta resolves higher. I saw that the config allows to set package filters, so I'll have to play with that and figure out the proper flow, but having a suggested "recipe" in the docs explaining the whole workflow: dev, local test, network install, network test, release, would be helpful, even if it's just a suggestion.

Another thing I would love to see in the documentation is some common mistakes and traps to avoid.
The studio I now work at tried to setup rez a few years ago, and gave up quickly. Looking at the setup, they would often set their dependencies to absolute versions, which turned into a nightmare every time they would want to release something as they had to go back and update all the chain of dependencies.

[nerdvegas]

Having any form of common storage between variants (that isn't simply a common package requirement) would have big implications for the project and would require a lot of effort to implement - effort that could be spent on other, more useful things. I don't mean to say that a reusable base is necessarily a bad idea, but in terms of small gain for large effort, this is probably right up there!

It also seems odd to me that a studio would be concerned with disk space taken up by multiple copies of python source, when they deal regularly with vastly larger files (ie renders)..?!

[herronelou]

It is odd to me as well, no need to go down the specifics on a public forum :)

[instinct-vfx]

I have read the wiki multiple times, yet I still have to try a few different pages every time I try to get back to some info I read before, I can never seem to find it right away, having the docs in a sphinx-like documentation which provides search would be nice. The studio is on Windows, and I definitely ran into a few issues mentioned above that made rez a bit more challenging to implement. The getting started rez-bind commands mostly failed, and those that succeeded put the exact windows version as a variant, which turned into a bit of a headache later on as windows likes to update all the time. I didn't even try setting up cmake on windows, and just went with a build.py.

rez-bind is to be deprectated, but i wholeheartly agree that getting started on windows has some pitfalls and also some documentation dead-ends like the quickstart.

As my needs started to get past what was provided by the command line tools, I saw myself wanting to use the API, but found no documentation for it besides the autodoc at https://nerdvegas.github.io/rez/index.html, having some pointers giving a few recipes on how to do common operations in API would be great (how do you list existing package families? how do you find the versions within these? how do you do a resolve? What kind of callbacks can perform?)

Agreeing with Allan's comment here, great point. A quick tip: The docstrings are generally really good, and one thing that particularly gave me a lot of insight is reading rezconfig.py's docstrings. Any feature that has a config is in there and they are usually described pretty well there.

I ran into quite a few issues with pip-rez, and the hashed variants. Pip-rez often puts incorrect dependencies or variants in the package.py that I mentioned in another post (like always putting the python version it ran on in the variants, even if the pip package is actually compatible with a range of versions). It's easy enough to change the package.py, but the hashed variants make moving the payloads quite painful, as there is no visibility into what the new hash should be (sure you can load the package, and it will print a message saying the path it tried to find, which gives the hash, but that's really time-consuming). I had to dig into the code to find the hash function and write a small utility that will regenerate the hashed payloads for me. Talking about rez-pip, I ran into an issue which I believe is due to pip itself, not rez, but on windows when it generates executable python files as .exe it encodes the path to the python executable in there, which makes it nearly impossible to run certain packages with a different python executable, I had to also write some tools that will go into these .exe and change the shebang pointing at an absolute path to just say "python", so that I can use these utilities with whichever python executable is in my rez env.

Hashed variants solve the issue that rez translates requirements into variants and that some requirement range syntax can not be used in folder names. It also shortens the path which can be a live saver on windows.
In regards to variants for rez-pip i agree. Before rez-pip was updated a lot we had an internal rez-pip implementation that derived the level of variant from multiple sources:

1. Scan all files. If any binaries (.dll, .pyd, .exe etc.) are found lock to minor
2. Scann the different meta data available to find out compatibility and use that as the variant range.

The main problem is that 2. is very unreliable and completely up to the author of the package and oftentimes horribly wrong and that dealing with the different meta data formats is the stuff of nightmares. That said i would still like to go back there and try and improve that area. It's not so much about code duplication but about having to re-release so much which can be a real pain.

There are also certain topics that I'm still confused about to this day, for example how to package up a dcc or some other external tools. Right now I create the package.py manually, generate a timestamp for it, and place the files in the correct path. Am I supposed to have some setup where I can do a rez-release for these? I'm still unsure how to manage these packages. I'm still not 100% certain what is the best way to make test releases so that some users can test certain fixes. If I release 1.2.beta, later it won't allow me to release 1.2 as that's a lower version, so I have to put an extra argument in the release command to allow that, but still 1.2.beta resolves higher. I saw that the config allows to set package filters, so I'll have to play with that and figure out the proper flow, but having a suggested "recipe" in the docs explaining the whole workflow: dev, local test, network install, network test, release, would be helpful, even if it's just a suggestion.

I am currently working on a tutorial to package up python on windows, that will give some insight for this too i think.

Another thing I would love to see in the documentation is some common mistakes and traps to avoid. The studio I now work at tried to setup rez a few years ago, and gave up quickly. Looking at the setup, they would often set their dependencies to absolute versions, which turned into a nightmare every time they would want to release something as they had to go back and update all the chain of dependencies.

That is also a very good idea. There are many gotchas. Oftentimes for good reasons, but not well documented and that can be very frustrating.

Thanks for all the feedback a lot of good info there!

[herronelou]

@instinct-vfx you mentioned you were working on a tutorial, any updates on that? I'd still be keen to see

[instinct-vfx]

Have not gotten to it yet sadly. I do have the first part - which is setting up rez on windows in a basic fashion. I did the python packaging one as a hands-on livestream that was not recorded. I want to turn that into a tutorial but am not sure when i will get to it.

[bpabel]

The current Rez Config section on Package Filters is a little confusing -- https://github.com/nerdvegas/rez/wiki/Configuring-Rez#package_filter

It includes an example that isn't valid python and references different rule types (e.g. glob(), regex()), yet it isn't clear how to actually use those types in the config.

[instinct-vfx]

As per discussion on Slack, we should improve documentation around build_requires vs. private_build_requires and possibly also adapt samples to use private_build_requires except specifically require otherwise

[ColinKennedy]

Adding relevant link, for posterity - https://academysoftwarefdn.slack.com/archives/C0321B828FM/p1655923271163899

[herronelou]

There is a bit of conversation about certain shell pitfalls to avoid in #1234, some of which might be worth including in the docs.

[herronelou]

There is almost nothing in the docs about 'has_plugins' and 'plugin_for' except that they exist.
What do they do? How to use them? Does marking X as a plugin for Y implicitly add a depend?

[ColinKennedy]

We should still update documentation but apparently it's used only for the rez-plugins command. So I guess it's only used for basic debugging at the moment(?)

[hughetop]

Since there's not really anywhere else to put something like this, I was told to add my photoshop-rez docs here:
https://github.com/Smurgledwerf/photoshop-rez-example

Using my personal github account because we're not allowed to have any public repos on our work account.

[JeanChristopheMorinPerso]

Thanks @hughetop !

[herronelou]

Looking at the variants page today, I notice that up until today every time I've made a package with variants I've assembled it manually.

Today I run into a case where I need to make a package with slightly different code destined for each of the variants, and I see nothing in the variants doc about how I would go about achieving that, so definitely could use some more documentation in this subject.

[rontown]

I've been trying to set rez up, following the wiki install page on Git Hub, but there is something incorrect in the guide or some significant steps missing.

In all likelihood, it's something I haven't done or understood fully, but I can't figure out how to get it working. I would have hoped the guide be clear enough or someplace to go for troubleshooting (discord or slack channel) but alas I cannot find it.

Any suggestions much appreciated.

[herronelou]

There is a slack channel at https://join.slack.com/t/academysoftwarefdn/shared_invite/zt-1zc8wlakn-65h2~ixmsvThd6oGJKB6VA

…

On Wed, Jul 19, 2023, 07:29 C ***@***.***> wrote: I've been trying to set rez up, following the wiki install page on Git Hub, but there is something incorrect in the guide or some significant steps missing. In all likelihood, it's something I haven't done or understood fully, but I can't figure out how to get it working. I would have hoped the guide be clear enough or someplace to go for troubleshooting (discord or slack channel) but alas I cannot find it. Any suggestions much appreciated. — Reply to this email directly, view it on GitHub <#1277 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAV5GXL6IEGIDXEQB2UQ3XQ7VNFANCNFSM5SXYUUIQ> . You are receiving this because you commented.Message ID: <AcademySoftwareFoundation/rez/repo-discussions/1277/comments/6489579@ github.com>

[rontown]

Thanks!

[JeanChristopheMorinPerso]

Hello everyone, I created a PR to move our wiki to ReadTheDocs. The PR is #1522 and you can see the preview at https://rez--1522.org.readthedocs.build/en/1522/.

I think it will improve the situation a bit and is also a good first step towards improving our documentation and it'll be easier to make contribution to it.

Please let me know what you think by commenting on the PR itself.

Thanks and I hope you'll like it!

[herronelou]

Documentation doesn't say anything about rez plugins, this could be a section to add

[JeanChristopheMorinPerso]

Yes, thanks for pointing that!

[JeanChristopheMorinPerso]

From @SitiSchu

maybe it'd be useful if he attributes under "Package definition" would mention if they can be @late or not, right now only the section on late binding functions mentions this

See https://academysoftwarefdn.slack.com/archives/C0321B828FM/p1693822267612699?thread_ts=1693697597.478139&cid=C0321B828FM.

[JeanChristopheMorinPerso]

The current --help on each of rez's command is kind of useless. The usage text is just the args on a single line, there isn't much description and some CLI flags could have better documentation. The help also doesn't tell us if the command accepts grouped arguments (--), etc.

I would like something like pip does:

[jcmorin@arch01 rez-pip]$ pip install --help

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:
  
  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.
  
  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Compare that to our rez-env help:

[jcmorin@arch01 rez-pip]$ /opt/rez/bin/rez/rez-env --help
usage: rez env [-h] [--shell {bash,csh,pwsh,sh,tcsh,zsh}] [--rcfile RCFILE] [--norc] [-c COMMAND] [-s] [--ni] [--nl] [-b] [--paths PATHS] [-t TIME] [--max-fails N] [--time-limit SECS]
               [-o FILE] [-i FILE] [--exclude RULE [RULE ...]] [--include RULE [RULE ...]] [--no-filters] [-p] [--strict] [--patch-rank N] [--no-cache] [-q] [--fail-graph]
               [--new-session] [--detached] [--no-passive] [--stats] [--no-pkg-cache] [-v]
               [PKG ...]

Open a rez-configured shell, possibly interactive.

[MrLixm]

Hello,
Linking #1569 here.
If ever an user guide is made for windows, it would be interesting to specify that the developer might want to check the pathed_env_vars config option, and suggest additional changes to extend it (like the *PATHS mentioned in the issue.)