Adding Artifact Repository

[ttrently]

UPDATE: Latest commit implements an artifact repository based on discussions found on #1393

Add support for artifact_repository plugins which allow for hosting of package payloads in remote servers. This works in conjunction with a MongoDB instance that contains the package information for querying and lookup.

Requirements

This implementation requires some additional packages that have not been included in rez.vendor.

• awscli
• pymongo

These can be installed via python -m pip install PKG --target path\to\rez\Lib\site-packages.

How To Use

In order to use you will need an existing S3 bucket and a MongoDB instance with a named database and a collection called "packages".

In your rezconfig.py you will need to set the following:

packages_path = [
    "mongo@localhost:27017/db_name",
]

artifacts_path = [
    "s3@s3://bucket-1/packages",
]

default_cachable = True
package_cache_same_device = True
package_cache_async  = False

If your MongoDB instance also requires a username / password, you can set it via:

REZ_MONGO_USERNAME=username
REZ_MONGO_PASSWORD=password

When these are set, it will format the packages path url as:

{REZ_MONGO_USERNAME}:{REZ_MONGO_PASSWORD}@localhost:27017/db_name

rez-build

The rez-build command has been extended to allow for installing to artifact repositories. This is done through the new -a --artifact-path argument from rez-build.

Example:

foo> rez-build --install -p mongo@uri:port/db_name -a s3@s3://bucket-1/packages

This build command will do the following:

1. Build the foo package normally
2. Install build results to a temporary location of $TEMP\rez\package_name\package_version, this is what will be pushed to the artifact repository.
3. Add a new variant entry to the specified mongo@ package repository
4. Push the installed build results to the s3@ artifact repository.

Missing / To-Do

• This uses a basic implementation of the artifact repository where they are defined separately from the package repository. On lookup, the artifact repositories are searched in order for the package payload, this seems to work well since the location stored in the database is the artifact_path qualifier, but may need to be more complex in the future.
• Artifact repository paths need to be extended to other cli's, such as move, release, and any others that interact with repositories.
• MongoDB authentication is still somewhat of an issue if you use multiple different databases that require different credentials.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ttrently (03b5ef7, cf887ff)

[JeanChristopheMorinPerso]

This is pretty exciting @ttrently! I haven't reviewed any of it yet, but I took some time to link to some resources.

Long ago, someone tried to create a MongoDB repository plugin and even created a PR, see #205. You could potentially re-use part of it or at maybe just take inspirations from it.

Additionally, there is REP-003 that tracks package caching. In the REP (Rez Enhancement Proposal) and its comments, there is some wording that makes me believe the package repository could say "I need all my packages to be locally cached". This would allow to mix and match multiple package repository types without forcing all resolved packages to be cached.

[JeanChristopheMorinPerso]

@ttrently is your latest commit implementing what you described in #1393?

[ttrently]

@ttrently is your latest commit implementing what you described in #1393?

Yes, to a degree, and primary comment updated with usage information. I'm still a bit unsure of how to handle complex paths, so for this I kept it close to how normal package repositories are handled, ie. Artifact repository paths are defined and lookups happen based on the order of those paths.

[JeanChristopheMorinPerso]

The new description is helpful and thanks for that. Can you update the description to mention that you implement part of that discussion, what's implemented, what's not, etc. I would also suggest that you mention which library it depends, etc.

(Note that I won't have time to review until next week).

[JeanChristopheMorinPerso]

Hey @ttrently , I'm curious if you made more progress on this work. Let us know if we can help in any way!