LakeFS support for binary distribution

[Lypsolon]

TODO

• Setting to acknowledge that this feature is not production-ready

• don't start the addon if the setting is not switched on.

• Windows title for download USD window

• Settings titles without underscores but with spaces

• Shorter addon name

• version info the addon class

• update settings so that lake fs can use the new naming system

• update the find resolver system so that it will find the resolver in the new layout

  • Linux
  • win

• Figure out why the prelaunch hook does not download under windows

• Currently, the download system does not take into account if the JSON has an entry but the file/folder is missing

• rename the usd-24.03_win64_py39.zip to usd-24.03_windows_py39.zip to allow ayon_usd to find it on Windows. (we will unify the naming via another PR)

• convert the string evaluation off settings keys into a tuple evaluation

• update ayon_bin_bridge_client to the newest version

• test if the new version of bin_client will run under Windows (the ne ci tests do run)

• better error message when get_element_info() cant find the element (check if element return is empty then print that no element was found)

• better error message when no resolver could be found (include: app_name)

• don't cache server settings (PreLaunchHook instances have access to the self.data["project_settings"])

• better create package py

  • git submodule init (wont fix for alpha, setup should change to AyonAutomator)
  • fix client site downloads folder zipping

• create package should ignore downloads, not delete it

Testing Notes

Getting started:

• Create ayon-usd package
• Upload to AYON server (this addon adds new settings to the server)
• Request credentials (currently only on Ynput-Discord) for the LakeFS distribution.
• Enter the credentials into the addon settings
  • ayon+settings://ayon_usd/LakeFs_Settings/access_key_id
  • ayon+settings://ayon_usd/LakeFs_Settings/secret_access_key
• Switch the branch from main to pinning-suport (this is because there is a compile error in Main that is fixed in this branch [Bug]_Py_Package_Init_does_not_define_PyInit__ayonUsdResolver ayon-usd-resolver#21)
  • this setting ayon+settings://ayon_usd/LakeFs_Settings/ayon_usd_lake_fs_server_repo
  • to this value: lakefs://ayon-usd/pinning-suport/ (default will be lakefs://ayon-usd/main/)
• Launch AYON
• Launch AYON USD Resolver supported DCC (Unreal, Houdini, Maya, AyonStandalone is not yet implemented).

Optionally, you can set the log level (ayon+settings://ayon_usd/Ayon_UsdResolver_Settings/ayon_log_lvl) to Info. When the resolver is initialised, this will print some init output in the Houdini console (Windows) or terminal (Linux).

What to test:

• AyonUsdLib should be downloaded on Launcher startup (all downloads are stored in the addon dir. You can find your addon dir by clicking the version number in the tray menu (this will open Ayon info). Then, you can open Environments and take the path from the AYON_ADDONS_DIR. Use this to navigate to the install dir of your ayon-usd-addon, and you will find a downloads dir)
• Resolver should get installed on Software Startup (the resolver will be downloaded in the same place as the UsdLib)
• Resolver should work in Houdini 19.5.xxx and 20.0.xxx, Maya2024, Maya2025, Unreal5.4

In case you don't have a loader that uses Ayon URI, do the following:

• Open Ayon Web view.
• Go to Projects-Browser, select a published USD asset
• select a representation and copy the entity URI
  • the URI looks like this ayon+entity://{ProjectName}/AssetName?product=usdAsset&version={VersionNumber}&representation=usd
  • the representation should be USD
• Launcher Software Via Ayon Launcher
  • Houdini
    • use a reference or sublayer node and put the URI into the file path.
  • Maya
    • Click Create/Universal Scene Description (USD)/Stage With New Layer
    • Your outliner's new stage, "Object", has a Stage-Source/File (Attribute Editor). Place the URI into the File Filed.
  • Unreal
    • create a Usd-Stage-Actor (you will need to activate the usdImport plugin in Unreal)
    • create a USD file where you reference a ayon uri (you can create this in Houdini or in Maya) load the USD file via its file path (see known issues if you want to know why)

Known Issues

• The downloaded file is currently blind to updates on lakeFs. This means that when you change your branch or upload a new resolver, this system will not redownload the resolver. (we should talk about how we want to fix this, (there was the idea to have build tags on the LakeFs zips and compare them, or we could write a JSON file with the data when we downloaded the resolver to see if a resolver with a newer time stamp exists. @antirotor @BigRoy @dee-ynput not sure any more who I should ask for a Greenligth on that decision))
• in unreal, when you create a stage actor, the resolver will not be passed the URI, but instead, it will be passed the project path + the Uri /home/lyonh/Documents/Unreal Projects/Dell/ayon+entity:/Usd_Base/trees?product=usdtree_106&version=v001&representation=usd this causes the resolve to fail. (there is no decision how we should fix this)

This Pr handles implementation of LakeFs into the Ayon-Usd addon.

Components

Ui

The ui is mainly build on the Server side, the addon itself dose not have any ui to configure
General Ui

LakeFs config ui (i would like to reduce this to a point where the lakeFs Package has a file that informs about the internal structure, maybe via an package_layout.json or similar so that we only need an alias config where we could say (key for Houdini is Hou20.0 but app_name is houdini20.0) but this will probably need to wait for this pr ayon-usd_lakefs-package_build )

Addon

the addon pulls the UsdLib by requesting the layout from the lakeFs server and then comparing the available UsdLibs with the platform name. (all of this should / could be handled by the package_layout.json at some point )

the resolver are pulled the same way but using the app_name and the mapping defined in the server settings.
Application overwrites will allays take control in case they are defined for a specific app_name in all other cases the LakeFs paths will be used in combination with the current defined branch.

[ynbot]

Task linked: AY-5722 Implement_versoin_off_LakeCtlPy_into_ayonUsdPlugin_excample

[BigRoy]

I can't seem to create the package @Lypsolon

ayon-usd/create_package.py --output E:\dev\tools_ayon\packages
Traceback (most recent call last):
  File "E:\dev\ayon-usd\create_package.py", line 389, in <module>
    main(args.output_dir, args.skip_zip, args.keep_sources)
  File "E:\dev\ayon-usd\create_package.py", line 347, in main
    zip_client_side(addon_output_dir, current_dir, log)
  File "E:\dev\ayon-usd\create_package.py", line 237, in zip_client_side
    shutil.rmtree(client_ayon_usd_download_dir)
  File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 759, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 610, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\User\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 607, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'E:\\dev\\ayon-usd\\client\\ayon_usd\\downloads'

• Note that I've specified an outputdir for the package with --output, like other packages do allow.
• However, also without that flag, it fails.
• When creating folder E:\dev\ayon-usd\client\downloads manually, and then retrying - it also fails. Nevermind - I forgot the extra ayon_usd folder in there.

[BigRoy]

Improve settings and addon name?

The addon name (see left side) and the Settings themselves can benefit from better titles to match with other addons:

• Shorter addon name
• Settings titles without underscores but with spaces

[BigRoy]

Requires to init the submodules

Launching the tray launcher with the package installed gives me this error in console:

*** WRN: >>> { AddonsLoader }: [  FAILED to import "ayon_usd"  ]
==============================
No module named 'ayon_usd.ayon_bin_client.ayon_bin_distro'
==============================
Traceback (most recent call last):
  File "E:\dev\ayon-core\client\ayon_core\addon\base.py", line 360, in _load_ayon_addons
    mod = __import__(basename, fromlist=("",))
  File "E:\dev\ayon-usd\client\ayon_usd\__init__.py", line 3, in <module>
    from .addon import USDAddon
  File "E:\dev\ayon-usd\client\ayon_usd\addon.py", line 7, in <module>
    from . import config, utils
  File "E:\dev\ayon-usd\client\ayon_usd\config.py", line 10, in <module>
    from ayon_usd.ayon_bin_client.ayon_bin_distro.lakectlpy import wrapper
ModuleNotFoundERROR: No module named 'ayon_usd.ayon_bin_client.ayon_bin_distro'
*** WRN: >>> { AddonsLoader }: [  Addon ayon_usd 1.0.4-dev.4 has no content to import  ]

Because this required:

git submodule update --init --recursive

Would be great if this could somehow be detected earlier :)

[BigRoy]

After building the package with all git submodules and re-uploading the package, etc.
On launch of tray I now get this error:

*** WRN: >>> { AddonsLoader }: [  FAILED to import "ayon_usd"  ]
=======================
No module named 'fcntl'
=======================
Traceback (most recent call last):
  File "E:\dev\ayon-core\client\ayon_core\addon\base.py", line 360, in _load_ayon_addons
    mod = __import__(basename, fromlist=("",))
  File "E:\dev\ayon-usd\client\ayon_usd\__init__.py", line 3, in <module>
    from .addon import USDAddon
  File "E:\dev\ayon-usd\client\ayon_usd\addon.py", line 7, in <module>
    from . import config, utils
  File "E:\dev\ayon-usd\client\ayon_usd\config.py", line 10, in <module>
    from ayon_usd.ayon_bin_client.ayon_bin_distro.lakectlpy import wrapper
  File "E:\dev\ayon-usd\client\ayon_usd\ayon_bin_client\ayon_bin_distro\lakectlpy\wrapper.py", line 7, in <module>
    import fcntl
ModuleNotFoundERROR: No module named 'fcntl'
*** WRN: >>> { AddonsLoader }: [  Addon ayon_usd 1.0.4-dev.4 has no content to import  ]

I'll stop here for now. ;) May be good to update the testing notes (or the PR itself) to solve this issue?

[Lypsolon]

@dee-ynput, I implemented the button you wanted to force the studio to accept that the AyonUsd add-on is experimental.

@iLLiCiTiT short question for you. The addon raises an Error when the admin does not accept that the addon is Experiments (in the Ayon server settings). Is that acceptable, or would you like the addon to communicate this to the Ayon Launcher differently?
I don't know if we have warning messages, but we may.

[BigRoy]

Approving so this can become the new state of develop branch.