Resources as artifacts and integration core components into Dash

[waralex]

This PR is a demonstration of the concept of using artifacts instead of hardoded resources as well as collecting resources from the python dash without manual copying.

Using artifacts:

• js files and metadata of the renderer and components are stored in a separate repository (now https://github.com/waralex/DashCoreResources)
• A tarball with this data (archive of the resource folder) is added to the release of this package
• In Dash, there is an Artifacts.toml associated with this archive:

[dash_resources]
git-tree-sha1 = "5221819aa12c212904a1922411cf8bbc9b787901"

    [[dash_resources.download]]
    sha256 = "0f129069bed005e92a8061c4a3b71ba0aedf54911e01103b3261d586995836a6"
    url = "https://github.com/waralex/DashCoreResources/releases/download/v1.20.0+0/DashCoreResources.v1.20.0.tar.gz"

• When perсompiling Dash, component functions are generated using a macro based on metadata from the artifact
• js and css files from the artifact are used for the Dash operation instead of the resources that were previously contained in the Dash itself

Collecting resources from python dash

• A separate folder gen_resources with the code for generating resources appeared in the root of the repository
• call to generate resources: julia --project=gen_resources gen_resources/generate.jl [--deploy]
• Resource generation occurs as follows:

1. Clone the python dash and check out the tag specified in gen_resources/Sources.toml
2. Install it using pip install <dir> in Julia's PyCall environment
3. Using PyCall, it import modules for dash_render and the main components (the modules that need to be imported are also specified in Sources.toml)
4. Based on the _js_dist and _css_dist of the renderer and modules, a list of resources corresponding to the module is formed
5. Based on metadata.json of modules it generate metadata for component functions
6. Repo for deploy resources (also specified in Sources.toml) is сloned and populated with data previous steps
7. A new artifact is created from content of the resources folder of resource repo
8. if --deploy command line argument is specified then:
   Using the OAuth app, the resource repository is pushed, a tarball with an artifact is created, and a new release is created in the resource repository to which the tarball is applied

9.Artifact bound into Artifacts.toml of Dash

[MichaelHatherly]

Thanks @waralex, yes from the looks of it the artifact approach used here should work out alright.

It can be a bit of a pain to have to do the artifact uploads (even if an automated script exists) if they do happen to change reasonably often. We recently came up with another approach (https://github.com/JuliaPackaging/RelocatableFolders.jl) which is working out reasonably well for this kind of requirement of shipping non-Julia assets that are still relocatable without having to go through the process of artifact creation and distribution. Whether it would actually make this specific PR less complex I couldn't say for certain.

[waralex]

Thanks @waralex, yes from the looks of it the artifact approach used here should work out alright.

It can be a bit of a pain to have to do the artifact uploads (even if an automated script exists) if they do happen to change reasonably often.

Thank you for participating @MichaelHatherly!
In this case, with the help of artifacts and an automatic script, I do not so much want to make the paths relocatable (this is a nice bonus), but I want to simplify the dependencies between the resources of the Python dash and Dash.jl as much as possible. The idea was as follows: when a new version of dash and the corresponding js resources are released, a script is run and adds these dependencies to the artifact in a separate repo. And Dash is linked to a specific version of resources only through the artifact id and different versions of Dash that use the same version of resources share it and do not store it in themselves

[MichaelHatherly]

Yeah, that definitely makes sense to just use artifacts then in that case.

I assume that custom component packages (https://github.com/facultyai/dash-bootstrap-components for example) will need to do their own artifact creation for their custom components. How much of the generator/upload scripts here will be re-useable by others? Would be great for them not to have to go copy-pasting scripts and altering things here and there to make it work.

[waralex]

I assume that custom component packages (https://github.com/facultyai/dash-bootstrap-components for example) will need to do their own artifact creation for their custom components. How much of the generator/upload scripts here will be re-useable by others? Would be great for them not to have to go copy-pasting scripts and altering things here and there to make it work.

Yesterday I suggested to @alexcjohnson to make an analog of JuliaBinaryWrappers. I.e. flow when any user (not only the components maintainer) can add a PR to a repository with the information necessary to run the generator. And after approve this PR bot from Plotly will release the artifacts and the wrapper package over them. What do you think of this idea?

[MichaelHatherly]

That sounds pretty neat, would definitely remove some friction in the process. Is this with all the language bindings (not just Julia) in mind, or specific to dealing with Julia?

[alexcjohnson]

We'll leave Python where it is, but yes it's definitely worth considering an approach like this for other languages.

R takes a good deal of language-specific knowledge to make a release, particularly if we try to have all of these packages available on CRAN; This is both a motivation in favor of relieving the component maintainer of this responsibility, and a concern that it may not be possible to automate the process to the extent we can in Julia.

.NET is still taking shape, but I've suggested adapting this kind of approach there as well.