Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[v1.8.x] Automate website artifacts uploading #19244

Merged
merged 19 commits into from
Oct 9, 2020

Conversation

waytrue17
Copy link
Contributor

@waytrue17 waytrue17 commented Sep 28, 2020

All versions of MXNet website artifacts are compressed into a zip file and stored in a private S3 bucket, with the following file structure:

version
----1.0.0
----1.1.0
...
----1.6.0
----1.7.0

For new MXNet versions released, we need to add the new website artifacts into the above zip file and upload it to the S3 bucket. Currently we have a pipeline that generates version specific artifacts, but it's still a manual process to upload them to s3 in the expected format shown above.
This PR aims to automate the website artifact uploading process.
We introduced pipeline Jenkinsfile_website_upload to build the new version website artifacts, prepare all versions of website artifacts into the expected format, and upload them to the target S3 bucket. This pipeline utilizes Jenkins environment variable FOLDER_NAME to specify the name of the new artifacts to be added.

@mxnet-bot
Copy link

Hey @waytrue17 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [windows-cpu, clang, sanity, unix-cpu, centos-gpu, edge, centos-cpu, miscellaneous, unix-gpu, website, windows-gpu]


Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@szha szha requested a review from ys2843 September 28, 2020 19:27
@mseth10
Copy link
Contributor

mseth10 commented Sep 28, 2020

After this PR gets merged, modify Step 3.2.1 Point 5 in
https://cwiki.apache.org/confluence/display/MXNET/Release+Process#ReleaseProcess-3.2.UpdatetheMXNetwebsite

@waytrue17 waytrue17 changed the title [WIP] Website release pipeline automation [v1.8.x] Website release pipeline automation Sep 29, 2020
Copy link
Contributor

@mseth10 mseth10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mseth10
Copy link
Contributor

mseth10 commented Sep 29, 2020

The PR changes are being tested by this Jenkins pipeline: http://jenkins.mxnet-ci-dev.amazon-ml.com/job/docs/job/sethman-website-artifacts/

@waytrue17
Copy link
Contributor Author

Hi @ys2843 Could you please review this PR? Thanks!

Copy link
Contributor

@ys2843 ys2843 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! Just one question:

ci/docker/runtime_functions.sh Outdated Show resolved Hide resolved
@waytrue17 waytrue17 changed the title [v1.8.x] Website release pipeline automation [v1.8.x] Automate website artifacts uploading Oct 1, 2020
@waytrue17
Copy link
Contributor Author

@mxnet-bot run ci [unix-cpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [unix-cpu]

@samskalicky
Copy link
Contributor

@marcoabreu can you please review again now that your feedback has been addressed?

@sandeep-krishnamurthy
Copy link
Contributor

@marcoabreu - Looks like your review comments are addressed. Can you take a look at this and approve / have other feedback?

Copy link
Contributor

@ys2843 ys2843 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sandeep-krishnamurthy
Copy link
Contributor

Merging this as it helps automating an important part of website building and all the requested changes by @marcoabreu are addressed.
@marcoabreu - Thank you for you review and inputs. Since we were unable to reach you, and I see all your comments were addressed by @waytrue17 merging this PR.
Please feel free to comment later, if there are any open items and we can cover it in another PR.

@sandeep-krishnamurthy sandeep-krishnamurthy merged commit 26486fe into apache:v1.8.x Oct 9, 2020
@sandeep-krishnamurthy
Copy link
Contributor

@waytrue17 / @mseth10 / @ys2843 => Should this be backported to 1.X branch please?

@mseth10
Copy link
Contributor

mseth10 commented Oct 9, 2020

Yes, this needs to be backported to 1.x and master. @waytrue17 can you create those PRs?

@waytrue17
Copy link
Contributor Author

Yes, this needs to be backported to 1.x and master. @waytrue17 can you create those PRs?

Sure, I'll backport it to 1.x and master

@ys2843
Copy link
Contributor

ys2843 commented Oct 9, 2020

@waytrue17 It doesn't need to be port to master, only to branches that will be built as static artifact.

@mseth10
Copy link
Contributor

mseth10 commented Oct 9, 2020

Hey @ys2843 , I thought we will need it for any release going forward, be it 1.9 or 2.0. How do we plan to build static artifacts for 2.0 once that branch is cut?

@waytrue17 waytrue17 deleted the website-release branch October 12, 2020 04:03
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants