Jenkins workflows to release -SNAPSHOT builds and to stage releases in maven #187

dblock · 2022-07-28T17:07:34Z

Description

On top of #186, build locally without maven local, sign, then release to maven staging.

We're misusing publishToMavenLocal, that takes a publishable set of artifacts, then actually publishes it to local maven (~/.m2). Then we try to get files from there and make them publishable again (by renaming maven-metadata-local for example). All that is too much work.

With this PR you can do ./gradlew --no-daemon publishPublishMavenPublicationToSnapshotRepoRepository for -SNAPSHOT builds, and VERSION=2.1.0 ./gradlew --no-daemon publishPublishMavenPublicationToLocalRepoRepository for producing local output that can be signed and uploaded to staging .

This just uploads -SNAPSHOTs.

ORG_GRADLE_PROJECT_snapshotsUsername=... ORG_GRADLE_PROJECT_snapshotsPassword=... ./gradlew publishPublishMavenPublicationToSnapshotRepoRepository

This produces local output which can then be uploaded to staging.

VERSION=2.1.0 ./gradlew --no-daemon publishPublishMavenPublicationToLocalRepoRepository

$ ls -la ./build/repository/org/opensearch/client/opensearch-java/2.1.0/
total 38136
drwxr-xr-x  27 dblock  staff       864 Jul 28 14:42 .
drwxr-xr-x   8 dblock  staff       256 Jul 28 14:42 ..
-rw-r--r--   1 dblock  staff  11518615 Jul 28 14:42 opensearch-java-2.1.0-javadoc.jar
-rw-r--r--   1 dblock  staff        32 Jul 28 14:42 opensearch-java-2.1.0-javadoc.jar.md5
-rw-r--r--   1 dblock  staff        40 Jul 28 14:42 opensearch-java-2.1.0-javadoc.jar.sha1
-rw-r--r--   1 dblock  staff        64 Jul 28 14:42 opensearch-java-2.1.0-javadoc.jar.sha256
-rw-r--r--   1 dblock  staff       128 Jul 28 14:42 opensearch-java-2.1.0-javadoc.jar.sha512
-rw-r--r--   1 dblock  staff   2941649 Jul 28 14:42 opensearch-java-2.1.0-sources.jar
-rw-r--r--   1 dblock  staff        32 Jul 28 14:42 opensearch-java-2.1.0-sources.jar.md5
-rw-r--r--   1 dblock  staff        40 Jul 28 14:42 opensearch-java-2.1.0-sources.jar.sha1
-rw-r--r--   1 dblock  staff        64 Jul 28 14:42 opensearch-java-2.1.0-sources.jar.sha256
-rw-r--r--   1 dblock  staff       128 Jul 28 14:42 opensearch-java-2.1.0-sources.jar.sha512
-rw-r--r--   1 dblock  staff   4964269 Jul 28 14:42 opensearch-java-2.1.0.jar
-rw-r--r--   1 dblock  staff        32 Jul 28 14:42 opensearch-java-2.1.0.jar.md5
-rw-r--r--   1 dblock  staff        40 Jul 28 14:42 opensearch-java-2.1.0.jar.sha1
-rw-r--r--   1 dblock  staff        64 Jul 28 14:42 opensearch-java-2.1.0.jar.sha256
-rw-r--r--   1 dblock  staff       128 Jul 28 14:42 opensearch-java-2.1.0.jar.sha512
-rw-r--r--   1 dblock  staff      7796 Jul 28 14:42 opensearch-java-2.1.0.module
-rw-r--r--   1 dblock  staff        32 Jul 28 14:42 opensearch-java-2.1.0.module.md5
-rw-r--r--   1 dblock  staff        40 Jul 28 14:42 opensearch-java-2.1.0.module.sha1
-rw-r--r--   1 dblock  staff        64 Jul 28 14:42 opensearch-java-2.1.0.module.sha256
-rw-r--r--   1 dblock  staff       128 Jul 28 14:42 opensearch-java-2.1.0.module.sha512
-rw-r--r--   1 dblock  staff      3530 Jul 28 14:42 opensearch-java-2.1.0.pom
-rw-r--r--   1 dblock  staff        32 Jul 28 14:42 opensearch-java-2.1.0.pom.md5
-rw-r--r--   1 dblock  staff        40 Jul 28 14:42 opensearch-java-2.1.0.pom.sha1
-rw-r--r--   1 dblock  staff        64 Jul 28 14:42 opensearch-java-2.1.0.pom.sha256
-rw-r--r--   1 dblock  staff       128 Jul 28 14:42 opensearch-java-2.1.0.pom.sha512

The JenkinsFile was renamed to jenkins/stage-maven-release.jenkinsfile to be consistent.

See http://jenki-jenki-fpgmrv2ryxko-1366042710.us-east-1.elb.amazonaws.com/job/dblock-opensearch-java-stage-maven-release/4/console for a run that succeeded all the way till uploading to sonatype that requires SONATYPE_USERNAME and SONATYPE_PASSWORD that I believe aren't available in staging (or shouldn't be :)).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: dblock <dblock@amazon.com>

jenkins/publish-snapshot.jenkinsfile

jenkins/stage-maven-release.jenkinsfile

gaiksaya · 2022-07-28T18:42:04Z

for a run that succeeded all the way till uploading to sonatype that requires SONATYPE_USERNAME and SONATYPE_PASSWORD that I believe aren't available in staging (or shouldn't be :)).

We do have it in staging as even though it is successful, a manual intervention is required (as of today) to actually publish the snapshots to maven central.

dblock · 2022-07-28T18:55:17Z

Updated.

gaiksaya · 2022-07-28T20:41:31Z

jenkins/publish-snapshot.jenkinsfile

+        }
+    }
+    stages {
+        stage('Publish to Maven Staging') {


Rename to Publish to sonatype snapshots repo or something similar. Quite confusing as maven staging is actually a maven central staging area

jenkins/stage-maven-release.jenkinsfile

gaiksaya · 2022-07-28T21:01:16Z

jenkins/stage-maven-release.jenkinsfile

+                        error('Missing REF and/or VERSION.')
+                    }
+                }
+            }


do you want to add triggers now or I can create a follow up PR. I have the code ready. Needs few PRs to add creds to jenkins infrastructure.

Let's do a followup.

gaiksaya · 2022-07-28T21:02:09Z

jenkins/stage-maven-release.jenkinsfile

+                script {
+                    signArtifacts(
+                        artifactPath: "${ARTIFACT_PATH}",
+                        type: 'maven',


My bad. Please do type: 'maven',

Yep. That's what I figured.

Signed-off-by: dblock <dblock@amazon.com>

dblock · 2022-07-28T21:32:02Z

More updates, thx. Let me know if you need anything else.

gaiksaya

LGTM!

gaiksaya · 2022-07-28T21:52:52Z

jenkins/stage-maven-release.jenkinsfile

+                // checkout the commit
+                git url: 'https://github.com/opensearch-project/opensearch-java.git', branch: 'main'
+                sh('git checkout ${REF}')


nit: Can be

checkout([$class: 'GitSCM', userRemoteConfigs: [[url: 'https://github.com/gaiksaya/opensearch-java.git']], branches: [[name: "$ref"]]])

dblock · 2022-07-28T22:12:13Z

@VachaShah care to +1? Thx

VachaShah

Thank you @dblock!

It might be helpful to also add the details about the release and snapshot workflow in https://github.com/opensearch-project/opensearch-java/blob/main/RELEASING.md may be?

saratvemulapalli · 2022-07-28T22:17:52Z

java-client/build.gradle.kts

+        if (version.toString().endsWith("SNAPSHOT")) {
+            maven("https://aws.oss.sonatype.org/content/repositories/snapshots/") {
+                name = "snapshotRepo"
+                credentials(PasswordCredentials::class)


For my learning, how are these creds passed?

See above, this asks for a specific environment variable that gets set by the wrapper.

saratvemulapalli · 2022-07-28T22:20:13Z

jenkins/publish-snapshot.jenkinsfile

@@ -0,0 +1,25 @@
+pipeline {


How does the Jenkins fleet pick this job/pipeline ?
Is there a corresponding file in opensearch-ci?

Also I see for every component, jenkins files are defined in opensearch-build. Why is this different?

@gaiksaya @peterzhuamazon ?

i think the job needs to be added manually once, but I could be wrong (I hope so)

We could move these to OpenSearch-build, I have no strong preference, I was replacing a file that was already here

Yeah for now we to manually added files, for releasing opensearch-java it already exists. @saratvemulapalli we are planning to move respective jenkins file for clients in their repo. Would be easier to manage. However, need to add regression tests for all jenkins file to see the behavior is as expected.

thanks that helps :).
One concern having these files in individual repositories is that the responsibility is delegated to the maintainers of the individual projects and make sure the infrastructure is not mis-used.

I believe the maintenance of these files will be delegated to respective maintainers of the project rather than @opensearch-project/engineering-effectiveness being the bottle neck for it. Regarding the mis-use of the infrastructure, we have to be careful what triggers the workflows. For example, if a commit push is gonna trigger the workflow no matter where the workflow lies it will be executed.

dblock · 2022-07-28T22:48:12Z

Thank you @dblock!

It might be helpful to also add the details about the release and snapshot workflow in https://github.com/opensearch-project/opensearch-java/blob/main/RELEASING.md may be?

Absolutely. @gaiksaya plans to hook this up as a GHA that kicks off this job in Jenkins, and we will need to document how to make the release then

(This workflow is in Jenkins because we don’t export credentials Into GHA)

gaiksaya · 2022-07-28T22:52:12Z

Actually @dblock , GHA is not in the picture as of now. Was experimenting and we can directly make a release using tags or pushes.
Will create a followup PR and add details then.
Thanks!

Publish snapshot to maven.

a42e97b

Signed-off-by: dblock <dblock@amazon.com>

dblock force-pushed the release-to-maven branch from cb86baa to 603b27a Compare July 28, 2022 17:15

dblock changed the title ~~Release to maven.~~ Simplify Jenkins workflow to release to maven staging Jul 28, 2022

dblock requested review from gaiksaya and VachaShah July 28, 2022 17:17

dblock marked this pull request as ready for review July 28, 2022 17:17

dblock requested review from a team and madhusudhankonda as code owners July 28, 2022 17:17

dblock force-pushed the release-to-maven branch from 603b27a to 50f177a Compare July 28, 2022 17:18

dblock mentioned this pull request Jul 28, 2022

Remove redundant SHA stage. #185

Closed

Release to maven.

144b722

Signed-off-by: dblock <dblock@amazon.com>

dblock force-pushed the release-to-maven branch from 50f177a to 144b722 Compare July 28, 2022 17:43

dblock mentioned this pull request Jul 28, 2022

Publish snapshot to maven. #186

Closed

dblock changed the title ~~Simplify Jenkins workflow to release to maven staging~~ Jenkins workflows to release -SNAPSHOT builds and to stage releases in maven Jul 28, 2022

dblock mentioned this pull request Jul 28, 2022

[PROPOSAL] Release v2.1.0 #174

Closed

gaiksaya requested changes Jul 28, 2022

View reviewed changes

dblock requested a review from gaiksaya July 28, 2022 18:55

gaiksaya requested changes Jul 28, 2022

View reviewed changes

Updated credentials and signing.

4d9e13c

Signed-off-by: dblock <dblock@amazon.com>

dblock force-pushed the release-to-maven branch from d6e7d94 to 4d9e13c Compare July 28, 2022 21:31

dblock requested a review from gaiksaya July 28, 2022 21:31

gaiksaya approved these changes Jul 28, 2022

View reviewed changes

VachaShah approved these changes Jul 28, 2022

View reviewed changes

saratvemulapalli reviewed Jul 28, 2022

View reviewed changes

dblock merged commit df34831 into opensearch-project:main Jul 28, 2022

dblock deleted the release-to-maven branch July 28, 2022 23:18

dblock mentioned this pull request Sep 30, 2022

Support release process for opensearch-testcontainers opensearch-project/opensearch-build#2656

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jenkins workflows to release -SNAPSHOT builds and to stage releases in maven #187

Jenkins workflows to release -SNAPSHOT builds and to stage releases in maven #187

dblock commented Jul 28, 2022 •

edited

Loading

gaiksaya commented Jul 28, 2022

dblock commented Jul 28, 2022

gaiksaya Jul 28, 2022

gaiksaya Jul 28, 2022

dblock Jul 28, 2022

gaiksaya Jul 28, 2022

dblock Jul 28, 2022

dblock commented Jul 28, 2022

gaiksaya left a comment

gaiksaya Jul 28, 2022

dblock commented Jul 28, 2022

VachaShah left a comment

saratvemulapalli Jul 28, 2022

dblock Jul 28, 2022

saratvemulapalli Jul 28, 2022

dblock Jul 28, 2022

dblock Jul 28, 2022

gaiksaya Jul 28, 2022

saratvemulapalli Jul 28, 2022

gaiksaya Jul 28, 2022 •

edited

Loading

dblock commented Jul 28, 2022

gaiksaya commented Jul 28, 2022 •

edited

Loading

Jenkins workflows to release -SNAPSHOT builds and to stage releases in maven #187

Jenkins workflows to release -SNAPSHOT builds and to stage releases in maven #187

Conversation

dblock commented Jul 28, 2022 • edited Loading

Description

gaiksaya commented Jul 28, 2022

dblock commented Jul 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblock commented Jul 28, 2022

gaiksaya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblock commented Jul 28, 2022

VachaShah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaiksaya Jul 28, 2022 • edited Loading

Choose a reason for hiding this comment

dblock commented Jul 28, 2022

gaiksaya commented Jul 28, 2022 • edited Loading

dblock commented Jul 28, 2022 •

edited

Loading

gaiksaya Jul 28, 2022 •

edited

Loading

gaiksaya commented Jul 28, 2022 •

edited

Loading