[Bug] Maven 1.1.0 failed Central Sync due to multiple errors #699

peterzhuamazon · 2021-10-05T22:31:32Z

Describe the bug

[Bug] Maven 1.1.0 failed Central Sync due to multiple errors.

When we try to release Maven Artifacts for 1.1.0 OpenSearch to Maven Central through sonatype, we encounter these errors that block us from closing.

Here are all the issues we see:

Event: Failed: Javadoc Validation:

Event: Failed: Javadoc Validation
Tuesday, October 05, 2021 17:07:20 EDT (GMT-0400)
typeId	javadoc-staging
failureMessage	Missing: no javadoc jar found in folder '/org/opensearch/performanceanalyzer-rca/1.1.0.0'

Event: Failed: Checksum Validation:

Event: Failed: Checksum Validation
Tuesday, October 05, 2021 17:07:21 EDT (GMT-0400)
typeId	checksum-staging
failureMessage	Required SHA-1: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0-javadoc.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0-javadoc.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.pom.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.pom.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0-sources.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0-sources.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.module.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.module.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0-sources.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0-sources.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0-javadoc.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0-javadoc.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0.pom.sha1'
failureMessage	Required MD5: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0.pom.md5'
failureMessage	Required SHA-1: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/common-utils/1.1.0.0/common-utils-1.1.0.0.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0.pom.sha1'
failureMessage	Required MD5: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0.pom.md5'
failureMessage	Required SHA-1: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0-javadoc.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0-javadoc.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0-sources.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/notification/1.1.0.0/notification-1.1.0.0-sources.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0-javadoc.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0-javadoc.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.module.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.module.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.pom.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.pom.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0-sources.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0-sources.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.pom.sha1'
failureMessage	Required MD5: '/org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.pom.md5'
failureMessage	Required SHA-1: '/org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.module.sha1'
failureMessage	Required MD5: '/org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.module.md5'
failureMessage	Required SHA-1: '/org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0-javadoc.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0-javadoc.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0-sources.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0-sources.jar.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0.pom.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0.pom.md5'
failureMessage	Required SHA-1: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0.jar.sha1'
failureMessage	Required MD5: '/org/opensearch/opensearch-job-scheduler-spi/1.1.0.0/opensearch-job-scheduler-spi-1.1.0.0.jar.md5'

Event: Failed: POM Validation:

Event: Failed: POM Validation
Tuesday, October 05, 2021 17:07:22 EDT (GMT-0400)
typeId	pom-staging
failureMessage	Invalid POM: /org/opensearch/opensearch-performance-analyzer/1.1.0.0/opensearch-performance-analyzer-1.1.0.0.pom: License information missing, Developer information missing
failureMessage	Invalid POM: /org/opensearch/opensearch-job-scheduler/1.1.0.0/opensearch-job-scheduler-1.1.0.0.pom: License information missing, Developer information missing
failureMessage	Invalid POM: /org/opensearch/performanceanalyzer-rca/1.1.0.0/performanceanalyzer-rca-1.1.0.0.pom: Project name missing, Project description missing, Project URL missing, License information missing, SCM URL missing, Developer information missing

Event: Failed: Sources Validation:

Event: Failed: Sources Validation
Tuesday, October 05, 2021 17:07:37 EDT (GMT-0400)
typeId	sources-staging
failureMessage	Missing: no sources jar found in folder '/org/opensearch/performanceanalyzer-rca/1.1.0.0'

Because of these errors, we are not able to close the repo, and proceed to publish the 1.1.0 release in Maven Central.

In 1.0.0, @peternied and @mch2 have removed PA in order to complete the release.
This time we are seeing more errors coming from plugin specifically.

Thanks.

To reproduce

See above.

Expected behavior

Click Close in Sonatype should close the repo.

Screenshots

Error screenshot:

Host / Environment

No response

Additional context

No response

Relevant log output

No response

The text was updated successfully, but these errors were encountered:

dblock · 2021-10-06T01:41:26Z

We should not be publishing job-scheduler, only SPI, that fix would be to remove https://github.com/opensearch-project/opensearch-build/blob/main/bundle-workflow/scripts/components/job-scheduler/build.sh#L63

peternied · 2021-10-06T19:30:31Z

Existing issues have been resolved but release was stopped with this error:

typeId | RepositoryWritePolicy
-- | --
failureMessage | Artifact updating: Repository ='releases:Releases' does not allow updating artifact='/org/opensearch/common-utils/maven-metadata-local.xml'
failureMessage | Artifact updating: Repository ='releases:Releases' does not allow updating artifact='/org/opensearch/notification/maven-metadata-local.xml'
failureMessage | Artifact updating: Repository ='releases:Releases' does not allow updating artifact='/org/opensearch/opensearch-job-scheduler-spi/maven-metadata-local.xml'

dblock · 2021-10-06T20:31:09Z

@peternied pls recap all the manual steps you did to unblock opensearch-1007 #close

peternied · 2021-10-07T02:51:39Z

Script to generate missing checksums, gen-checksum.sh

echo "Deleting any existing artifacts from $1"
find $1 -type f -name '*.md5' | xargs rm
find $1 -type f -name '*.sha1' | xargs rm
find $1 -type f -name '*.sha256' | xargs rm
find $1 -type f -name '*.sha512' | xargs rm


echo "Generated new checksums"
for file in $(find $1 -type f)
do
  if [ ${file##*.} != "asc" ]
  then
  echo "Creating checksum for $file"
  (md5sum $file | cut -d ' ' -f 1) > $file.md5 
  (sha1sum $file | cut -d ' ' -f 1) > $file.sha1
  (sha256sum $file | cut -d ' ' -f 1) > $file.sha256
  (sha512sum $file | cut -d ' ' -f 1) > $file.sha512
  fi
done

Note: This is documentation of the process used for this release, not what we should do and many of these steps have already been mitigated with other issues resolved in this and component repositories

Prereqs:

Install maven
export env variables for the sonatype release see stage-maven-release.sh
Download the signed artifacts from s3://artifact-bucket-stack-buildbucket-9omh0hnpg12q/builds/1.1.0/405/x64/ to the local disk

Remediation steps:

Delete artifacts that should not be published
- rm -rf ./org/opensearch-performance-analyzer
- rm -rf ./org/opensearch/opensearch-job-scheduler
- rm -rf ./org/opensearch/performanceanalyzer-rca
Rename the maven-metadata-local.xml to maven-metadata.xml
- mv ./org/opensearch/common-utils/maven-metadata-local.xml ./org/opensearch/common-utils/maven-metadata.xml
- mv ./org/opensearch/notifications/maven-metadata-local.xml ./org/opensearch/notifications/maven-metadata.xml
- mv ./org/opensearch/opensearch-job-scheduler-spi/maven-metadata-local.xml ./org/opensearch/opensearch-job-scheduler-spi/maven-metadata.xml
Generate all of the missing checksums
- gen-checksum.sh ./org/opensearch/common-utils
- gen-checksum.sh ./org/opensearch/notifications
- gen-checksum.sh ./org/opensearch/opensearch-job-scheduler-spi
Make sure there is no properties file in the root
- rm ./*.properties
Deploy the modified artifacts to sonatype
- ./publish/stage-maven-release.sh signing-fix/artifacts

Deploy Artifacts

Go to https://aws.oss.sonatype.org/index.html#stagingRepositories
Find the staging repo created with the deployment
close the repository, wait up to 20 minutes while checksums and signatures are validated
release the repository, wait up to 24 hours for the artifacts to be visible in public, see

dblock · 2021-10-07T12:14:53Z

Delete artifacts that should not be published

This was fixed in opensearch-project/job-scheduler#71 and #703

Rename the maven-metadata-local.xml to maven-metadata.xml

The maven-metadata-local.xml comes from publishing to ~/.m2, and it's being removed when we publish via publishShadowPublicationToMavenLocal to fix signatures. So neither of these files will be present in 1.2. Is that going to be an issue? @peternied did you try to delete the -local.xml as a first solution?

Generate all of the missing checksums

Fixed as above.

Make sure there is no properties file in the root

Where do we see one in the build today? I can't find any.

peternied · 2021-10-07T16:43:28Z

Make sure there is no properties file in the root

Where do we see one in the build today? I can't find any.

While the stage-maven-release.sh release script runs it puts a XXXXXXXXXXX.properties in the same directory as the maven artifacts. When I had to rerun the script this artifact would be uploaded and I would be unable to remove via the Sonatype UX, prevent closing the repository. Tracking this with #712

peternied · 2021-10-07T16:45:17Z

Rename the maven-metadata-local.xml to maven-metadata.xml

The maven-metadata-local.xml comes from publishing to ~/.m2, and it's being removed when we publish via publishShadowPublicationToMavenLocal to fix signatures. So neither of these files will be present in 1.2. Is that going to be an issue? @peternied did you try to delete the -local.xml as a first solution?

I did not attempt to remove the file. Do we need the maven-metadata.xml at all, if not, we can alter the publishing process to filter them out before starting. Tracking with #713

peterzhuamazon added bug Something isn't working untriaged Issues that have not yet been triaged labels Oct 5, 2021

peterzhuamazon assigned dblock Oct 5, 2021

peterzhuamazon added release v1.1.0 labels Oct 5, 2021

peterzhuamazon mentioned this issue Oct 5, 2021

Release jars on Maven Central for 1.1.0 #685

Closed

peternied mentioned this issue Oct 5, 2021

Check maven central distribution requirements #700

Closed

4 tasks

This was referenced Oct 6, 2021

Attempting to publish to Sonatype staging fails to produce checksums GradleUp/shadow#718

Open

Fixing maven artifacts for 1.1 #703

Merged

peternied self-assigned this Oct 6, 2021

dblock mentioned this issue Oct 6, 2021

Document publishing maven checksums. opensearch-project/opensearch-plugins#89

Merged

1 task

dblock closed this as completed in #703 Oct 7, 2021

peternied mentioned this issue Oct 7, 2021

Document publishing to maven central #701

Closed

dblock mentioned this issue Oct 12, 2021

[Build] Add OpenSearch Dashboards to Jenkinsfile #717

Merged

1 task

dblock mentioned this issue Nov 2, 2021

[Bug]: Automate maven publication to Sonatype staging for validation #855

Closed

mch2 mentioned this issue Nov 8, 2021

Update instructions for including checksums with maven publications. opensearch-project/opensearch-plugins#99

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Maven 1.1.0 failed Central Sync due to multiple errors #699

[Bug] Maven 1.1.0 failed Central Sync due to multiple errors #699

peterzhuamazon commented Oct 5, 2021

dblock commented Oct 6, 2021

peternied commented Oct 6, 2021

dblock commented Oct 6, 2021

peternied commented Oct 7, 2021 •

edited

Loading

dblock commented Oct 7, 2021 •

edited

Loading

peternied commented Oct 7, 2021 •

edited

Loading

peternied commented Oct 7, 2021 •

edited

Loading

[Bug] Maven 1.1.0 failed Central Sync due to multiple errors #699

[Bug] Maven 1.1.0 failed Central Sync due to multiple errors #699

Comments

peterzhuamazon commented Oct 5, 2021

Describe the bug

To reproduce

Expected behavior

Screenshots

Host / Environment

Additional context

Relevant log output

dblock commented Oct 6, 2021

peternied commented Oct 6, 2021

dblock commented Oct 6, 2021

peternied commented Oct 7, 2021 • edited Loading

dblock commented Oct 7, 2021 • edited Loading

peternied commented Oct 7, 2021 • edited Loading

peternied commented Oct 7, 2021 • edited Loading

peternied commented Oct 7, 2021 •

edited

Loading

dblock commented Oct 7, 2021 •

edited

Loading

peternied commented Oct 7, 2021 •

edited

Loading

peternied commented Oct 7, 2021 •

edited

Loading