[BUG] Use snapshots from distribution/maven for dependencies

[dblock]

Describe the bug

There are -SNAPSHOT builds checked into https://github.com/opensearch-project/cross-cluster-replication/tree/main/src/test/resources/security/plugin. This is error prone at the very least, use a distribution build.

[saikaranam-amazon]

There are -SNAPSHOT builds checked into https://github.com/opensearch-project/cross-cluster-replication/tree/main/src/test/resources/security/plugin. This is error prone at the very least, use a distribution build.

we have addressed this in this PR: #296 but keeping this issue open as original intent is to use distribution builds from maven to use in CI workflow.

[rursprung]

why was this closed? it's not yet using opensearch-security from maven:

cross-cluster-replication/.github/workflows/security-tests.yml

Lines 26 to 36 in a3d0bf0

	- name: Checkout security
	uses: actions/checkout@v2
	with:
	repository: 'opensearch-project/security'
	path: security
	ref: 'main'
	- name: Build security
	working-directory: ./security
	run: |
	./gradlew clean build -Dbuild.snapshot=false -x test
	cp build/distributions/opensearch-security-*.zip ../src/test/resources/security/plugin/opensearch-security.zip

=> this ticket should IMHO be re-opened as it isn't resolved.
note that this cannot be done before opensearch-project/security#1801 is implemented

[prudhvigodithi]

Hey @ankitkala @rursprung just to connect the dots once the issue is resolved opensearch-project/security#1801, the required zip files can be fetched from maven, if then can this issue be resolved?
@dblock @bbarani
Example (once the zips are published to maven) as
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://aws.oss.sonatype.org/content/repositories/snapshots/ -Dartifact=org.opensearch.plugin:opensearch-security:1.3.0.0-SNAPSHOT:zip

[rursprung]

you shouldn't have to manually run mvn in the GitHub Action - instead you should probably have the dependency in your build.gradle so that it also works in any other place where you build the project. otherwise you wouldn't really have to use maven but could just download the ZIP from any download location.

[prudhvigodithi]

you shouldn't have to manually run mvn in the GitHub Action - instead you should probably have the dependency in your build.gradle so that it also works in any other place where you build the project. otherwise you wouldn't really have to use maven but could just download the ZIP from any download location.

Hey @rursprung, I have just added as an example, but yes its should be fetched as dependency in gradle.

[prudhvigodithi]

Hey Starting with release 2.1.0, we now have plugin zips in maven repo
opensearch-project/opensearch-build#1916 (comment)
SecurityZip maven: https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-security/
@saikaranam-amazon @dblock @rursprung @ankitkala @bbarani
Thank you

[rursprung]

@prudhvigodithi: see my comment here: opensearch-project/index-management#403 (comment)

i think what's still missing to resolve this is the publishing of the ZIP files to the snapshot maven repo during snapshot builds. unless i was looking in the wrong place...? but then the solution for index-management is IMHO still wrong.

[prudhvigodithi]

Hey @rursprung, official release zips are part of maven repo, snapshots can be fetched from nexus maven repo
Example for job-scheduler snapshots

Quick test:

mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get -DrepoUrl=https://aws.oss.sonatype.org/content/repositories/snapshots/ -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0-SNAPSHOT:zip

Picked up JAVA_TOOL_OPTIONS: -Dlog4j2.formatMsgNoLookups=true
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:get (default-cli) @ standalone-pom ---
Downloading from temp: https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-job-scheduler/2.1.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from temp: https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-job-scheduler/2.1.0.0-SNAPSHOT/maven-metadata.xml (804 B at 1.6 kB/s)
Downloading from temp: https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-job-scheduler/2.1.0.0-SNAPSHOT/opensearch-job-scheduler-2.1.0.0-20220630.215242-39.zip
Downloaded from temp: https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-job-scheduler/2.1.0.0-SNAPSHOT/opensearch-job-scheduler-2.1.0.0-20220630.215242-39.zip (210 kB at 344 kB/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.557 s
[INFO] Finished at: 2022-07-10T08:30:10-04:00
[INFO] ------------------------------------------------------------------------

[bbarani]

@saikaranam-amazon @ankitkala whats the next step here?

[ankitkala]

We'll take this up in upcoming 1-2 weeks. @bbarani Are there any timelines you're looking at?
cc: @manishdev-amzn

[ankitkala]

Closing the issue now.