Release SQL JDBC driver

[Yury-Fridlyand]

Is your feature request related to a problem? Please describe

JDBC driver was never released on Maven. The one published on artifacts page is a bit old already and should be updated.

Describe the solution you'd like

Publish most recent JDBC driver on Maven and on artifacts page.
Driver build is available as GHA artifact: https://github.com/opensearch-project/sql/actions/workflows/sql-jdbc-test-and-build-workflow.yml?query=branch%3A2.x

Describe alternatives you've considered

N/A

Additional context

#794
opensearch-project/sql#401

CC: @anirudha

[acarbonetto]

Before releasing new drivers, we need to update the JDBC version number in the gradle build file:
https://github.com/opensearch-project/sql/blob/2.x/sql-jdbc/build.gradle#L27
It is currently set at 1.1.0.1

[anirudha]

@bbarani can we move this to maven and setup a process for this, thnks

[prudhvigodithi]

Hey @Yury-Fridlyand we have some publish scripts handy, could use them, but quick question, do you have a proper project or gradle task when executed publishes SQL JDBC driver jar to local maven repo (~/.m2) with proper maven group Id like org.opensearch.driver? may be if we have that task or job we can use it part of the CI to publish it to actual maven repo with right creds.
Example for opensearch-job-scheduler plugin zip

<groupId>org.opensearch.plugin</groupId>
<artifactId>opensearch-job-scheduler</artifactId>
<version>2.3.0.0</version>

Thank you

[Yury-Fridlyand]

Thanks, I will have a look and make all preparations. We need to do few minor fix in the driver prior to release.
I'll ping you once everything will be ready.

[MaxKsyunz]

@prudhvigodithi yes there's a publishToMavenLocal task in jdbc driver.

One quirk with JDBC driver is that it can build two jars -- normal one, and one with all dependencies shadow copied. The second one is what can be used right now by just dropping the jar in the classpath. We want to publish the first one.

[MaxKsyunz]

Notes from our discussion.

Examples to follow:

• opensearch-java publishes with Jenkins like this.
• opensearch-testcontainers publishes with Jenkins like this.
  JDBC driver will be set up similarly.

Someone with admin permissions on sql repo will need to setup a webhook used by Jenkins to publish releases when a release tag is made.

[MaxKsyunz]

@peterzhuamazon since JDBC driver is in the sql repo, what git tag pattern should its releases have? The Jenkins pipelines currently assume refs/tags/v

[MaxKsyunz]

Also @peterzhuamazon do you know if JDBC driver will be kept in the plugin repo or split out? The discussion happened here but it's not clear what will happen to drivers.

If JDBC driver stays in a monorepo with something else, will there be one Jenkinsfile for each product in the repo or one Jenkinsfile for all of repo?

[peterzhuamazon]

Hi @MaxKsyunz I added several comments to @Yury-Fridlyand PR there.
Would love to discussion more later on the strategies of the repo.

Thanks.

[dblock]

@MaxKsyunz Is there an issue on extracting the JDBC driver into its own repo with pros/cons already? Care to open one so we can discuss that there?

My uneducated opinion is that it looks like the JDBC driver should follow its own semver, and have its own CI/CD that publishes it to maven similar to how we do this now in opensearch-java and other repos. Sounds to me like it should live in its own repo, but I don't know what its dependencies are.

[acarbonetto]

My uneducated opinion is that it looks like the JDBC driver should follow its own semver, and have its own CI/CD that publishes it to maven similar to how we do this now in opensearch-java and other repos. Sounds to me like it should live in its own repo, but I don't know what its dependencies are.

Raised:

• Split JDBC Driver into it's own repository  sql#964
• Split ODBC Driver into it's own repository sql#965
• Split CLI Tool into it's own repository sql#966
  for specific discussion and execution @dblock - thanks!

[anirudha]

is repo split a blocker for maven release now on this issue or is it a separate issue ?

[acarbonetto]

is repo split a blocker for maven release now on this issue or is it a separate issue ?

Separate issue. At least until I'm informed otherwise.

[dblock]

AFAIK if we were able to release the driver before I don't see why splitting should be a blocker, however to setup automation you'll need to queue off a tag, and that becomes more difficult in a monorepo (you'd have to queue of a tag format, unclear which branch to tag, etc.).

[peterzhuamazon]

Hi,

Have new discussion with @MaxKsyunz @Yury-Fridlyand @penghuo here is the method we are going to use:

1. Tag creation trigger a github action to clone the code, build maven artifacts, and push to an s3 bucket.
2. Github action will trigger jenkins workflow with the webhook.
3. Jenkins workflow will download the maven artifacts from s3 bucket, sign them, stage to sonatype.
4. Build repo members login to sonatype to release artifacts to maven central.

Thanks.

[anirudha]

thanks peter, can you provide n estimate on this.

[Yury-Fridlyand]

Release automation is tracked in #2505. This ticket is to track the current JDBC release.
The deadline is ~ Nov 9, 2022.

---

I created a JDBC driver pre-release on SQL repo: https://github.com/opensearch-project/sql/releases/tag/JDBC-2.0.0.0
Please proceed with manual publish on the Maven (no-shadow-copy-jar.zip).

[YANG-DB]

We recently separated SQL into multiple repositories - this issue is now referred to the appropriate (SQL-JDBC) repo
https://github.com/opensearch-project/sql-jdbc

[gaiksaya]

Hi,
I am looking into integrating the workflows for releasing SQL-JDBC. Publishing snapshots integration is depending on #1182
For releasing artifacts to maven central, I'll create PR with above repo.
Thanks!

[gaiksaya]

Hi,

@MaxKsyunz @YANG-DB @anirudha,

Looks like ./gradlew build test shadowJar is just generating a single jar. Can you follow the instructions posted here: #2692 (comment) and it get it in so that we can publish to maven?
Example: https://aws.oss.sonatype.org/content/repositories/releases/org/opensearch/opensearch-job-scheduler/2.4.1.0/
This has source java doc and application jar under same folder. Ideally it should be this way so that a user can download all 3 of them with same groupID. Also the gradle command should generate it all in a folder org/opensearch/driver. Can someone look into that?
Thanks!

[YANG-DB]

hi
can we remove the next jenkins file
https://github.com/opensearch-project/sql-jdbc/blob/main/jenkins/jdbc-stage-maven-release.jenkinsfile

and only have the next workflow ?
https://github.com/opensearch-project/spring-data-opensearch/blob/main/.github/workflows/release-drafter.yml

[gaiksaya]

The jenkins file will be replaced similar by the ones similar to https://github.com/opensearch-project/spring-data-opensearch/blob/main/jenkins/release.jenkinsFile
For generating release artifacts we need some gradle command similar to this line https://github.com/opensearch-project/spring-data-opensearch/blob/main/.github/workflows/release-drafter.yml#L24 for jbdc that will take care of everything stated here #2692 (comment)

[YANG-DB]

OK I will replace the jenkins file with a similar to the one you suggested - can u plz take care for the creation of the appropriate tokenIdCredential: 'jenkins-sql-jdbc-opensearch-generic-webhook-token' ? Thanks

[gaiksaya]

Hey! Looks like there is some confusion. Sorry for not being clear. I'll take care of adding release workflows (GHA workflow as well as jenkins), also will set up other things like webhook and stuff. However, before we proceed with any of that, jdbc team needs to fix what I mentioned here in above comment.
Please update this issue once its done.
Thanks!

[Yury-Fridlyand]

@gaiksaya,
Try using

./gradlew publishPublishMavenPublicationToLocalRepoRepository

It produces following files in build directory:

├── repository
│   └── org
│       └── opensearch
│           └── client
│               └── opensearch-sql-jdbc
│                   ├── 2.0.0.0
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.md5
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.jar.sha1
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom
│                   │   ├── opensearch-sql-jdbc-2.0.0.0.pom.md5
│                   │   └── opensearch-sql-jdbc-2.0.0.0.pom.sha1
│                   ├── maven-metadata.xml
│                   ├── maven-metadata.xml.md5
│                   └── maven-metadata.xml.sha1

ref: opensearch-project/sql#957
I can modify the script to produce sources.jar and javadoc.jar too, if needed.

[gaiksaya]

Thanks @Yury-Fridlyand , that would be great! So as far I understand this is what we want to publish on maven right?
Got confused because of the shadow jar conversation.

[Yury-Fridlyand]

@gaiksaya, please, see opensearch-project/sql-jdbc#45

[YANG-DB]

@gaiksaya - plz add the 'release_drafter.yml' workflow to the following branches

• main
• 2.x
  Thanks

[gaiksaya]

Hey @YANG-DB ,
Github actions only triggers workflows on default branch (which I assume is main). The triggers are at repo level. So it you want to publish something on 2.x branch, just checkout the branch, create a tag and push it. It will trigger the workflow and checkout the git ref accordingly.
Will work on adding the release workflows. Thanks!

[gaiksaya]

Few questions:

1. What should be the trigger for the release? The ones followed by various other repos in this org is using a tag. However, I had offline discussion with @YANG-DB that they want to control publishing snapshots (in future) via tag too.
2. Please confirm the below artifacts needs to be published to maven:

build/repository
└── org
    └── opensearch
        └── driver
            └── opensearch-sql-jdbc
                ├── 1.2.0.0
                │   ├── opensearch-sql-jdbc-1.2.0.0.jar
                │   ├── opensearch-sql-jdbc-1.2.0.0.jar.md5
                │   ├── opensearch-sql-jdbc-1.2.0.0.jar.sha1
                │   ├── opensearch-sql-jdbc-1.2.0.0.pom
                │   ├── opensearch-sql-jdbc-1.2.0.0.pom.md5
                │   └── opensearch-sql-jdbc-1.2.0.0.pom.sha1
                ├── maven-metadata.xml
                ├── maven-metadata.xml.md5
                └── maven-metadata.xml.sha1

[YANG-DB]

hi

1. tag is fine
2. yes @Yury-Fridlyand can you approve ?

[gaiksaya]

Thanks,
Please make sure you do not use tag for snapshots as well then as it will draft a release, triggering the release workflows. For publishing snapshots I believe push is a good way to go. Your call.

One more question:
What should be the java version? Need it to specify here https://github.com/opensearch-project/spring-data-opensearch/blob/main/.github/workflows/release-drafter.yml#L19

[YANG-DB]

lets go with java 11
@Yury-Fridlyand - sound correct ?

[Yury-Fridlyand]

lets go with java 11
@Yury-Fridlyand - sound correct ?

Yes

[gaiksaya]

Hi @YANG-DB , @joshuali925 , @anirudha
Was the security review completed for this version? Since the release template was not used, asking this question here.
Please confirm. Thanks!
cc: @bbarani

[gaiksaya]

Got confirmation from @dblock @bbarani that security review is not required since its not a new product just code shifted to new repo. All the set up is done. We are good for the release. Will sync up with @YANG-DB offline to cut a tag.
Thanks!

[gaiksaya]

sql-jdbc 1.2.0.0 is out https://aws.oss.sonatype.org/content/repositories/releases/org/opensearch/driver/opensearch-sql-jdbc/1.2.0.0/
Thanks everyone involved in this release. The process is end to end automated. You can check more details about universal release mechanism in this issue #1234.
Regarding releases see the documentation in the repo https://github.com/opensearch-project/sql-jdbc/blob/main/RELEASING.md

[YANG-DB]

The SQL 2.5 release, 2.x and main branches are now depended on this maven artifact

see https://github.com/opensearch-project/sql/blob/050e7b6b1e54ceaf1f33ec37c9ecd91e2d040186/integ-test/build.gradle#L87

[gaiksaya]

Hi @YANG-DB, @joshuali925

You can now publish snaphots to maven using GHA directly. See the PR[1] as an example on what all needs to be added to make that happen. You can add me as a reviewer once PR is ready so that I can make changes at the backend and add the required role to retrieve the credentials.
Thanks!

[1] opensearch-project/opensearch-java#325