Creating Distribution for OpenSearch Plugin for mac for unreleased version

[navneet1v]

Is your feature request related to a problem? Please describe

I am trying to create a new OpenSearch plugin which will be dependent on OpenSearch K-NN plugin. Now while enabling the integration test task, I need to install the k-NN plugin on the cluster but for macOS there is no release for k-NN plugin(version 3.0 and 2.4). This is resulting in usage of 2.3 version of plugin and also OpenSearch core.

Problem:
There is no artifact for unreleased versions of plugin for macOS.

Describe the solution you'd like

The solution that I am recommending is we should have something like OpenSearch-3.0.0-SNAPSHOT version for plugins so that other plugins can take dependency on that version of plugin and run the integration tests and development on mac.

Describe alternatives you've considered

Alternative is to only use the older released version of plugin like right now I am using the 2.3 version of plugin rather than 2.4 or 3.0 which this new plugin should use.

Additional context

No response

[dblock]

@navneet1v I am a bit confused, do you want to a) an opensearch-3.0.0-SNAPSHOT published for darwin, or b) k-nn plugin -SNAPSHOT published for Darwin (which would require a I believe), or c) something else?

Mac distribution is tracked in #38. That already includes both of these asks I believe, but please correct me if I am wrong. If I am right, let's close this as a dup?

If you want to help, start by enabling OSX builds in OpenSearch core, which is opensearch-project/OpenSearch#1449.

[navneet1v]

@dblock I am looking for 3.0.0-SNAPSHOT and 2.4.0-SNAPSHOT distribution of the K-NN plugin for macOS which is not present now. opensearch-3.0.0-SNAPSHOT is available as per my understanding.

[dblock]

@navneet1v I think for k-nn that's specifically blocked by opensearch-project/k-NN#157 ? so dup of that? sorry my bad windows vs. mac

I meant to say opensearch-project/k-NN#158. Start there?

Looking at https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.0.0/opensearch-2.0.0.yml#L39 that plugin may have been working for MacOS when those snapshot builds were being made. But either way start with CI within that plugin.

[dblock]

@peterzhuamazon What's the status of MacOS builds? It sounds like we're publishing 2.x and 3.0 MacOS snapshots somewhere and those only include some components? Where?

[peterzhuamazon]

As of now we dont push any snapshots version of min artifacts + plugin zips to ci.opensearch.org.
We only push the min artifacts to artifacts.opensearch.org/snapshots for the purpose of using them as snapshots.
If we want to push snapshots to ci.opensearch.org as well we need some changes to jenkinsfile.

Let me know.

[gaiksaya]

More details:
We just have snapshot stage for min here: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L172-L201
and nothing else for macos as of now.

[peterzhuamazon]

The snapshot stage was initially added for the sake of distribution downloader.
The problem being at the time we dont have ci.opensearch.org.
And we did not build -SNAPSHOT in ci.opensearch.org because there is no point, since we can just build without -SNAPSHOT and push them into ci.opensearch.org and treat as snapshot.

I propose we just add a stage of normal macOS build.

[navneet1v]

@bbarani is there a timeline that I can track when this can be picked up? Reason being we are developing a plugin that depends on K-NN and ML-Commons, and we are planning to release for 2.4. If we don't have that it will create problems while doing the release as the checks will keep on failing because of non-existent artifacts of other plugins.

[bbarani]

@navneet1v This is not on our priority list at this point in time but feel free to submit a PR with the changes and we can help with review.

[navneet1v]

@bbarani I can go ahead and do the changes, but problem is with the testing of those changes. If there is setup that I can use to test my changes I can go ahead and do that.

@peterzhuamazon I am not sure of the architecture here, but I can see a stage as pointed by @gaiksaya here: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch/distribution-build.jenkinsfile#L172-L201 . If that is the case then what else is required?

[navneet1v]

Closing this issue, as I we are able to build with 3.0.0 version of OpenSearch with neural plugin.