Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Windows Support #157

Closed
1 of 2 tasks
dblock opened this issue Oct 27, 2021 · 17 comments · Fixed by #595
Closed
1 of 2 tasks

Add Windows Support #157

dblock opened this issue Oct 27, 2021 · 17 comments · Fixed by #595
Assignees
Labels
Enhancements Increases software capabilities beyond original client specifications k-NN roadmap v2.4.0 'Issues and PRs related to version v2.4.0' windows

Comments

@dblock
Copy link
Member

dblock commented Oct 27, 2021

Coming from opensearch-project/opensearch-plugins#95, add Windows support.

  • Passing CI on Windows
  • Documentation
@bbarani
Copy link
Member

bbarani commented Jun 29, 2022

@vamshin We have started the work on adding support for Windows distribution and need your inputs on the timeline to support KNN on Windows.

@setiah
Copy link

setiah commented Jul 8, 2022

Hey @vamshin, do we have any updates here?

@vamshin
Copy link
Member

vamshin commented Jul 8, 2022

Hi @setiah @bbarani , we are prioritizing this task. Will update timeline soon. Thanks

@vamshin vamshin added v2.4.0 'Issues and PRs related to version v2.4.0' roadmap k-NN labels Jul 27, 2022
@navneet1v navneet1v added the Enhancements Increases software capabilities beyond original client specifications label Jul 27, 2022
@setiah
Copy link

setiah commented Aug 17, 2022

Hi @bbarani @peterzhuamazon, since we are tracking Windows effort with milestones, is there a specific milestone we would like to map the KNN effort with, to ensure we have these artifacts in time for the 2.4 release? That should help @vamshin and team to plan accordingly. cc @elfisher

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Sep 13, 2022

Current issues:

  1. Windows mingw libgfortran-5.dll is version 5 and we are currently renaming it to libgfortran-3.dll to use.
  2. The OpenBlas lib for windows is old on binary https://sourceforge.net/projects/openblas/files/v0.2.9.rc2/ seems like the official releases have not release any win version after 0.2.19. We need to deep dive a bit into seeing how to compile openblas, possibly, on windows to match the version on linux.
  3. Add lib env vars to user specific env var section, or we can set it to global.

@peterzhuamazon
Copy link
Member

New issues, the lib files provided by the knn team is invalid:

# Yours
$ file lib*
libopensearchknn_faiss.dll:  data
libopensearchknn_nmslib.dll: data

# Download a random windows valid dll online
$ file A3D.dll
A3D.dll: PE32 executable (DLL) (GUI) Intel 80386, for MS Windows

@peterzhuamazon
Copy link
Member

New issues:

How to make sure the plugin/libs folder present in java.library.path:

java.library.path = 
        C:\Windows\Sun\Java\bin
        C:\Windows\system32
        C:\Windows
        C:\Windows\system32
        C:\Windows
......

Probably will try to append the path like linux version through the OS startup batch script.

@peterzhuamazon
Copy link
Member

I am able to add the path to java.library.path

On windows as long as you expose the path into Path variable it will expose to the java.library.path

We can add a batch file through cmd commands to dynamically update the Path variable during startup

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Sep 29, 2022

We are able to successfully install knn on 2.3.0 snapshot windows zip and complete the KNN integTest run.

Notes:

  1. Remove the prefix lib in front of the faiss and nmslib .dll file. As the lib prefix is specifically used in linux. In windows, it seems to look for the exact naming conversion you defined in the code:
    https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/jni/FaissService.java#L34
    https://github.com/opensearch-project/k-NN/blob/main/src/main/java/org/opensearch/knn/common/KNNConstants.java#L63-L95

  2. As long as the path is added to Path env vars, it will show up in the java.library.path variable and will be loaded into the process.

  3. Use .bat to run everything and ignore the bash version.

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Sep 29, 2022

$  ./gradlew integTest -Dopensearch.version=2.3.0-SNAPSHOT -Dbuild.snapshot=true -Dtests.rest.cluster="localhost:9200" -Dtests.cluster="localhost:9200" -Dtests.clustername="opensearch-integrationtest" -Dhttps=false -Duser=admin -Dpassword=admin --console=plain
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.5
  OS Info               : Windows Server 2019 10.0 (amd64)
  JDK Version           : 17 (Eclipse Temurin JDK)
  JAVA_HOME             : C:\Users\Administrator\scoop\apps\temurin17-jdk\17.0.4-101
  Random Testing Seed   : 400D094C45C29453
  In FIPS 140 mode      : false
=======================================
> Task :processResources UP-TO-DATE
> Task :processTestFixturesResources NO-SOURCE
> Task :generateNotice UP-TO-DATE
> Task :copyPluginPropertiesTemplate UP-TO-DATE
> Task :pluginProperties UP-TO-DATE
> Task :processTestResources UP-TO-DATE
> Task :qa:compileJava NO-SOURCE
> Task :qa:processResources NO-SOURCE
> Task :qa:classes UP-TO-DATE
> Task :qa:processTestResources NO-SOURCE
> Task :qa:restart-upgrade:compileJava NO-SOURCE
> Task :qa:restart-upgrade:processResources NO-SOURCE
> Task :qa:restart-upgrade:classes UP-TO-DATE
> Task :qa:restart-upgrade:processTestResources NO-SOURCE
> Task :qa:rolling-upgrade:compileJava NO-SOURCE
> Task :qa:rolling-upgrade:processResources NO-SOURCE
> Task :qa:rolling-upgrade:classes UP-TO-DATE
> Task :qa:rolling-upgrade:processTestResources NO-SOURCE
> Task :generateEffectiveLombokConfig
> Task :compileJava UP-TO-DATE
> Task :classes UP-TO-DATE
> Task :jar UP-TO-DATE
> Task :bundlePlugin UP-TO-DATE
> Task :generateTestFixturesEffectiveLombokConfig
> Task :compileTestFixturesJava UP-TO-DATE
> Task :generateTestEffectiveLombokConfig
> Task :testFixturesClasses UP-TO-DATE
> Task :testFixturesJar UP-TO-DATE
> Task :compileTestJava UP-TO-DATE
> Task :testClasses UP-TO-DATE

> Task :integTest
OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0

> Task :qa:compileTestJava NO-SOURCE
> Task :qa:testClasses UP-TO-DATE
> Task :qa:integTest SKIPPED
> Task :qa:restart-upgrade:compileTestJava UP-TO-DATE
> Task :qa:restart-upgrade:testClasses UP-TO-DATE
> Task :qa:restart-upgrade:integTest SKIPPED
> Task :qa:rolling-upgrade:compileTestJava UP-TO-DATE
> Task :qa:rolling-upgrade:testClasses UP-TO-DATE
> Task :qa:rolling-upgrade:integTest SKIPPED

BUILD SUCCESSFUL in 11m 57s
17 actionable tasks: 4 executed, 13 up-to-date

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Sep 30, 2022

@naveentatikonda Could you help comment on what is the issue with BWC test challenges on KNN for windows support?

cc: @setiah

@naveentatikonda
Copy link
Member

@naveentatikonda Could you help comment on what is the issue with BWC test challenges on KNN for windows support?

cc: @setiah

@peterzhuamazon We might need to make few changes to the BWC framework to make it compatible with windows. For example url of the tar file. To test this and make those changes we need an artifact of any BWC version with k-NN plugin bundled in that artifact.

Also, as per our previous conversation I know we need to push our windows changes to 2.x branch for you to include k-NN plugin into that artifact. But, for now will you be able to pick those changes from any feature branch as we are still trying to make our script more organic without any work arounds?

@peterzhuamazon
Copy link
Member

Have some discussion with @naveentatikonda and he would provide two branches for 2.3.0 and 2.2.1 respectively. I will use those branches to build bundles on my fork.

Then he can try to implement the bwc test.

Thanks.

@naveentatikonda
Copy link
Member

@peterzhuamazon windows_support_v4 this feature branch on my fork points to OS version 2.3.0 and ws_os_version_2.2.1 points to OS version 2.2.1. Please let me know if you need anything else, Thanks!

@naveentatikonda
Copy link
Member

@peterzhuamazon windows_support_v4 this feature branch on my fork points to OS version 2.3.0 and ws_os_version_2.2.1 points to OS version 2.2.1. Please let me know if you need anything else, Thanks!

@peterzhuamazon do we have any update on this ? Also, just want to remind you about the changes that you want to make for the jenkins build script

@peterzhuamazon
Copy link
Member

@naveentatikonda sorry for the late reply I would have some time tomorrow to get a bundle.

Thanks.

@peterzhuamazon
Copy link
Member

Working on building 2.3.0 with windows_support_v4 branch now and also editing the build script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancements Increases software capabilities beyond original client specifications k-NN roadmap v2.4.0 'Issues and PRs related to version v2.4.0' windows
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants