[Discuss] OpenSearch/Dashboards Windows Distribution Design Document

[peterzhuamazon]

OpenSearch/Dashboards Windows Distribution Design Document

Hi everyone,

We want to give you more details on how we currently experiment/implement the assemble workflows of Windows ZIP/Installer for the future releases on OpenSearch/Dashboards.

Please feel free to use this as guidance for contributing on Windows, as well as raising questions about our approach.

Introduction/Problem Statement

As of 2022/06, OpenSearch project has published Tarball/Docker/RPM as part of the release artifacts. We have noticed a strong demand for Windows artifacts from the community. Community is currently requesting Windows distribution for OpenSearch and OpenSearch-Dashboards to deploy them on Windows Platform. We have completed the initial planning and design for the generation of Windows distributions as listed below,

Meta issue for tracking the Windows Distribution progress:

• [Meta] OpenSearch / OpenSearch Dashboards - Windows distribution (X64) #33
• [META] Add Windows Support opensearch-plugins#95

Milestones

• [WIN M1] WIN (ZIP/MSI) Build/Assemble Process  #2306
• [WIN M2] WIN (ZIP/MSI) Final Release Artifact Structure #2307
• [WIN M3] WIN (ZIP/MSI) Validation #2308
• [WIN M4] WIN (ZIP/MSI) Publishing #2309
• [WIN M5] WIN (ZIP/MSI) Installation #2310
• [WIN M6] WIN (ZIP/MSI) Signing #2311

Requirements

Here we explain the details of Windows artifacts that our users can expect during release:

• Naming:
  • ZIP: WIN artifact name should follow the binary file naming scheme %{name}-%{version}-%{platform}-%{architecture}.%{extension} so we have a clear path for upgrade/re-release (e.g. opensearch-1.2.0-windows-x64.zip).
  • Installer: Same but opensearch-1.2.0-windows-x64.exe or opensearch-1.2.0-windows-x64.msi.
• Install/Format:
  • ZIP: Just like tarball for advanced usages, user can extract zip at any location with sufficient permission to run the binary with batch scripts.
  • Installer: Our current plan is to provide .msi (Windows Installer) package, but we are open to feedback if .exe or any other installation format is most preferred for Windows users. Please comment on this issue to share your feedback. With installer, user can choose installation path, setup windows service, etc. through a GUI.
• Runtime Dependencies:
  • Both ZIP and Installer should have their dependencies bundled them in the artifact, such as JDK, NodeJS, etc.
• Pre/Post Scripts:
  • ZIP: WIN-ZIP will have tar-install equivalent batch script to handle min and plugin specific configurations upon start.
  • Installer: Installer artifact will have pre/post install/remove scripts to handle min and plugin specific configurations.
• Folder Structure:
  • ZIP: Like in Tarball, ZIP should have fixed folder structures pre-configured for users.
  • Installer: Installer executable should handle the creation and removal of user/group/dir during installation / removal, and apply any of these logic from the pre/post scripts.
• Size:
  • Both artifacts should be as slim as possible, without duplicate libs and related dependencies
• Logs:
  • ZIP: ZIP archive should store logs within the extracted location /log directory.
  • Installer: Installer should redirect logs to corresponding log location per Windows standard.
• Uninstall:
  • ZIP: Uninstall zip archive depend all on users.
  • Installer: User can go to control panel to remove the application. It will not remove user data from the Windows host, unless user specify when running the uninstaller.
• Metadata:
  • Both artifacts should have a clear description of its content in metadata.
• License:
  • Both artifacts should have the LICENSE file included in the zip/installer.
• Service:
  • ZIP: ZIP archive does not have windows service setup.
  • Installer: Installer should contain windows service related service file, so user can manage the software with ease.
• Plugin Initialization:
  • Both zip and msi installers should handle the initialization of certain plugins automatically, such as default security plugin configurations, setup demo certs, etc.

Frequently Asked Questions:

Q1. What types of artifacts will you release for Windows distribution?
A1: We will release both the ZIP archive and the Installer(.exe/.msi) artifacts of Windows distribution.

Q2: What plugins will you include in the Windows artifacts?
A2: All the plugins will be included in Windows artifacts, except k-NN and Performance-Analyzer as a minimum deliverable. We have opened issues with k-NN and PA already on their Windows support, and will try to add them into the final deliverable if possible.

• Add Windows Support k-NN#157
• Add Windows Support performance-analyzer#75

Q3: Which product of opensearch project would you include in this Windows distribution?
A3: Both OpenSearch and OpenSearch-Dashboards artifacts will be included.

Q4: What CPU architecture do you support in the Windows distribution?
A4: We are aiming to release x64 architecture support 1st, arm64 architecture for Windows is still in research.

Q5: How do you ensure the quality of the Windows distribution?
A5: All of the deliverables will have their corresponding integration test created to ensure production quality, you can access these test logs and results through out public Jenkins instance.

Q6: How can someone contribute to the Windows distribution?
A6: We welcome everyone to take part in the contribution. We will publish our milestone issues soon and start the development. Please share your feedback in the meta issue for Windows distribution here.

Q7: What is the communication platform for users to communicate with the dev team?
A7: Please communicate through the Windows distribution meta issue here.

Additional Notes

• We are going to assemble the code on Windows Server 2019 due to support WSL1:
  • For our usecase of running Python code and do assemble, there is no difference between 2019 and 2022.
  • 2022 using EC2Launch which removed a lot of scripts such as SysprepInstance.ps1, and stuck on stopping the instance for creating AMI.
  • LTS:
    • 2019 LTS 2029
    • 2022 LTS 2031

---

Thanks,
OpenSearch Project.

[rursprung]

i don't personally need a windows installer (we're not running OpenSearch on Windows and for local development i can just play with a ZIP file which i'm not installing) and i'm not super familiar with windows installers, but if i remember correctly Microsoft wanted to push developers from MSI towards MSIX. it seems that it also supports services, so it might fit here.
this is something you might want to look into, though i don't know all the implications of it.

[dblock]

I've used https://github.com/wixtoolset/ extensively (which seems to not support msix), and have authored much of https://github.com/dblock/msiext which would make building an installer for OpenSearch a couple of days' job, except that technology is ten years old and probably obsolete. So someone who has recently looked into Windows installers would need to give some strong opinions about the technology available today (MSI vs. MSIX) before choosing something.

A property written Windows installer would support install, uninstall, in-place upgrades, would have optional UX to enter a bunch of inputs such as a port number to run OpenSearch on, or a cluster to join, where to store data. It would correctly install OpenSearch as a service, optionally using an account other than LocalSystem. What else?

[bbarani]

@dblock @peterzhuamazon @rursprung Can't we not use the official MSIX tool to create MSIX from generated executable?

[dblock]

@bbarani I don't know I've never used it. There's a lot more to an installer than turning an executable into an installable executable (see my comment above).

[HoffmannTom]

I used WIX-Toolset for my needs and everything worked so far, even if it can get quite complex.

Here is a comparison chart which might help to decide what to use:
https://pacesuite.com/blog/msi-vs-msix-overview-and-comparison/
msix is more leightweight but has some restrictions, like when you need admin privileges or access to some reg-keys.

[dblock]

Re: EXE vs. MSI, typically EXEs are bootstrappers/bundles that contain multiple components (see https://wixtoolset.org/documentation/manual/v3/bundle/ or https://github.com/dotnetinstaller/dotnetinstaller/), while MSIs are single installer packages. Today an OpenSearch distribution is a core + plugins, so the question is: is the distribution 1 big MSI with preinstalled plugins, or is it 1 MSI for -min and MSIs for each plugin that can be installed/uninstalled in sequence, and a distribution that bootstraps multiple MSIs. I would prefer it if each plugin were its own MSI because it would be able to specify its own dependencies, and is closer to the extensions model where you can release an MSI for a plugin/extension separately.

[MaxKsyunz]

Generally, the bootstrap installer for a complex application is responsible for providing the UI for managing plugins/features.

As a decent example, here's what SQL Server installer looks like -- it walks the user through pre-requisites, features/extension selection, and service configuration.

Powershell DSC is also worth considering. It allows for repeatable unattended installs on multiple nodes.

[peterzhuamazon]

Windows already release.