[Design] OpenSearch/Dashboards DEB Distribution Design Document

[mnin]

Coming from #28

---

OpenSearch DEB Distribution Design Document

Hey everyone,

I want to explain all details for the workflow of providing the DEB packages for the future releases on OpenSearch/Dashboards.

In parts, this topic has already been dealt with in Issue Design OpenSearch/Dashboards DEB/RPM Distribution Design Document · Issue #1452 · opensearch-project/opensearch-build · GitHub.

Please feel free to comment and raising questions about it.

To be transparent, I am currently working for the company Graylog, who would like to provide a DEB packages to their customers as well.

Introduction

As 2022/08, OpenSearch project published Tarball, Docker Images, RPM packages and YUM packages as part of the release artifacts. As a result, the community asked in multiple issues for DEB packages for OpenSearch.

This design issue is based on the issue created in 2022/01 Design OpenSearch/Dashboards DEB/RPM Distribution Design Document · Issue #1452 · opensearch-project/opensearch-build · GitHub. This design issue has already covered parts of the DEB packages build. But then mainly the building of the RPM package was described in the issue.

Requirements

[Milestone 1] OpenSearch build process for DEB package generation

• The current workflow which already generates Tarball and RPM packages should be modified to also generate DEB packages.
• The way the current packages are built should also be feasible for building the DEB packages by using the same behavior and arguments.
• The build process which already exists should be used to create the contents of the DEB packages.
• Create one DEB package for OpenSearch.
• Create one DEB package for OpenSearch-Dashboards.
  • Post scripts should register the plugins.
• Should include scripts for pre and post stuff.
• Should include systemd and sysvinit configuration files.

[Milestone 2] DEB Package Signing

• The provided DEB packages should be signed with GPG to allow verify the source.
• Should provide a GPG public keyring to verify the source of the packages.
• Likewise, a checksum file with SHA512 should be provided.
• Maybe debsign is an option to use (man page is debsign(1) — devscripts — Debian bullseye — Debian Manpages) to sign the packages.

[Milestone 3] DEB Package Design

• The Debian package should consider the following the naming schema opensearch_1.0.0_amd64.deb and opensearch-dashboards_1.0.0_amd64.deb.
• As for now, we should provide two DEB packages, one which contains OpenSearch and another for the Dashboards check Opensearch 2.1.0 · OpenSearch
• The package should be built for the architectures amd64 and arm64.
• The packages should use the prerm, postinst features to store a script here which will be executed after the package has been unpacked.
  • With the help of these scripts, in the case when the packages have to be uninstalled, for example, the services should be stopped before removing files from the filesystem.
  • These scripts should also initialise plugins automatically.
  • We should be able to adopt the existing scripts which are already used for the RPM package.
• Include a description, LICENSE and README file.
• When assembling the DEB packages, dependencies should be added to the package definition. And also define to be incompatible with other versions of the same DEB packages.
• The package should create user/group for the daemon.
• The dashboards plugin package should require the opensearch package as an dependency.
• The dashboards plugin package should automatically register the plugins in OpenSearch.

[Milestone 4] DEB Package Publishing

• Publish DEB packages files for direct downloading
• Publish via DEB repository
  • Publish a DEB repository which can be created by for example aptly - Debian repository management tool see also DebianRepository - Debian Wiki.
  • The DEB packages and the DEB repository should be uploaded to the existing artifacts.opensearch.org with the same process which already exists.
  • Also the generated e.g. Packages files for the repository should be signed with PGP, aptly are able to sign the files.

[Milestone 5] Installation

The DEB packages should work on the current Debian (11.4 Bullseye) and Ubuntu (22.04.1 LTS) versions.

• Users should be able to download the DEB packages directly from Opensearch 2.1.0 · OpenSearch and installing them with:

dpkg -i opensearch_2.0.0_amd64.deb opensearch-dashboards_2.0.0_amd64.deb

• Users should be able to add the APT repository and install the DEB packages with:

apt-get install opensearch opensearch-dashboards

[Milestone 6] Validation

The build system should apply tests to validate the quality of the delivered product before releasing to public.

These tests should cover the latest Debian and Ubuntu distribution.

• Validate DEB packages file downloading.
• Validate DEB packages downloading over APT repository.
• Validate DEB packages signing.
• Validate DEB packages installing.
• Validate DEB packages upgrading/downgrading.
• Validate DEB packages uninstalling.

Implementation

The already existing workflow should be adapted and extended to build the DEB packages as well as to deploy them in the existing infrastructure to serve the files.

To build the packages, the build system should use Debian Bullseye or Ubuntu 22.04.1 for this.

Assemble the DEB distribution

The existing scripts that already work for RPM packages should be adapted so that they also work for DEB packages.

• DEB packages build should also based on the tar and plugin installation.
• Scripts for pre and post hooks, systemd and sysvinit files should be included.

Usage

Installation

With dpkg:

sudo dpkg -i opensearch_<version>_amd64.deb
sudo dpkg -i opensearch-dashboards_<version>_amd64.deb

With the APT repository:

# download the gpg key
wget -qO - https://artifacts.opensearch.org/GPG-KEY-opensearch | sudo gpg --dearmor -o /usr/share/keyrings/opensearch-keyring.gpg

# install dependencies
sudo apt-get install apt-transport-https

# add the opensearch repository
echo "echo "deb [signed-by=/usr/share/keyrings/opensearch-keyring.gpg] https://artifacts.opensearch.org/packages/2.x/apt stable main" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list"

# install opensearch
sudo apt-get update && sudo apt-get install opensearch opensearch-dashboards

Test

Once the OpenSearch daemon is installed, the status can be queried from the cluster, checkout the API https://opensearch.org/docs/latest/opensearch/popular-api/

Uninstallation

With dpkg or APT repository:

sudo dpkg --remove opensearch_<version>_amd64.deb
sudo dpkg --remove opensearch-dashboards_<version>_amd64.deb

Usage with systemd to manipulate the daemon state

Start

sudo systemctl enable --now opensearch
sudo systemctl enable --now opensearch-dashboards

Stop

sudo systemctl disable --now opensearch-dashboards
sudo systemctl disable --now opensearch

Status

sudo systemctl status opensearch
sudo systemctl status opensearch-dashboards

Reference links

• https://manpages.debian.org/bullseye/devscripts/debsign.1.en.html
• https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf
• HOWTO: GPG sign and verify deb packages and APT repositories | Packagecloud Blog
• aptly - Debian repository management tool

[bbarani]

@peterzhuamazon @gaiksaya @prudhvigodithi @zelinh @dblock Please provide your feedback and comments on this design issue.

[peterzhuamazon]

Hi @mnin thanks for the design document.

A few points:

1. We need to think about whether the repo metadata should be signed as well, apart from pkg, such as repomd.xml in the yum repo for rpm.
2. M3 the name should be matching rpm such as opensearch-dashboards-1.3.2-linux-x64.deb, the old naming has been replaced by us during the release of rpm.
3. M3 the pre/post scripts have already been created for rpm, thus can be re-used for deb I think.
4. M4 we should mention the tools to use for apt repo creation, such as aptly.

Thanks.

[mnin]

Thanks @peterzhuamazon for reviewing this issue.

1. We need to think about whether the repo metadata should be signed as well, apart from pkg, such as repomd.xml in the yum repo for rpm.

I added a comment to the milestone 4 section to clarify this.

2. M3 the name should be matching rpm such as opensearch-dashboards-1.3.2-linux-x64.deb, the old naming has been replaced by us during the release of rpm.

I would like to keep the naming convention from Debian: https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html#pkgname

Or does that not make sense?

3. M3 the pre/post scripts have already been created for rpm, thus can be re-used for deb I think.

I added another point to the milestone 3 to adopt the existing scripts which are used for the RPM package.

4. M4 we should mention the tools to use for apt repo creation, such as aptly.

I already mentioned aptly twice times, one in the milestone 4 section and in the reference links section. Might be enough, or what you think?

Thank you.

[bbarani]

@elfisher @krisfreedain @dblock @setiah Do you have any inputs on the naming part?

2. M3 the name should be matching rpm such as opensearch-dashboards-1.3.2-linux-x64.deb, the old naming has been replaced by us during the release of rpm.

I would like to keep the naming convention from Debian: https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html#pkgname

Or does that not make sense?

[setiah]

I like following the standard Debian naming convention as suggested by @mnin. The package naming conventions do vary across platforms so it doesn't seem like a problem to me.

[dblock]

+1 on following what Debian wants, even though I find it aesthetically displeasing ;) I found this interesting too.

[peterzhuamazon]

RPM is already not following the rpm naming scheme, this requires a lot of changes to our build scripts just to consume a new naming scheme.

[elfisher]

+1 to following the Debian naming convention.

[peterzhuamazon]

Hi,

Just to compare, all of the current artifacts in TAR/RPM/ZIP are following this naming convention:

opensearch-dashboards-1.3.2-linux-x64.<extension>

If we make debian to follow the debian naming scheme it would become an outlier compares to other distribution.

Tho not a blocker but definitely something we need to be careful when approaching the renaming.

Thanks.

[bbarani]

@elfisher @dblock @setiah @mnin I assume you all are good with following different naming convention for Debian packages compared to existing OpenSearch artifacts as mentioned above by @peterzhuamazon ? Can we proceed based on that assumption?

[dblock]

I think you should make this call @peterzhuamazon. My opinion is not very uneducated.

[mnin]

I followed the recommendation of keeping the naming scheme as it currently is with OpenSearch packages.

Maybe this can still lead to problems when building the APT repository.

[bertvandepoel]

I'm not sure where to ask, but I guess this is the most appropriate issue: what exactly is the plan for migrations? Many people have simply frozen their ELK stacks on the latest non-SSPL version, hoping that opensearch will mean a direct upgrade path. Based on the fact that names are different and there's no mention of transition/meta packages, I wonder if there are any plans for people wishing to migrate, or will new installations be presumed?

[dblock]

@bertvandepoel See https://opensearch.org/docs/latest/upgrade-to/upgrade-to/. Do you have a scenario not covered by that? We'd like to help!

[peterzhuamazon]

We have merged the #2526 PR thanks to @mnin contribution.

@setiah @bbarani We will put it onto Jenkins pipeline as the next step.

Thanks.

[bbarani]

Closing this issue as we have designed and added Debian support in 2.5.0 release