DEB packages build

[mnin]

Description

Introduce deb package build.

Issues Resolved

#2460

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dblock]

Looking good. Finish it up.

[codecov-commenter]

Codecov Report

Merging #2526 (6af7f23) into main (918ae0d) will decrease coverage by 0.54%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    opensearch-project/OpenSearch-Dashboards#2526      +/-   ##
==========================================
- Coverage   93.72%   93.17%   -0.55%     
==========================================
  Files         158      167       +9     
  Lines        4414     4602     +188     
==========================================
+ Hits         4137     4288     +151     
- Misses        277      314      +37     

Impacted Files	Coverage Δ
src/manifests/build_manifest.py	100.00% <ø> (ø)
src/manifests/bundle_manifest.py	97.43% <ø> (ø)
src/assemble_workflow/bundle_linux_deb.py	100.00% <100.00%> (ø)
src/assemble_workflow/bundle_linux_rpm.py	93.02% <100.00%> (ø)
src/assemble_workflow/dist.py	96.15% <100.00%> (+3.09%)	⬆️
src/assemble_workflow/dists.py	100.00% <100.00%> (ø)
src/build_workflow/build_args.py	100.00% <100.00%> (ø)
src/sign_workflow/signer_pgp.py	100.00% <100.00%> (ø)
src/system/os.py	100.00% <100.00%> (ø)
src/test_workflow/integ_test/distribution_deb.py	100.00% <100.00%> (ø)
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[setiah]

Hi @dblock @bbarani @peterzhuamazon , requesting PR review on behalf of @mnin. Thanks!

[dblock]

Solid work.

[dblock]

Thanks. After a more careful review I think there's a temp folder and a bit of a class hierarchy problem where it's easily fixable, see below, please.

[peterzhuamazon]

Thanks @mnin very much for the PR.
Apart from the reviews coming from @dblock I only have one tweak suggested.
We will keep this in draft until our next 2.4.0 version is out.

Thanks.

[peterzhuamazon]

Potential conflict right here due to we added a change to support windows.
Would need to resolve the conflict after 2.4.0.

#2845

[peterzhuamazon]

@mnin I am running into some assemble error here:

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2:-:internal"
HOME_URL="https://amazonlinux.com/"
VARIANT="internal"
Amazon Linux release 2 (Karoo)

% ./assemble.sh deb/builds/opensearch/manifest.yml
Installing dependencies in . ...
Installing dependencies from Pipfile.lock (b36c9c)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 6/6 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Running ./src/run_assemble.py deb/builds/opensearch/manifest.yml ...
2022-11-21 20:32:24 INFO     Bundling OpenSearch (x64) on linux into /local/home/zhujiaxi/releases/opensearch-build-matt/deb/dist/opensearch ...
2022-11-21 20:32:24 INFO     Copied min bundle to /tmp/tmpez6p46ss/opensearch-min_2.4.0_amd64.deb.
2022-11-21 20:32:24 INFO     Start creating distribution deb for OpenSearch.
2022-11-21 20:32:24 INFO     Extracting dist into /tmp/tmpez6p46ss.
2022-11-21 20:32:24 INFO     Extract data.tar.gz from deb for extraction: /tmp/tmpez6p46ss/opensearch-min_2.4.0_amd64.deb to /tmp/tmpez6p46ss/data.tar.gz
ar: unrecognized option '--output=/tmp/tmpez6p46ss'
Usage: ar [emulation options] [-]{dmpqrstx}[abcDfilMNoPsSTuvV] [--plugin <name>] [member-name] [count] archive-file file...
       ar -M [<mri-script]
 commands:
  d            - delete file(s) from the archive
  m[ab]        - move file(s) in the archive
  p            - print file(s) found in the archive
  q[f]         - quick append file(s) to the archive
  r[ab][f][u]  - replace existing or insert new file(s) into the archive
  s            - act as ranlib
  t            - display contents of archive
  x[o]         - extract file(s) from the archive
 command specific modifiers:
  [a]          - put file(s) after [member-name]
  [b]          - put file(s) before [member-name] (same as [i])
  [D]          - use zero for timestamps and uids/gids
  [U]          - use actual timestamps and uids/gids (default)
  [N]          - use instance [count] of name
  [f]          - truncate inserted file names
  [P]          - use full path names when matching
  [o]          - preserve original dates
  [u]          - only replace files that are newer than current archive contents
 generic modifiers:
  [c]          - do not warn if the library had to be created
  [s]          - create an archive index (cf. ranlib)
  [S]          - do not build a symbol table
  [T]          - make a thin archive
  [v]          - be verbose
  [V]          - display the version number
  @<file>      - read options from <file>
  --target=BFDNAME - specify the target object format as BFDNAME

[mnin]

@mnin I am running into some assemble error here:

This should be fixed with efcd9ba please try again.

[peterzhuamazon]

Assuming this is because debmake does not install by default on AL2, will install and check again soon:

2022-11-22 18:13:57 INFO     Execute debmake -f "OpenSearch Team" -e "opensearch@amazon.com" -i debuild -p opensearch -n -r 1 -u 2.4.0 in /tmp/tmpt2gg_mul
/bin/sh: debmake: command not found

[mnin]

Assuming this is because debmake does not install by default on AL2, will install and check again soon:

2022-11-22 18:13:57 INFO     Execute debmake -f "OpenSearch Team" -e "opensearch@amazon.com" -i debuild -p opensearch -n -r 1 -u 2.4.0 in /tmp/tmpt2gg_mul
/bin/sh: debmake: command not found

As we discussed via Slack, we need to have a build system based on Debian or Ubuntu to build the DEB package, because debmake isn't available for the AL2 distribution.

[peterzhuamazon]

I will need to have a new PR adding debian/ubuntu build image before this PR can be tested again.

Thanks.

[peterzhuamazon]

New issue.

As I have some issues with deb11 on docker (still fixing), I use an older image of ubuntu2004.

See this error:

I: debuild
 dpkg-buildpackage -us -uc -ui
dpkg-buildpackage: info: source package opensearch
dpkg-buildpackage: info: source version 2.4.0
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by OpenSearch Team <opensearch@amazon.com>
 dpkg-source --before-build .
dpkg-buildpackage: info: host architecture amd64
dpkg-checkbuilddeps: error: Unmet build dependencies: debhelper-compat (= 12)
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui failed
E: failed to build Debian package(s).
2022-11-28 23:39:18 INFO     Removing /tmp/tmptp441aej

[mnin]

New issue.

As I have some issues with deb11 on docker (still fixing), I use an older image of ubuntu2004.

And install the package debhelper-compat on the Ubuntu image?

[peterzhuamazon]

There are a few output that is not clear but overall the build is success, we can fix the nitpik parts in new PRs.

2022-11-29 20:39:50 INFO     Found deb file: None

2022-11-29 20:39:50 INFO     Published /opensearch-build/deb/dist/opensearch/opensearch-2.4.0-linux-x64.deb.
2022-11-29 20:39:50 INFO     Removing /tmp/tmpe3qs0uc5
2022-11-29 20:39:50 INFO     Done.

root@a4c4778c7cb6:/opensearch-build/deb/dist/opensearch# ll
total 218116
drwxr-xr-x 2 root root      4096 Nov 29 20:39 ./
drwxr-xr-x 3 root root      4096 Nov 28 22:09 ../
-rw-r--r-- 1 root root       515 Nov 29 20:39 manifest.yml
-rw-r--r-- 1 root root 223334216 Nov 29 20:39 opensearch-2.4.0-linux-x64.deb

Thanks @mnin we are happy to merge this code.
And thanks very much for your contribution.

PS: also confirm that RPM is not touched and still runs well.

Thanks.

@mnin has resolved the issues raised by @dblock, and I have co-review / test the code on a Ubuntu2004 Server.

[kavilla]

Seems like a new line is causing build failures: #2942

[mnin]

@kavilla thanks for reporting, fixed in #2944

[dblock]

We should look at getting rid of scripts/components/OpenSearch-Dashboards/build.sh and moving it into the OpenSearch Dashboards repo branches so it can change release after release. @mnin WDYT?

[mnin]

@dblock would make sense to me to move this stuff into the official repository.

[GezimSejdiu]

Hey everyone,

this is great and thanks a lot for pushing it up to also distribute deb packages.

Was this already part of the release? Or it will come only within the next release? I'm getting:

gpg: no valid OpenPGP data found

when I try to install it usign APT repository:

# download the gpg key
wget -qO - https://artifacts.opensearch.org/GPG-KEY-opensearch | sudo gpg --dearmor -o /usr/share/keyrings/opensearch-keyring.gpg

# install dependencies
sudo apt-get install apt-transport-https

# add the opensearch repository
echo "echo "deb [signed-by=/usr/share/keyrings/opensearch-keyring.gpg] https://artifacts.opensearch.org/packages/2.x/apt stable main" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list"

# install opensearch
sudo apt-get update && sudo apt-get install opensearch opensearch-dashboards

Also, based on the convention I also can't access: https://artifacts.opensearch.org/releases/bundle/opensearch-build/deb/dist/opensearch/opensearch-2.4.0-linux-x64.deb -- it says Access Denied.

Once again thanks a lot and looking forward to having it.

[peterzhuamazon]

Hey everyone,

this is great and thanks a lot for pushing it up to also distribute deb packages.

Was this already part of the release? Or it will come only within the next release? I'm getting:

gpg: no valid OpenPGP data found

when I try to install it usign APT repository:

# download the gpg key
wget -qO - https://artifacts.opensearch.org/GPG-KEY-opensearch | sudo gpg --dearmor -o /usr/share/keyrings/opensearch-keyring.gpg

# install dependencies
sudo apt-get install apt-transport-https

# add the opensearch repository
echo "echo "deb [signed-by=/usr/share/keyrings/opensearch-keyring.gpg] https://artifacts.opensearch.org/packages/2.x/apt stable main" | sudo tee /etc/apt/sources.list.d/opensearch-2.x.list"

# install opensearch
sudo apt-get update && sudo apt-get install opensearch opensearch-dashboards

Also, based on the convention I also can't access: https://artifacts.opensearch.org/releases/bundle/opensearch-build/deb/dist/opensearch/opensearch-2.4.0-linux-x64.deb -- it says Access Denied.

Once again thanks a lot and looking forward to having it.

Hi @GezimSejdiu this is only the python code to build on your local host.
We havent produce anything on Jenkins nor did we create any apt repo or promote to production bucket for you to consume.
So your command would not run.

We are trying to see if we can get it up for a release but it takes time to get it properly automated on our system.
Please be patient and stay tune to the progress.

Thanks.