Automating Docker Release for Opensearch and Opensearch Dashboards

[abhinavGupta16]

Is your feature request related to a problem? Please describe

https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/README.md

Currently the docker release build process is manual and is done on the server. This needs to be automated and added to the Jenkins workflow.

Describe the solution you'd like

Adding the automated build release for docker to the Jenkins workflow.

Describe alternatives you've considered

No response

Additional context

No response

Outstanding Tasks (Outdated):

• Make the script commands section a fixed list or retrieve value from manifest.
• Add scm to monitor docker dir and auto trigger docker builds.

[VijayanB]

Can this be made generic to support #726 as well?

[peterzhuamazon]

Can this be made generic to support #726 as well?

It is.

[peterzhuamazon]

Create a bot account as part of the prod automation.

[peterzhuamazon]

Adding a EC2 image builder to create Jenkins Docker Build AMI.

[peterzhuamazon]

V458352478

[peterzhuamazon]

Some issues pop up as x64 host not able to run logstash-plugin on arm64 through buildx.

The command timed out and even failed with logstash-plugin --help.

 => ERROR [linux/arm64 3/3] RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch                                                                                                                                                                                                           564.9s
------
 > [linux/arm64 3/3] RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch:
#11 0.149 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#11 0.191 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#11 0.349 Using bundled JDK: /usr/share/logstash/jdk
#11 4.249 /usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#11 4.731 OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
#11 473.7 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
#11 564.5 ERROR: Something went wrong when installing logstash-output-opensearch, message: Socket closed
------
Dockerfile:18
--------------------
  16 |     USER logstash
  17 |     COPY --chown=logstash:logstash ./logstash-opensearch-sample.conf /usr/share/logstash/config/
  18 | >>> RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch
  19 |
--------------------
error: failed to solve: process "/bin/sh -c /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch" did not complete successfully: exit code: 1

[peterzhuamazon]

Trying to see if running on arm64 host would help.

[peterzhuamazon]

See an issue:

 => => # qemu: uncaught target signal 11 (Segmentation fault) - core dumped

This only shows after running buildx on arm64 host and the amd64 shows this, errors shows opposite on the buildx docker based on the host arch.

Assuming this is related to binary translation issues with qemu.

[peterzhuamazon]

Seems common issue: docker/for-mac#5123

[peterzhuamazon]

Seems like the best building platform for issues with this is still on docker desktop on macOS x64.

https://stackoverflow.com/questions/68862313/qemu-uncaught-target-signal-11-segmentation-fault-core-dumped-in-docker-con

[peterzhuamazon]

Seems like this is specific to Ubuntu2004 which is the one I am using.
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1928075

[peterzhuamazon]

Tried qemu 5 installation but doesnt seems resolving the issue:
docker/buildx#328 (comment)

[peterzhuamazon]

Multiple comments have indicate the switching back from 11 to 10 debian, aka 20 to 19 Ubuntu would fix the issue:

Ubuntu            Debian  
21.04  hirsute    bullseye/ sid   - 11
20.10  groovy     bullseye/ sid
20.04  focal      bullseye/ sid
19.10  eoan       buster  / sid   - 10
19.04  disco      buster  / sid
18.10  cosmic     buster  / sid
18.04  bionic     buster  / sid

However, Ubuntu 19 is already out of support.
We can try to switch to Debian 10 since it will be supported until 2024.

[peterzhuamazon]

Debian10 has docker 18 which does not recognize the cli plugin buildx, despite it is running API over 1.25 version.

admin@ip-10-0-37-16:~/.docker/cli-plugins$ docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.11.6
 Git commit:        4c52b90
 Built:             Sun, 21 Feb 2021 18:18:35 +0100
 OS/Arch:           linux/amd64
 Experimental:      true

Server:
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.6
  Git commit:       4c52b90
  Built:            Sun Feb 21 17:18:35 2021
  OS/Arch:          linux/amd64
  Experimental:     true

https://docs.docker.com/engine/reference/commandline/plugin/

[peterzhuamazon]

Also, Ubuntu 1804 despite is using Debian 10 it is not possible as binfmt-support version 2.1.43 or later is required.

[peterzhuamazon]

After manually install docker to the higher version on Debian 10 I am able to see docker buildx:

-rw-r--r-- 1 admin admin 24733166 Aug 17 12:11 containerd.io_1.4.9-1_amd64.deb
-rw-r--r-- 1 admin admin 38815666 Nov  2 10:07 docker-ce-cli_20.10.9~3-0~debian-buster_amd64.deb
-rw-r--r-- 1 admin admin 21184082 Nov  2 10:07 docker-ce_20.10.9~3-0~debian-buster_amd64.deb
-rw-r--r-- 1 admin admin  3520264 Nov  2 10:07 docker-scan-plugin_0.9.0~debian-buster_amd64.deb

https://download.docker.com/linux/debian/dists/buster/pool/stable/amd64/

github.com/docker/buildx v0.4.1 bda4882a65349ca359216b135896bddc1d92461c

github.com/docker/buildx v0.6.3 266c0eac611d64fcc0c72d80206aa364e826758d

[peterzhuamazon]

After a long battle with all the combinations between debian 10 x64/arm64, ubuntu 2004 x64/arm64, the conclusion is, you need to properly upgrade Ubuntu 2004 x64 host qemu to 5.0 (not available on any other 3 combination):

qemu-user-static/focal,now 1:5.0-5ubuntu6~ppa0 amd64 [installed]
qemu/focal,now 1:5.0-5ubuntu6~ppa0 amd64 [installed]

It works:

 => [linux/arm64 3/3] RUN /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch                                                                                                                                                                                                                 161.1s
 => => # Using bundled JDK: /usr/share/logstash/jdk
 => => # /usr/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
 => => # OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
 => => # /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
 => => # Validating logstash-output-opensearch
 => => # Installing logstash-output-opensearch

[peterzhuamazon]

-rw-r--r-- 1 ubuntu ubuntu    58232 Nov  2 17:06 binfmt-support_2.2.0-2_amd64.deb
-rw-r--r-- 1 ubuntu ubuntu 21253472 Nov  2 17:06 qemu-user-static_1%3a4.2-3ubuntu6.18_amd64.deb
-rw-r--r-- 1 ubuntu ubuntu 16892560 Nov  2 17:06 qemu-user_1%3a5.0-5ubuntu6~ppa0_amd64.deb

[peterzhuamazon]

More improvements:

• Make the script commands section a fixed list or retrieve value from manifest.
• Add scm to monitor docker dir and auto trigger docker builds.

[peterzhuamazon]

Create #952 for the improvements, we can close this issue for now.
Thanks.