Development: Unify docker compose settings

[4ludwig4]

⚠️ This is a PR to the develop-deployment-wg branch which is WIP, there might be still open TODOs which will be fixed in different PRs, but reviews and feedbacks are welcome!

Checklist

General

• This is a small issue that I tested locally and was confirmed by another developer on a test server.
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Motivation and Context

We would like to keep the Artemis container to a minimum and not integrate things needed in Docker, but maybe not in Kubernetes.
Also, the Docker Compose setup should be unified and easier to use.

Description

This PR unifies the Docker Compose settings and setups by doing the following:

• new Docker Compose specification, which merges v2 and v3 of the old Docker Compose specification.
• unified volume and network names
• no hard-coded IP addresses
• reusing duplicated docker services through the extends functionality

This PR also improves the current Docker setup with these changes:

• first iteration of multi-stage Dockerfile for Artemis
• refactored the bootstrap.sh for Artemis to make it rootless according to docker best practices
• introduced wait-for.sh shell script for docker-compose setups to control service startup order(this is something docker compose doesn't support out of the box)
• basic docker yaml config setup
• added some in-code TODOs which are either still thoughts or real tasks for which I didn't have time yet (limited to /src/main/docker and /docs/dev)
• added saml-test service for basic SAML manual testing/debugging
• added mailhog service for basic Mail testing/debugging
• added Java Remote Debugger settings/profiles for easy in Container Debugging
• gradle cleanup

Steps for Testing

Play a bit with the different docker-compose setups:

Build Artemis locally:
docker-compose -f src/main/docker/artemis-dev-mysql.yml build artemis-app

Run Artemis (dev profile) and MySQL:
docker-compose -f src/main/docker/artemis-dev-mysql.yml up

Stop and remove the volumes for this instance:
docker-compose -f src/main/docker/artemis-dev-mysql.yml down -v


For gitlab and jenkins setups, you can keep for instance the jenkins and gitlab volumes by stopping the containers:
docker-compose -f src/main/docker/artemis-dev-mysql-gitlab-jenkins.yml down

Then just remove the artemis-app and artemis-mysql volume:
docker volume rm artemis-data artemis-mysql-data

Also, maybe test the following:

• adding mailhog or saml (probalby not fully working yet) as an extra service with and additional -f mailhog.yaml
• other addtional services or docker-compose setups
• read the documentation about docker-compose and docker

Code Review

• Review 1
• Review 2

Manual Tests

• Test 1
• Test 2

[dfuchss]

Some comments

[dfuchss]

Instead of wait-for, you can use something like this:

  gitlab:
    build:
      context: ./docker/gitlab
      args:
          GITLAB_VERSION: ${GITLAB_VERSION}
    restart: unless-stopped
    volumes:
      - ./data/gitlab/config:/etc/gitlab
      - ./data/gitlab/logs:/var/log/gitlab
      - $BACKUP_DIR/gitlab:/var/opt/gitlab/backups
      - ./data/gitlab/data:/var/opt/gitlab
    ports:
      - "22:22"
    networks:
      - artemis-net
    healthcheck:
      test: curl --fail -s -o /dev/null http://gitlab:80 || exit 1
      interval: 60s
      retries: 10
      start_period: 120s
      timeout: 10s

[4ludwig4]

I just remember these two stackoverflows threads [1] [2] when I looked into health checks for the Dockerfile.

Both say that the healthchecks of docker are not triggering anything unless you hack something together with killing certain things in your healthcheck CMD etc. ...

It seems like Kubernetes and Podman have better integrations for this feature.

They also sadly don't provide other probes like Kubernetes does. That's why I thought the mounting of the wait-for script is the easiest way without a lot of own nc and curl scripting?! But I'm open for better solutions 😉

[dfuchss]

mmh ok, the command is waiting until gitlab is ready ; I did't mentioned any problems on our systems 😂

[4ludwig4]

After writing you a PM on Slack about your prod setup I now have more of an idea of an alternative to the wait-for script but just added it for now as a todo.

[4ludwig4]

@b-fein @dfuchss added your suggestions either as To-dos or directly. If the PR suits you this way, I would be happy to get an approval to merge it to our develop-deployment-wg branch and hopefully soon deploy it on the dedicated test server!

[b-fein]

Looks good for the initial iteration of the Docker cleanup. The remaining ToDos can be solved incrementally in the future.

Added just one more non-blocking question about the Artemis container build.