Development: Restructure docker compose setups

[4ludwig4]

Checklist

General

• I tested these configs locally (Artemis) and with multiple docker setups (Artemis, MySQL, Jenkins, GitLab, Atlassian Stack).
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Motivation and Context

So far docker compose files were mainly written as little helpers for development but not for the usage in production, development, and the combination of multiple docker compose services.

This PR structures the docker compose files and documents the setups and services.

The Artemis Dockerfile was also refactored with the goal to build the whole application without the need of a .war file beforehand and therefore documenting the build process.

Description

This PR lead to the idea of restructuring the configuration files in the following PR, but is NOT depended on it:

• General: Clean up Spring configurations #5944

Migration Path

Bamboo Tests

I re-added and updated the E2E bamboo scripts which @TheZoker and I debugged manually last week to get the E2E tests stable again. So these scripts have to be activated again (load from version control instead of inline) after merging this PR.

Links to the bamboo plans:
https://bamboo.ase.in.tum.de/build/admin/edit/editBuildTasks.action?buildKey=ARTEMIS-AETG-DA
https://bamboo.ase.in.tum.de/build/admin/edit/editBuildTasks.action?buildKey=ARTEMIS-AECF-DA

The tasks are 2x CLEANUP (has to point to the corresponding ./.bamboo/.../cleanup.sh) and Deploy Artemis and run Cypress tests (has to point to the corresponding ./.bamboo/.../execute.sh) for each plan. After switching the script location from Inline to File the corresponding paths are probably already set.

Docker volumes

As some docker volumes were renamed, you might have to rename these if you want to keep your state of your docker containers.

Do so by looking up the current volume names:

docker volume ls

The old volume names might have prefixes like docker_.
This shows a table with the old_volume name and the new_volume name:

old_volume	new_volume
[PREFIX]artemis-bamboo	artemis-bamboo-data
[PREFIX]artemis-bitbucket	artemis-bitbucket-data
[PREFIX]artemis-jira	artemis-jira-data
[PREFIX]artemis-gitlab-data	artemis-gitlab-data
[PREFIX]artemis-gitlab-logs	artemis-gitlab-logs
[PREFIX]artemis-gitlab-config	artemis-gitlab-config
[PREFIX]artemis-jenkins-data	artemis-jenkins-data
[PREFIX]artemis-mysql-data	artemis-mysql-data
[PREFIX]artemis-gitlabci-runner-config	artemis-gitlabci-runner-config

Then the following procedure should be done for each volume as there is no other way to rename a volume apparently in docker:

docker volume create --name <new_volume>
docker run --rm -it -v <old_volume>:/from -v <new_volume>:/to alpine ash -c "cd /from ; cp -av . /to"
docker volume rm <old_volume>

Hard-coded Docker IPs

Remove hard-coded docker IP addresses from your application-local.yml.

Steps for Testing

• Check/Read the docs concerning the Dockerfile and the Docker Compose setups
• Test each or a subset of the following setups:
  • Artemis-Prod-MySQL (docker-compose.yml in the docker directory)
  • Atlassian (separate Atlassian stack with build agent, check that builds complete and start in a separate docker container on the docker host)
  • GitLab-Jenkins (separate GitLab-Jenkins stack)
  • MySQL (separate MySQL container)
  • Prometheus-Grafana (separate Prometheus Grafana stack)
• Test the Bamboo E2E Setup (can be tested in this customized Bamboo Build pipeline)

Note: If you do not have time to build the Docker Image by yourself, you can also pull it from:
ghcr.io/ls1intum/artemis:pr-5915

Review Progress

Code Review

• Review 1
• Review 2

Manual Tests of these setups:

• Artemis-Prod-MySQL 1
• Artemis-Prod-MySQL 2
• Atlassian 1
• Atlassian 2
• GitLab-Jenkins 1
• GitLab-Jenkins 2
• MySQL 1
• MySQL 2
• Prometheus-Grafana 1
• Prometheus-Grafana 2
• Bamboo E2E Setup 1
• Bamboo E2E Setup 2

Exam Mode Test

• Test

[4ludwig4]

@Mtze I participated in a testing session as discussed and got some more feedback which I already incorporated, but in general it ran if the students were on a x86_64 platform. Would be cool if you could also give this PR your approval if it's ready enough.

@dfuchss and @b-fein Thx that you already took a look at this PR in the past! I hope I was able to address all of your and the other people's feedback. Would be cool if you could retry at least the ./docker/artemis-prod-mysql.yml setup which now also has a Nginx in front and bind mounts for the volumes and minor other improvements.

@Hialus Thx for already integrating and testing this with the test servers. Would be cool if you could test it there and then, depending on how it goes, also give this PR an approval. If you need help ping me on Slack!

[frederik-kt]

Tried creating the mysql container using docker compose. Worked as expected.

[sleiss]

Tried the mysql & production setup - both work as expected.
Had issues with Gitlab & Jenkins - will retest it later.

[Hialus]

Tested the production setup (Artemis, MySQL, Nginx) extensively over the last couple of weeks and continuously gave Ludwig feedback. This stack will be used as a basis for the test servers soon, so I tested them specifically in that context. The current docker compose files work as expected in that regard now.
The setup isn't really ideal, but thanks to docker compose 2.16 this is sadly required.

I also reviewed the code for the related files and they look good to me now. The changes to the build-deploy GitHub workflow also look good to me, though I hope ARM support comes soon 😉

Overall I approve of this PR, as it greatly cleans up the docker compose files and makes dev and prod deployment way easier. 🚀
Good job 👍

Now for those interested:
The test server setup now looks like this in the working directory:

• .env file for the secrets
• docker-compose.yml to override and set some additional values for the test server (e.g. which database to use)
  • This also includes a way to check out a specific PR. More on that soon
• The Artemis repository to have these files available
• Some other folders that are passed to Artemis
• Due to docker compose 2.16 the command for interacting with this is rather lengthy and has the form docker compose --project-directory $PROJECT_DIR -f $COMPOSE_FILE --env-file $ENV_FILE $ARGS. The project dir points to the docker folder in the Artemis repository. $ARGS would be something like up -d

All of these files and folders can be automatically created with ansible using the artemis-ansible-collection after the related PR there gets merged.

[b-fein]

Retested mysql prod setup again 👍

One small code comment, looks good otherwise.

[dfuchss]

The dockerfile works for us. I've added two minor comments. But overall, this PR looks good to me :)

[TheZoker]

Docker image build and run without any obvious issues for me. Also the E2E tests still seem to pass, so LGTM 👍