Simplify GitLab CI documentation

[TjeuKayim]

Last week, I spend some time figuring out the best configuration for GitLab CI and DinD. At first, I used the example in the docs. Then I tried other ways.

As it turns out, the documented GitLab configuration is over-complicated. It uses three containers (docker:latest, gradle:3.4, docker:dind) and runs docker in a script, but that extra layer is unnecessary. By setting the DOCKER_HOST environment variable, you only need two containers.

This is a simpler alternative configuration:

# we need DinD service
services:
  - docker:dind

variables:
  # to improve performance
  DOCKER_DRIVER: overlay2
  # connect testcontainers to DinD service
  DOCKER_HOST: "tcp://docker:2375"

test:
 image: gradle:3.4
 stage: test
 script: ./gradlew test

This configuration is more readable, and has better performance. That's why I suggest to update the documentation for using Gitlab CI.

How the previous configuration operates

Disclaimer: I'm not a Docker expert.

Let's take this configuration as an example:

# complex configuration that I don't recommend
services:
  - docker:dind

gradle check:
  image: docker:latest
  script:
    - echo $DOCKER_HOST
    # returns tcp://docker:2375
    - >
      docker run -t --rm 
      -v /var/run/docker.sock:/var/run/docker.sock
      -v "$(pwd)":"$(pwd)"
      -w "$(pwd)"
      -u 0:0
      openjdk:11 ./gradlew check

The docker run command uses the daemon specified by the DOCKER_HOST environment variable.

At Gitlab CI, DOCKER_HOST defaults to tcp://docker:2375 if you define a docker:dind service (proof).
If you don't define such service, it will be empty.

So in the example above, docker run will use the Docker daemon inside the docker:dind service (with default hostname docker).

The documentation explains it this way:
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-docker-in-docker-executor

When using dind service we need to instruct docker, to talk with the
daemon started inside of the service. The daemon is available with
a network connection instead of the default /var/run/docker.sock socket.

The 'docker' hostname is the alias of the service container as described at
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services

Note that if you're using Kubernetes executor, the variable should be set to
tcp://localhost:2375 because of how Kubernetes executor connects services
to the job container

As I understand it, docker run will delegate the command to the docker:dind service. That service will spawn an openjdk container. In turn, Testcontainers will use the Docker daemon of docker:dind through the /var/run/docker.sock binding.

In conclusion, this configuration is over-complicated, because the docker container does nothing but delegating its work to docker:dind.

[TjeuKayim]

I just created this example project on Gitlab.
https://gitlab.com/TjeuKayim/testcontainers-gitlab-ci/blob/master/.gitlab-ci.yml

[kiview]

Hey @TjeuKayim,
this looks pretty nice an is indeed much simpler.

I'm normally just a Gitlab on-prem user, using Docker socket mounting, so I was never sure what's the best way to achieve this in Gitlab cloud. But I think your example is definitely nicer as the current one, so thanks.

[TjeuKayim]

Another working example:
https://gitlab.com/sivalabs/spring-boot-2-sample/merge_requests/1/diffs?view=parallel

In reply to #967.

[TjeuKayim]

Maybe we should recommend this new configuration in the docs, but also mention the old configuration as an alternative.
Or just remove the old configuration entirely?

[kiview]

I'm kind of wondering how file mounting works for both configs. I kind of assume it's not working for both examples, since the DinD is another container?

[TjeuKayim]

It’s indeed a bit mysterious how that old configuration worked. I’m not a Docker expert either.

[bsideup]

@kiview either Gitlab configures current directory and one-to-one path mapping or that example project is not mounting any files

[TjeuKayim]

Both the previous and the new Gitlab CI configurations support the use of withClasspathResourceMapping().
I created a working example at https://gitlab.com/TjeuKayim/testcontainers-gitlab-ci/merge_requests/3/.

[kiview]

@bsideup I think Gitlab has some special hacks for DinD service, that will automatically mount stuff from build container into DinD container.

I'll try to check in the Gitlab sources later.

Nevertheless, I think this example config is better for the general Gitlab DinD user, so I'd like to merge it.
WDYT @bsideup?

[kiview]

Okay, found the explanation in the Gitlab docs:

Since version 1.5 GitLab Runner mounts a /builds directory to all shared services.

See:
https://gitlab.com/gitlab-org/gitlab-runner/issues/1520

[bsideup]

@kiview looks good to me 👍

@TjeuKayim extra points for testing it 👍

[rnorth]

Nice! Thanks a lot for this @TjeuKayim - it's much nicer to read. 🙇