Skip to content

Latest commit

 

History

History
executable file
·
216 lines (166 loc) · 7.17 KB

autoscaling.md

File metadata and controls

executable file
·
216 lines (166 loc) · 7.17 KB
Raw

Install and configure AlloyCI Runner for autoscaling

For an overview of the auto-scale architecture, take a look at the comprehensive documentation on autoscaling.

Prepare the environment

In order to use the auto-scale feature, Docker and AlloyCI Runner must be installed in the same machine:

  1. Login to a new Linux-based machine that will serve as a bastion server and where Docker will spawn new machines from
  2. Install AlloyCI Runner following the AlloyCI Runner installation documentation
  3. Install Docker Machine following the Docker Machine installation documentation

Prepare the Docker Registry and Cache Server

To speedup the builds we advise to setup a personal Docker registry server working in proxy mode. A cache server is also recommended.

Install Docker Registry

Note: Read more in Distributed registry mirroring.

  1. Login to a dedicated machine where Docker registry proxy will be running

  2. Make sure that Docker Engine is installed on this machine

  3. Create a new Docker registry:

    docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    --restart always \
    --name registry registry:2

    You can modify the port number (6000) to expose Docker registry on a different port.

  4. Check the IP address of the server:

    hostname --ip-address

    You should preferably choose the private networking IP address. The private networking is usually the fastest solution for internal communication between machines of a single provider (DigitalOcean, AWS, Azure, etc,) Usually the private networking is also not accounted to your monthly bandwidth limit.

  5. Docker registry will be accessible under MY_REGISTRY_IP:6000.

Install the cache server

Note: You can use any other S3-compatible server, including Amazon S3. Read more in Distributed runners caching.

  1. Login to a dedicated machine where the cache server will be running

  2. Make sure that Docker Engine is installed on that machine

  3. Start minio, a simple S3-compatible server written in Go:

    docker run -it --restart always -p 9005:9000 \
        -v /.minio:/root/.minio -v /export:/export \
        --name minio \
        minio/minio:latest server /export

    You can modify the port 9005 to expose the cache server on different port.

  4. Check the IP address of the server:

    hostname --ip-address

  5. Your cache server will be available at MY_CACHE_IP:9005

  6. Read the Access and Secret Key of minio with: sudo cat /.minio/config.json

  7. Create a bucket that will be used by the Runner: sudo mkdir /export/runner. runner is the name of the bucket in that case. If you choose a different bucket then it will be different

  8. All caches will be stored in the /export directory

Configure AlloyCI Runner

  1. Register a AlloyCI Runner, selecting the docker+machine executor (Look into runners documentation to learn how to obtain a token):

    sudo alloy-runner register

    Example output:

    Please enter the coordinator URL (e.g. https://alloy-ci.com )
https://alloy-ci.com
Please enter the token for this runner
xxx
Please enter the description for this runner
my-autoscale-runner
INFO[0034] fcf5c619 Registering runner... succeeded
Please enter the executor: shell, docker, docker-ssh, docker+machine, docker-ssh+machine, ssh?
docker+machine
Please enter the Docker image (eg. ruby:2.1):
ruby:2.1
INFO[0037] Runner registered successfully. Feel free to start it, but if it's
running already the config should be automatically reloaded!

  2. Edit config.toml. You need to fill in the options for [runners.machine] and [runners.cache] and configure the MachineDriver selecting your provider. Also configure MachineOptions, limit and IdleCount.

    For more information visit the dedicated page covering detailed information about AlloyCI Runner Autoscaling.

    Example configuration using DigitalOcean:

    concurrent = 20

[[runners]]
executor = "docker+machine"
limit = 20
[runners.docker]
  image = "ruby:2.1"
[runners.machine]
  IdleCount = 5
  MachineDriver = "digitalocean"
  MachineName = "auto-scale-runners-%s.my.domain.com"
  MachineOptions = [
      "digitalocean-image=coreos-stable",
      "digitalocean-ssh-user=core",
      "digitalocean-access-token=MY_DIGITAL_OCEAN_TOKEN",
      "digitalocean-region=nyc2",
      "digitalocean-private-networking",
      "engine-registry-mirror=http://MY_REGISTRY_IP:6000"
    ]
[runners.cache]
  Type = "s3"
  ServerAddress = "MY_CACHE_IP:9005"
  AccessKey = "ACCESS_KEY"
  SecretKey = "SECRET_KEY"
  BucketName = "runner"
  Insecure = true # Use Insecure only when using with Minio, without the TLS certificate enabled

  3. Try to build your project. In a few seconds, if you run docker-machine ls you should see a new machine being created.

Upgrading the Runner

  1. Ensure your operating system isn't configured to automatically restart the runner if it exits (which is the default configuration on some platforms).

  2. Stop the runner:

    killall -SIGQUIT alloy-runner

    Sending the SIGQUIT signal will make the Runner to stop gracefully. It will stop accepting new jobs, and will exit as soon as the current builds are finished.

  3. Wait until the Runner exits. You can check its status with alloy-runner status or await a graceful shutdown for up to 30 minutes with:

    for i in `seq 1 180`; do # 1800 seconds = 30 minutes
    alloy-runner status || break
    sleep 10
done

  4. You can now safely upgrade the Runner without interrupting any builds

Manage the Docker Machines

  1. Ensure your operating system isn't configured to automatically restart the runner if it exits (which is the default configuration on some platforms).

  2. Stop the Runner:

    killall -SIGQUIT alloy-runner

  3. Wait until the Runner exits. You can check its status with: alloy-runner status or await a graceful shutdown for up to 30 minutes with:

    for i in `seq 1 180`; do # 1800 seconds = 30 minutes
    alloy-runner status || break
    sleep 10
done

  4. You can now manage (upgrade or remove) any Docker Machines with the docker-machine command