Loadbalaning issues with traefik and Docker Swarm cluster

[prateek2408]

Hi,

I am using the latest traefik binary (v1.1.0-rc1) :-

Docker swarm which I am using is -

I performed the following activity:-

1. Create a swarm cluster of 7 nodes.
2. Created an overlay network.
3. Created a traefik container using docker service create
4. Created 4 instances of 'emilevauge/whoami' via docker service create and docker service scale =4

Now if I go into the traefik container and do a curl on the whoami instance IP i get different host names signifying that the load is getting balanced, here are the snippets:-

However the traefik binary which is running in the same container and reaching the whoami service via the same internal IP is giving me same hostnames telling that load is going only to a single container.

Seems it is stuck to a single instance.

In the traefik logs I see that it is redirecting internally to the same IP so ideally I should get a different hostname, moreover load balancing is now taken care by swarm and has nothing to do with traefik but still this issue is coming for me when I am using traefik . Here is the snippet of traefik log

I am clueless here... Can anyone help.

Regards,
Prateek

[mthird]

Are you using DNSRR or VIP mode for your backend service?

[emilevauge]

@prateek2408 could you give us your traefik config (TOML, args)?

[prateek2408]

@mthird - Nopes I am just using plain Swarm cluster and under that I created a service and then scaled it to 4 instances, goes something like this: -

1. docker network create --driver overlay mynet
2. docker service create --network mynet emilevauge/whoami .....
3. docker service scale =
4. docker service inspect

Now the service inspect output will give me a single IP from the overlay network. If i internally do a curl to this IP( screenshots attached above) I get different hostnames, but when traefik uses it same hostnames are being displayed signifying that load balancing of swarm is not working

@emilevauge - I ran traefik binary in the container in the following way -

./traefik_linux-amd64 --web --docker.swarmmode --docker.watch

It runs and as the container on which it is running has docker unix socket shared so it connects to the docker service and starts listening on port 8080 for WEBUI and 80 for front end.

[vovimayhem]

I've also noticed this.

My cluster was created following the v1.1 Swarm Mode Guide with some minor corrections, as I was deploying on Azure.

This is how I created the Traefik service on the swarm (I changed just the network name and the domain when I ran this command):

docker service create \
    --name traefik \
    --constraint 'node.role==manager' \
    --publish 80:80 \
    --publish 8080:8080 \
    --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
    --network traefik-net \
    traefik:camembert \
    --docker \
    --docker.swarmmode \
    --docker.domain=traefik \
    --docker.watch \
    --logLevel=DEBUG \
    --web

Then this is how I created the service:

docker service create \
  --name whoami0 \
  --label 'traefik.port=80' \
  --network traefik-net \
  emilevauge/whoami

And this is how I scaled the service:

docker service scale whoami0=3

Tested with the following command:

curl -H Host:whoami0.traefik http://$(docker-machine ip manager)

Instead of seeing each request being processed by each of the three containers (doing round robin), it just kept being processed by the first container.

[vovimayhem]

And it looks like the sticky behavior is at traefik level. If I try curl-ing the site after a few minutes, another container starts processing the requests

[sunnysideocean]

I tried reproducing this problem but the load balancing works fine for me under the swarm mode setup. I tested it in swarm mode with labels (in which case the IP is added to traefik). I also tested it using a file config where I added the host name of the container.

However, I noticed that the traffic is first redirected to container A and after many requests, traffic is routed to container B for a while. So with a load test I can see that the traffic switches between the 2 containsers every few seconds (~7 seconds in my test).

I did the same setup with nginx as reverse proxy and there the traffic was distributed evenly between the containers at all time.

Nevertheless, eben with traefik the load balancing works - altough it happens in batches and of course if may work due to the docker ingress load balancing.

[emilevauge]

This seems duplicate of #718.
Closing this one in favor of #727