consul: all services being re-registered all the time

[shoenig]

While looking into #3935, it looks like we are improperly re-registering every service on every iteration of the reconciliation loop. This is quite inefficient, we should only re-register a service if the service definition has changed on the Consul side from what the Nomad job definition declares.

E.g.

GET_SERVICES namespace=default why=sync
got services keys=[_nomad-server-o5s5vxlab3cgpyi3x2i5cm24xa2r2hbz _nomad-server-tl5dgauqxnxsq7pgcvloyq7d6gf75mj7 _nomad-client-rxajoy6bie7cqxttksivmlv2ucz7qxfl _nomad-server-icsfum2knxzafc7jtsv5hmelx35hkufm]
different H
SERVICE_REGISTER id=_nomad-server-o5s5vxlab3cgpyi3x2i5cm24xa2r2hbz name=nomad why=missing exists=true
different H
SERVICE_REGISTER id=_nomad-server-icsfum2knxzafc7jtsv5hmelx35hkufm name=nomad why=missing exists=true
different H
SERVICE_REGISTER id=_nomad-client-rxajoy6bie7cqxttksivmlv2ucz7qxfl name=nomad-client why=missing exists=true
different H
SERVICE_REGISTER id=_nomad-server-tl5dgauqxnxsq7pgcvloyq7d6gf75mj7 name=nomad why=missing exists=true
GET_CHECKS namespace=default
GET_SERVICES namespace=default why=sync
got services keys=[_nomad-server-tl5dgauqxnxsq7pgcvloyq7d6gf75mj7 _nomad-client-rxajoy6bie7cqxttksivmlv2ucz7qxfl _nomad-server-icsfum2knxzafc7jtsv5hmelx35hkufm _nomad-server-o5s5vxlab3cgpyi3x2i5cm24xa2r2hbz]
different H
SERVICE_REGISTER id=_nomad-server-tl5dgauqxnxsq7pgcvloyq7d6gf75mj7 name=nomad why=missing exists=true
different H
SERVICE_REGISTER id=_nomad-server-o5s5vxlab3cgpyi3x2i5cm24xa2r2hbz name=nomad why=missing exists=true
different H
SERVICE_REGISTER id=_nomad-server-icsfum2knxzafc7jtsv5hmelx35hkufm name=nomad why=missing exists=true
different H
SERVICE_REGISTER id=_nomad-client-rxajoy6bie7cqxttksivmlv2ucz7qxfl name=nomad-client why=missing exists=true

The cause of course being the mis-use of reflect.DeepEqual to compare slices of strings.

case !reflect.DeepEqual(wanted.TaggedAddresses, existing.TaggedAddresses):
    netlog.Cyan("different H")
    return true

edit: on closer look, it's actually also because of the default tagged addresses being set by Consul on our agent registrations

LOOP id=_nomad-server-tl5dgauqxnxsq7pgcvloyq7d6gf75mj7 service=nomad
DIFFERENT
H wanted=map[] existing=map[lan_ipv4:{127.0.0.1 4647} wan_ipv4:{127.0.0.1 4647}]
LOOP id=_nomad-server-o5s5vxlab3cgpyi3x2i5cm24xa2r2hbz service=nomad
DIFFERENT
H wanted=map[] existing=map[lan_ipv4:{127.0.0.1 4648} wan_ipv4:{127.0.0.1 4648}]
LOOP id=_nomad-server-icsfum2knxzafc7jtsv5hmelx35hkufm service=nomad
DIFFERENT
H wanted=map[] existing=map[lan_ipv4:{127.0.0.1 4646} wan_ipv4:{127.0.0.1 4646}]
LOOP id=_nomad-client-rxajoy6bie7cqxttksivmlv2ucz7qxfl service=nomad-client
DIFFERENT
H wanted=map[] existing=map[lan_ipv4:{127.0.0.1 4646} wan_ipv4:{127.0.0.1 4646}]
LOOP id=_nomad-task-0a45376d-cfd3-54b8-d801-99d5f93e4bd7-group-group-s1-8080 service=s1
DIFFERENT
SAME

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.