Setting group.service.tags causes connect service churn

[shoenig]

$ nomad version 
Nomad v1.0.3

$ consul version 
Consul v1.9.2+ent

job "simple" {
  datacenters = ["dc1"]

  group "api" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"

      connect {
        sidecar_service {} # this gets re-registered
      }

      tags = ["foo"] # if this is set 
    }

    task "web" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
      }
    }
  }
}

Consul logs (note 30 second interval, suggesting mis-comparison in our service reconciliation loop)

2021-02-12T17:45:06.744-0600 [DEBUG] agent: updated local registration for service: service=_nomad-task-d2090c07-529e-a51a-241f-a6ff193a1270-group-api-count-api-9001-sidecar-proxy
2021-02-12T17:45:36.747-0600 [DEBUG] agent: updated local registration for service: service=_nomad-task-d2090c07-529e-a51a-241f-a6ff193a1270-group-api-count-api-9001-sidecar-proxy
2021-02-12T17:46:06.749-0600 [DEBUG] agent: updated local registration for service: service=_nomad-task-d2090c07-529e-a51a-241f-a6ff193a1270-group-api-count-api-9001-sidecar-proxy

Removing the service.tags makes the problem go away

[tgross]

@shoenig I wasn't able to replicate this on the HEAD of master, at least as of this afternoon. Here's the Consul logs from my Consul 1.8.3 when Nomad does its periodic sync, and I got the same with Consul 1.9.3+ent

2021-02-18T19:49:03.395Z [DEBUG] agent: Node info in sync
2021-02-18T19:49:03.395Z [DEBUG] agent: Service in sync: service=_nomad-server-pq5rsq6x2i2t3ut3vydstlugdscquqbp
2021-02-18T19:49:03.395Z [DEBUG] agent: Service in sync: service=_nomad-client-s44sa35slmqquz3mjjwlsrbsykzelomz
2021-02-18T19:49:03.395Z [DEBUG] agent: Service in sync: service=_nomad-task-8febffc2-9738-e31a-e05d-0c782b3cd4fe-group-api-count-api-9001
2021-02-18T19:49:03.395Z [DEBUG] agent: Service in sync: service=_nomad-task-8febffc2-9738-e31a-e05d-0c782b3cd4fe-group-api-count-api-9001-sidecar-proxy
2021-02-18T19:49:03.395Z [DEBUG] agent: Service in sync: service=_nomad-server-7xv2uqz2ytiqpkroboa4p5gwbedto76a
2021-02-18T19:49:03.395Z [DEBUG] agent: Service in sync: service=_nomad-server-big7o7tunda3xgsvo75u6ckmu2t4o5hd
2021-02-18T19:49:03.395Z [DEBUG] agent: Check in sync: check=_nomad-check-6ccf53474f0808def17ece1e05b7d82cc8a39023
2021-02-18T19:49:03.395Z [DEBUG] agent: Check in sync: check=_nomad-check-0121591bd043f3acf4169301ec05612dc162a341
2021-02-18T19:49:03.395Z [DEBUG] agent: Check in sync: check=_nomad-check-4b7a819a6103c5ab3ecd56a5bc72cedb2ae48e9e
2021-02-18T19:49:03.395Z [DEBUG] agent: Check in sync: check=_nomad-check-f1202e59a5c72eb25a0a1d3ce8f75b532ba0431c
2021-02-18T19:49:03.395Z [DEBUG] agent: Check in sync: check=service:_nomad-task-8febffc2-9738-e31a-e05d-0c782b3cd4fe-group-api-count-api-9001-sidecar-proxy:1
2021-02-18T19:49:03.395Z [DEBUG] agent: Check in sync: check=service:_nomad-task-8febffc2-9738-e31a-e05d-0c782b3cd4fe-group-api-count-api-9001-sidecar-proxy:2

[shoenig]

@tgross those logs are about syncing from the Consul agent to Consul server (agent <-> catalog). The log line we're looking for occurs during Nomad's periodic sync onto Consul agent, being triggered on Nomad's periodic schedule.

I understand the bug: whenever you set service.tags but not sidecar_service.tags, Consul will helpfully overwrite sidecar_service.tags in its own copy of the service definitions so that they "inherit" service.tags. On the Nomad side we don't do that, leaving sidecar_service.tags empty. When it comes time to sync the sidecar_service, they are now out of sync, comparing an empty list on Nomad's side to a non-empty list on the Consul side.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.