Connect sidecar and check management based on false invariant

[shoenig]

The Consul synchronization path attempts to ignore Connect sidecar service and check definitions. From Nomad's perspective these are implicitly defined via the connect.sidecar_service stanza, and therefor are not explicitly added or removed (i.e. via WorkloadServices methods) - creating a false sense of atomicity between the sidecar service + checks and its parent service + checks. In Consul the sidecar service and the parent service are independent.

The problem is the implementation and use of isNomadSidecar, which refers to Nomad's internal tracking of services to determine if a service or check belongs to a parent service. Nomad may have removed it's tracking of the parent service, but Consul still has a handle on its definition of the sidecar service or checks, which means Nomad could get into a state of never removing them even though Nomad [implicitly] created them.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.