System jobs require restart after adding upstreams

[AnthonySendra]

Nomad version

0.12.3

Issue

When a proxy upstream is added to a job of type = "system", the container is not restarted but the new configuration is not taken into account. The new upstream is unreachable from the container and we need to restart the job to make it work.
When the job type is set to service everything works as expected: the container is not restarted and the new upstream can be reached.

Reproduction steps

1. Run job api1
2. Run job api2 without upstream
3. Run job api2 with upstream to api1 (uncomment the upstream part of the following job)
4. Enter the job api2 container and try to curl api1: curl http://127.0.0.1:7777

Job file (if appropriate)

api1

job "api" {
  datacenters = ["dc1"]

  group "api1" {

    count = 1

    network {
      mode ="bridge"
      port "http" {
        to     = 8080
      }
    }

    service {
      name = "api1"
      port = 8080

      connect {
        sidecar_service {}
      }

      check {
        name     = "alive"
        port     = "http"
        type     = "http"
        path     = "/"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "api1" {
      driver = "docker"

      env {
        PORT = 8080
        NAME = "api1"
      }

      config {
        image = "patatra/test-connections-server:latest"
      }
    }
  }
}

api2

job "api2" {
  datacenters = ["dc1"]
  type = "system"

  group "api2" {

    network {
      mode ="bridge"
      port "http" {
        to     = 8081
      }
    }

    service {
      name = "api2"
      port = 8081

      check {
        name     = "alive"
        port     = "http"
        type     = "http"
        path     = "/"
        interval = "10s"
        timeout  = "2s"
      }

      connect {
        sidecar_service {
          proxy {
//            upstreams {
//              destination_name = "api1"
//              local_bind_port = 7777
//            }
          }
        }
      }
    }

    task "api2" {
      driver = "docker"

      env {
        PORT = 8081
        NAME = "api2"
      }

      config {
        image = "patatra/test-connections-server:latest"
      }
    }
  }
}

[tgross]

Hi @AnthonySendra that certainly doesn't look right to me. I've tagged this so the appropriate folks can dig into it. Thanks for reporting this issue!

[AnthonySendra]

Hello,
FYI I still have the problem with Nomad 1.0.1.
And I have also noticed that Consul's meta are not updated either in case of type system.

[shoenig]

Apologies for the extremely long delay @AnthonySendra, I'm looking into this now

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.