Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consul/connect: check connect group and service names for uppercase characters #10455

Merged
merged 1 commit into from
Apr 27, 2021

Conversation

shoenig
Copy link
Member

@shoenig shoenig commented Apr 27, 2021

This PR adds job-submission validation that checks for the use of uppercase characters
in group and service names for services that make use of Consul Connect. This prevents
attempting to launch services that Consul will not validate correctly, which in turn
causes tasks to fail to launch in Nomad.

Underlying Consul issue: hashicorp/consul#6765

Closes #7581 #10450

@shoenig
Copy link
Member Author

shoenig commented Apr 27, 2021

Demonstration of old behavior, what we're trying to prevent.

Submitting a job with a connect service with uppercase characters.
job "example" {
  datacenters = ["dc1"]

  group "group" {
    network {
      mode = "bridge"
    }

    service {
      name = "Count-Api"
      port = "9001"

      connect {
        sidecar_service {}
      }
    }

    task "web" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
      }
    }
  }
}

Job submits fine

$ nomad job run example.nomad 
==> Monitoring evaluation "f2de25f6"
    Evaluation triggered by job "example"
    Allocation "d734fe27" created: node "5d66187c", group "group"
==> Monitoring evaluation "f2de25f6"
    Evaluation within deployment: "877fe3c4"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "f2de25f6" finished with status "complete"

Alloc will always fail as Consul silently assumes lower case characters

$ nomad alloc status d73
ID                  = d734fe27-6856-ac82-6c34-6f1c000ad98a
Eval ID             = f2de25f6
Name                = example.group[0]
Node ID             = 5d66187c
Node Name           = nuc10
Job ID              = example
Job Version         = 0
Client Status       = pending
Client Description  = No tasks have started
Desired Status      = run
Desired Description = <none>
Created             = 31s ago
Modified            = 2s ago
Deployment ID       = 877fe3c4
Deployment Health   = unset

Allocation Addresses (mode = "bridge")
Label                     Dynamic  Address
*connect-proxy-Count-Api  yes      192.168.1.53:29725 -> 29725

Task "connect-proxy-Count-Api" (prestart sidecar) is "pending"
Task Resources
CPU      Memory   Disk     Addresses
250 MHz  128 MiB  300 MiB  

Task Events:
Started At     = N/A
Finished At    = N/A
Total Restarts = 2
Last Restart   = 2021-04-27T10:53:58-06:00

Recent Events:
Time                       Type              Description
2021-04-27T10:53:58-06:00  Restarting        Task restarting in 17.846644841s
2021-04-27T10:53:58-06:00  Task hook failed  envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: exit status 1
2021-04-27T10:53:37-06:00  Restarting        Task restarting in 15.607083387s
2021-04-27T10:53:37-06:00  Task hook failed  envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: exit status 1
2021-04-27T10:53:30-06:00  Task Setup        Building Task Directory
2021-04-27T10:53:29-06:00  Received          Task received by client
Submitting a job with a connect service group with uppercase characters.

Job file

job "example" {
  datacenters = ["dc1"]

  group "Group" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"

      connect {
        sidecar_service {}
      }
    }

    task "web" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
      }
    }
  }
}

Job submits fine

$ noamd job run example.nomad 
==> Monitoring evaluation "be412281"
    Evaluation triggered by job "example"
    Allocation "50652988" created: node "5d66187c", group "Group"
==> Monitoring evaluation "be412281"
    Evaluation within deployment: "b70c0338"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "be412281" finished with status "complete"

Alloc always failing

$ nomad alloc status 506
ID                  = 50652988-3157-1172-8d99-107c5db4376c
Eval ID             = be412281
Name                = example.Group[0]
Node ID             = 5d66187c
Node Name           = nuc10
Job ID              = example
Job Version         = 0
Client Status       = pending
Client Description  = No tasks have started
Desired Status      = run
Desired Description = <none>
Created             = 31s ago
Modified            = 23s ago
Deployment ID       = b70c0338
Deployment Health   = unset

Allocation Addresses (mode = "bridge")
Label                     Dynamic  Address
*connect-proxy-count-api  yes      192.168.1.53:24679 -> 24679

Task "connect-proxy-count-api" (prestart sidecar) is "pending"
Task Resources
CPU      Memory   Disk     Addresses
250 MHz  128 MiB  300 MiB  

Task Events:
Started At     = N/A
Finished At    = N/A
Total Restarts = 1
Last Restart   = 2021-04-27T10:57:01-06:00

Recent Events:
Time                       Type              Description
2021-04-27T10:57:01-06:00  Restarting        Task restarting in 17.740798267s
2021-04-27T10:57:01-06:00  Task hook failed  envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: exit status 1
2021-04-27T10:56:55-06:00  Task Setup        Building Task Directory
2021-04-27T10:56:54-06:00  Received          Task received by client

Demonstration of new behavior, preventing submission of jobs that can't work

Submitting job with connect service with uppercase characters

Job file

job "example" {
  datacenters = ["dc1"]

  group "group" {
    network {
      mode = "bridge"
    }

    service {
      name = "Count-Api"
      port = "9001"

      connect {
        sidecar_service {}
      }
    }

    task "web" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
      }
    }
  }
}

Submission is blocked

$ nomad job run example.nomad 
Error submitting job: Unexpected response code: 500 (1 error occurred:
	* Consul Connect service name "Count-Api" in group "group" must not contain uppercase characters

)
Submitting job with connect service in group with uppercase characters

Job file

job "example" {
  datacenters = ["dc1"]

  group "Group" {
    network {
      mode = "bridge"
    }

    service {
      name = "count-api"
      port = "9001"

      connect {
        sidecar_service {}
      }
    }

    task "web" {
      driver = "docker"

      config {
        image = "hashicorpnomad/counter-api:v3"
      }
    }
  }
}

Submission is blocked

$ nomad job run example.nomad 
Error submitting job: Unexpected response code: 500 (1 error occurred:
	* Consul Connect group "Group" with service "count-api" must not contain uppercase characters

)

…haracters

This PR adds job-submission validation that checks for the use of uppercase characters
in group and service names for services that make use of Consul Connect. This prevents
attempting to launch services that Consul will not validate correctly, which in turn
causes tasks to fail to launch in Nomad.

Underlying Consul issue: hashicorp/consul#6765

Closes #7581 #10450
Copy link
Member

@tgross tgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 24, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using uppercase characters in the service name causes the envoy proxy to fail
2 participants