gRPC and TCP/HTTP listeners can't work together in the same gateway-ingress

[Dgotlieb]

Nomad version
v1.2.6 (a6c6b47)

Operating system and Environment details
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

Issue
When placing gRPC listener and a TCP/HTTP listener in the same gateway-ingress an error is thrown

Reproduction steps
Place different listener types under the same gateway-ingress entry

Expected Result
Allow different listener types to be located under the same gateway-ingress entry

Actual Result
Error submitting job: Unexpected response code: 500 (rpc error: Unexpected response code: 500 (service "service-a-http" has protocol "tcp", which does not match defined listener protocol "http"))

Workaround
Separate the job into 2 groups:

• 1st group for gRPC listeners
• 2nd group for non-gRPC (TCP/HTTP) listeners

Job file (before workaround):

job "ingress" {
  type        = "service"
  datacenters = ["dc1"]

  group "ingress" {

    count = 3
    
    network {
      mode = "bridge"

      port "service-a-http" {
        static = 8000
        to     = 8000
      }

      port "service-a-grpc" {
        static = 8001
        to     = 8001
      }
    }

    service {
      name = "ingress"      
      connect {
        gateway {
          proxy {}
          ingress {
            listener {
              port = 8000
              service {
                name = "service-a-http"
              }
            }

             listener {
               port     = 8011
               protocol = "grpc"
               service {
                 name  = "service-a-grpc"
                 hosts = ["*"]
               }
             }
          }
        }
      }
    }
  }
}

Job file (after workaround):

job "ingress" {
  type        = "service"
  datacenters = ["dc1"]

  group "ingress" {

    count = 3

    network {
      mode = "bridge"
      port "service-a-http" {
        static = 8000
        to     = 8000
      }
    }

    service {
      name = "ingress"

      connect {
        gateway {
          proxy {}
          ingress {
            listener {
              port = 8000
              service {
                name = "service-a-http"
              }
            }
          }
        }
      }
    }
  }

  group "ingress-grpc" {
    count = 3

    network {
      mode = "bridge"

      port "service-a-grpc" {
        static = 8011
        to     = 8011
      }
    }

    service {
      name = "ingress-grpc"

      connect {
        gateway {
          proxy {}
          ingress {
             listener {
               port     = 8011
               protocol = "grpc"
               service {
                 name  = "service-a-grpc"
                 hosts = ["*"]
               }
             }
          }
        }
      }
    }
  }
}

This issue is possibly related to #8647

[lgfa29]

Hi @Dgotlieb 👋

I believe this is a characteristic of Consul. From their ingress gateway documentation:

For listeners with a protocol other than tcp, multiple services can be specified for a single listener.

So tcp listeners must be set on their own.

Would you mind opening an issue in the Consul repo? You can link back to this one.

Thanks!

[Dgotlieb]

Hi @lgfa29 😄
It seems like this issue could be on Consul side (and not Nomad).
That being said, based on the docs and as you mentioned:

tcp listeners must be set on their own

which I think means that I can add multiple services when using a "non-tcp" protocol (in this case grpc):

 ...
  ingress {
     listener {
       port     = 8011
       protocol = "grpc"
       service {
         name  = "service-a"
         hosts = ["foo"]
       }
       service {
         name  = "service-b"
         hosts = ["bar"]
       }
     }
...

But can't add more than one service when using tcp protocol:

 ...
  ingress {
     listener {
       port     = 8011
       protocol = "tcp"
       service {
         name  = "service-a"
       }
       service {
         name  = "service-b"
       }
     }
...

My issue is focused on the fact that when using different listeners, it works only when they are separated into different (Nomad) groups.

Thanks!

[shoenig]

Hey @Dgotlieb, the error message coming back from Consul contains service "a-service", but the services named in your Nomad config refer to a-service-http and a-service-grpc. Are you sure you have your service names all lined up in the service definition you're trying to register?

[Dgotlieb]

OK, after debugging a little more I realized this is 100% related to #8647
The issue is that Nomad can't register non-tcp services, and requires to first define service-defaults externally (via CLI / API) and then register the non-tcp listener via Nomad.

$ cat 1.json

{
  "Kind": "service-defaults",
  "Name": "service-a-grpc",
  "Protocol": "grpc"
}

$ consul config write 1.json

This is has nothing to do with the groups separation as I thought, but a Nomad issue nonetheless.
Would be nice to get an update...

Sorry for the hassle.
Thank you

[shoenig]

Ah okay, thanks @Dgotlieb! I added some thoughts on #8647, and will go ahead and close this as a duplicate.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.