Proxy DNS Does not function properly when using an external endpoint

[samzph]

Describe the bug
When attempting to connect via the proxy using an ingress with a custom URL, an error is thrown: Resolve error: asio.netdb:1 : Host not found (authoritative), presumably due to the DNS being switched from the custom URL to an internal Kubernetes URL.

Failure presumably at the changeover:
http://dev-pulsar-proxy.<redacted>.com:80 -> pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650

See below for more error details w/ full stack trace.

To Reproduce
Install Apache Pulsar Helm Chart using these configs:

values.yaml (relevant snippets)

components:
  #broker
  broker: true
  # proxy
  proxy: true

broker:
    repository: apachepulsar/pulsar-all
    tag: 2.8.2
proxy:
    repository: apachepulsar/pulsar-all
    tag: 2.8.2

broker:
  ports:
    http: 80
  nodeSelector:
      Name: messaging-nodes
  configData:
    PULSAR_MEM: >
      -Xms128m -Xmx256m -XX:MaxDirectMemorySize=256m
    PULSAR_GC: >
      -XX:+UseG1GC
      -XX:MaxGCPauseMillis=10
      -Dio.netty.leakDetectionLevel=disabled
      -Dio.netty.recycler.linkCapacity=1024
      -XX:+ParallelRefProcEnabled
      -XX:+UnlockExperimentalVMOptions
      -XX:+DoEscapeAnalysis
      -XX:ParallelGCThreads=4
      -XX:ConcGCThreads=4
      -XX:G1NewSizePercent=50
      -XX:+DisableExplicitGC
      -XX:-ResizePLAB
      -XX:+ExitOnOutOfMemoryError
      -XX:+PerfDisableSharedMem
      -XX:+IgnoreUnrecognizedVMOptions

proxy:
  nodeSelector:
      Name: messaging-nodes
  service:
    type: ClusterIP
  ports:
    http: 80
    pulsar: 8080
  configData:
    PULSAR_MEM: >
      -Xms64m -Xmx64m -XX:MaxDirectMemorySize=64m
    PULSAR_GC: >
      -XX:+UseG1GC
      -XX:MaxGCPauseMillis=10
      -Dio.netty.leakDetectionLevel=disabled
      -Dio.netty.recycler.linkCapacity=1024
      -XX:+ParallelRefProcEnabled
      -XX:+UnlockExperimentalVMOptions
      -XX:+DoEscapeAnalysis
      -XX:ParallelGCThreads=4
      -XX:ConcGCThreads=4
      -XX:G1NewSizePercent=50
      -XX:+DisableExplicitGC
      -XX:-ResizePLAB
      -XX:+ExitOnOutOfMemoryError
      -XX:+PerfDisableSharedMem
      -XX:+IgnoreUnrecognizedVMOptions

pulsar-ingress.yaml

# pulsar-broker
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: pulsar-broker
  namespace: pulsar
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/group.name: pulsar-alb
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/success-codes: 200,302,301
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
    alb.ingress.kubernetes.io/certificate-arn: <redacted> 
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: enabled
    alb.ingress.kubernetes.io/ssl-policy: <redacted>
    alb.ingress.kubernetes.io/security-groups: <redacted>
spec:
  rules:
    - host: dev-pulsar-broker.<redacted>.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: ssl-redirect
              servicePort: use-annotation
          - path: /*
            backend:
              serviceName: pulsar-broker
              servicePort: 80
---
# pulsar-proxy
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: pulsar-proxy
  namespace: pulsar
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/group.name: pulsar-alb
    alb.ingress.kubernetes.io/healthcheck-path: /metrics
    alb.ingress.kubernetes.io/success-codes: 200,302,301
    alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
    alb.ingress.kubernetes.io/certificate-arn: <redacted>
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: enabled
    alb.ingress.kubernetes.io/ssl-policy: <redacted>
    alb.ingress.kubernetes.io/security-groups: <redacted>
spec:
  rules:
    - host: dev-pulsar-proxy.<redacted>.com
      http:
        paths:
          - path: /*
            backend:
              serviceName: pulsar-proxy
              servicePort: 80
---

Test the Proxy
note: Tenants, Namespaces, Topics and Subscriptions replaced with for universality.

In [1]: import pulsar

In [2]: client = pulsar.Client('http://dev-pulsar-proxy.<redacted>.com:80')

In [3]: consumer = client.subscribe('<Tenant>/<Namespace>/<Topic>', '<Subscription>')
   ...: while True:
   ...:     msg = consumer.receive()
   ...:     print(msg.data())
   ...:     consumer.acknowledge(msg)
   ...:
   ...: client.close()
2022-03-25 15:21:40.839 INFO  [0x105274580] Client:88 | Subscribing on Topic :<Tenant>/<Namespace>/<Topic>
2022-03-25 15:21:40.840 INFO  [0x16bf53000] HTTPLookupService:235 | Curl Lookup Request sent for http://dev-pulsar-proxy.<redacted>.com:80/admin/v2/persistent/<Tenant>/<Namespace>/<Topic>/partitions?checkAllowAutoCreation=true
2022-03-25 15:21:41.184 INFO  [0x16bf53000] HTTPLookupService:249 | Response received for url http://dev-pulsar-proxy.<redacted>.com:80/admin/v2/persistent/<Tenant>/<Namespace>/<Topic>/partitions?checkAllowAutoCreation=true code 200
2022-03-25 15:21:41.184 INFO  [0x16bf53000] HTTPLookupService:293 | parsePartitionData = { LookupDataResult [brokerUrl_ = ] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
2022-03-25 15:21:41.188 INFO  [0x16bf53000] HandlerBase:64 | [persistent://<Tenant>/<Namespace>/<Topic>, <Subscription>, 0] Getting connection from pool
2022-03-25 15:21:41.188 INFO  [0x16bf53000] HTTPLookupService:235 | Curl Lookup Request sent for http://dev-pulsar-proxy.<redacted>.com:80/lookup/v2/topic/persistent/<Tenant>/<Namespace>/<Topic>
2022-03-25 15:21:41.418 INFO  [0x16bf53000] HTTPLookupService:249 | Response received for url http://dev-pulsar-proxy.<redacted>.com:80/lookup/v2/topic/persistent/<Tenant>/<Namespace>/<Topic> code 200
2022-03-25 15:21:41.419 INFO  [0x16bf53000] HTTPLookupService:328 | parseLookupData = { LookupDataResult [brokerUrl_ = pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
2022-03-25 15:21:41.419 INFO  [0x16bf53000] ClientConnection:181 | [<none> -> pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650] Create ClientConnection, timeout=10000
2022-03-25 15:21:41.419 INFO  [0x16bf53000] ConnectionPool:96 | Created connection for pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650
2022-03-25 15:21:46.424 ERROR [0x16c0f7000] ClientConnection:530 | [<none> -> pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650] Resolve error: asio.netdb:1 : Host not found (authoritative)
2022-03-25 15:21:46.425 INFO  [0x16c0f7000] ClientConnection:1535 | [<none> -> pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650] Connection closed
2022-03-25 15:21:46.425 INFO  [0x16c0f7000] ClientConnection:255 | [<none> -> pulsar://pulsar-broker-2.pulsar-broker.pulsar.svc.cluster.local:6650] Destroyed connection

Expected behavior
A connection is made through the pulsar proxy on port 80 or 443 using HTTP/HTTPS that gets converted into usable Pulsar commands.

[lhotari]

Related to the advertised listeners feature in Pulsar. Please see PIP-61, PIP-95 and PIP-338 in Pulsar. Closing this.
You can also check #423