Issue to get Hazelcast cluster up and running using DNS discovery mode

[abx-firez]

Hi there,
I am facing an issue to get an cluster up and running on Azure Kubernetes Service since I am not able to contact the service from outside. I am using Vert.x 3.9.5 with Hazelcast Kubernetes in version 1.5.1. on Java 11.
What I did so far...:

1. Adding dependencies to the pom.xml and created a cluster.xml

  <dependency>
  	<groupId>io.vertx</groupId>
  	<artifactId>vertx-hazelcast</artifactId>
  	<version>${vertx.version}</version>
  </dependency>`

  <dependency>
  	<groupId>com.hazelcast</groupId>
  	<artifactId>hazelcast-kubernetes</artifactId>
  	<version>${hazelcast-kubernetes.version}</version>
  </dependency>

cluster.xml:

<hazelcast
   xsi:schemaLocation="http://www.hazelcast.com/schema/config http://www.hazelcast.com/schema/config/hazelcast-config-3.12.xsd"
   xmlns="http://www.hazelcast.com/schema/config"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
   <properties>
     <property name="hazelcast.logging.type">slf4j</property>
     <property name="hazelcast.discovery.enabled">true</property>
   </properties>
   <network>
     <join>
       <multicast enabled="false"/>
       <tcp-ip enabled="false" />
       <discovery-strategies>
         <discovery-strategy enabled="true"
                             class="com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy">
           <properties>
 			<property name="service-dns">service-hazelcast-server.default.svc.cluster.local</property> 
           </properties>
         </discovery-strategy>
       </discovery-strategies>
     </join>
   </network>
 </hazelcast>

2. Created a headless service (as described here: https://vertx.io/docs/3.9.6/vertx-hazelcast/java/#_configuring_for_kubernetes)

apiVersion: v1
kind: Service
metadata:
 namespace: default
 name: service-hazelcast-server
spec:
 selector:
   component: service-hazelcast-server
 clusterIP: None
 ports:
 - name: hz-port-name
   port: 5701
   targetPort: 5701
   protocol: TCP

3. Applied the headless service above to the Kubernetes cluster.
   service/service-hazelcast-server ClusterIP None <none> 5701/TCP 3d23h

4. Created a deployment for a client

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-gateway-deployment
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
        component: service-hazelcast-server
    spec:
      containers:
      - name: api-gateway
        image: someImageRessourceURL
      restartPolicy: Always

5. And a service for the client

apiVersion: v1
kind: Service
metadata:
  name: api-gateway
  namespace: default
spec:
  type: LoadBalancer
  selector: 
    app: api-gateway
  ports:
  - port: 8787
    targetPort: 8787
    name: rest-verticle-port
  - name: hazelcast-port
    port: 5701  
    targetPort: 5701

The client service is deployed correctly and gives me an external ip. Let's assume this external ip is 12.34.56.78 with ports 8787:31236/TCP,5701:31667/TCP. My Dockerfile is exposing port 8787.

Also the log shows that the cluster consists of one member (I added the complete log, just in case...)
2021-03-22 14:50:00.236  INFO 1 --- [           main] com.tsystems.aqm.gateway.Application     : Starting Application v0.0.1-SNAPSHOT on api-gateway-deployment-6c95bc56df-qkw4r with PID 1 (/usr/verticles/api-gateway-service-0.0.1-SNAPSHOT.jar started by root in /usr/verticles)
2021-03-22 14:50:00.240  INFO 1 --- [           main] com.tsystems.aqm.gateway.Application     : No active profile set, falling back to default profiles: default
2021-03-22 14:50:01.238  INFO 1 --- [           main] com.tsystems.aqm.gateway.Application     : Started Application in 1.575 seconds (JVM running for 2.163)
2021-03-22 14:50:01.544  WARN 1 --- [worker-thread-0] c.h.instance.HazelcastInstanceFactory    : Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
 --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
2021-03-22 14:50:01.604  INFO 1 --- [worker-thread-0] com.hazelcast.instance.AddressPicker     : [LOCAL] [dev] [3.12.5] Prefer IPv4 stack is true, prefer IPv6 addresses is false
2021-03-22 14:50:01.616  INFO 1 --- [worker-thread-0] com.hazelcast.instance.AddressPicker     : [LOCAL] [dev] [3.12.5] Picked [10.244.2.14]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
2021-03-22 14:50:01.645  INFO 1 --- [worker-thread-0] com.hazelcast.system                     : [10.244.2.14]:5701 [dev] [3.12.5] Hazelcast 3.12.5 (20191210 - 294ff46) starting at [10.244.2.14]:5701
2021-03-22 14:50:01.645  INFO 1 --- [worker-thread-0] com.hazelcast.system                     : [10.244.2.14]:5701 [dev] [3.12.5] Copyright (c) 2008-2019, Hazelcast, Inc. All Rights Reserved.
2021-03-22 14:50:01.896  INFO 1 --- [worker-thread-0] c.h.s.i.o.impl.BackpressureRegulator     : [10.244.2.14]:5701 [dev] [3.12.5] Backpressure is disabled
2021-03-22 14:50:02.372  INFO 1 --- [worker-thread-0] c.h.s.d.integration.DiscoveryService     : [10.244.2.14]:5701 [dev] [3.12.5] Kubernetes Discovery properties: { service-dns: service-hazelcast-server.default.svc.cluster.local, service-dns-timeout: 5, service-name: null, service-port: 0, service-label: null, service-label-value: true, namespace: default, resolve-not-ready-addresses: false, use-node-name-as-external-address: false, kubernetes-api-retries: 3, kubernetes-master: https://kubernetes.default.svc}
2021-03-22 14:50:02.375  INFO 1 --- [worker-thread-0] c.h.s.d.integration.DiscoveryService     : [10.244.2.14]:5701 [dev] [3.12.5] Kubernetes Discovery activated with mode: DNS_LOOKUP
2021-03-22 14:50:02.438  INFO 1 --- [worker-thread-0] com.hazelcast.instance.Node              : [10.244.2.14]:5701 [dev] [3.12.5] Activating Discovery SPI Joiner
2021-03-22 14:50:02.683  INFO 1 --- [worker-thread-0] c.h.s.i.o.impl.OperationExecutorImpl     : [10.244.2.14]:5701 [dev] [3.12.5] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
2021-03-22 14:50:02.686  INFO 1 --- [worker-thread-0] c.h.internal.diagnostics.Diagnostics     : [10.244.2.14]:5701 [dev] [3.12.5] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
2021-03-22 14:50:02.690  INFO 1 --- [worker-thread-0] com.hazelcast.core.LifecycleService      : [10.244.2.14]:5701 [dev] [3.12.5] [10.244.2.14]:5701 is STARTING
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (jar:file:/usr/verticles/api-gateway-service-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/hazelcast-3.12.5.jar!/) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-03-22 14:50:03.117  WARN 1 --- [worker-thread-0] c.h.s.d.integration.DiscoveryService     : [10.244.2.14]:5701 [dev] [3.12.5] Cannot fetch the current zone, ZONE_AWARE feature is disabled
2021-03-22 14:50:08.336  INFO 1 --- [worker-thread-0] c.h.internal.cluster.ClusterService      : [10.244.2.14]:5701 [dev] [3.12.5] 

Members {size:1, ver:1} [
        Member [10.244.2.14]:5701 - 2ef9b0fc-6d74-4df9-8c4d-7dc625dd4f76 this
]

2021-03-22 14:50:08.367  INFO 1 --- [worker-thread-0] com.hazelcast.core.LifecycleService      : [10.244.2.14]:5701 [dev] [3.12.5] [10.244.2.14]:5701 is STARTED
2021-03-22 14:50:08.734  INFO 1 --- [worker-thread-3] c.h.i.p.impl.PartitionStateManager       : [10.244.2.14]:5701 [dev] [3.12.5] Initializing cluster partition table arrangement...
2021-03-22 14:50:09.354  INFO 1 --- [ntloop-thread-0] com.tsystems.aqm.gateway.Application     : Rest Controller Verticle started: e91e44bf-e754-40bc-b92b-168f23d11ef4

I should now be able to call 12.34.56.78:8787/someRessource. But this just gives me a timeout. What am I doing wrong? Is there anything missing or badly configured? I would really appreciate any kind of hint.

Thank you in advance.

[leszko]

Please check the related guide: Connect To Hazelcast Running on Kubernetes from Outside.

If it does not work, please add the steps to reproduce without Vertx.

[abx-firez]

Thank you for your reply.
I've checked the guide before. Unfortunately, the guide doesn't seem to fit for my purposes since it is not using the DNS discovery mode. Furthermore I don't want to connect a client from outside, since Hazelcast and it's clients are running in the same Kubernetes cluster.
Regarding Vertx: It should not make a big difference here, since it directly uses Hazelcast and - as I described - the verticle is starting correctly.
@leszko: Can you please check if my config seems correct for you regardless the Vertx dependencies? Thank you!

[leszko]

@abx-firez As mentioned earlier, please attach the reproducer without Vertx, just Hazelcast. Then, I'll try to have a look.

[Lucas-Yu-Chen]

Hi @abx-firez
I just meet same issue...
And after checking , i found it maybe need setup the "namespace" (btw the port number also required if you customized)
the default namespace is "default" it may cause can not find that "Service"