Akka remoting bootstrap probing

[chbatey]

Probing is currently via HTTP.

For applications that don't use HTTP e.g. communication via Kafka it may be inconvenient to open up a HTTP port for cluster bootstrap.

Probing via remoting also means that bootstrap could move to core Akka.

Low priority until users request a non HTTP version of bootstrap.

[patriknw]

I agree, it would be possible but since we have a working solution it's not top priority to change it.

I'm trying to recall why we selected http for the probing. I can't find any strong reasons. A few that comes to mind but not anything critical:

• we already had cluster http management, and that is what we prefer as the management/insight to the cluster, tagging along on that http server that we prefer to have active anyway
• http needed for health checks anyway
• remoting connections are long lived while http requests fits the probing use case better, but the end goal is to join the discovered nodes anyway and then the remoting connections will be established anyway
• maybe some fear that some of the discovered nodes would not be real candidates for the cluster and that could only be observed after probing, and therefore it would be unnecessary to open "heavy" remoting connections to those nodes (I don't think we have that feature currently, all discovered nodes should be real candidates for the cluster)
• for large clusters, many discovered nodes, there might be a performance problem to create remote connections to all of them at the same time at startup (once joining the cluster those connections are created more gradually as needed)