Raft Reverse-Proxy doesn't work when UseSSL = true

[akatashev]

This issue is similar to #873

Actual behaviour:

When orchestrator.conf.json is configured that way:

{
  "HTTPAdvertise": "http://{{ orchestrator_hostname }}:{{ orchestrator_port }}",
  "UseSSL": false
}

Then reverse-proxy works fine and curl - D - http://{orchestrator_ip}:{orchestrator_port}/api/clusters request returns correct information about known clusters both for a raft leader and its followers.

But when it is configured that way:

{
  "HTTPAdvertise": "https://{{ orchestrator_hostname }}:{{ orchestrator_port }}",
  "UseSSL": true
}

Then curl -D - -k https://{orchestrator_ip}:{orchestrator_port}/api/clusters request returns correct information only if it was sent to a raft leader. If it's sent to a follower, it returns 502 Bad Gateway and Orchestrator logs say: remote error: tls: bad certificate.

Expected behaviour:

reverse-proxy should work properly, so any follower should be able to provide requested information from its leader.

Root cause analysis:

orchestrator/go/http/raft_reverse_proxy.go

Lines 46 to 47 in f818d8a

	proxy := httputil.NewSingleHostReverseProxy(url)
	proxy.ServeHTTP(w, r)

Creates a new proxy without any HTTP transport and starts serving HTTP . Since no transport is configured, a default HTTP transport is created automatically:

   func (p *ReverseProxy) ServeHTTP(rw http.ResponseWriter, req *http.Request) {
   	transport := p.Transport
    	if transport == nil {
    		transport = http.DefaultTransport
    	}

This works when SSL is disabled, but it breaks routing logic in the middle when it is enabled.

Workaround:

Sending all requests directly to a raft leader works pretty well both with and without SSL.

Solution proposal:

A suitable HTTP transport is already being created for raft health requests:

orchestrator/go/raft/http_client.go

Lines 37 to 68 in f818d8a

	httpTimeout := time.Duration(config.ActiveNodeExpireSeconds) * time.Second
	dialTimeout := func(network, addr string) (net.Conn, error) {
	return net.DialTimeout(network, addr, httpTimeout)
	}
	tlsConfig := &tls.Config{
	InsecureSkipVerify: config.Config.SSLSkipVerify,
	}
	if config.Config.UseSSL {
	caPool, err := ssl.ReadCAFile(config.Config.SSLCAFile)
	if err != nil {
	return err
	}
	tlsConfig.RootCAs = caPool
	if config.Config.UseMutualTLS {
	var sslPEMPassword []byte
	if ssl.IsEncryptedPEM(config.Config.SSLPrivateKeyFile) {
	sslPEMPassword = ssl.GetPEMPassword(config.Config.SSLPrivateKeyFile)
	}
	if err := ssl.AppendKeyPairWithPassword(tlsConfig, config.Config.SSLCertFile, config.Config.SSLPrivateKeyFile, sslPEMPassword); err != nil {
	return err
	}
	}
	}
	httpTransport := &http.Transport{
	TLSClientConfig: tlsConfig,
	Dial: dialTimeout,
	ResponseHeaderTimeout: httpTimeout,
	}
	httpClient = &http.Client{Transport: httpTransport}

This code can be reused to get a proper HTTP transport for a reverse-proxy as well.

[shlomi-noach]

also possibly related to #979?

[akatashev]

also possibly related to #979?

I don't really think so. It is about authentication and this issue is about wrong TLSClientConfig for Proxy transport.
Followers' logs were saying remote error: tls: bad certificate and leader's logs were saying: x509: certificate signed by unknown authority. And I don't have AuthenticationMethod set at all, so it's basically none.

The proposed PR was tested "manually" on a 3-members cluster and it solved the issue for me.

[shlomi-noach]

fixed by #1344