SSL client auth: Client cert is not submitted from the 2nd time when upgrading from 1.8 to 1.9

[jeffpeiyt]

Appreciate your help!

After upgrading to 1.9.10 from 1.8.14, the SSLcontext that have set the client authentication (mutual authentication) only works when hitting the target URL the first time. Then from the 2nd call the client is not submitting the cert anymore.

Is there any pool or settings that reset the client auth with the ssl engine / ssl parameters?

When <=1.8 we had the same issue if using AsyncHttpClientConfigBean, but when <=1.8, AsyncHttpClientConfig.Builder() actually do not have this issue.

code: from the 2nd time using the fastClient will not submit the client cert..

//set ssl context
sslContext.init(keyManagers, trustManager, secureRandom);

com.ning.http.client.AsyncHttpClientConfig.Builder builderFastClient = new AsyncHttpClientConfig.Builder();
builderFastClient.setSSLContext(sslContext);
builderFastClient.setAcceptAnyCertificate(true);

builderFastClient.setConnectTimeout(**);
builderFastClient.setRequestTimeout(**);
fastClient = new AsyncHttpClient(builderFastClient.build());

[slandelle]

Correct me if I'm wrong, but mutual auth only happens during the TLS handshake, not on every request. So if you're reusing a connection from the pool, this handshake is already done.

Then, I don't see anything in your sample related to mutual auth. I'd expect you to set SSLParameters.needClientAuth on the SSLContext.

[slandelle]

Could you please provide the exceptions that you have that made you think that the client cert was not "sent"?

Are you sure you're not bitten by SNI, as SSLEngines are now properly created with the peer host and port?

[jeffpeiyt]

Stephane Thank you so much for your prompt response and help!! Really appreciate it!

We did not put all our code to set the client cert part here. It is basically set the keystore KeyManager with loading the client cert, then sslContext.init(keyManagers, trustManager, secureRandom);
Then builderFastClient.setSSLContext(sslContext);

OK. my server may not support keep alive / reusing the connection. So I should disable the reusing existing connection? How to do this or enforce a TLS handshake on every request?

I am reusing the sslContext but did not create a sslengine or set the SSLParameters...

I tried to create a sslenginee and set the ssl parameters but did not work. Maybe I did it incorrectly. I need to set it for teh ssl context but I do not know how to do it without creating a SSLengine. but that engine seems to target each request.

...
KeyStore ks = KeyStore.getInstance("JKS");
...
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(ks, certificatePassword);
KeyManager[] keyManagers = kmf.getKeyManagers();
...
sslContext.init(keyManagers, trustManager, secureRandom);

com.ning.http.client.AsyncHttpClientConfig.Builder builderFastClient = new AsyncHttpClientConfig.Builder();
builderFastClient.setSSLContext(sslContext);

We are doing massive HTTP calls on massive servers. with AHC 1.8 on 0.1% of stuck bad servers, mutual auth fails with socket not able to close forever (netstats in established state but not FIN) error was: java.net.ConnectException: Handshake did not complete within 10000ms. unclosed socket leak memory

With upgrade to AHC 1.9, this cannot close socket issue is gone. However, now after the first call, the later call to the same URL will not honor the ssl context we set with the keystone from the 2nd time....

[slandelle]

Check for yourself: if you pass a SSLContext to the config, it's used to create every SSLEngine: https://github.com/AsyncHttpClient/async-http-client/blob/master/api/src/main/java/org/asynchttpclient/SSLEngineFactory.java#L47

Except for constructing the SSLEngines with the peer hostname and port, I don't think I've changed anything.
If you really think there's an issue with AHC, please provide a reproducer.

[jeffpeiyt]

We can reproduce the issue. However it is a little tricky with the cert to send it here....
Since I mark isAcceptAnyCertificate as true. Could this be the root cause.

should accepting any server cert lead to ignoring all SSLParameters? (Para has useClientAuth)

SSLEngine sslEngine = sslContext.createSSLEngine(peerHost, peerPort);
if (!config.isAcceptAnyCertificate()) {
    SSLParameters params = sslEngine.getSSLParameters();
    params.setEndpointIdentificationAlgorithm("HTTPS");
    sslEngine.setSSLParameters(params);
}

[jeffpeiyt]

My bad.... I found the issue is gone when I remove the -source.jar from the lib. Currently we keep both jar for easy read source code. This issue haunted me whole week....

Thanks so much for your help!!

[jeffpeiyt]

hey I was messed with this issue for a long time. My previous test was false negative. I did not use the right target servers. Still a 401 after 1st attempt. If you can think of any thing is related to this. Please let me know

[slandelle]

You getting 401 means that your issue has nothing to do with HTTPS/TLS. Otherwise you'd be getting an SSL related IOException.
I really suspect that the issue is with your application. You have to find out on which condition it replies with such a 401.

I'm afraid I can do much if you don't provide a simple reproducer (I don't mean access to your application, I mean some simple mock application that displays the issue).

Closing for now. Feel free to ping if you can provide a reproducer. I'll reopen then.

[jeffpeiyt]

Sure. Understandable.

hmm. just validated that 1.8.15 is good but 1.9.1 is bad.

So just the changes between (Nov 28, 2014) (Dec 09, 2014) , any easy way can diff?

Knowing this is a big release to dump JDK6....

[jeffpeiyt]

actually 1.9.0 fails.

so just check diff on (Nov 28, 2014) (Dec 01, 2014)

[slandelle]

AHC 1.9 is a complete rewrite (AHC 2.0 preview). It's way faster and improves a lot of things like connection pooling. It might just be there's a race condition in your application that you didn't hit before.

[jagte]

I am having the same issue. I posted a response here: https://groups.google.com/forum/#!topic/gatling/QEMYnZsVI-k

[jeffpeiyt]

+1,
looking forward to fixing it and trying 1.9

[slandelle]

@jeffpeiyt Best way to get this fixed is to provide a way to reproduce.

[jeffpeiyt]

@slandelle, agreed.
We are able to reproduce just as @jagte however in a proprietary environment. As this issue concerns sensitive data. We need to find a public available mutual auth server with newly generated server key pair, and then the client key pair in order to show it here.... The mutual auth server we have right now is not open sourced yet....

It just needs quite some efforts to do it.

@jagte, we can do this http://stackoverflow.com/questions/27362588/mutual-authentication-with-tomcat-7 and generate unsigned key pairs, install them. and then show the trace....

[srirama-rayaprolu]

@slandelle
In my case, it is not sending even in the first time also. Due to this SSL handshake is failing.

Do I need to do this setting by default? I am also using 1.9.
context.getSupportedSSLParameters().setNeedClientAuth(true);

[slandelle]

No, needClientAuth is a server side parameter. You must configure a proper keystore.