Fix leak of connection to OpenShift

[sleshchenko]

When OpenShift workspace is started we have following output in logs of tomcat:

2017-08-03 15:44:15,559[aceSharedPool-0]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 411]    - Workspace 'che:testx19' with id 'workspace8go3un4udnwr1pfd' started by user 'undefined'
Aug 03, 2017 3:49:01 PM okhttp3.internal.platform.Platform log
WARNING: A connection to https://192.168.42.246:8443/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE);

Creation of all OpenShift clients placed in try with resources section.
So it is needed to investigate why leak is done and fix it.

[l0rd]

This issue has some analysis about various problems (including memory leaks) in the OpenShift client. And since we don't use JIRA anymore we are tracking that here.

[sleshchenko]

@l0rd As far as I understand described issues mostly are not in OpenShift connector but in fabric8 kubernetes client. Some of them have the corresponding issue for kubernetes-client, some no. Also, there is no described problem that is described here.
Do you think this is our incorrect usage of kubernetes client or they have a bug? If they have, then should I create issue for kubernetes-client and link it with this issue?

[l0rd]

@sleshchenko I have no idea. @amisevsk @ibuziuk and @snjeza have worked on that and may know if it's a kubernetes client bug or how we use it in OpenShiftConnector.

[sleshchenko]

@amisevsk @ibuziuk and @snjeza Do you have any thought about it?

[snjeza]

@sleshchenko I have described the issue in https://issues.jboss.org/browse/CHE-180?focusedCommentId=13396715&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13396715

[sleshchenko]

@snjeza I see. Thanks

[akorneta]

Code which interacts with OpenShift client produces connection leaks. We also investigated that the kubernetes client version does not affect this. We tried combinations on ocp instance v3.6.0:

                                            current version
openshift-client:   v3.1.0  v3.0.3  v2.6.3     v2.2.8
kubernetes-client:  v3.1.0  v3.0.3  v2.6.3     v2.2.8
kubernetes-model:   v2.0.4  v2.0.4  v1.1.4     v1.0.67
okhttp:             v3.8.1  v3.8.1  v3.8.1     v3.6.0 

reproduce steps:

• run any che workspace
• wait about ~ 10 minutes
• check che server logs

Also we investigated that for start of one che workspace we create about 16 OpenShiftClients which is turn contain one instance of the OkHttpClient that holds the pool of connections, executor service e.t.c.
We decided to replace the approach and create 1 OpenShiftClient for the whole application and reuse it. After that connection leaks did not appear. But, another problem occurs:
on several cycles of normal start/stop of che workspaces, bootstrapper execs are freezes and falls out on timeouts, as result workspaces cannot be run.

All the studies were carried out on branch che6 continue to work on this.

[akorneta]

In che6 branch we had the one OpenShiftClient for one interaction with OpenShift API, it means that for start workspace flow we created at least 16 new OpenShiftClients. With this approach serves logs was full of warnings about connections leaks.
Snjeza described that happens because of OpenShiftOAuthInterceptor. If request fails with 403 or 401 response code, it tries to resend request with oauth token(which is configured or with fetched one if username and password are configured) . Before resending of request it performs the close of response body, but for websocket connections it does not release the connection. So when we send first websocket connection using newly created OpenShiftClient instance with oauth token from configuration, we had the connection leaks after 5 minutes. The warnings about connection leaks appear after 5 minutes because of the cleanup job that is performed by ConnectionPool(where the real connection is placed) in 5 minutes.
How it works now:
We create only one instance of OpenShiftClient whole Che server and reuse it for all the workspaces. No warnings about connections leaks appear because of initialization of oauth token during first http(not websocket) connection.