Broken pipe on API call with Minikube #1003

zenbones · 2020-06-24T19:24:00Z

I have a pretty basic minikube setup running in hyperv. Just trying to make sure the basics work, so this is copied from the examples. This code running inside the image of a deployment...

ApiClient client = ClientBuilder.cluster().build();

Configuration.setDefaultApiClient(client);

OkHttpClient httpClient = client.getHttpClient().newBuilder().readTimeout(0, TimeUnit.SECONDS).writeTimeout(0, TimeUnit.SECONDS).build();
client.setHttpClient(httpClient);

CoreV1Api api = new CoreV1Api();

V1Pod pod =
  new V1PodBuilder()
    .withNewMetadata()
    .withName("apod")
    .endMetadata()
    .withNewSpec()
    .addNewContainer()
    .withName("www")
    .withImage("nginx")
    .endContainer()
    .endSpec()
    .build();

api.createNamespacedPod("default", pod, null, null, null);

...at the point of api.createNamespacedPod(), leads to...

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.smallmind.spark.singularity.boot.SingularityEntryPoint.main(SingularityEntryPoint.java:65)
Caused by: io.kubernetes.client.openapi.ApiException: java.net.SocketException: Broken pipe (Write failed)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:898)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPodWithHttpInfo(CoreV1Api.java:7902)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPod(CoreV1Api.java:7876)
at com.forio.epicenter.k8s.operator.worker.WorkerController.main(WorkerController.java:58)
... 5 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at okio.Okio$1.write(Okio.java:79)
at okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
at okio.RealBufferedSink.flush(RealBufferedSink.java:224)
at okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:185)
at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:499)
at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:489)
at okhttp3.internal.connection.RealConnection.startHttp2(RealConnection.java:315)
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:304)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
at okhttp3.RealCall.execute(RealCall.java:81)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:894)
... 8 more

What's the best way to determine why I'm getting that error? Anything obvious I'm missing?

The text was updated successfully, but these errors were encountered:

alek-sys · 2020-06-25T14:48:28Z

The same problem exists with kind cluster (kind v0.8.1 go1.14.2 darwin/amd64), however it works on GKE.

zenbones · 2020-06-25T16:35:47Z

If it helps, my problem is a minikube env on hyperv , win 10/amd64, docker 19.0.3.8 using ubuntu 18.0.4 containers

brendandburns · 2020-06-27T05:37:30Z

I'm not sure how minikube sets up it's networking. I suspect that the kubernetes service isn't setup properly, but I will try to reproduce this.

habuma · 2020-06-29T12:49:57Z

I've recreated this on Kind (v0.8.1 go1.14.2 darwin/amd64, running on Docker version 19.03.8, build afacb8b), doing nothing more than making a call to api.getAPIResources() (which, if I'm not mistaken, makes a GET request to /api/v1). Same exception and stack trace.

So, in the same pod, I tried to bypass the Kubernetes client by using Spring's RestTemplate to make a GET request to /api/v1. I had to set it up to ignore SSL problems and to fetch the token from /var/run/secrets/kubernetes.io/serviceaccount/token and put it into the Authorization header. But with that setup, the request came back successful.

But knowing that the Kubernetes client uses OkHttp and not RestTemplate, I tried making the request using the OkHttp client, setting up the SSL context and token the same as I had for RestTemplate (and having to truncate the trailing new line from the token file), and it failed with the same stack trace as when going through the Kubernetes client.

A little deeper digging makes it seem that the exception is thrown while dealing with the SSL handshake, as per the following stack trace. It's deep in some low-level socket code where it happens, but if I were guessing, I'd say that OkHttp is fine and Kind/Minikube are both fine. But the combination isn't so great. I guess that at some point OkHttp sends or does something that the K8s API on Kind/Minikube handle by closing the connection...but OkHttp expecting to continue using that connection throws the Broken Pipe exception because the connection has been closed by the K8s API on those clusters. My guess continues to suppose that whatever OkHttp does to cause the K8s API to close, doesn't result in a closed connection on GKE (or other K8s platforms)...and whatever it is that OkHttp does isn't done by RestTemplate so that even on Kind/Minikube, the connection stays open. Again...all guesses in this paragraph.

habuma · 2020-06-29T17:04:05Z

I've found a way to make this work!

In short, I was reading that there are some TLS 1.3 changes that might impact this and that aren't available until more recent versions of Java. So...long story short...I realized that I was still targeting Java 1.8 in my project's build. I bumped it up to target Java 11 in the build (which also involved me switching to Java 11 in SDKMan...for "reasons" I still have 1.8 as the default in SDKMan), and suddenly it works! What's more, it works on both Kind and Minikube!

@zenbones Any chance you are building to target Java 1.8? If so, maybe try targeting Java 11? Would like to hear someone else try this and confirm that it works.

(Note, that I'm building using Spring Boot 2.3.1 and using its support for creating a container image. Java 11 works fine, but Java 12 and 13 not so much because at build time it can't find compatible dependencies for the Paketo BellSoft Liberica Buildpack 2.8.0 while building the image. It does seem to build fine with Java 14, though it fails at runtime with a new exception: javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request.

zenbones · 2020-06-29T19:44:51Z

Wow, that's good detective work. Thanks. Our build does target Java 8, and we do pan to move to Java 11, but Oracle has given control of J2EE to the Eclipse Foundation, but not the javax.* namespace. So Eclipse is in the process of moving javax.* to jarakta.*, and we (and most everyone) is using pieces of the javax.* modules. We also use libraries with javax.* dependencies, so if we move it'll be wrong fro some our transient dependencies, and if we don't move we get caught by others that have shifted. This is all working itself out by the end of the year (I believe), so I was hoping to wait until then before making the jump to Java 11.

Sounds like this is not anyone's fault and there's probably little to be done and it's on us to make the Java11 move... but I'm wondering if forking the project and rebuilding it under Java 8 might also be a temporary solution... maybe going so far as to swap out the OKHttp client for something else (maybe the Apache client). I may be able to try that.

habuma · 2020-06-29T20:24:37Z

Swapping out OKHttp for Apache (or anything else) seems like quite a chore. The ApiClient class appears to be coupled to OkHtttp. Of course, it depends on how much of ApiClient you need. Ultimately, it just provides a convenience for making HTTP requests against the K8s API, so you could write your own convenient abstraction over the API that uses whatever HTTP client you want. I happen to know that Spring's RestTemplate with an Apache HttpClient client factory works fine in Java 8.

But yeah, I don't think this is a bug as much as a compatibility issue. As @alek-sys noted above, I also know it to work on GKE, regardless of the Java version in play (even Java 14 plays nice there). Just Kind and Minikube sorta require that your app that's using ApiClient be on Java 11.

brendandburns · 2020-06-30T06:09:36Z

fwiw, all of this code is generated, so swapping out the client library isn't a workable solution since we need the ability to re-generate the library.

For now, I would just use kind to create a Kubernetes 1.17 cluster instead of 1.18 (unless you really need 1.18 APIs) and I believe it should work w/ Java 8, kubernetes 1.18 made some TLS changes.

zenbones · 2020-07-01T18:02:15Z

Not sure what I can or can't do for now, but I can live with transient issues and find workarounds. I'm more concerned with ongoing problems. Are there any plans to make this compatible with Java 8, Minikube and kubernetes 1.18, or is the plan to let people upgrade out of Java 8 in order to find a solution?

brendandburns · 2020-07-02T18:15:42Z

This library is compatible w/ Java8 and Kubernetes 1.18. This library is tested and working with both JDK8 and JDK11 and both cloud (Azure, GKE) and on-premise (my home cluster) with Kubernetes 1.16, 1.17 and 1.18. So I'm confident that it works correctly.

There was a bug in JDK 11 (https://bugs.openjdk.java.net/browse/JDK-8236039) that has been fixed in recent versions that may have caused the problem cited w/ JDK 14 above (#1003 (comment)) as well as other certificate problems depending on JDK version.

I have not done recent testing w/ minikube so it's possible that there is some problem in there certificate generation. I will try to repro when I have time.

If you have problems w/ Java8 + this library and a complete install of Kubernetes (e.g. not minikube) let me know, in the meantime I think it's a problem specifically w/ minikube and we'll try to reproduce when we have cycles.

zenbones · 2020-07-04T00:04:19Z

And in the meantime I've sorted out a minimal Java 11 version of our code so I'll try this again and let you know.

zenbones · 2020-07-06T19:08:04Z

Whatever's happening is not just Java 11. I get the same broken pipe under Minikube with...

openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment (build 11.0.7+10-post-Ubuntu-2ubuntu218.04)
OpenJDK 64-Bit Server VM (build 11.0.7+10-post-Ubuntu-2ubuntu218.04, mixed mode, sharing)

I guess the next step is to try this in a non-Minikube environment. Back to the drawing board...

zenbones · 2020-07-08T00:29:52Z

And running in EKS on kubernetes 1.16 I get...

exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.smallmind.spark.singularity.boot.SingularityEntryPoint.main(SingularityEntryPoint.java:65)
Caused by: io.kubernetes.client.openapi.ApiException:
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPodWithHttpInfo(CoreV1Api.java:7902)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPod(CoreV1Api.java:7876)
at com.forio.epicenter.k8s.operator.worker.WorkerController.main(WorkerController.java:58)
... 5 more

This is copied from your example code and was about the simplest example I could find. Is this tested against anything more than GKE? I don't have an account but maybe I'll try to set one up there.

Oh, do you have that example you used to test on your home cluster? Maybe I could just drop that into a container and see if it works in Minikube or EKS?

sarveshkaushal · 2020-07-08T07:07:27Z

I tried to repro this on my machine (MacOS) without any success. Here are the steps I followed:

Installed Minikube(v1.11.0) with virtual box(6.1.10) on MacOS(10.15.5)
Created a Java(1.8.0_212 openjdk jdk alpine) Spring Boot(2.4.0-SNAPSHOT) application which has a controller mapped to /run URI.
Have the above example code (given in first comment) run when a request for /run resource is made.
Created a deployment to run my java spring boot application on single node K8 cluster created by Minikube.
Started port forwarding on port 8080. (Spring boot app runs on 8080 in my case)
Send a request to http://localhost:8080/run. I am able to create the pod (apod) as given in example.

zenbones · 2020-07-08T16:57:48Z

Odd, as the second report was for...

The same problem exists with kind cluster (kind v0.8.1 go1.14.2 darwin/amd64), however it works on GKE.

...but I'm game. I'll update minikube to the latest win10 version, and put the code into a Jersey resource, run the 1.8 JVM starting a Grizzly instance on Ubuntu, none of which should be a significant difference from your setup. Fingers crossed.

zenbones · 2020-07-09T00:15:48Z

Replicated with...

Windows 10

minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd

In container...

Ubuntu 18.04

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1~18.04-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

Running a Jersey resource under Grizzly, same broken pipe. Maybe an interaction between OS, Minikube and the Java client. Would love to be able to debug into Minikube. Or get an error log at least.

sarveshkaushal · 2020-07-09T00:22:50Z

@zenbones -Are you sure it's a broken pipe error or something else? I got SocketTimeout initially but that was due to wrong configs.

zenbones · 2020-07-09T00:38:45Z

Yes, I'm sure. See the error above, it has not changed. And I have debug statements in the java code so I can watch it approach the call to createNamespacePod(), so it's getting to that line, and that's the line in the stack trace. I did check the minikube logs. The only output is...

Trace[267839508]: [2m19.517418668s] [2m19.515862333s] Transformed response object
Jul 09 00:31:53 minikube kubelet[4466]: I0709 00:31:53.617174 4466 log.go:172] http: superfluous response.WriteHeader call from k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader (httplog.go:205)

...which occurs before the java client code ever reaches createNamespacePod(). I'll check and see if GoLang has a remote debugger I can attach to.

sarveshkaushal · 2020-07-09T00:46:49Z

As @brendandburns mentioned above, looks like some problem with your setup. I wish I could help you more on this but I am not able to reproduce this.

franck102 · 2020-07-09T08:20:48Z

I have the exact same problem on this environment, with Java 8 operators that have been working fine until a recent minikube upgrade. The problem affect several containers, and multiple operations (list, watch ...):

minikube version: v1.11.0
commit: 57e2f55f47effe9ce396cea42a1e0eb4f611ebbd

$ minikube config view
- coredns: false
- dashboard: true
- disk-size: 30G
- kube-dns: true
- memory: 16384
- metrics-server: true
- WantReportError: true
- cpus: 6
- heapster: true
- ingress: true
- vm-driver: virtualbox

$ kc version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-04T14:53:16Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Here is a sample exception;

Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Namespace]  with name: [null]  in namespace: [dmp-system]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:155)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:608)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)
	at com.fico.dmp.planning.client.KubernetesClientFactory.getKubernetesClient(KubernetesClientFactory.java:37)
	at com.fico.dmp.planning.KubernetesPlanRunner.exec(KubernetesPlanRunner.java:31)
	at com.fico.dmp.planning.openshift.ManageComponentInstanceInOpenshiftOperator.reconcile(ManageComponentInstanceInOpenshiftOperator.java:172)
	at com.fico.dmp.container.plan.PlansController.doStart(PlansController.java:62)
	at com.fico.dmp.container.plan.PlansController.startPlan(PlansController.java:48)
	at com.fico.dmp.container.services.ComponentController.update(ComponentController.java:315)
	... 68 common frames omitted
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
	at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:894)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:865)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at okio.Okio$1.write(Okio.java:79)
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:180)
	at okio.RealBufferedSink.flush(RealBufferedSink.java:224)
	at okhttp3.internal.http2.Http2Writer.settings(Http2Writer.java:203)
	at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:514)
	at okhttp3.internal.http2.Http2Connection.start(Http2Connection.java:504)
	at okhttp3.internal.connection.RealConnection.startHttp2(RealConnection.java:299)
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:288)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:111)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
	at okhttp3.RealCall.execute(RealCall.java:93)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:412)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:151)
	... 76 common frames omitted

franck102 · 2020-07-09T17:24:32Z

I just reproduced the issue with Kubernetes 1.17.8:

kc version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.8", GitCommit:"35dc4cdc26cfcb6614059c4c6e836e5f0dc61dee", GitTreeState:"clean", BuildDate:"2020-06-26T03:43:27Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.8", GitCommit:"35dc4cdc26cfcb6614059c4c6e836e5f0dc61dee", GitTreeState:"clean", BuildDate:"2020-06-26T03:36:03Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

This is critical for us, any pointer would be really appreciated!

sarveshkaushal · 2020-07-09T17:43:56Z

@franck102 - Can you give me more information on your setup. I will try to repro it.

sarveshkaushal · 2020-07-09T17:47:04Z

io.fabric8.kubernetes.client looks like a different client.

franck102 · 2020-07-09T19:37:50Z

io.fabric8.kubernetes.client looks like a different client.

Good point, to be honest I had focused on the okhttp3 stack trace which is exactly the same and I didn’t realize the client was different.
So this is clearly an okhttp3 vs. minikube Pb... but not sure who can help :(

I will reproduce with SSL debugging enabled and post the stack trace, I did that before and it didn’t seem to be a certificate problem.

zenbones · 2020-07-09T19:44:17Z

I finally have client code working. It's against Minikube but running from outside kubernetes. I was able to reproduce the problems this way, and to get a solution, I'm setting up to try from with a container now. My experience is that the client using Java 8 does not work. Using Java 11 does not work, unless you add...

System.setProperty("jdk.tls.client.protocols", "TLSv1.2")

...and then it works. I got this notion from golang/go#35722, so I suspect it's related and a TLS issue. I'll let you know if the solution works from within the container.

sarveshkaushal · 2020-07-09T20:07:06Z

io.fabric8.kubernetes.client looks like a different client.

Good point, to be honest I had focused on the okhttp3 stack trace which is exactly the same and I didn’t realize the client was different.
So this is clearly an okhttp3 vs. minikube Pb... but not sure who can help :(

I will reproduce with SSL debugging enabled and post the stack trace, I did that before and it didn’t seem to be a certificate problem.

You can check with maintainers of this repo.

franck102 · 2020-07-09T20:28:03Z

Thanks, since this affect both this client and the io.fabric8 client (@zenbones above) I will post in the minikube issue I mentioned above.

zenbones · 2020-07-09T22:17:51Z

BTW, using System.setProperty("jdk.tls.client.protocols", "TLSv1.2") in a container running in the cluster leads to...

SEVERE: service exception:
javax.servlet.ServletException: javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: io.kubernetes.client.openapi.ApiException: okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:116)
at org.glassfish.grizzly.servlet.FilterChainImpl.invokeFilterChain(FilterChainImpl.java:83)
at org.glassfish.grizzly.servlet.ServletHandler.doServletService(ServletHandler.java:202)
at org.glassfish.grizzly.servlet.ServletHandler.service(ServletHandler.java:149)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.servlet.ServletException: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: io.kubernetes.client.openapi.ApiException: okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:128)
at org.smallmind.nutsnbolts.lang.web.PerApplicationContextFilter.doFilter(PerApplicationContextFilter.java:61)
at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:114)
... 7 more
Caused by: javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: io.kubernetes.client.openapi.ApiException: okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
at org.glassfish.grizzly.servlet.FilterChainImpl.doFilter(FilterChainImpl.java:124)
... 9 more
Caused by: org.glassfish.jersey.server.ContainerException: io.kubernetes.client.openapi.ApiException: okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
at org.glassfish.jersey.servlet.internal.ResponseWriter.rethrow(ResponseWriter.java:254)
at org.glassfish.jersey.servlet.internal.ResponseWriter.failure(ResponseWriter.java:236)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:436)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
... 14 more
Caused by: io.kubernetes.client.openapi.ApiException: okhttp3.internal.http2.StreamResetException: stream was reset: NO_ERROR
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:898)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPodWithHttpInfo(CoreV1Api.java:7902)
at io.kubernetes.client.openapi.apis.CoreV1Api.createNamespacedPod(CoreV1Api.java:7876)
at com.forio.epicenter.k8s.operator.worker.WorkerControllerResource.start(WorkerControllerResource.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
... 23 more
at okhttp3.internal.http2.Http2Stream.checkOutNotClosed(Http2Stream.java:650)
at okhttp3.internal.http2.Http2Stream$FramingSink.emitFrame(Http2Stream.java:575)
at okhttp3.internal.http2.Http2Stream$FramingSink.close(Http2Stream.java:622)
at okio.ForwardingSink.close(ForwardingSink.java:47)
at okhttp3.internal.connection.Exchange$RequestBodySink.close(Exchange.java:253)
at okio.RealBufferedSink.close(RealBufferedSink.java:248)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:70)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at io.kubernetes.client.openapi.ApiClient$2.intercept(ApiClient.java:1232)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221)
at okhttp3.RealCall.execute(RealCall.java:81)
at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:894)
... 39 more

This NO_ERROR is treated by java as an error, and I don't see any result from the call, so this is different but still not good.

zenbones · 2020-07-09T22:23:45Z

But from outside this cluster, this works perfectly...
System.setProperty("jdk.tls.client.protocols", "TLSv1.2");

String kubeConfigPath = "<home dir>/.kube/config";
ApiClient client = ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath))).build();
Configuration.setDefaultApiClient(client);
CoreV1Api api = new CoreV1Api();

V1Pod pod =
  new V1PodBuilder()
    .withNewMetadata()
    .withName("apod")
    .endMetadata()
    .withNewSpec()
    .addNewContainer()
    .withName("www")
    .withImage("nginx")
    .endContainer()
    .endSpec()
    .build();

api.createNamespacedPod("default", pod, null, null, null);

So it's not Minikube itself, but the configuration being loaded inside the cluster?

franck102 · 2020-07-10T04:45:22Z

So it's not Minikube itself, but the configuration being loaded inside the cluster?
It seems to be Kubernetes, I found 2 combinations that work:

Java 11 (amazoncorretto:11) with Kubernetes 1.17.8 (no tls switch needed for me)
Java 8 (amazoncorretto:8) with Kubernetes 1.15.11.(but not on minikube)

zenbones · 2020-07-10T17:21:58Z

I agree, in that I think it's something basic in the way Java versions and Kubernetes or Go versions have implemented the TLS specification. The problem is that sometimes working, on very particular versions and implementations is very close to not working. I need to find a way through I can depend on. Not sure what that is. At least this works from outside kubernetes, so I could implement a controller process that sat outside the kubernetes framework... which seems ridiculous.

zenbones · 2020-07-10T23:59:34Z

I created a little client using the apache http client...

  <dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.4.13</version>
  </dependency>

...and it works, in that calls complete, with TLS 1.3 (or at least no restriction).

byte[] tokenBytes = Files.readAllBytes(Paths.get("/var/run/secrets/kubernetes.io/serviceaccount/token"));

K8SCallback callback;
HttpGet httpReq = new HttpGet("/api/v1/pods");

httpReq.setHeader("Authorization", "Bearer " + new String(tokenBytes));
HttpClient.execute(HttpHost.create("https://kubernetes.default.svc"), httpReq, callback = new K8SCallback(), 3);

So, this may be an OKHTTP3 / Kubernetes interaction. I do think it's TLS related, but apache has no problems.

franck102 · 2020-07-11T06:42:33Z

FYI Java 8 works with no special changes against Kubernetes 1.15.11.

zenbones · 2020-07-13T19:14:06Z

So, getting proper error messages from own little client, I could see I was getting a 403 on calls trying to do anything real with the API. That gave me the clue I needed to fix my RBAC. Once I did that everything worked with my client, so I flipped back to the standard client and everything worked. And, for whatever reason, it worked with TLS 1.3. So all of this may have been RBAC. Why this translates into broken pipes as opposed to some helpful error message I don't know. But as I have a working client, I'm going to close this issue.

If anything help can come from this, if you're having problems, I'd say create your own working client so you can see what's going on. If I get problems again I'll expand the one I have with the swagger apis and open source it.

Thanks for all the great input.

Navodar22 · 2020-07-22T06:24:51Z

Hello.
Update to java 11 (correto) helped resolved this issue.

KevinWang15 · 2021-01-29T07:46:33Z

I can confirm that this method mentioned here works

client.setHttpClient(client.getHttpClient().newBuilder().protocols(Arrays.asList(Protocol.HTTP_1_1)).build());

The easiest way to reproduce this is to use docker 8u252-jre image, and use the okhttp client to send any request to a kubernetes 1.18 server. (it doesn't even have to be authenticated)

funky-eyes · 2022-07-12T08:12:05Z

I can confirm that this method mentioned here works
client.setHttpClient(client.getHttpClient().newBuilder().protocols(Arrays.asList(Protocol.HTTP_1_1)).build());
The easiest way to reproduce this is to use docker 8u252-jre image, and use the okhttp client to send any request to a kubernetes 1.18 server. (it doesn't even have to be authenticated)

#1370

franck102 mentioned this issue Jul 9, 2020

java kubernetes client error kubernetes/minikube#8584

Closed

zenbones closed this as completed Jul 13, 2020

fabiokung mentioned this issue Aug 11, 2020

okhttp3 issues with openjdk8 > 8u251 and http/2 #1149

Closed

kezhenxu94 mentioned this issue Oct 25, 2020

failure enabling TLS in Docker distro of OAP server 8.1.0-es7 apache/skywalking#5518

Closed

4 tasks

Broken pipe on API call with Minikube #1003

Broken pipe on API call with Minikube #1003

Comments

zenbones commented Jun 24, 2020

alek-sys commented Jun 25, 2020

zenbones commented Jun 25, 2020

brendandburns commented Jun 27, 2020

habuma commented Jun 29, 2020 • edited Loading

habuma commented Jun 29, 2020

zenbones commented Jun 29, 2020 • edited Loading

habuma commented Jun 29, 2020

brendandburns commented Jun 30, 2020

zenbones commented Jul 1, 2020

brendandburns commented Jul 2, 2020 • edited Loading

zenbones commented Jul 4, 2020

zenbones commented Jul 6, 2020

zenbones commented Jul 8, 2020

sarveshkaushal commented Jul 8, 2020

zenbones commented Jul 8, 2020 • edited Loading

zenbones commented Jul 9, 2020 • edited Loading

sarveshkaushal commented Jul 9, 2020

zenbones commented Jul 9, 2020

sarveshkaushal commented Jul 9, 2020

franck102 commented Jul 9, 2020

franck102 commented Jul 9, 2020

sarveshkaushal commented Jul 9, 2020

sarveshkaushal commented Jul 9, 2020

franck102 commented Jul 9, 2020 • edited Loading

zenbones commented Jul 9, 2020

sarveshkaushal commented Jul 9, 2020 • edited Loading

franck102 commented Jul 9, 2020

zenbones commented Jul 9, 2020

zenbones commented Jul 9, 2020

franck102 commented Jul 10, 2020

zenbones commented Jul 10, 2020

zenbones commented Jul 10, 2020

franck102 commented Jul 11, 2020

zenbones commented Jul 13, 2020

Navodar22 commented Jul 22, 2020

KevinWang15 commented Jan 29, 2021

funky-eyes commented Jul 12, 2022

habuma commented Jun 29, 2020 •

edited

Loading

zenbones commented Jun 29, 2020 •

edited

Loading

brendandburns commented Jul 2, 2020 •

edited

Loading

zenbones commented Jul 8, 2020 •

edited

Loading

zenbones commented Jul 9, 2020 •

edited

Loading

franck102 commented Jul 9, 2020 •

edited

Loading

sarveshkaushal commented Jul 9, 2020 •

edited

Loading