[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate | Rancher EKS Cluster #1767
Comments
|
This is how I was able to connect: Not ideal, as SSL looks totally broken. But glad I was able to get it to work. I suppose I was expecting this to be handled for me. :( |
|
/assign @yliaog |
|
there seem to be issues in your use of the client library, could you try the given example in https://github.com/kubernetes-client/python#examples? |
|
I am seeing the exact same issue pipfile Local kubectl get all pods on a remote EKS cluster after setting up .kube/config Confirmed this also worked for me -> #1767 (comment) |
|
Maybe it's a bug in rancher ? I have similar issue on premise. |
|
I have the same issue and i am not using rancher. kubectl is able to use while the python client seems to ignore these information. i disabled validation directly in the kubeconfig in the end: with open(os.environ["KUBECONFIG"], "r") as fd:
kubeconfig = yaml.load(fd, Loader=yaml.FullLoader)
kubeconfig["clusters"][0]["cluster"]["insecure-skip-tls-verify"] = True
with open(os.environ["KUBECONFIG"], "w") as fd:
yaml.dump(kubeconfig, fd, default_flow_style=False)
config.load_kube_config(os.environ['KUBECONFIG'])note that these modifications break the use of i would provide a patch as well, but i am not sure about the logic where to set the ca here to evaluate this if statement here to |
|
Thanks, I had the same problem as well and wanted to keep kubectl working. I generate my client now like this until this is fixed: import os
import yaml
import tempfile
from pathlib import Path
from kubernetes import client, config
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def gen_client():
kube_config_orig = f'{Path.home()}/.kube/config'
tmp_config = tempfile.NamedTemporaryFile().name
with open(kube_config_orig, "r") as fd:
kubeconfig = yaml.load(fd, Loader=yaml.FullLoader)
for cluster in kubeconfig["clusters"]:
cluster["cluster"]["insecure-skip-tls-verify"] = True
with open(tmp_config, "w") as fd:
yaml.dump(kubeconfig, fd, default_flow_style=False)
config.load_kube_config(tmp_config)
os.remove(tmp_config)
return client.CoreV1Api()
v1 = gen_client() |
|
I just started with this module and come across this issue. Is this the standard behaviour? |
|
The client just does not seem to read EDIT: It does seem to read it correctly but somehow it's broken for some clusters. With a remote cluster on AKS it works just fine. The broken cluster is an in LAN cluster created with kubeadm. The one difference I can detect is that AKS uses 4096 bits RSA and kubeadm uses 2048 and the kubeadm CA cert has the following extra extensions: And it connects to a nonstandard port, which might also cause the issue. configuration.verify_ssl = False
configuration.client_side_validation = FalseHas no effect which is curious |
|
Just to help any others who come across this, I had a similar error message while using ansible with the Likely not the same cause as above, but the certificate that I provided in
The value of This doesn't make valid certificate chain, but usage of an intermediary as your I resolved my issue by setting this wouldn't be ideal for inter-worker communication (broadens the trust scope a great deal), but it will be fine for my local config. I wouldn't be surprised if other tooling (a la Only thing left that is curious to me is why it didn't resolve the root certificate from the operating store (since Hope this helps someone :) |
|
I also encountered this problem, following is some debug info which might be useful, seems like a problem with urllib3 Reproduction steps:
Expected
Actual
|
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
as there seems to be no traction here or by the looks of the investigation done by @OriHoch the issue might not even relate to this repository, could we maybe document the workaround provided by @dheeg in #1767 (comment) @yliaog ? |
|
This seems to really be an issue. We ran into it some days ago after renewing our cluster certificate. @yliaog also the list-all-pods example from https://github.com/kubernetes-client/python#examples does not work. Is there any incentive to fix this? |
|
Besides the rootCA we also had to add the issuingCA to the system, which fixed the issue. |
|
This is still an issue with kubernetes module on latest stable version too. Help us with the resolution. |
|
error :- urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='***', port=443): Max retries exceeded with url: ****/api/v1/nodes (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1131)'))) |
非常好的解决了我的问题，大赞👍 |
|
I found a permanent fix for this problem which should also be secure, the following script fixes kube config files certificate-authority-data to a valid value based on what is actually used when connecting to the relevant server. https://raw.githubusercontent.com/Kamatera/kamateratoolbox-iac/main/bin/fix_kubeconfig_ca_certs.py explanation: after a lot of research into ssl, I found the source of the problem in my case to be a mismatch in root certificate. I'm using Let's Encrypt certificate and the certificate chain which is served by the server differs from the actual chain used in the client (e.g. when using curl / web browser). For some reason the Python ssl library does not take into account the client certificate store when providing a cafile, and I think that's the reason it fails. My fix, uses Python code to fetch the certificate chain which is actually used by the client, and then use that as the certificate-authority-data. |
This is not working . Below are the error details.
|
|
@OriHoch I have the kubeconfig file generated with your python script and used it to fetch data using kubernetes provider and also with kubectl, but is getting cert error.
@yliaog I am looking for a way to move further with kubernetes provider, Why can't we skip SSL verification in latest kubernetes module ? |
|
While troubleshooting with wireshark and comparing the kubectl request, I discovered that the plugin doesn't set the SNI value in the TLS handshake. This value should be derived from the tls-server-name property in kubeconfig. I considered opening PR, but I found that there's already an open one (#1933) addressing the same issue. As a temporary solution, I directly added the addition_pool_args['server_hostname'] with the value of tls-server-name in the init method of the client/rest.py file until that PR is merged or released. |
|
Hi everyone. Any guidance on how to resolve this issue when using config.load_incluster_config? |
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close not-planned |
|
@k8s-triage-robot: Closing this issue, marking it as "Not Planned". In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What happened (please include outputs or screenshots):
Here's me using
kubectl. I'm just expecting the equiv. Python to work.My attempt at doing the same thing in Python and getting different results.
OK, doesn't work. Let's try loading the kube_config directly.
OK, digging through Github issues I came across #1622 which leads to #36
I can try that...
OK, what if we just disable SSL verify instead?
I can't actually get this client to work in any way that I can try...
What you expected to happen:
Anything to work.
How to reproduce it (as minimally and precisely as possible):
Many examples above.
Anything else we need to know?:
Environment:
kubectl version):python --version)pip list | grep kubernetes)I originally tried the very latest of this library, but I decided to test a downgrade to see if v23 would work too. I downgraded to have the client version match the server version.
The text was updated successfully, but these errors were encountered: