certificate verify failed

[misrn]

python 3.7
client 10.0.1

from kubernetes import client, config
config.kube_config.load_kube_config(config_file="F:\\config.yaml")
class Kubernetes:

    def __init__(self):
        self.Connect = client.CoreV1Api()

    def list_namespaces(self):
        data = []
        for ns in self.Connect.list_namespace().items:
            data.append(ns)
        return data
k = Kubernetes()
print(k.list_namespaces())

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.18.56.103', port=6443): Max retries exceeded with url: /api/v1/namespaces (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1056)')))

Can't verify the certificate. What should I do?

[roycaihw]

which OS are you running? SSL verification failures usually heavily depends on your environment setup. Please check out the SSL troubleshooting in readme: https://github.com/kubernetes-client/python#troubleshooting

you could also try disabling SSL verification if you're doing some testing. Ref kubernetes-client/python-base#177

[micw523]

Looking at his file path I think it’s some version of Windows

[nettyxiong]

I miss the same problem. It seems that the packages of requirements.txt is statisfied. And I find that use python3 is working but not with python2.
The enviroment is as follows:
CentOS 7.3

[root@VM_centos /data/code/python]# python -c "import ssl; print ssl.OPENSSL_VERSION"
OpenSSL 1.0.1e-fips 11 Feb 2013

[root@VM_centos /data/code/python]# pip freeze -r requirements.txt
google-auth==1.10.0
ipaddress==1.0.23
websocket-client==0.57.0
requests==2.6.0
requests-oauthlib==1.3.0
urllib3==1.25.7
## The following requirements were added by pip freeze:
Babel==0.9.6
backports.ssl-match-hostname==3.4.0.2
cachetools==4.0.0
chardet==2.2.1
Cheetah==2.4.4
Click==7.0
cloud-init==17.1
configobj==4.7.2
decorator==3.4.0
Flask==1.1.1
iniparse==0.4
iotop==0.6
IPy==0.75
itsdangerous==1.1.0
javapackages==1.0.0
Jinja2==2.10.3
jsonpatch==1.2
jsonpointer==1.9
jsonschema==2.3.0
kitchen==1.1.1
kubernetes===11.0.0-snapshot
lxml==3.2.1
M2Crypto==0.21.1
Markdown==2.4.1
MarkupSafe==1.1.1
nose==1.3.0
numpy==1.7.1
oauthlib==3.1.0
perf==0.1
pexpect==2.3
Pillow==2.0.0
policycoreutils-default-encoding==0.1
prettytable==0.7.2
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycrypto==2.6.1
pycurl==7.19.0
Pygments==1.4
pygobject==3.14.0
pygpgme==0.3
pyliblzma==0.5.3
pyparsing==1.5.6
python-ldap==2.4.15
pyudev==0.15
pyxattr==0.5.1
PyYAML==3.12
rsa==4.0
seobject==0.1
sepolicy==1.1
slip==0.4.0
slip.dbus==0.4.0
urlgrabber==3.10
Werkzeug==0.16.0
yum-metadata-parser==1.1.4

[root@VM_centos /data/code]# pip install kubernetes
Requirement already satisfied (use --upgrade to upgrade): kubernetes in /usr/lib/python2.7/site-packages/kubernetes-11.0.0_snapshot-py2.7.egg
Requirement already satisfied (use --upgrade to upgrade): certifi>=14.05.14 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.5.3 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): setuptools>=21.0.0 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.12 in /usr/lib64/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): google-auth>=1.0.1 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): requests in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): requests-oauthlib in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): urllib3>=1.24.2 in /usr/lib/python2.7/site-packages/urllib3-1.25.7-py2.7.egg (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): ipaddress>=1.0.17 in /usr/lib/python2.7/site-packages (from kubernetes)
Requirement already satisfied (use --upgrade to upgrade): rsa<4.1,>=3.1.4 in /usr/lib/python2.7/site-packages (from google-auth>=1.0.1->kubernetes)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules>=0.2.1 in /usr/lib/python2.7/site-packages (from google-auth>=1.0.1->kubernetes)
Requirement already satisfied (use --upgrade to upgrade): cachetools<5.0,>=2.0.0 in /usr/lib/python2.7/site-packages (from google-auth>=1.0.1->kubernetes)
Requirement already satisfied (use --upgrade to upgrade): oauthlib>=3.0.0 in /usr/lib/python2.7/site-packages (from requests-oauthlib->kubernetes)
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.3 in /usr/lib/python2.7/site-packages (from rsa<4.1,>=3.1.4->google-auth>=1.0.1->kubernetes)

[ikogan]

I'm having this issue on Fedora 31. The CA is in my system trust anchors, Chrome likes it, kubectl likes it. I've tried it with the certificate-authority-data in my kubeconfig and not in my kubeconfig. In both cases, kubectl likes it but the Python module does not:

OpenSSL version is OpenSSL 1.1.1d FIPS 10 Sep 2019
Python version is 3.7.5.

# pip install kubernetes
Requirement already satisfied: kubernetes in /usr/local/lib/python3.7/site-packages (10.0.1)
Requirement already satisfied: six>=1.9.0 in /usr/lib/python3.7/site-packages (from kubernetes) (1.12.0)
Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /usr/local/lib/python3.7/site-packages (from kubernetes) (0.57.0)
Requirement already satisfied: requests in /usr/lib/python3.7/site-packages (from kubernetes) (2.22.0)
Requirement already satisfied: urllib3>=1.24.2 in /usr/lib/python3.7/site-packages (from kubernetes) (1.25.7)
Requirement already satisfied: google-auth>=1.0.1 in /usr/local/lib/python3.7/site-packages (from kubernetes) (1.10.0)
Requirement already satisfied: pyyaml>=3.12 in /usr/lib64/python3.7/site-packages (from kubernetes) (5.1.2)
Requirement already satisfied: requests-oauthlib in /usr/local/lib/python3.7/site-packages (from kubernetes) (1.3.0)
Requirement already satisfied: certifi>=14.05.14 in /usr/local/lib/python3.7/site-packages (from kubernetes) (2019.9.11)
Requirement already satisfied: setuptools>=21.0.0 in /usr/lib/python3.7/site-packages (from kubernetes) (41.2.0)
Requirement already satisfied: python-dateutil>=2.5.3 in /usr/lib/python3.7/site-packages (from kubernetes) (2.8.0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.7/site-packages (from requests->kubernetes) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python3.7/site-packages (from requests->kubernetes) (2.8)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/site-packages (from google-auth>=1.0.1->kubernetes) (0.2.8)
Requirement already satisfied: rsa<4.1,>=3.1.4 in /usr/local/lib/python3.7/site-packages (from google-auth>=1.0.1->kubernetes) (4.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/site-packages (from google-auth>=1.0.1->kubernetes) (4.0.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/site-packages (from requests-oauthlib->kubernetes) (3.1.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes) (0.4.8)

CONNECTED(00000003)
depth=1 O = IPA.MYDOMAIN.ORG, CN = Certificate Authority
verify return:1
depth=0 O = IPA.MYDOMAIN.ORG, CN = rancher.ipa.mydomain.org
verify return:1
---
Certificate chain
 0 s:O = IPA.MYDOMAIN.ORG, CN = rancher.ipa.mydomain.org
   i:O = IPA.MYDOMAIN.ORG, CN = Certificate Authority
---
Server certificate
                                                         SNIP
-----END CERTIFICATE-----
subject=O = IPA.MYDOMAIN.ORG, CN = rancher.ipa.mydomain.org

issuer=O = IPA.MYDOMAIN.ORG, CN = Certificate Authority

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1846 bytes and written 431 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 45207113145BB2DF5F37383E2ECE42BE98ABD14F39BF10A55BC3E9E8B0853E21
    Session-ID-ctx: 
    Master-Key: 4726C53457BD224D3846E522C52864AB0C4C22B2589D84A3FE5849C170CEFE91315C1FD2201B6A90ED8DB0D8071B07D8
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - 25 59 89 e3 b9 f2 1c 33-4e 28 3d 10 8b 52 0c 9d   %Y.....3N(=..R..
    0010 - cd a3 9c 82 1f 8e 40 f1-ea 17 35 92 a7 88 63 53   ......@...5...cS
    0020 - 83 d9 13 f8 78 39 ab 61-d6 58 3e c2 92 0b 9a 45   ....x9.a.X>....E
    0030 - c7 9a b8 13 f7 16 40 f7-ef c5 39 9a e9 eb 3e 4b   ......@...9...>K
    0040 - e5 df 31 db 99 e7 8c a7-07 d4 48 47 a6 38 bc f8   ..1.......HG.8..
    0050 - f5 cc 4d 10 98 99 9d 1f-86 ea b2 a2 00 e8 e5 2a   ..M............*
    0060 - f0 fe 0f 24 6c 7a 30 21-fe c4 0d 8b 14 d5 fc 90   ...$lz0!........
    0070 - 77 3d 18 7a 4a 76 dd d7-                          w=.zJv..

    Start Time: 1578624966
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

depth=1 O = IPA.MYDOMAIN.ORG, CN = Certificate Authority
verify return:1
depth=0 O = IPA.MYDOMAIN.ORG, CN = rancher.ipa.mydomain.org
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 46 (0x2e)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = IPA.MYDOMAIN.ORG, CN = Certificate Authority
        Validity
            Not Before: Jun 18 18:31:53 2019 GMT
            Not After : Jun 18 18:31:53 2021 GMT
        Subject: O = IPA.MYDOMAIN.ORG, CN = rancher.ipa.mydomain.org
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                                    SNIP
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                keyid:7 SNIP

            Authority Information Access: 
                OCSP - URI:http://ipa-ca.ipa.mydomain.org/ca/ocsp

            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://ipa-ca.ipa.mydomain.org/ipa/crl/MasterCRL.bin
                CRL Issuer:
                  DirName:O = ipaca, CN = Certificate Authority

            X509v3 Subject Key Identifier: 
                SNIP
            X509v3 Subject Alternative Name: 
                othername:<unsupported>, othername:<unsupported>, DNS:rancher.ipa.mydomain.org
    Signature Algorithm: sha256WithRSAEncryption
         SNIP

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[degerrit]

Had the same issue here, but found a solution for me - which may help some.

Previously, even though curl worked fine for me - python 3.5 & client kubernetes==11.0.0 threw up an SSL: CERTIFICATE_VERIFY_FAILED error when using the exact same bearer token and CA file (which in my case is simply the public, Rancher self-signed, Kube API server cert itself).

SSL connection worked for curl:
# curl --noproxy chq20 --url https://chq20:6443/healthz --cacert ./check_qa.pem --header 'Authorization: Bearer deadbeef' --max-time 1.0

But SSL connection failed as above, using the example code in python/examples/remote_cluster.py and the same token/CA file as worked above.

(I had also checked that python is connecting to the correct host, and not our corporate proxy -which is another common problem here- using strace -e trace=network python3 : connection destination looks good)

In the end, I found openssl s_client -connect chq20:6443 -CAfile ./check_qa.pem indicated a few errors (unlike @ikogan, thanks for the tip though):

CONNECTED(00000003)
depth=0 CN = kube-apiserver
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = kube-apiserver
verify error:num=21:unable to verify the first certificate
verify return:1

So in my case the problem was using an unexpected CA file (and assuming all ssl client implementations behave the same!), and therefore an inability for more strict SSL clients to check the complete trust chain (which the API server cert does refer to).

I fetched the right Rancher CA from a Rancher container in the right cluster /etc/kubernetes/ssl/kube-ca.pem and am now using that instead.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thomashoell]

We're having the same issue. Unless the whole trust chain is in the kubeconfig's certificate-authority-data, the module will throw SSL: CERTIFICATE_VERIFY_FAILED.
This is not the expected behaviour.