[X509: CERT_ALREADY_IN_HASH_TABLE] cert already in hash table (_ssl.c:2635)

[ixuuux]

Using proxy through httpx.AsyncClient, when concurrent (the same proxy) will occasionally cause ssl.SSLError: [X509: CERT_ALREADY_IN_HASH_TABLE] cert already in hash table (_ssl.c:2635). The target website is https.
But httpx.Client No problem.

It feels like sharing the same variable (hash table) in the same thread

proxy use examples:

proxies = 'http://...'  # note: http proxy

async httpx.AsyncClient(proxies=proxies) as client:
    # The target website is `https`.
    # post upload file
    resp = await client.post('https://...', files=...)

system：CentOS Linux release 7.7.1908 (Core)
Python 3.8
httpx info:

httpx==0.21.1
  - certifi [required: Any, installed: 2021.10.8]
  - charset-normalizer [required: Any, installed: 2.0.9]
  - httpcore [required: >=0.14.0,<0.15.0, installed: 0.14.3]
    - anyio [required: ==3.*, installed: 3.4.0]
      - idna [required: >=2.8, installed: 3.3]
      - sniffio [required: >=1.1, installed: 1.2.0]
    - certifi [required: Any, installed: 2021.10.8]
    - h11 [required: >=0.11,<0.13, installed: 0.12.0]
    - sniffio [required: ==1.*, installed: 1.2.0]
  - rfc3986 [required: >=1.3,<2, installed: 1.5.0]
  - sniffio [required: Any, installed: 1.2.0]

[Brightcells]

I also encountered this problem and investigated it.

• https://bugs.ruby-lang.org/issues/11033

• ruby/openssl@v2.0.3...v2.0.4

  

• https://github.com/openssl/openssl/pull/2830/files

  Before 1.1.1 return 0, Since 1.1.1 return 1

  

• https://github.com/python/cpython/blob/main/Modules/_ssl.c#L4018

  

The CPython SSL module calls the OpenSSL function X509_STORE_ add_cert. When X509_R_CERT_ALREADY_IN_HASH_TABLE appears, before OpenSSL 1.1.1, X509_STORE_ add_cert returns 0 to indicate failure, and since OpenSSL 1.1.1 returns 1 to indicate success.

Solution

• Upgrade OpenSSL to version 1.1.1 or beyond
• Recompile Python with upgraded OpenSSL
  • Check the OpenSSL version used in Python

    Python 3.7.2 (default, May 29 2020, 10:57:47)
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import ssl
    >>> ssl.OPENSSL_VERSION
    'OpenSSL 1.0.2k-fips  26 Jan 2017'
    >>> ssl.OPENSSL_VERSION_INFO
    (1, 0, 2, 11, 15)
    >>>