Numerous test failures with libressl

[qolii]

I observe test failures when building against libressl(2.7.4).

I don't know anything about this codebase, but looking briefly at the error for, say, test_set_session_id_fail, I think the test is expecting a certain specific error string, with libressl is not exactly providing, despite it appearing to fail in the correct way.

Are these errors mistakes in the tests, or do you consider them to be mistakes in libressl?

My full test output follows:

running install tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.15, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
OpenSSL: LibreSSL 2.7.4
cryptography: 2.3
rootdir: /build/pyOpenSSL-18.0.0, inifile: setup.cfg
plugins: flaky-3.1.0
collected 496 items

tests/test_crypto.py ................................................... [ 10%]
............................FF.......................................... [ 24%]
........................................................................ [ 39%]
............................F..................................          [ 52%]
tests/test_debug.py .                                                    [ 52%]
tests/test_rand.py ....                                                  [ 53%]
tests/test_ssl.py ..........F........................................... [ 63%]
.............FFF...........................FFFF......................... [ 78%]
...............s..................................................F..... [ 92%]
.................................                                        [ 99%]
tests/test_tsafe.py .                                                    [ 99%]
tests/test_util.py .                                                     [100%]

=================================== FAILURES ===================================
_________________________ TestX509.test_set_notBefore __________________________

self = <tests.test_crypto.TestX509 instance at 0x7fffe8c0f7a0>

    def test_set_notBefore(self):
        """
            `X509.set_notBefore` takes a string in the format of an
            ASN1 GENERALIZEDTIME and sets the beginning of the certificate's
            validity period to it.
            """
>       self._setBoundTest("Before")

tests/test_crypto.py:1601:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_crypto.py:1573: in _setBoundTest
    set(when)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/crypto.py:1391: in set_notBefore
    return self._set_boundary_time(_lib.X509_get_notBefore, when)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/crypto.py:1378: in _set_boundary_time
    return _set_asn1_time(which(self._x509), when)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

boundary = <cdata 'struct asn1_string_st *' 0x976fa0>
when = '20040203040506+0530'

    def _set_asn1_time(boundary, when):
        """
        The the time value of an ASN1 time object.

        @param boundary: An ASN1_TIME pointer (or an object safely
            castable to that type) which will have its value set.
        @param when: A string representation of the desired time value.

        @raise TypeError: If C{when} is not a L{bytes} string.
        @raise ValueError: If C{when} does not represent a time in the required
            format.
        @raise RuntimeError: If the time value cannot be set for some other
            (unspecified) reason.
        """
        if not isinstance(when, bytes):
            raise TypeError("when must be a byte string")

        set_result = _lib.ASN1_TIME_set_string(boundary, when)
        if set_result == 0:
>           raise ValueError("Invalid string")
E           ValueError: Invalid string

/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/crypto.py:161: ValueError
__________________________ TestX509.test_set_notAfter __________________________

self = <tests.test_crypto.TestX509 instance at 0x7fffe8c0fc20>

    def test_set_notAfter(self):
        """
            `X509.set_notAfter` takes a string in the format of an ASN1
            GENERALIZEDTIME and sets the end of the certificate's validity period
            to it.
            """
>       self._setBoundTest("After")

tests/test_crypto.py:1609:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_crypto.py:1573: in _setBoundTest
    set(when)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/crypto.py:1417: in set_notAfter
    return self._set_boundary_time(_lib.X509_get_notAfter, when)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/crypto.py:1378: in _set_boundary_time
    return _set_asn1_time(which(self._x509), when)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

boundary = <cdata 'struct asn1_string_st *' 0x9ce240>
when = '20040203040506+0530'

    def _set_asn1_time(boundary, when):
        """
        The the time value of an ASN1 time object.

        @param boundary: An ASN1_TIME pointer (or an object safely
            castable to that type) which will have its value set.
        @param when: A string representation of the desired time value.

        @raise TypeError: If C{when} is not a L{bytes} string.
        @raise ValueError: If C{when} does not represent a time in the required
            format.
        @raise RuntimeError: If the time value cannot be set for some other
            (unspecified) reason.
        """
        if not isinstance(when, bytes):
            raise TypeError("when must be a byte string")

        set_result = _lib.ASN1_TIME_set_string(boundary, when)
        if set_result == 0:
>           raise ValueError("Invalid string")
E           ValueError: Invalid string

/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/crypto.py:161: ValueError
_______________________ TestCRL.test_verify_with_revoked _______________________

self = <tests.test_crypto.TestCRL object at 0x7fffea2d5350>

    def test_verify_with_revoked(self):
        """
            `verify_certificate` raises error when an intermediate certificate is
            revoked.
            """
        store = X509Store()
        store.add_cert(self.root_cert)
        store.add_cert(self.intermediate_cert)
        root_crl = self._make_test_crl(
            self.root_cert, self.root_key, certs=[self.intermediate_cert])
        intermediate_crl = self._make_test_crl(
            self.intermediate_cert, self.intermediate_key, certs=[])
        store.add_crl(root_crl)
        store.add_crl(intermediate_crl)
        store.set_flags(
            X509StoreFlags.CRL_CHECK | X509StoreFlags.CRL_CHECK_ALL)
        store_ctx = X509StoreContext(store, self.intermediate_server_cert)
        with pytest.raises(X509StoreContextError) as err:
            store_ctx.verify_certificate()
>       assert err.value.args[0][2] == 'certificate revoked'
E       AssertionError: assert 'format error...tUpdate field' == 'certificate revoked'
E         - format error in CRL's lastUpdate field
E         + certificate revoked

tests/test_crypto.py:3432: AssertionError
_____________________ TestContext.test_set_session_id_fail _____________________

self = <tests.test_ssl.TestContext object at 0x7fffe8910510>
context = <OpenSSL.SSL.Context object at 0x7fffe8910250>

    def test_set_session_id_fail(self, context):
        """
            `Context.set_session_id` errors are propagated.
            """
        with pytest.raises(Error) as e:
            context.set_session_id(b"abc" * 1000)

>       assert [
            ("SSL routines",
             "SSL_CTX_set_session_id_context",
             "ssl session id context too long")
        ] == e.value.args[0]
E       AssertionError: assert [('SSL routin...xt too long')] == [('SSL routine...xt too long')]
E         At index 0 diff: ('SSL routines', 'SSL_CTX_set_session_id_context', 'ssl session id context too long') != ('SSL routines', '(UNKNOWN)SSL_internal', 'ssl session id context too long')
E         Use -v to get the full diff

tests/test_ssl.py:465: AssertionError
____________________ TestContext.test_add_extra_chain_cert _____________________

self = <tests.test_ssl.TestContext object at 0x7fffe891b850>
tmpdir = local('/build/pytest-of-nixbld/pytest-0/test_add_extra_chain_cert0')

    def test_add_extra_chain_cert(self, tmpdir):
        """
            `Context.add_extra_chain_cert` accepts an `X509`
            instance to add to the certificate chain.

            See `_create_certificate_chain` for the details of the
            certificate chain tested.

            The chain is tested by starting a server with scert and connecting
            to it with a client which trusts cacert and requires verification to
            succeed.
            """
        chain = _create_certificate_chain()
        [(cakey, cacert), (ikey, icert), (skey, scert)] = chain

        # Dump the CA certificate to a file because that's the only way to load
        # it as a trusted CA in the client context.
        for cert, name in [(cacert, 'ca.pem'),
                           (icert, 'i.pem'),
                           (scert, 's.pem')]:
            with tmpdir.join(name).open('w') as f:
                f.write(dump_certificate(FILETYPE_PEM, cert).decode('ascii'))

        for key, name in [(cakey, 'ca.key'),
                          (ikey, 'i.key'),
                          (skey, 's.key')]:
            with tmpdir.join(name).open('w') as f:
                f.write(dump_privatekey(FILETYPE_PEM, key).decode('ascii'))

        # Create the server context
        serverContext = Context(TLSv1_METHOD)
        serverContext.use_privatekey(skey)
        serverContext.use_certificate(scert)
        # The client already has cacert, we only need to give them icert.
        serverContext.add_extra_chain_cert(icert)

        # Create the client
        clientContext = Context(TLSv1_METHOD)
        clientContext.set_verify(
            VERIFY_PEER | VERIFY_FAIL_IF_NO_PEER_CERT, verify_cb)
        clientContext.load_verify_locations(str(tmpdir.join("ca.pem")))

        # Try it out.
>       self._handshake_test(serverContext, clientContext)

tests/test_ssl.py:1370:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_ssl.py:1248: in _handshake_test
    s.do_handshake()
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/SSL.py:1907: in do_handshake
    self._raise_ssl_error(self._ssl, result)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/SSL.py:1639: in _raise_ssl_error
    _raise_current_error()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

exception_type = <class 'OpenSSL.SSL.Error'>

    def exception_from_error_queue(exception_type):
        """
        Convert an OpenSSL library failure into a Python exception.

        When a call to the native OpenSSL library fails, this is usually signalled
        by the return value, and an error code is stored in an error queue
        associated with the current thread. The err library provides functions to
        obtain these error codes and textual error messages.
        """
        errors = []

        while True:
            error = lib.ERR_get_error()
            if error == 0:
                break
            errors.append((
                text(lib.ERR_lib_error_string(error)),
                text(lib.ERR_func_error_string(error)),
                text(lib.ERR_reason_error_string(error))))

>       raise exception_type(errors)
E       Error: [('SSL routines', 'CONNECT_CR_CERT', 'certificate verify failed')]

/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/_util.py:54: Error
______________ TestContext.test_use_certificate_chain_file_bytes _______________

self = <tests.test_ssl.TestContext object at 0x7fffe891b4d0>
tmpfile = '/build/pytest-of-nixbld/pytest-0/tmp87s4A3'

    def test_use_certificate_chain_file_bytes(self, tmpfile):
        """
            ``Context.use_certificate_chain_file`` accepts the name of a file (as
            an instance of ``bytes``) to specify additional certificates to use to
            construct and verify a trust chain.
            """
        self._use_certificate_chain_file_test(
>           tmpfile + NON_ASCII.encode(getfilesystemencoding())
        )

tests/test_ssl.py:1417:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_ssl.py:1408: in _use_certificate_chain_file_test
    self._handshake_test(serverContext, clientContext)
tests/test_ssl.py:1248: in _handshake_test
    s.do_handshake()
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/SSL.py:1907: in do_handshake
    self._raise_ssl_error(self._ssl, result)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/SSL.py:1639: in _raise_ssl_error
    _raise_current_error()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

exception_type = <class 'OpenSSL.SSL.Error'>

    def exception_from_error_queue(exception_type):
        """
        Convert an OpenSSL library failure into a Python exception.

        When a call to the native OpenSSL library fails, this is usually signalled
        by the return value, and an error code is stored in an error queue
        associated with the current thread. The err library provides functions to
        obtain these error codes and textual error messages.
        """
        errors = []

        while True:
            error = lib.ERR_get_error()
            if error == 0:
                break
            errors.append((
                text(lib.ERR_lib_error_string(error)),
                text(lib.ERR_func_error_string(error)),
                text(lib.ERR_reason_error_string(error))))

>       raise exception_type(errors)
E       Error: [('SSL routines', 'CONNECT_CR_CERT', 'certificate verify failed')]

/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/_util.py:54: Error
_____________ TestContext.test_use_certificate_chain_file_unicode ______________

self = <tests.test_ssl.TestContext object at 0x7fffe891bc90>
tmpfile = '/build/pytest-of-nixbld/pytest-0/tmpKqbnDH'

    def test_use_certificate_chain_file_unicode(self, tmpfile):
        """
            ``Context.use_certificate_chain_file`` accepts the name of a file (as
            an instance of ``unicode``) to specify additional certificates to use
            to construct and verify a trust chain.
            """
        self._use_certificate_chain_file_test(
>           tmpfile.decode(getfilesystemencoding()) + NON_ASCII
        )

tests/test_ssl.py:1427:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_ssl.py:1408: in _use_certificate_chain_file_test
    self._handshake_test(serverContext, clientContext)
tests/test_ssl.py:1248: in _handshake_test
    s.do_handshake()
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/SSL.py:1907: in do_handshake
    self._raise_ssl_error(self._ssl, result)
/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/SSL.py:1639: in _raise_ssl_error
    _raise_current_error()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

exception_type = <class 'OpenSSL.SSL.Error'>

    def exception_from_error_queue(exception_type):
        """
        Convert an OpenSSL library failure into a Python exception.

        When a call to the native OpenSSL library fails, this is usually signalled
        by the return value, and an error code is stored in an error queue
        associated with the current thread. The err library provides functions to
        obtain these error codes and textual error messages.
        """
        errors = []

        while True:
            error = lib.ERR_get_error()
            if error == 0:
                break
            errors.append((
                text(lib.ERR_lib_error_string(error)),
                text(lib.ERR_func_error_string(error)),
                text(lib.ERR_reason_error_string(error))))

>       raise exception_type(errors)
E       Error: [('SSL routines', 'CONNECT_CR_CERT', 'certificate verify failed')]

/nix/store/m20m5p792gcz0w8lwcad0hly1655q5rw-python2.7-pyOpenSSL-18.0.0/lib/python2.7/site-packages/OpenSSL/_util.py:54: Error
__________________ TestNextProtoNegotiation.test_npn_success ___________________

self = <tests.test_ssl.TestNextProtoNegotiation object at 0x7fffe9116c90>

    def test_npn_success(self):
        """
            Tests that clients and servers that agree on the negotiated next
            protocol can correct establish a connection, and that the agreed
            protocol is reported by the connections.
            """
        advertise_args = []
        select_args = []

        def advertise(conn):
            advertise_args.append((conn,))
            return [b'http/1.1', b'spdy/2']

        def select(conn, options):
            select_args.append((conn, options))
            return b'spdy/2'

        server_context = Context(TLSv1_METHOD)
        server_context.set_npn_advertise_callback(advertise)

        client_context = Context(TLSv1_METHOD)
        client_context.set_npn_select_callback(select)

        # Necessary to actually accept the connection
        server_context.use_privatekey(
            load_privatekey(FILETYPE_PEM, server_key_pem))
        server_context.use_certificate(
            load_certificate(FILETYPE_PEM, server_cert_pem))

        # Do a little connection to trigger the logic
        server = Connection(server_context, None)
        server.set_accept_state()

        client = Connection(client_context, None)
        client.set_connect_state()

        interact_in_memory(server, client)

>       assert advertise_args == [(server,)]
E       assert [] == [(<OpenSSL.SSL.Connection object at 0x7fffe9116e50>,)]
E         Right contains more items, first extra item: (<OpenSSL.SSL.Connection object at 0x7fffe9116e50>,)
E         Use -v to get the full diff

tests/test_ssl.py:1771: AssertionError
________________ TestNextProtoNegotiation.test_npn_client_fail _________________

self = <tests.test_ssl.TestNextProtoNegotiation object at 0x7fffe90fe290>

    def test_npn_client_fail(self):
        """
            Tests that when clients and servers cannot agree on what protocol
            to use next that the TLS connection does not get established.
            """
        advertise_args = []
        select_args = []

        def advertise(conn):
            advertise_args.append((conn,))
            return [b'http/1.1', b'spdy/2']

        def select(conn, options):
            select_args.append((conn, options))
            return b''

        server_context = Context(TLSv1_METHOD)
        server_context.set_npn_advertise_callback(advertise)

        client_context = Context(TLSv1_METHOD)
        client_context.set_npn_select_callback(select)

        # Necessary to actually accept the connection
        server_context.use_privatekey(
            load_privatekey(FILETYPE_PEM, server_key_pem))
        server_context.use_certificate(
            load_certificate(FILETYPE_PEM, server_cert_pem))

        # Do a little connection to trigger the logic
        server = Connection(server_context, None)
        server.set_accept_state()

        client = Connection(client_context, None)
        client.set_connect_state()

        # If the client doesn't return anything, the connection will fail.
        with pytest.raises(Error):
>           interact_in_memory(server, client)
E           Failed: DID NOT RAISE <class 'OpenSSL.SSL.Error'>

tests/test_ssl.py:1814: Failed
________________ TestNextProtoNegotiation.test_npn_select_error ________________

self = <tests.test_ssl.TestNextProtoNegotiation object at 0x7fffe90fe6d0>

    def test_npn_select_error(self):
        """
            Test that we can handle exceptions in the select callback. If
            select fails it should be fatal to the connection.
            """
        advertise_args = []

        def advertise(conn):
            advertise_args.append((conn,))
            return [b'http/1.1', b'spdy/2']

        def select(conn, options):
            raise TypeError

        server_context = Context(TLSv1_METHOD)
        server_context.set_npn_advertise_callback(advertise)

        client_context = Context(TLSv1_METHOD)
        client_context.set_npn_select_callback(select)

        # Necessary to actually accept the connection
        server_context.use_privatekey(
            load_privatekey(FILETYPE_PEM, server_key_pem))
        server_context.use_certificate(
            load_certificate(FILETYPE_PEM, server_cert_pem))

        # Do a little connection to trigger the logic
        server = Connection(server_context, None)
        server.set_accept_state()

        client = Connection(client_context, None)
        client.set_connect_state()

        # If the callback throws an exception it should be raised here.
        with pytest.raises(TypeError):
>           interact_in_memory(server, client)
E           Failed: DID NOT RAISE <type 'exceptions.TypeError'>

tests/test_ssl.py:1854: Failed
______________ TestNextProtoNegotiation.test_npn_advertise_error _______________

self = <tests.test_ssl.TestNextProtoNegotiation object at 0x7fffe90fec90>

    def test_npn_advertise_error(self):
        """
            Test that we can handle exceptions in the advertise callback. If
            advertise fails no NPN is advertised to the client.
            """
        select_args = []

        def advertise(conn):
            raise TypeError

        def select(conn, options):  # pragma: nocover
            """
                Assert later that no args are actually appended.
                """
            select_args.append((conn, options))
            return b''

        server_context = Context(TLSv1_METHOD)
        server_context.set_npn_advertise_callback(advertise)

        client_context = Context(TLSv1_METHOD)
        client_context.set_npn_select_callback(select)

        # Necessary to actually accept the connection
        server_context.use_privatekey(
            load_privatekey(FILETYPE_PEM, server_key_pem))
        server_context.use_certificate(
            load_certificate(FILETYPE_PEM, server_cert_pem))

        # Do a little connection to trigger the logic
        server = Connection(server_context, None)
        server.set_accept_state()

        client = Connection(client_context, None)
        client.set_connect_state()

        # If the client doesn't return anything, the connection will fail.
        with pytest.raises(TypeError):
>           interact_in_memory(server, client)
E           Failed: DID NOT RAISE <type 'exceptions.TypeError'>

tests/test_ssl.py:1895: Failed
_____________________ TestConstants.test_op_no_compression _____________________

self = <tests.test_ssl.TestConstants object at 0x7fffe8cc1cd0>

    @pytest.mark.skipif(
        OP_NO_COMPRESSION is None,
        reason="OP_NO_COMPRESSION unavailable - OpenSSL version may be too old"
    )
    def test_op_no_compression(self):
        """
            The value of `OpenSSL.SSL.OP_NO_COMPRESSION` is 0x20000, the
            value of `SSL_OP_NO_COMPRESSION` defined by `openssl/ssl.h`.
            """
>       assert OP_NO_COMPRESSION == 0x20000
E       assert 0 == 131072

tests/test_ssl.py:3283: AssertionError
===Flaky Test Report===

test_gmtime_adj_notBefore passed 1 out of the required 1 times. Success!
test_gmtime_adj_notAfter passed 1 out of the required 1 times. Success!
test_export_text passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
============== 12 failed, 483 passed, 1 skipped in 10.76 seconds ===============

[davehayes]

This is an attempt to get to the bottom of this issue with synapse.

I managed to get pytest to run pyopenssl's tests on the installed version. This produced this test suite output. I'm on FreeBSD 12 with the 2021Q1 ports tree versions of all this software (pytest, flaky, pyopenssl, etc.)

Is there any chance of actually addressing these issues? The "certificate verify failed" is most pressing for me, but I'm sure some of these actually require fixes?

[tiran]

Semi-related question:
Since when does FreeBSD use LibreSSL? I was under the impression that FreeBSD uses OpenSSL in core and ports. Or is this a custom ports build with LibreSSL instead of OpenSSL?

[davehayes]

This is a custom ports build with LibreSSL of course.

[tiran]

Thanks for the information!

Over the years the paths of OpenSSL and LibreSSL have separated. Although LibreSSL initially started as a fork of OpenSSL 1.0.1g, it's no longer a drop-in replacement for OpenSSL. There several differences in API and even in behavior. It's become increasingly complicated and painful to write software that works with OpenSSL and LibreSSL. Several Linux distros (Alpine, Gentoo, Void) have given up and moved back to OpenSSL recently. It's unfortunate.

IMHO the best course of action is to work with OpenSSL and LibreSSL upstream development on a common API. If you can get both projects (and maybe even BoringSSL) to reconcile their APIs, then projects like PyOpenSSL can support both more easily.

[davehayes]

I am very familiar with the background of this, including the ongoing efforts (some of which you mention) by many well-meaning developers to passively require using "one SSL stack to rule them all" if you want anything to actually work. I do not believe this is a healthy policy myself, but that's a debate for another time.

So I provide this test result as non-zero incentive to try to be compatible with both LibreSSL and OpenSSL. That being said, I completely understand if you take no action and/or close this ticket. I would like it to be clearly understood that I do not wish to force any work on any other developer. :)

If it helps any, the certificate verification part of the failing tests (my major issue) was a problem with the FreeBSD pkg system and not PyOpenSSL.