On debian stretch with libssl1.1, synapse has trouble connecting with some home servers

[ptman]

I get messages like these in synapse log:

synapse.http.outbound - 239 - INFO - - {PUT-O-27084} [irc.snt.utwente.nl] Result: ResponseNeverReceived - Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]  

It can be reproduced with the attached script ssltest.py:

[Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]]                  

The theory is that twisted on debian stretch with libssl1.1 only offers one EC cipher in the handshake. Not my theory.

I realise this may not be the best place to track this issue, but I'm not sure what is, and I think users of synapse would like to know about it and how it proceeds.

[lordievader]

I ran into this issue as well.
Ultimately I solved the issue by changing distro's... not something every user wants-to/can do.

I did find out that on Debian (9) connecting to irc.snt.utwente.nl using OpenSSL's s_client (openssl s_client -connect matrix-irc.snt.utwente.nl:443) things worked fine. In the packet capture many elliptic curves were listed.

Using the ssltest.py script, as above, the only listed elliptic curve was scep256r1.

On my Gentoo host the ssltest.py script works fine and many elliptic curves were in the handshake.

The only real difference I could quickly see between the two hosts is the OpenSSL version, Debian 9 uses OpenSSL 1.1 whereas Gentoo uses OpenSSL 1.0.1.

[silkeh]

Because the server has a secp384r1 certificate, secp384r1 needs to be supported by the client.

I have tested the following (with my own ssltest.py):

Distribution	OpenSSL	Client	Works?	Curves
Debian Jessie	1.0.1t	Python 3.4.2	Yes	25
Debian Jessie	1.0.1t	s_client	Yes	25
Debian Stretch	1.1.0f	Python 3.5.3	No	1
Debian Stretch	1.1.0f	s_client	Yes	4
Arch Linux	1.1.0f	Python 3.6.1	Yes	4

Only secp256r1 is offered by the tested Python applications on Debian Stretch. When using openssl s_client on Debian Stretch ecdh_x25519, secp256r1, secp521r1, and secp384r1 are offered.

Edit: updated the client to reflect the Python versions and added test script.

[silkeh]

I believe the issue is fixed by python/cpython#395 (and related PRs). See the related issue for more information.

[ptman]

Thank you @silkeh for tracking it down. I filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868143

[ptman]

Btw, synapse uses python 2.7, not 3.x

[silkeh]

Correct, and both Python 2.7 and 3.x are affected by the same bug. The only Python release with the fix is 3.6.1.

[lordievader]

Yesterday Gentoo updated Openssl to 1.0.1k. This broke my homeserver with the above mentioned issues.
The ssl testing script of Silke is able to make a connection to https://matrix-irc.snt.utwente.nl with 13 available curves. However, if the same is done through the Twisted testing script, it fails with only 1 curve available.

There is a ticket for this issue within Twisted: https://twistedmatrix.com/trac/ticket/9210
The 'fix' mentioned in this ticket works for me, it seems.
Change _defaultCurveName = u"prime256v1" to _defaultCurveName = u"secp384r1" in twisted.internet._sslverify.py.

[ptman]

The debian bug was just updated. A patch for the problem is included in the version in sid/unstable. No idea how soon it will make it to an update in stable.

[richvdh]

some notes on this, largely for my own benefit, though I hope others will find them useful:

This appears to have been a bug in the OpenSSL 1.1 code in cPython - specifically https://bugs.python.org/issue29697. It's therefore nothing particularly specific to Debian, although Stretch is probably the first distribution to use OpenSSL 1.1 by default.

It is now fixed in cPython 2.7.14 (python/cpython@f1a696e), 3.5.4 (python/cpython@564ace8) and 3.6.1 (python/cpython@784ba7c).

Since Stretch is still currently using an unfixed Python 2.7.13 with OpenSSL 1.1, I would consider it fundamentally broken with respect to TLS support and therefore inadequate for hosting a Synapse server. @ptman 's bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868143 is the place this is being tracked.

The issue mentioned by @lordievader is, I suspect, somewhat different because (a) he mentions using OpenSSL 1.0.1k whereas this issue affects specifically OpenSSL 1.1.x; (b) this issue makes silkeh's ssltest fail whereas lordievader reports that it works ok. If you're still having problems please report a separate issue.

[kroeckx]

So I build a version of python2.7 for Debian with the patch applied and prepared this for a stable upload to Debian, and a simple test program shows that the supported groups that are send are changed, so I believe that I've fixed it in the python2.7 package for Debian.

However, on my server with the fixed version of python2.7 installed, after restarting synapse I still see it sending the P-256 curve as the only supported group and so the error doesn't go away, and I currently have no idea where it's coming from.

[lordievader]

This might be Twisted doing a wrong fallback. Have you tried the work around I posted in my last comment?

[kroeckx]

I've commented out that code, and it seems to work. I'll file some more bugs.

[kroeckx]

So looking at this, synapse isn't using the python code for ssl but twisted, and it's not the python bug that needs to be fixed but the twisted one. I've filed this bug in Debian for it: #881420

[kroeckx]

Just to clarify, synapse works when both python and twisted are fixed, both need to be fixed.

[kroeckx]

Now I'm confused again, the error has come back.

[kroeckx]

So looking at the real packets again, I see that with the broken version of python but the fixed version of twisted the packets are what they should be and that it works.

[ptman]

I see that the debian bug was closed, so a fixed package will be available soon. But is that enough?

[kroeckx]

A fixed python package does not change anything. You need a fixed twisted package.

[ptman]

Thank you for confirming. Does a twisted release with the fix already exist, or is patching necessary?

[richvdh]

The relevant twisted bug is https://twistedmatrix.com/trac/ticket/9210. As you can see, it's not yet fixed, though it does have a link to @kroeckx 's PR which patches it.

(Thanks for doing all this investigation btw @kroeckx )

[rkfg]

I have a similar issue with keytown.me. The script in the first comment now works fine for matrix-irc.snt.utwente.nl (after adding one more argument to cbResponse) but this one throws an error:

from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.web.http_headers import Headers

agent = Agent(reactor)

d = agent.request(
    'GET',
    'https://keytown.me',
    Headers({'User-Agent': ['Twisted Web Client Example']}),
    None)

def cbResponse(ignored, ignored2):
    print 'Response received'

def failure(failure_args):
    print failure_args

def stop(ignored):
    print ignored
    reactor.stop()

d.addCallback(cbResponse, failure).addBoth(stop)

def cbShutdown(ignored):
    reactor.stop()
d.addBoth(cbShutdown)

reactor.run()

The exact issue is that there are no previews for any link from that site. If this is not relevant I can post a new issue.

[ptman]

I believe the snt homeserver switched to rsa keys because of this issue.

[silkeh]

Yes, matrix-irc.snt.utwente.nl is using RSA keys now. https://keytown.me also seems to be using RSA, so I don't think this is related.

[richvdh]

nope, keytown.me also appears to reject connections because of the limited range of elliptic curves.

[Note that, aiui, RSA certificate keys and ECDH key exchange are not mutually exclusive, and indeed keytown.me and many other servers use them together]

[richvdh]

The underlying bug here, https://twistedmatrix.com/trac/ticket/9210, is fixed, as of Twisted 18.4.0.

However, debian stretch still seems to have twisted 16.2.

Can any stretch users who have installed from the packages comment on whether this is still a problem?

[rkfg]

For the record, I upgraded Twisted to 18.4.0 long ago and it seems this issue is fixed as of now. Current Debian stable has 16.6.0, not 16.2, but I don't want to downgrade just to test it.

[DirtyJerz]

It seems stretch stable is still 2.7.13-2+deb9u2. Is the best recourse to build python 2.7.15 from source and update-alternatives?

[richvdh]

@DirtyJerz: the problem is an incompatibility between between libssl1.1 and Twisted < 18.4. There is no need to upgrade python.

[richvdh]

Now that we have backported packages of Twisted 18.4 for stretch, this bug should be fixed for everyone.

[DirtyJerz]

I was on 2.7.13 (stretch stable) and was referring to the bugs that you were mentioning here. Build from source resolved the issue. It was just you're normal "let me google this error" that lead me here. It was probably a different dependency. Apologies if i was necromancing.