Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Synapse uses TLS1.0 for smtp which is rejected by some mail servers #6211

Closed
gjabell opened this issue Oct 17, 2019 · 46 comments
Closed

Synapse uses TLS1.0 for smtp which is rejected by some mail servers #6211

gjabell opened this issue Oct 17, 2019 · 46 comments
Labels
z-bug (Deprecated Label) z-p2 (Deprecated Label) Z-Upstream-Bug This issue requires a fix in an upstream dependency.

Comments

@gjabell
Copy link

gjabell commented Oct 17, 2019

Description

Requesting a password reset from a brand-new Synapse installation returns a 500 error, with the error twisted.mail._except.SMTPConnectError: Unable to connect to server.

Steps to reproduce

  • On a vanilla homeserver, add the following configuration to homeserver.yaml:
email:
  enable_notifs: false
  smtp_host: [hostname or ip]
  smtp_port: 587
  smtp_user: [username]
  smtp_pass: [password]
  notif_from: "Your friendly %(app)s Home Server <[email]>"
  app_name: Matrix
  • Restart synapse to apply changes
  • Using riot, change the homeserver url and then select "Set a new password"
  • Enter the valid email address and a new password
  • Select "Send Reset Email"

After the last step, the server will respond with a 500 error, and the following will be displayed in synapse's log:

Oct 17 15:19:00 [hostname] synapse[11936]: synapse.handlers.identity: [POST-49] Error sending threepid validation email to [email]
                                                Traceback (most recent call last):
                                                  File "/nix/store/1al2bnj8f2y66jxmzhi00aw3a7wp1jgw-matrix-synapse-1.4.0/lib/python3.7/site-packages/synapse/handlers/identity.py", line 347, in send_threepid_validation
                                                    yield send_email_func(email_address, token, client_secret, session_id)
                                                twisted.mail._except.SMTPConnectError: Unable to connect to server.

And this is displayed in the postfix log of the receiving server:

Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: connect from unknown[ip]
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: SSL_accept error from unknown[ip]: -1
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: warning: TLS library problem: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:ssl/statem/statem_srvr.c:1661:
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: lost connection after STARTTLS from unknown[ip]
Oct 17 15:19:00 [hostname] postfix/smtpd[2546]: disconnect from unknown[ip] ehlo=1 starttls=0/1 commands=1/2

I've tested this configuration with both require_transport_security: false and require_transport_security: true. Also worth mentioning that the username / password are correct, as logging into the mail server from a mail program and sending a test email from there works fine.

Version information

New personal homeserver running synapse.

  • Version: 1.4.0

  • Install method: Package Manager

  • Platform: NixOS running on Hetzner Cloud VM for both Matrix and mail server
@gjabell
Copy link
Author

gjabell commented Oct 17, 2019

After some more testing, this appears to be an issue with Synapse (or rather Twisted) not using an up-to-date TLS protocol when communicating with the mail server. My server is set to not accept TLSv1 or SSLv2 & 3, which explains the unsupported protocol:ssl error above. Is there a way to force Synapse/Twisted to use newer TLS protocols?

@madpsy
Copy link

madpsy commented Oct 18, 2019

I second this - seeing exactly the same issue.

Interestingly, require_transport_security doesn't seem to do anything. When set to true or false it still attemps STARTTLS which is obviously wrong.

@fadenb
Copy link

fadenb commented Oct 20, 2019

I am experiencing the exact same issue.

@neilisfragile neilisfragile added z-bug (Deprecated Label) z-p2 (Deprecated Label) labels Oct 20, 2019
@elasticroentgen
Copy link

same here.

@ghost
Copy link

ghost commented Nov 23, 2019

Yes, I have the same behavior here as well.

@sbiberhofer
Copy link

While looking at this bug w/ @henry-nicolas yesterday, we did some testing which might be helpful in pinpointing the bug:

  • It seems like synapse's support of STARTTLS is hardcoded to TLSv1.0. Regardless of federation_client_minimum_tls_version, the TLS Client Hello following STARTTLS in an smtp connection is using TLSv1.0 and is thus rejected by any mailserver not supporting such an old TLS version.
  • As @madpsy already noticed, twisted is using STARTTLS in an opportunistic manner by default. The only thing that require_transport_security controls is whether or not STARTTLS is a hard requirement. This isn't "wrong" per se but it would be nice to have more control over this behaviour. Additionally, synapse doesn't support smtps (i.e. smtp wrapped in TLS), which should probably be at least mentioned in the documentation.
  • I'm not sure this is actually synapse's fault. It might well be a bug in twisted since - as far as I can tell - synapse simply relies on twisted's sendmail() functionality to "do the right thing".

@phtmgt
Copy link

phtmgt commented Nov 27, 2019

+1

@richvdh richvdh changed the title Synapse fails to send password reset emails to external SMTP server Synapse uses TLS1.0 for smtp which is rejected by some mail servers Nov 27, 2019
@vmario89
Copy link

vmario89 commented Nov 27, 2019

same problem here. I am running Plesk which controls Postfix/Dovecot. The systems are configured to only use TLS 1.2 or higher

the log output:

2019-11-27 12:55:05,982 - synapse.handlers.identity - 357 - ERROR - POST-1239- Error sending threepid validation email to
Traceback (most recent call last):
File "/opt/venvs/matrix-synapse/lib/python3.5/site-packages/synapse/handlers/identity.py", line 354, in send_threepid_validation
yield send_email_func(email_address, token, client_secret, session_id)
twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.

hopefully it belongs to this issue. Some releases before everything worked fine

@vmario89
Copy link

i was able to validate that its a TLS problem. i re-enabled old TLS 1.0 and 1.1 to test:
plesk bin server_pref -u -ssl-protocols 'TLSv1 TLSv1.1 TLSv1.2'

that makes it work directly after enabling TLS 1.0

@a-fellerer
Copy link

a-fellerer commented Dec 2, 2019

Same here on ubuntu server 18.04. Please provide a solution.

2019-12-02 10:30:12,234 - synapse.http.site - 203 - WARNING - POST-14- Error processing request <XForwardedForRequest at 0x7f76febbcfd0 method='POST' uri='/_matrix/client/r0/account/3pid/email/requestToken' clientproto='HTTP/1.1' site=8008>: <class 'twisted.internet.error.ConnectionDone'> Connection was closed cleanly.
2019-12-02 10:30:12,325 - twisted - 172 - INFO - - SMTP Client retrying server. Retry: 5

@MisterAlainDev
Copy link

MisterAlainDev commented Dec 2, 2019

the problem was that twisted mail smtp wrapper does not call SSL connect at all (inside synapse virtual env > /twisted/mail/smtp.py)
in this python file

  1. add this import from twisted.internet.ssl import optionsForClientTLS
  2. inside the class ESMTPSender(SenderMixin, ESMTPClient) > fix the minimal tls version (for example : context.method = ssl.SSL.TLSv1_1_METHOD)
  3. inside the def sendmail > replace the line connectTCP with

if (requireTransportSecurity):
    contextFactory = optionsForClientTLS(smtphost)
    connector = reactor.connectSSL(smtphost, port, factory, contextFactory)

else:  
    connector = reactor.connectTCP(smtphost, port, factory)

@schwukas
Copy link

schwukas commented Dec 3, 2019

It seems that @sbiberhofer is right with the hardcoded part.
I noticed that in line 2038 the smtp module has the context method overwritten to a lower TLS version.
The context factory which is called by the smtp part actually provides the more modern TLS versions 2 and 3. However it states to also allow for TLSv1, which doesn't explain why the below works.

At any rate, commenting line 2038 in the smtp module where the method gets overwritten to TLSv1 worked for me.
I don't know if this has other implications. It might be there for a reason. I will try to find someone in the #twisted matrix room to ask.

I have created a ticket on twisted's issue tracker, that you can find here.

@kaiyou
Copy link
Contributor

kaiyou commented Dec 20, 2019

For what is worth, here is a temporary fix on our side, thank you so much for pinpointing the actual culprit in Twisted sources: https://forge.tedomum.net/tedomum/synapse/blob/aac748e3720e001a2fc9e42ef2add49ce815443e/docker/Dockerfile#L57

If required, I can PR this ugly jewel.

@babolivier
Copy link
Contributor

Since I don't see it mentioned here, here's the ticket on Twisted's side for tracking: https://twistedmatrix.com/trac/ticket/9740

@mmilata
Copy link
Contributor

mmilata commented Mar 3, 2020

Submitted a patch to Twisted. If it's not accepted we can make Synapse use ESMTPSenderFactory (with explicit ClientContextFactory context returned by optionsForClientTLS) directly instead of the higher-level function sendmail() (with hardcoded TLSv1.0 ClientContextFactory).

@n3m3s1s
Copy link

n3m3s1s commented Apr 6, 2020

The problem seems to be fixed. Could you update the dependency or is it more complicated to get the fix into 1.12.4 maybe?

@clokep
Copy link
Member

clokep commented Apr 6, 2020

@n3m3s1s The fix is not yet in a release version of Twisted. After they've done their next release it should just be a matter of ensuring the version of Twisted used is the latest.

@n3m3s1s
Copy link

n3m3s1s commented Apr 9, 2020

@clokep Right, sorry. Got a little confused by the 1k branches they have. Looking forward to their next release.

@Half-Shot

This comment has been minimized.

@mmilata

This comment has been minimized.

@babolivier babolivier added the Z-Upstream-Bug This issue requires a fix in an upstream dependency. label Apr 28, 2020
@exarkun

This comment has been minimized.

@babolivier
Copy link
Contributor

@Neustradamus Please don't harass people. The Twisted devs already answered your question on Twitter (https://twitter.com/twistedmatrix/status/1256852143425773568) giving you a viable solution if you really can't wait for a release (https://twitter.com/twistedmatrix/status/1256852515586338816), this should be more than enough while waiting for them to release a new version.

To reiterate and elaborate a bit on the advice the Twisted devs gave, here's how you install the patch if you really really really can't wait for a release:

# Activate Synapse's virtualenv
source env/bin/activate
# Uninstall the mainline version of twisted[tls]
pip uninstall twisted[tls]
# Reinstall twisted[tls] from github at the commit provided by the Twisted folks
pip install https://github.com/twisted/twisted/archive/8c251edc95b48d578660343c5de072691ff75e8b.zip#egg=twisted[tls]

Let me just empathise that this isn't a recommended procedure as unreleased patches are known to be less stable than releases.

Note that you'll need to run pip uninstall twisted[tls] && pip install twisted[tls] when the fix is released.

@matrix-org matrix-org locked as too heated and limited conversation to collaborators Aug 11, 2020
@clokep
Copy link
Member

clokep commented Mar 2, 2021

This should be fixed if you update Twisted to the newly released version (21.2.0). I'm going to close this issue and unlock it. If you upgrade and are still seeing this issue we can re-open and see if Synapse needs any changes.

@Dirk23
Copy link

Dirk23 commented Jul 28, 2021

with "tlsv1 alert unknown ca" most probably the SMTP client is using a client certificate that is not signed by a CA accepted by the server.

So the Client ist synapse and the Server is my Mailserver?
I don't have anything like a client certificate in synapse configured and I don't have any troubles with my Mailserver. Where should this problem come from then?

@adiroiban
Copy link

I don't know how the system is configured.

AFAIK, the original error was "unsupported protocol"
This was generated because the Twisted SMTP client was hardcoded to use TLS v1.0 and TLSv1.1 while the server was only accepting TLSv1.2 or TLSv1.3

I think that the "tlsv1 alert unknown ca" is not related to this ticket.

Cheers

@Dirk23
Copy link

Dirk23 commented Jul 28, 2021

well, Matrix can't send E-Mails to my Mailserver, but everything else can. I don't think it is a problem with my Mailserver.

@Dirk23
Copy link

Dirk23 commented Jul 29, 2021

Ok, my problem was the cert of the Mailserver. Didn't have full chain there, never mind.

@richvdh
Copy link
Member

richvdh commented Aug 12, 2021

see also #9566

@Linuxine
Copy link

Hi,

I have the same issue on my homeserver, version 1.40.0. Every mail for new user or password reset fails with the error "cannot connect to server".

I guess this is related to my version of twisted being 20.3.0. But how can I upgrade the Twisted version to the right one ?
My homeserver is installed using pip in a virtualenv.

I tried a "pid install -U twisted" but it seems to break my matrix instance, I get the following error when launching synapse:

Aug 18 18:32:36  python3[4142027]:    from twisted.python.compat import _PY3, unicode
Aug 18 18:32:36  python3[4142027]: ImportError: cannot import name '_PY3'

as related to YunoHost-Apps/synapse_ynh#247, the version for twisted is forced to 20.3.0 ?

So how can I resolve this issues without upgrading Twisted ? I am a bit lost 😞

@richvdh
Copy link
Member

richvdh commented Aug 18, 2021

Aug 18 18:32:36 python3[4142027]: ImportError: cannot import name '_PY3'

sounds like you have another package in your virtualenv which breaks with newer Twisted. Try sharing the whole stacktrace so we can see which it is.

@voegelas
Copy link

So how can I resolve this issues without upgrading Twisted ? I am a bit lost disappointed

You have to remove a line from smtp.py. See twisted/twisted@d427cbd

If Twisted is updated, treq has to be updated too. That's probably where the "cannot import name '_PY3'" error message comes from. Welcome to the wonderful world of Python and non-existing backward compatibility.

@richvdh
Copy link
Member

richvdh commented Aug 18, 2021

Welcome to the wonderful world of Python and non-existing backward compatibility.

yes, because using a private interface in another library, and then being surprised when that private interface is removed, could never happen in any other language.

@voegelas
Copy link

Welcome to the wonderful world of Python and non-existing backward compatibility.

yes, because using a private interface in another library, and then being surprised when that private interface is removed, could never happen in any other language.

It just happens in Python much more often then in any other language. I still have a copy of van Rossums "Internet Programming with Python" where he lies on page 5: "New versions of the interpreter will always run programs written for old versions of the interpreter". The packages are an even bigger mess. But never mind. We've mostly completed our migration from Matrix to Mattermost.

@Linuxine
Copy link

Hi @richvdh , thanks for the reply !
The full stack trace is as following:

Aug 18 17:56:30 linuxine3 python3[4139948]: Traceback (most recent call last):
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
Aug 18 17:56:30 linuxine3 python3[4139948]:    "__main__", mod_spec)
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
Aug 18 17:56:30 linuxine3 python3[4139948]:    exec(code, run_globals)
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/app/homeserver.py", line 37, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.app import _base
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/app/_base.py", line 40, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.events.spamcheck import load_legacy_spam_checkers
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/events/spamcheck.py", line 31, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.rest.media.v1._base import FileInfo
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/rest/__init__.py", line 31, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.rest.client.v2_alpha import (
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/rest/client/v2_alpha/register.py", line 38, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.handlers.auth import AuthHandler
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/handlers/auth.py", line 65, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.module_api import ModuleApi
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/module_api/__init__.py", line 35, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from synapse.http.client import SimpleHttpClient
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/synapse/http/client.py", line 32, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    import treq
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/treq/__init__.py", line 5, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from treq.api import head, get, post, put, patch, delete, request
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/treq/api.py", line 5, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from treq.client import HTTPClient
Aug 18 17:56:30 linuxine3 python3[4139948]:  File "/home/matrix/synapse/env/lib/python3.6/site-packages/treq/client.py", line 11, in <module>
Aug 18 17:56:30 linuxine3 python3[4139948]:    from twisted.python.compat import _PY3, unicode
Aug 18 17:56:30 linuxine3 python3[4139948]: ImportError: cannot import name '_PY3'

@richvdh
Copy link
Member

richvdh commented Aug 18, 2021

@Linuxine: as @voegelas says, you'll also need to update treq.

@Linuxine
Copy link

@Linuxine: as @voegelas says, you'll also need to update treq.

Okays, thanks a lot, I will do it tomorrow (Paris time) and let you know !

@Linuxine
Copy link

Hi,

I tried using pip to update twisted and treq at the same time, and it worked ! I have successfully received an email for a new account creation and also to reset a password. Thanks a lot for you help @richvdh and @voegelas \o/

mguentner added a commit to mguentner/nixpkgs that referenced this issue Aug 22, 2021
twisted is used in matrix-synapse for smtp handling.
Mostly this is used for password resets, but also notifications
are delivered that way.

older versions of twisted require the e-mail server to
have TLS1.0 enabled.

Obviously, quite a lot of servers have this disabled which means
synapse won't be able to deliver mails using such servers.

matrix-synapse issue:

matrix-org/synapse#6211
mguentner added a commit to mguentner/nixpkgs that referenced this issue Aug 22, 2021
mguentner added a commit to mguentner/nixpkgs that referenced this issue Aug 22, 2021
twisted is used in matrix-synapse for smtp handling.
Mostly this is used for password resets, but also notifications
are delivered that way.

older versions of twisted require the e-mail server to
have TLS1.0 enabled.

Obviously, quite a lot of servers have this disabled which means
synapse won't be able to deliver mails using such servers.

matrix-synapse issue:

matrix-org/synapse#6211
@vmario89
Copy link

vmario89 commented Oct 1, 2021

it still does not work for my. on my server i always get, no matter what i configure.
postfix log:

SSL3 alert read:fatal:unknown CA
Okt 02 00:17:05 <redacted> postfix/smtpd[3054606]: SSL_accept:error in error
Okt 02 00:17:05  <redacted> postfix/smtpd[3054606]: SSL_accept error from localhost[127.0.0.1]: -1
Okt 02 00:17:05  <redacted> postfix/smtpd[3054606]: warning: TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1543:SSL alert number 48:
Okt 02 00:17:05  <redacted> postfix/smtpd[3054606]: lost connection after STARTTLS from localhost[127.0.0.1]

the log /var/log/matrix-synapse/homeserver.log

2021-10-02 00:49:25,542 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 5
2021-10-02 00:49:25,562 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 4
2021-10-02 00:49:25,582 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 3
2021-10-02 00:49:25,593 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 2
2021-10-02 00:49:25,613 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 1
2021-10-02 00:49:25,733 - synapse.handlers.identity - 415 - ERROR - POST-145 - Error sending threepid validation email to <redacted>@<redacted>.de
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/identity.py", line 413, in send_threepid_validation
    await send_email_func(email_address, token, client_secret, session_id)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/push/mailer.py", line 201, in send_add_threepid_mail
    await self.send_email(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/push/mailer.py", line 318, in send_email
    await self.send_email_handler.send_email(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/send_email.py", line 175, in send_email
    await self._sendmail(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/send_email.py", line 111, in _sendmail
    await make_deferred_yieldable(d)
twisted.mail._except.SMTPConnectError: Unable to connect to server.
2021-10-02 00:49:25,734 - synapse.http.server - 88 - INFO - POST-145 - <XForwardedForRequest at 0x7fb2743e2130 method='POST' uri='/_matrix/client/r0/account/3pid/email/requestToken' clientproto='HTTP/1.0' site='8008'> SynapseError: 500 - An error was encountered when sending the email

i checked CA cert which is fine. All things with mail clients work properly, like thunderbird or roundcube or other services which connect to the mail server from exernal or localhost

my mail server only supports TLS 1.2 and TLS 1.3. SSLv2/v3 is completely disabled for smtp/smtpd.

i updated twisted[tls] und removed the older apt package too python3-twisted

also tried synapse connecting to mail server locally or by public mail server address

any idea what to do?

some infos:

  • ii openssl 1.1.1f-1ubuntu2.8 amd64 Secure Sockets Layer toolkit - cryptographic utility
  • ii postfix 3.4.13-0ubuntu1.2 amd64 High-performance mail transport agent
  • ii matrix-synapse-py3 1.43.0+focal1 amd64 Open federated Instant Messaging and VoIP server
  • twisted 21.7.0
  • recent version of 21.5.0

some more stuff i tried which also looks good, but without solving or modifying the problem:

update-ca-certificates

openssl verify -CAfile /etc/ssl/certs/ISRG_Root_X1.pem /etc/letsencrypt/live/mymailserver.de/chain.pem

openssl s_client -showcerts -servername mail.mymailserver.org -connect smtp.mymailserver.org:587
openssl s_client -starttls smtp -showcerts -servername mymailserver.org -connect smtp.mymailserver.org:587
openssl s_client -CApath /etc/ssl/certs -CAfile /etc/ssl/certs/ca-certificates.crt -starttls smtp -showcerts -servername mail.mymailserver.org -connect smtp.mymailserver.org:587
openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -starttls smtp -showcerts -servername mail.mymailserver.org -connect mail.mymailserver.org:25

apt remove python3-openssl
sudo pip install --upgrade pyOpenSSL

@vmario89
Copy link

vmario89 commented Oct 2, 2021

i recognized it seems to belong to #9599 too. same behaviour:

openssl s_client -connect matrix..de:443

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
z-bug (Deprecated Label) z-p2 (Deprecated Label) Z-Upstream-Bug This issue requires a fix in an upstream dependency.
Projects
None yet
Development

No branches or pull requests