(0.7 / 0.8) SslContext::new() is not returning (until thread interrupted)

[mikedilger]

I've got a weird problem that I've been struggling with off and on for the last few days.

I've got a project that uses hyper and lettre. Both use ssl. hyper still requires 0.7, but lettre uses 0.8. Anyhow, they seemed to work with those separate versions, but just in case I back-coded a branch of lettre to use 0.7 again, and this made no difference.

The problem is that in lettre::transport::smtp::SmtpTransportBuilder::new(), the call to SslContext::new(SslMethod::Tlsv1) does not return. When I ^C the entire program, it does return and runs the next few commands but is quickly killed thereafter.

To be sure, this works in general. Something about my specific case is causing the problem but I cannot for the life of me imagine what it might be.

My system openssl library is 1.0.2j

I'm using an older rust nightly because I haven't upgraded plugin-related deps yet: rustc 1.14.0-nightly (a7bfb1aba 2016-10-09)

[sfackler]

That's a weird place for it to block. Can you grab a stack trace when it's blocked?

[mikedilger]

How can I do that?

[sfackler]

The easiest way would probably be to run it in your debugger of choice, pause it at that point and take a backtrace there.

[mikedilger]

This is very weird. Why is it going into python!? At this point I think this is an issue on my system.

Thread 6 (Thread 0x7ffff41ff700 (LWP 12074)):
#0  0x00007ffff68dd318 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ffff6e0058f in ?? () from /usr/lib64/libpython3.4m.so.1.0
#2  0x00007ffff6e009b9 in PyEval_RestoreThread () from /usr/lib64/libpython3.4m.so.1.0
#3  0x00007ffff6e24f8b in PyGILState_Ensure () from /usr/lib64/libpython3.4m.so.1.0
#4  0x00007fffe96750c5 in ?? () from /usr/lib64/python3.4/site-packages/_cffi_backend.cpython-34m.so
#5  0x00007fffe967cd21 in ?? () from /usr/lib64/python3.4/site-packages/_cffi_backend.cpython-34m.so
#6  0x00007fffe98cd62c in ?? ()
   from /usr/lib64/python3.4/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so
#7  0x00007ffff7815119 in SSL_CTX_new () from /usr/lib64/libssl.so.1.0.0
#8  0x0000555556432fb1 in openssl::ssl::SslContext::new (method=openssl::ssl::SslMethod::Tlsv1)
    at /home/mike/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/openssl-0.8.3/src/ssl/mod.rs:709
#9  0x00005555563f02cb in lettre::transport::smtp::SmtpTransportBuilder::new<&std::net::addr::SocketAddr> (addr=0x7ffff3c30000)
    at /home/mike/.cargo/git/checkouts/lettre-53652803723a9045/master/src/transport/smtp/mod.rs:104
#10 0x00005555563fa20d in mailstrom::worker::smtp::smtp_delivery (envelope=..., 
---Type <return> to continue, or q <return> to quit---
    smtp_server=0x7ffff3c30000, helo=..., attempt=1) at /work/mailstrom/src/worker/smtp.rs:58
#11 0x00005555563feac4 in mailstrom::worker::deliver (email=0x7ffff41fd070, 
    internal_status=0x7ffff41fd038, helo_name=...) at /work/mailstrom/src/worker/mod.rs:389
#12 0x0000555555ac771a in mailstrom::worker::Worker<joist_app::mail_store::MailStore>::send_email<joist_app::mail_store::MailStore> (self=0x7ffff41fe340, email=..., internal_status=..., initial=false)
    at /work/mailstrom/src/worker/mod.rs:215
#13 0x0000555555ac862f in mailstrom::worker::Worker<joist_app::mail_store::MailStore>::handle_task<joist_app::mail_store::MailStore> (self=0x7ffff41fe340, task=0x7ffff3c0f030)
    at /work/mailstrom/src/worker/mod.rs:282
#14 0x0000555555acacfa in mailstrom::worker::Worker<joist_app::mail_store::MailStore>::run<joist_app::mail_store::MailStore> (self=0x7ffff41fe340) at /work/mailstrom/src/worker/mod.rs:167
#15 0x0000555555d1317e in mailstrom::{{impl}}::new::{{closure}}<joist_app::mail_store::MailStore> ()
    at /work/mailstrom/src/lib.rs:141
#16 0x0000555555bd5ae9 in std::panic::{{impl}}::call_once<(),closure> (self=...)
    at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/obj/../src/libstd/panic.rs:255
#17 0x0000555555aba228 in std::panicking::try::do_call<std::panic::AssertUnwindSafe<closure>,()> (
    data=0x7ffff41fe900 "")
    at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/obj/../src/libstd/panicking.rs:356
#18 0x0000555556600a27 in __rust_maybe_catch_panic ()
#19 0x0000555555ab9e07 in std::panicking::try<(),std::panic::AssertUnwindSafe<closure>> (f=...)
    at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/obj/../src/libstd/panicking.rs:332
#20 0x0000555555ab8fbd in std::panic::catch_unwind<std::panic::AssertUnwindSafe<closure>,()> (f=...)
    at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/obj/../src/libstd/panic.rs:311
#21 0x0000555555d13adb in std::thread::{{impl}}::spawn::{{closure}}<closure,()> ()
    at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/obj/../src/libstd/thread/mod.rs:277
#22 0x0000555555b3876f in alloc::boxed::{{impl}}::call_box<(),closure> (self=0x7ffff5a35060, args=())
    at /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/obj/../src/liballoc/boxed.rs:595
#23 0x00005555565f6791 in std::sys::thread::Thread::new::thread_start::h50b05608a499d2b2 ()
#24 0x00007ffff68d73b4 in start_thread () from /lib64/libpthread.so.0
#25 0x00007ffff7b28aed in clone () from /lib64/libc.so.6

[sfackler]

That is a very bizarre stack trace. I'm assuming your thing is intentionally linking to Python? My hunch right now is that it could be cryptography-installed OpenSSL locking callbacks. I'm poking some of the devs to see if that's right.

[mikedilger]

There is one thread in my program that links to python, written by another developer, to use the python dropbox api. I forgot about that.

Yeah, I think its in the locking callbacks

[mikedilger]

Looking at the python dropbox code he wrote, it acquires the GIL and never drops it. I guess he figured he was the only python consumer. I would have thought so too until now. He even writes:

        Dear Future Alan,

        If you end up with issues where multiple threads fight over the global interpreter lock
        then sorry, my bad. Hopefully it won't turn out to be a problem but yeah, it might.

        Regards,
        Past Alan

So yeah, I think I can sort this.

[sfackler]

Haha, that'd do it!

[sfackler]

Those callbacks have been moved into C in cryptography 1.6, so upgrading should avoid the GIL issue entirely if keeping it released while in Rust becomes hard.