[druid] Fixing issue 3894 multi-processing w/ Gunicorn

[john-bodley]

This PR fixes issue #3894.

It seems that using multi-processing for fetching Druid datasource metadata asynchronously fails under a Gunicorn environment. Switching to multi-threading seems to remedy the problem.

Note previously Pool() created only 4 processes and I'm not overly certain why this part of the code needs to leverage multi-processing from a performance standpoint, i.e., I would be supportive of remove the multi-threading/processing logic entirely.

Also it seems like we're using the synchronous PyClient and thus refresh_async seems like a misnomer to me.

to: @mistercrunch @Mogball @xrmx

[Mogball]

Multithreaded execution was introduced because refreshing a cluster with many datasources took... a long time. Metadata queries were issued sequentially, but multithreading this makes refreshing lots of datasources much faster.

[john-bodley]

Thanks @Mogball for the explanation. Are you opposed to using multi-threading rather than multi-processing? Also could you provide context on the refresh_async method name?

[Mogball]

Looks okay to me. I'm pretty sure it should be thread-safe.

refresh_async was the name of the function when it was suppose to be actually asynchronous but I never changed the name afterwards (i.e. a bit of a misnomer).

[john-bodley]

@Mogball per @xrmx's comment in the issue this may actually be a Gevent issue. I'm not certain whether we want to disable that or go with the approach outlined in this PR, i.e., moving from multi-processing to multi-threading.

[xrmx]

If you are just waiting on druid IO threads instead of processes should be fine. To avoid the possible creation of ton of threads you can initialize the pool outside the function so the number is fixed.

[john-bodley]

@mistercrunch @Mogball @xrmx I'm not overly familiar with this portion of the code base, and the interactions with Gevent, I'm merely trying to fix an issue which is plaguing us in production.

I can't fathom the scale @Mogball of your Druid datasources, but for us this request takes mere seconds and thus I'm unsure whether the 4x speedup (maximum) of a multi-threaded environment outweighs the additional complexity, potential thread-safeness etc.

[Mogball]

The speed up is more than just 4x (or whatever the number of available threads). Most of the time is spent waiting for Druid to respond, which means that the metadata requests are issued more or less simultaneously for each datasource, and then processed when they all start coming back. This makes a difference of like 40 seconds to 3 seconds (when hard refreshing all datasources).

This should be thread-safe since there is no interaction outside of the datasource object and with the Superset backend during refresh.

[mistercrunch]

Heads up that there are caveats with thread-safety around SQLAlchemy

[Mogball]

latest_metadata doesn't make any calls to SQLAlchemy 👍