Scanning or Refreshing Druid Datasource fails whilst using Gevent with Gunicorn

[john-bodley]

Make sure these boxes are checked before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if any
• I have reproduced the issue with at least the latest released version of superset
• I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

0.21.0dev

Expected results

Both 'Scan New Datasources' and 'Refresh Druid Metadata' should function whilst running Gevent with Gunicorn.

Actual results

We're seeing the following error in NewRelic when running on production,

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 536, in run
File "/usr/local/lib/python2.7/dist-packages/gevent/baseserver.py", line 26, in _handle_and_close_when_done
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 56, in handle
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/async.py", line 107, in handle_request
File "/usr/local/lib/python2.7/dist-packages/newrelic/api/web_transaction.py", line 1309, in _nr_wsgi_application_wrapper_
File "/usr/local/lib/python2.7/dist-packages/newrelic/api/web_transaction.py", line 837, in __init__
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
File "/usr/local/lib/python2.7/dist-packages/werkzeug/contrib/fixers.py", line 152, in __call__
File "/usr/local/lib/python2.7/dist-packages/newrelic/api/web_transaction.py", line 1198, in _nr_wsgi_application_wrapper_
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
File "/usr/local/lib/python2.7/dist-packages/newrelic/hooks/framework_flask.py", line 45, in _nr_wrapper_handler_
File "/usr/local/lib/python2.7/dist-packages/superset/utils.py", line 599, in wraps
File "/usr/local/lib/python2.7/dist-packages/superset/connectors/druid/views.py", line 308, in scan_new_datasources
File "/usr/local/lib/python2.7/dist-packages/superset/utils.py", line 599, in wraps
File "/usr/local/lib/python2.7/dist-packages/superset/connectors/druid/views.py", line 285, in refresh_datasources
File "/usr/local/lib/python2.7/dist-packages/superset/connectors/druid/models.py", line 125, in refresh_datasources
File "/usr/local/lib/python2.7/dist-packages/superset/connectors/druid/models.py", line 162, in refresh_async
File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
File "/usr/lib/python2.7/multiprocessing/pool.py", line 561, in get
File "/usr/lib/python2.7/multiprocessing/pool.py", line 556, in wait
File "/usr/lib/python2.7/threading.py", line 340, in wait
File "src/gevent/_semaphore.py", line 198, in gevent._semaphore.Semaphore.acquire (src/gevent/gevent._semaphore.c:4541)
File "src/gevent/_semaphore.py", line 226, in gevent._semaphore.Semaphore.acquire (src/gevent/gevent._semaphore.c:4367)
File "src/gevent/_semaphore.py", line 166, in gevent._semaphore.Semaphore._do_wait (src/gevent/gevent._semaphore.c:3562)
File "/usr/local/lib/python2.7/dist-packages/gevent/hub.py", line 630, in switch

Whilst testing I was able to successfully refresh the Druid datasources without Gevent, i.e.,

gunicorn -w 1 --timeout 120 -b 0.0.0.0:8888 --limit-request-line 0 --limit-request-field_size 0 --statsd-host localhost:8125 superset:app

though when running Gevent with Gunicorn, i.e.,

gunicorn -w 1 -k gevent --timeout 120 -b 0.0.0.0:8888 --limit-request-line 0 --limit-request-field_size 0 --statsd-host localhost:8125 superset:app

the system hangs and eventually returns a 502. The issue seems to be with using multi-processing in conjunction with Gevent.

Steps to reproduce

See above.

[xrmx]

Could you reproduce without a gevent worker?

[john-bodley]

@xrmx it seems like this is actually a Gevent issue as opposed to Gunicorn. I'm updated the description of the issue.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.