Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"TypeError: 'str' does not support the buffer interface" from async queries with redis backend on python3 #2079

Closed
3 tasks done
rumbin opened this issue Jan 31, 2017 · 2 comments

Comments

@rumbin
Copy link
Contributor

rumbin commented Jan 31, 2017

Make sure these boxes are checked before submitting your issue - thank you!

  • I have checked the superset logs for python stacktraces and included it here as text if any
  • I have reproduced the issue with at least the latest released version of superset
  • I have checked the issue tracker for the same issue and I haven't found one similar

Superset version

0.15.4 on python3.4.5

Expected results

Async query succeeds; fetch results succeeds and shows result table.

Actual results

Async queries result in an error in the superset worker:
"TypeError: 'str' does not support the buffer interface"

This seems to be caused by python3 which requires the payload to be encoded.
For testing, I did the following adjustments which seem to do the job:

File sql_lab.py:

#results_backend.set(key, zlib.compress(payload))  
results_backend.set(key, zlib.compress(payload.encode('utf-8')))  

File views.py:

#json_payload = zlib.decompress(blob)
json_payload = zlib.decompress(blob).decode('utf-8')

Steps to reproduce

  1. Have a redis server running.
  2. Install superset on python3 with celery[redis] and redis,
  3. activate them in the config:
class CeleryConfig(object):
  BROKER_URL = 'redis://localhost:6379/0'
  CELERY_IMPORTS = ('superset.sql_lab', )
  CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
  CELERY_ANNOTATIONS = {'tasks.add': {'rate_limit': '10/s'}}
CELERY_CONFIG = CeleryConfig

from werkzeug.contrib.cache import RedisCache
RESULTS_BACKEND = RedisCache(host='localhost', port=6379, password=None, db=0, default_timeout=300, key_prefix='cel_result')
  1. Start superset server and worker
  2. Run async query in SQL-Lab and fetch results

Questions

Supposing, my adjustments for python3 are the right means for fixing the issue, I got the following questions:
a. Are there more than these two locations where we need to encode a string? Can anyone point me there?
b. What is the right way to fix this python3-specific issue so that we do not compromise python2 installations? (I have absolutely no experience in this regard…)
c. I'm also not experienced with github and pull requests. Should I still try to fix it myself? I'm perfectly fine if someone else wants to deal with this issue. However, if needed, I can try to make a pull request.

@xrmx
Copy link
Contributor

xrmx commented Jan 31, 2017

There's already a pull request for this #2034 and it'll also answer all of your questions :)

@rumbin
Copy link
Contributor Author

rumbin commented Apr 5, 2017

Unfortunately, this very pull request is in limbo for more than 2 months now :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants