Async chart not rendering

[mkramer5454]

Hello, I have tried to turn on GAQ on the latest helm/superset release (0.6.1, 1.5.0, respectively). When I try to run a query from a chart, the chart spins until it times out. However, I can see the query execute on the data warehouse and the results are returned to redis. In fact, if I refresh the chart page after seeing the load_chart_data_into_cache task complete successfully on the worker, then the chart pulls from redis and loads as expected. I believe that I've configured superset as outlined here.

Async queries are running as expected in sql labs.

How to reproduce the bug

Deploy superset to k8s using helm with the following config overrides:

configOverrides:
  enable_gaq: |
    GLOBAL_ASYNC_QUERIES_JWT_SECRET = env('GLOBAL_ASYNC_QUERIES_JWT_SECRET')
    GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
      "port": env('REDIS_PORT'),
      "host": env('REDIS_HOST'),
      "password": env('REDIS_PASSWORD'),
      "db": 0,
      "ssl": False
     }

    FEATURE_FLAGS = { 'GLOBAL_ASYNC_QUERIES': True }

Expected results

I expect the chart to render results after the query completes.

Actual results

The chart never renders.

Screenshots

Environment

(please complete the following information):

• browser type and version: Firefox 99.0.1 (64-bit)
• superset version: 1.5.0
• python version: 3.8.12
• node.js version: node -v
• any feature flags active: Global async queries

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Add any other context about the problem here.

[ensky]

+1, the async event in polling api is always empty([]). The charts never renders.

[wulfuric]

I'm seeing this same behavior as well, the celery worker seems to load the query fine

2022-07-27 16:15:45,322: INFO/MainProcess] Task load_chart_data_into_cache[4470045e-893b-4591-b9f0-e137680061f1] received
[2022-07-27 16:15:45,323: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7fd54dd5b790> (args:('load_chart_data_into_cache', '4470045e-893b-4591-b9f0-e137680061f1', {'lang': 'py', 'task': 'load_chart_data_into_cache', 'id': '4470045e-893b-4591-b9f0-e137680061f1', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, 21600], 'root_id': '4470045e-893b-4591-b9f0-e137680061f1', 'parent_id': None, 'argsrepr': "({'channel_id': '672a8009-1db5-45ca-b2eb-6cd2d8f2713b', 'job_id': 'eb001c6c-d2e3-43b4-9b43-af5a84f6fa26', 'user_id': 1, 'status': 'pending', 'errors': [], 'result_url': None}, {'datasource': {'id': 182, 'type': 'table'}, 'force': False, 'queries': [{...}], 'form_data': {'datasource': '182__table', 'viz_type': 'big_number_total', 'slice_id': 305, 'url_params': {...}, 'granularity_sqla': 'data_dt', 'time_grain_sqla': 'P1D', 'time_range': 'No filter', 'metric': {...}, 'adhoc_filters': [...], 'header_font_size': 0.4, 'subheader_font_size': 0.15, 'y_axis_format': 'SMART_NUMBER', 'time_format': 'smart_date', 'extra_form_data': {...}, 'force':... kwargs:{})
The field `timeseries_limit` is deprecated, please use `series_limit` instead.
[2022-07-27 16:15:45,341: WARNING/MainProcess] The field `timeseries_limit` is deprecated, please use `series_limit` instead.
[2022-07-27 16:15:45,645: DEBUG/MainProcess] Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
[2022-07-27 16:15:45,648: DEBUG/MainProcess] Making request: POST https://oauth2.googleapis.com/token
[2022-07-27 16:15:45,652: DEBUG/MainProcess] Starting new HTTPS connection (1): oauth2.googleapis.com:443
[2022-07-27 16:15:45,702: DEBUG/MainProcess] https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
[2022-07-27 16:15:45,706: DEBUG/MainProcess] Starting new HTTPS connection (1): bigquery.googleapis.com:443
[2022-07-27 16:15:46,143: DEBUG/MainProcess] https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects//jobs?prettyPrint=false HTTP/1.1" 200 None
[2022-07-27 16:15:46,338: DEBUG/MainProcess] https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects//queries/c24b2e77-f8e9-441e-9008-366628029d54?maxResults=0&location=US&prettyPrint=false HTTP/1.1" 200 None
[2022-07-27 16:15:48,358: INFO/MainProcess] Task load_chart_data_into_cache[4470045e-893b-4591-b9f0-e137680061f1] succeeded in 3.0331106479279697s:

But the web server keeps repeatedly hitting the async endpoint, and not getting any results.

Configs are

class CeleryConfig(object):
    broker_url = 'redis://{0}:{1}/0'.format(REDIS_HOST, REDIS_PORT)
    imports = (
        'superset.sql_lab',
        'superset.tasks',
    )
    result_backend = 'redis://{0}:{1}/0'.format(REDIS_HOST, REDIS_PORT)
    worker_log_level = 'DEBUG'
    worker_prefetch_multiplier = 1
    task_acks_late = False
    task_annotations = {
        'sql_lab.get_sql_results': {
            'rate_limit': '10000/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }

CELERY_CONFIG = CeleryConfig

# On Redis
from cachelib.redis import RedisCache
RESULTS_BACKEND = RedisCache(host=REDIS_HOST, port=REDIS_PORT, key_prefix='superset_results')


# import redis
CACHE_DEFAULT_TIMEOUT = 21600 # Default timeout

GLOBAL_ASYNC_QUERIES_JWT_SECRET = ''
GLOBAL_ASYNC_QUERIES_POLLING_DELAY = 4000

GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = {
    "port": REDIS_PORT,
    "host": REDIS_HOST,
    "password": "",
    "db": 0,
    "ssl": False,
}
GLOBAL_ASYNC_QUERIES_TRANSPORT = "polling"

CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

DATA_CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

EXPLORE_FORM_DATA_CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

FILTER_STATE_CACHE_CONFIG = {
    'CACHE_TYPE':'redis', # Use Redis
    'CACHE_REDIS_PORT': REDIS_PORT, # Configure port number
    'CACHE_REDIS_URL':'redis://{0}:{1}'.format(REDIS_HOST, REDIS_PORT)
}

FEATURE_FLAGS = {
"ENABLE_TEMPLATE_PROCESSING": True,
"DASHBOARD_NATIVE_FILTERS": True,
"DASHBOARD_CROSS_FILTERS": True,
"VERSIONED_EXPORT": True,
"GLOBAL_ASYNC_QUERIES": True,
}

Celery worker running with "celery --app=superset.tasks.celery_app:app worker --pool=threads -c 20 --loglevel=DEBUG",

This is running superset 2.0.0, we may look into using websockets but at this point hard to invest the time if global async queries might not work.

[D3nn3]

Did anyone investigate this further? Having the same problem with 2.0.0.

[fmusayev]

Having the same issue, tried different versions, no luck

[wulfuric]

I'm seeing similar behavior, particularly in embeds, it seems to be that there is auth / permission issues with the POST to the chart endpoint to fetch the polling data. Captured here - #18814

[anhtuanluu36]

I have the same issue. I tried version 1.3.2, 1.5.2 and 2.0.0 but no luck

[phongvt322]

+1, any update?

[etadelta222]

+1

[rohitpawar2811]

+1

[rohitpawar2811]

I am getting results successfully no problem

[RobLucchi]

@rohitpawar2811 What did you do differently?

[okayhooni]

I got same problem.. is there an update related this issue..?

[rusackas]

I'm not sure if this is still an issue in current versions of Superset (3.x). If it is, we can re-open this, or feel free to open a new issue with updated context and a reproducible case. We're no longer supporting Superset 2.x or prior, and it's been a while since this thread saw any activity, so I'm closing this as stale.

[smisch]

Can confirm that async chart queries work with @mkramer5454 's config. We're running chart version 0.12.6 and I just enabled it successfully.

I'd recommend adding his config to the documentation of this feature. Without this issue I don't think I'd have succeeded in enabling this feature.

Cheers!

[rusackas]

@smisch would you be willing to open a docs PR? There's an "edit this page" button in the bottom right of the docs site now which I hope makes it easier. Or if anyone wants to spell out what the instructions should be here on this thread, I can open the PR.

[smisch]

@rusackas I'll check it out. Haven't been contributing on Github yet, but looks easy enough. Fork, clone, change, commit, push, propose as PR to you.

[rusackas]

Yep! And I think the Github website will do like half of that pretty seamlessly/smoothly if you do it from the website. Of course, forking/cloning and making edits locally is fantastic if you want to contribute to additional pages or areas of the codebase!