fix: url_params cache miss with global async query #23641

kekwan · 2023-04-10T23:46:33Z

SUMMARY

Fixes initial cache miss when global async queries are used.

Currently, whenever a Jinja context is used in a query such as url_params with async queries enabled, a call to get_form_data is made. form_data which has the url_params is used to generate extra cache keys. Right now, form_data is empty as there is no request context in async queries. This means that we are always encountering a cache miss on the first query.

Since async queries won't have request context, we need to fallback to setting Flask global. https://github.com/apache/superset/blob/master/superset/views/utils.py#L182

BEFORE/AFTER SCREENSHOTS OR

BEFORE

Screen.Recording.2023-04-10.at.4.45.16.PM.mov

AFTER

Screen.Recording.2023-04-10.at.4.19.37.PM.mov

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue: Jinja template is not working when async query is enabled #16650
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

villebro

LGTM - thanks for fixing this long standing bug!

superset/charts/data/api.py

villebro

@kekwan I was too quick to approve - there's a missing import, and some linting errors. Let's see if fixing those clears up CI

codecov · 2023-04-11T16:48:19Z

Codecov Report

Merging #23641 (91a84fc) into master (95d71ff) will increase coverage by 0.00%.
The diff coverage is 100.00%.

❗ Current head 91a84fc differs from pull request most recent head 6e0ff2b. Consider uploading reports for the commit 6e0ff2b to get more accurate results

@@           Coverage Diff           @@
##           master   #23641   +/-   ##
=======================================
  Coverage   68.08%   68.08%           
=======================================
  Files        1920     1920           
  Lines       73990    73992    +2     
  Branches     8091     8091           
=======================================
+ Hits        50374    50376    +2     
  Misses      21547    21547           
  Partials     2069     2069

Flag	Coverage Δ
hive	`53.18% <66.66%> (-0.01%)`	⬇️
mysql	`79.21% <100.00%> (+<0.01%)`	⬆️
postgres	`79.29% <100.00%> (+<0.01%)`	⬆️
presto	`53.09% <66.66%> (-0.01%)`	⬇️
python	`83.14% <100.00%> (+<0.01%)`	⬆️
sqlite	`77.79% <100.00%> (+<0.01%)`	⬆️
unit	`53.02% <33.33%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/charts/data/api.py	`88.95% <100.00%> (+0.06%)`	⬆️
superset/views/core.py	`75.29% <100.00%> (+0.01%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

kekwan · 2023-04-11T17:08:57Z

thanks @villebro, fixed all the linting issues.

villebro · 2023-04-11T17:46:36Z

@kekwan I restarted the Presto test suite - if it keeps coming back red we should consider disabling it in the Presto test suite (there's quite a few others that have had to be disabled, too)

kekwan · 2023-04-11T21:16:21Z

@villebro seems like its been failing consistently on latest builds on master. let me know if you want me to disable it as part of this PR.

villebro · 2023-04-13T08:09:20Z

@kekwan I believe it was fixed here: #23666 So a rebase should fix it

…orm_data import flask g fix linting fix linting 2

kekwan · 2023-04-13T20:26:49Z

thanks @villebro rebased

villebro · 2023-04-13T20:34:57Z

Perfect, thanks for all the work and patience here @kekwan ! ❤️

betodealmeida · 2023-05-02T18:50:29Z

superset/charts/data/api.py

@@ -299,6 +299,9 @@ def data_from_cache(self, cache_key: str) -> Response:
        """
        try:
            cached_data = self._load_query_context_form_from_cache(cache_key)
+            # Set form_data in Flask Global as it is used as a fallback
+            # for async queries with jinja context
+            setattr(g, "form_data", cached_data)


Is there a reason to do this instead of g.form_data = cached_data?

pull-request-size bot added the size/XS label Apr 10, 2023

kekwan mentioned this pull request Apr 10, 2023

fix: apply url_params in global async query #18921

Closed

9 tasks

villebro approved these changes Apr 11, 2023

View reviewed changes

villebro reviewed Apr 11, 2023

View reviewed changes

superset/charts/data/api.py Show resolved Hide resolved

villebro reviewed Apr 11, 2023

View reviewed changes

kekwan force-pushed the url-param-dashboard-cache-fix branch from f6b31eb to fb60d4b Compare April 11, 2023 16:20

pull-request-size bot added size/S and removed size/XS labels Apr 11, 2023

kekwan force-pushed the url-param-dashboard-cache-fix branch from b4f5e4d to 3059f23 Compare April 11, 2023 16:27

pull-request-size bot added size/XS and removed size/S labels Apr 11, 2023

kekwan force-pushed the url-param-dashboard-cache-fix branch from cbe6899 to 6dbbad2 Compare April 11, 2023 16:35

fix url_params cache miss with global async query by setting global f…

6e0ff2b

…orm_data import flask g fix linting fix linting 2

kekwan force-pushed the url-param-dashboard-cache-fix branch from 6dbbad2 to 6e0ff2b Compare April 13, 2023 16:55

villebro merged commit 19404bc into apache:master Apr 13, 2023

kekwan deleted the url-param-dashboard-cache-fix branch April 13, 2023 22:28

sebastianliebscher pushed a commit to sebastianliebscher/superset that referenced this pull request Apr 28, 2023

fix: url_params cache miss with global async query (apache#23641)

53522f4

betodealmeida reviewed May 2, 2023

View reviewed changes

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: url_params cache miss with global async query #23641

fix: url_params cache miss with global async query #23641

kekwan commented Apr 10, 2023

villebro left a comment

villebro left a comment

codecov bot commented Apr 11, 2023 •

edited

Loading

kekwan commented Apr 11, 2023

villebro commented Apr 11, 2023

kekwan commented Apr 11, 2023

villebro commented Apr 13, 2023

kekwan commented Apr 13, 2023

villebro commented Apr 13, 2023

betodealmeida May 2, 2023

fix: url_params cache miss with global async query #23641

fix: url_params cache miss with global async query #23641

Conversation

kekwan commented Apr 10, 2023

SUMMARY

BEFORE/AFTER SCREENSHOTS OR

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

villebro left a comment

Choose a reason for hiding this comment

villebro left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 11, 2023 • edited Loading

Codecov Report

kekwan commented Apr 11, 2023

villebro commented Apr 11, 2023

kekwan commented Apr 11, 2023

villebro commented Apr 13, 2023

kekwan commented Apr 13, 2023

villebro commented Apr 13, 2023

betodealmeida May 2, 2023

Choose a reason for hiding this comment

codecov bot commented Apr 11, 2023 •

edited

Loading