Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve presto query perf #11069

Merged

Conversation

etr2460
Copy link
Member

@etr2460 etr2460 commented Sep 25, 2020

SUMMARY

The latest_partition function results shouldn't change from second to second, yet it could be called many times simultaneously if a dashboard has many charts dependent on the same table. Conservatively memoize the result of this function for 60 seconds to prevent an extra roundtrip to the Presto db. Perhaps this could be extended to 5 minutes in the future, but wanted to start out conservatively

TEST PLAN

CI, add print statements to the function and see that the function isn't called when run for the 2nd or 3rd time in a minute

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Requires DB Migration.
  • Confirm DB Migration upgrade and downgrade tested.
  • Introduces new feature or API
  • Removes existing feature or API

to: @villebro @john-bodley @betodealmeida

@codecov-commenter
Copy link

codecov-commenter commented Sep 25, 2020

Codecov Report

Merging #11069 into master will increase coverage by 0.07%.
The diff coverage is 50.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #11069      +/-   ##
==========================================
+ Coverage   65.79%   65.87%   +0.07%     
==========================================
  Files         816      816              
  Lines       38422    38511      +89     
  Branches     3621     3621              
==========================================
+ Hits        25280    25369      +89     
  Misses      13034    13034              
  Partials      108      108              
Flag Coverage Δ
#cypress 57.08% <ø> (+<0.01%) ⬆️
#javascript 62.00% <ø> (+0.23%) ⬆️
#python 61.40% <50.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/presto.py 81.95% <50.00%> (+0.04%) ⬆️
...perset-frontend/src/views/CRUD/chart/ChartList.tsx 82.05% <0.00%> (-0.86%) ⬇️
superset-frontend/src/components/FaveStar.tsx 100.00% <0.00%> (ø)
superset-frontend/src/components/Icon/index.tsx 100.00% <0.00%> (ø)
...erset-frontend/src/dashboard/components/Header.jsx 52.71% <0.00%> (+0.36%) ⬆️
...set-frontend/src/dashboard/util/getDropPosition.js 93.65% <0.00%> (+1.58%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 052d7a0...fcff034. Read the comment docs.

@etr2460 etr2460 force-pushed the erik-ritter--memoize-helper-presto-queries branch from 5c8f3ab to fcff034 Compare September 25, 2020 21:58
@etr2460 etr2460 merged commit 0f7e90f into apache:master Sep 28, 2020
auxten pushed a commit to auxten/incubator-superset that referenced this pull request Nov 20, 2020
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.38.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XS 🚢 0.38.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants