refactor: remove unnecessary dataset queries from dashboard requests

[graceguo-supercat]

SUMMARY

In airbnb we did some investigation for server-side processing time for dashboard requests, try to find perf bottleneck and improve the dashboard performance, especially for those large ones with multiple datasets and tabs.

Currently when user opens a dashboard we need 3 different API to fetch data:

• GET /api/v1/dashboard/<id_or_slug>: to get dashboard metadata
• GET /api/v1/dashboard/<id_or_slug>/charts: to get charts metadata for a given dashboard id
• GET /api/v1/dashboard/<id_or_slug>/datasets: to get dataset metadata for a given dashboard id

Since this PR, we found the queries to tables is perf bottleneck, so we already made datasets request non-blocking for dashboard render. And given we already split big blob of dashboard data into 3 different APIs, it seems not necessary for each of request to query tables. This PR is to remove unnecessary dataset queries from these 2 requests.

We run some profiling for /dashboard/id and /dashboard/id/charts requests in Datadog. I used a heavy dashboard in airbnb as example, which has 80 datasets and about 300 charts.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before:

You will see that even each query to tables table only takes 1.2 ~ 1.9 ms, but the queries was called a few hundred times, it cause the overall response time took ~ 1 seconds (for both dashboard and dashboard/charts request)

After:

After removing unnecessary queries to datasets, pymysql.query operation is reduced a lot.

TESTING INSTRUCTIONS

CI and manual test.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #16110 (0b05c27) into master (423ff50) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #16110   +/-   ##
=======================================
  Coverage   76.83%   76.84%           
=======================================
  Files         995      995           
  Lines       52884    52881    -3     
  Branches     6721     6721           
=======================================
  Hits        40636    40636           
+ Misses      12023    12020    -3     
  Partials      225      225           

Flag	Coverage Δ
hive	81.33% <100.00%> (+0.03%)	⬆️
javascript	71.23% <ø> (+0.01%)	⬆️
mysql	81.59% <100.00%> (-0.01%)	⬇️
postgres	81.61% <100.00%> (-0.01%)	⬇️
presto	81.40% <100.00%> (-0.01%)	⬇️
python	82.12% <100.00%> (-0.01%)	⬇️
sqlite	81.25% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/charts/schemas.py	100.00% <ø> (ø)
superset/dashboards/schemas.py	99.37% <ø> (-0.01%)	⬇️
superset/models/dashboard.py	76.69% <ø> (-0.30%)	⬇️
superset/views/dashboard/mixin.py	95.00% <100.00%> (ø)
...rontend/src/components/Select/DeprecatedSelect.tsx	85.71% <0.00%> (-1.03%)	⬇️
.../src/dashboard/components/RefreshIntervalModal.tsx	89.47% <0.00%> (ø)
...ontrols/DndColumnSelectControl/DndColumnSelect.tsx	46.42% <0.00%> (ø)
...ntend/src/dashboard/components/CssEditor/index.jsx	96.42% <0.00%> (+0.59%)	⬆️
...uperset-frontend/src/components/Menu/MenuRight.tsx	92.15% <0.00%> (+1.59%)	⬆️
...ols/DndColumnSelectControl/utils/optionSelector.ts	46.15% <0.00%> (+11.53%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 423ff50...0b05c27. Read the comment docs.

[ktmud]

table_names is still used here (looks like for FAB CRUD views), should we clean up it as well?

[ktmud]

I think it's also safe to remove

superset/superset/models/dashboard.py

Lines 164 to 167 in b3e699b

	@property
	def table_names(self) -> str:
	# pylint: disable=no-member
	return ", ".join(str(s.datasource.full_name) for s in self.slices)

as that method is not particularly efficient (should've used batch query instead of looping through all slices).

[suddjian]

Looks familiar :)