feat: Add deck.gl Heatmap Visualization #23551

Mattc1221 · 2023-04-02T02:52:22Z

SUMMARY

Adds the heatmap layer from deck.gl here.

Related SIP: #23553

Rationale:

Superset already has supported functionality for other deck.gl visualization layers (e.g. Arc, Geojson, Grid, Hex, Path, Polygon, Scatter, Screengrid). Since Superset already contains the logic to handle deck.gl layers, it is straightforward and easy to maintain to add new deck.gl layers.

By expanding the range of available deck.gl visualization layers, users will have more options to choose from when creating their visualizations. This will allow them to tailor their visualizations to their specific needs and explore their data in different ways. It will also allow Superset to better compete with other data visualization tools on the market that offer a wider range of visualization options.

Overall, I think adding more deck.gl layers to Superset is a great idea and has the potential to significantly improve the user experience and the insights that users can gain from their data.

Design Decisions

The addition of the Heatmap layer follows the structure of existing deck.gl layers to maintain readability and consistency within the codebase.

Structure:
The directory superset-frontend/plugins/legacy-preset-chart-deckgl/src/layers/Heatmap/ contains the front end structure for the heat map

controlPanel configures the form to grab relevant information specified by the user to create and tune the heatmap.
Heatmap.jsx contains functions that define the Heatmap Layer, control the tooltip when hover the chart, and to get relevant data
index.js serves as the wrapper for the heatmap component and provides a metadata function to grab information to descrive what this visualization is/does

The Back End code for this feature follows the back end code for other deck.gl visualizations as well, providing functions to structure queried data for the Heatmap component.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

New visualization recording:

Screen.Recording.2023-04-01.at.9.20.50.PM.mov

TESTING INSTRUCTIONS

Start Superset
Click on the Charts tab at the top of the screen
Click the + Chart button at the top right
Load a dataset with geo-positional data (longitude/latitude)
Search the charts for Deck.gl Heatmap or click on the Map chart category
Select deck.gl Heatmap and hit Create New Chart
Update the Longitude & Latitude field to the correct columns
Select a Weight function
Select Create Chart
There should now be a heat map layer present on the screen
- Note: the underlying map component may not be visible if the MAP_BOX_API_KEY is not set. To set it follow the instructions here
Change the options in the control panel, check that the information tooltips are correct/make sense for each field
Check that when hovering a point on the map, the latitude/longitude shows up correctly

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

github-actions

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

villebro · 2023-04-04T06:54:43Z

This is really exciting work @Mattc1221 ! I'm marking myself as a reviewer and will do my best to review+test this later this week.

Mattc1221 · 2023-04-16T19:41:56Z

Hi @villebro I believe this PR should now be ready for review!

However, I noticed the tests have not run and I was wondering if I needed to close the PR and open a new one?

rusackas · 2023-04-17T14:54:21Z

Running the tests... thanks for this PR! I hope to review/test as well in a day or three :)

codecov · 2023-04-17T15:02:20Z

Codecov Report

Merging #23551 (a25f82d) into master (1670275) will decrease coverage by 0.02%.
The diff coverage is 36.11%.

❗ Current head a25f82d differs from pull request most recent head f37ea67. Consider uploading reports for the commit f37ea67 to get more accurate results

@@            Coverage Diff             @@
##           master   #23551      +/-   ##
==========================================
- Coverage   68.23%   68.21%   -0.02%     
==========================================
  Files        1942     1945       +3     
  Lines       75205    75241      +36     
  Branches     8145     8150       +5     
==========================================
+ Hits        51314    51327      +13     
- Misses      21806    21829      +23     
  Partials     2085     2085

Flag	Coverage Δ
hive	`53.18% <66.66%> (+<0.01%)`	⬆️
javascript	`54.50% <25.92%> (-0.02%)`	⬇️
mysql	`78.93% <66.66%> (-0.01%)`	⬇️
postgres	`79.01% <66.66%> (-0.01%)`	⬇️
presto	`53.10% <66.66%> (+<0.01%)`	⬆️
python	`82.78% <66.66%> (-0.01%)`	⬇️
sqlite	`77.53% <66.66%> (-0.01%)`	⬇️
unit	`53.05% <66.66%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...i-core/src/color/colorSchemes/sequential/common.ts	`100.00% <ø> (ø)`
...plugins/legacy-preset-chart-deckgl/src/factory.tsx	`0.00% <ø> (ø)`
...preset-chart-deckgl/src/layers/Heatmap/Heatmap.tsx	`0.00% <0.00%> (ø)`
...d/plugins/legacy-preset-chart-deckgl/src/preset.js	`100.00% <ø> (ø)`
...cy-preset-chart-deckgl/src/layers/Heatmap/index.ts	`66.66% <66.66%> (ø)`
superset/viz.py	`61.34% <66.66%> (+0.02%)`	⬆️
...et-chart-deckgl/src/layers/Heatmap/controlPanel.ts	`83.33% <83.33%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Mattc1221 · 2023-04-19T05:47:06Z

@rusackas Thanks for re-running the tests :)

The failing tests should now be fixed!

kgabryje · 2023-04-19T07:45:11Z

superset-frontend/plugins/legacy-preset-chart-deckgl/src/layers/Heatmap/Heatmap.jsx

@@ -0,0 +1,82 @@
+/**


Could you please refactor this file to typescript? I know that the rest of deckgl plugins are written in javascript, but our policy is to create the new files in typescript and SOME day refactor the old js files to ts too 🙂 (correct me if I'm wrong @rusackas)

kgabryje · 2023-04-19T07:46:27Z

superset-frontend/plugins/legacy-preset-chart-deckgl/src/layers/Heatmap/index.js

@@ -0,0 +1,45 @@
+/**


Same as above - js -> ts 🙏

kgabryje · 2023-04-19T07:47:25Z

Awesome first contribution, congrats! I left a comment about refactoring js to ts, but otherwise the code looks great 🎉

Mattc1221 · 2023-04-19T07:49:41Z

Thanks for the review @kgabryje 😀I'll update those JavaScript files to typescript sometime tomorrow evening!

Mattc1221 · 2023-04-29T23:01:37Z

@kgabryje I really appreciate your first review! The changes requests should now be addressed and I would love to get another round of review!

kgabryje · 2023-05-04T06:04:19Z

@Mattc1221 Sorry for the long delay and thank you for implementing the changes 🙂 Will merge as soon as the CI passes

kgabryje · 2023-05-04T06:26:27Z

@Mattc1221 looks like there are some linting issues.
Also, to fix the failed Python integration test, please rebase on the latest master

Mattc1221 · 2023-05-15T01:22:45Z

@kgabryje Thanks again for the review, and I apologize for the delayed response! I've rebased this branch to include the latest changes on the master branch and fixed the linting errors for the failing file plugins/legacy-preset-chart-deckgl/types/external.d.ts!

Mattc1221 · 2023-05-20T19:19:24Z

Hey @kgabryje! could we run the CI tests again? With the latest changes, the tests pass locally and also on my GitHub actions page!

villebro · 2023-05-21T08:00:22Z

@Mattc1221 I started CI and will review today 👍

villebro

LGTM, really excited to see geospatial heatmaps arrive in Superset 🎉

v1bg · 2023-05-22T17:40:41Z

Hi! This is really cool. Just deployed out and tested and everything looks good with the new heatmap chart type, but just FYI when you try to add the heatmap to the "deck.gl Multiple Layers" chart type it doesn't appear to be working. Wanted to mention this to see if it's a known issue.

Thanks again!

Mattc1221 · 2023-05-23T05:41:58Z

Big thanks to @kgabryje @villebro and @v1bg for reviewing this PR!

Hi! This is really cool. Just deployed out and tested and everything looks good with the new heatmap chart type, but just FYI when you try to add the heatmap to the "deck.gl Multiple Layers" chart type it doesn't appear to be working. Wanted to mention this to see if it's a known issue.

This may have been an oversight and may be a missing import in the Multiple Layers chart type. I'll look into this further later this week and open a corresponding issue!

v1bg · 2023-05-23T23:41:58Z

Awesome, appreciate it @Mattc1221! Great work here 🙏🏾

salvatorecolori · 2024-05-24T13:18:29Z

Hi, does this plugin/chart have extraProps enabled for javascript tooltips generator? When I try to use js code to generate tooltpis, I receive a javascript Uncaught TypeError: Cannot read properties of undefined (reading 'extraProps')".
Please check (url)#25871 (comment) too

pull-request-size bot added the size/L label Apr 2, 2023

github-actions bot reviewed Apr 2, 2023

View reviewed changes

Mattc1221 changed the title ~~[WIP]feat: Add deck.gl Heatmap Visualization~~ feat: [WIP] Add deck.gl Heatmap Visualization Apr 2, 2023

villebro self-requested a review April 4, 2023 06:54

Mattc1221 changed the title ~~feat: [WIP] Add deck.gl Heatmap Visualization~~ feat: Add deck.gl Heatmap Visualization Apr 16, 2023

Mattc1221 marked this pull request as ready for review April 16, 2023 07:14

rusackas requested review from rusackas and kgabryje April 17, 2023 14:56

kgabryje reviewed Apr 19, 2023

View reviewed changes

Mattc1221 requested a review from kgabryje April 23, 2023 18:35

kgabryje approved these changes May 4, 2023

View reviewed changes

Mattc1221 added 8 commits May 14, 2023 20:00

added deck gl heatmap

f97c686

added thumbnails

d0221c4

fix: fix linting errors

a15a305

fix: remove unused variable

8b78c58

fix: remove testing code

e1f58e0

chore: remove commented out code

982cb98

chore: add translations

3597b0d

chore: remove TODO comments

2d580bd

Mattc1221 added 2 commits May 14, 2023 20:00

fix: Convert js to ts

e77a188

chore: remove comment

a700bed

Mattc1221 force-pushed the deck_gl_heat_map branch from 8974271 to a700bed Compare May 15, 2023 01:00

fix: use prettier formatter to fix linting issues

f37ea67

villebro approved these changes May 22, 2023

View reviewed changes

villebro merged commit fc8c537 into apache:master May 22, 2023

Mattc1221 mentioned this pull request Jun 10, 2023

feat: Add Deck.gl Contour Layer #24154

Merged

9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add deck.gl Heatmap Visualization #23551

feat: Add deck.gl Heatmap Visualization #23551

Mattc1221 commented Apr 2, 2023 •

edited

Loading

github-actions bot left a comment

villebro commented Apr 4, 2023

Mattc1221 commented Apr 16, 2023

rusackas commented Apr 17, 2023

codecov bot commented Apr 17, 2023 •

edited

Loading

Mattc1221 commented Apr 19, 2023

kgabryje Apr 19, 2023 •

edited

Loading

kgabryje Apr 19, 2023

kgabryje commented Apr 19, 2023

Mattc1221 commented Apr 19, 2023

Mattc1221 commented Apr 29, 2023

kgabryje commented May 4, 2023

kgabryje commented May 4, 2023 •

edited

Loading

Mattc1221 commented May 15, 2023

Mattc1221 commented May 20, 2023

villebro commented May 21, 2023

villebro left a comment

v1bg commented May 22, 2023

Mattc1221 commented May 23, 2023

v1bg commented May 23, 2023

salvatorecolori commented May 24, 2024 •

edited

Loading

feat: Add deck.gl Heatmap Visualization #23551

feat: Add deck.gl Heatmap Visualization #23551

Conversation

Mattc1221 commented Apr 2, 2023 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

github-actions bot left a comment

Choose a reason for hiding this comment

villebro commented Apr 4, 2023

Mattc1221 commented Apr 16, 2023

rusackas commented Apr 17, 2023

codecov bot commented Apr 17, 2023 • edited Loading

Codecov Report

Mattc1221 commented Apr 19, 2023

kgabryje Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

kgabryje Apr 19, 2023

Choose a reason for hiding this comment

kgabryje commented Apr 19, 2023

Mattc1221 commented Apr 19, 2023

Mattc1221 commented Apr 29, 2023

kgabryje commented May 4, 2023

kgabryje commented May 4, 2023 • edited Loading

Mattc1221 commented May 15, 2023

Mattc1221 commented May 20, 2023

villebro commented May 21, 2023

villebro left a comment

Choose a reason for hiding this comment

v1bg commented May 22, 2023

Mattc1221 commented May 23, 2023

v1bg commented May 23, 2023

salvatorecolori commented May 24, 2024 • edited Loading

Mattc1221 commented Apr 2, 2023 •

edited

Loading

codecov bot commented Apr 17, 2023 •

edited

Loading

kgabryje Apr 19, 2023 •

edited

Loading

kgabryje commented May 4, 2023 •

edited

Loading

salvatorecolori commented May 24, 2024 •

edited

Loading