feat: Database Connection UI #14881

hughhhh · 2021-05-27T18:56:23Z

SUMMARY

Fully revamped the DB Connection Flow for users to easily add their databases to the metastore. With the new DB Connection Modal, we are able to build custom forms (dynamic forms) for whatever databases/engines we want. If their isn't a dynamic form available all dbs default to sqlalchemy for connecting.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Screen.Recording.2021-06-22.at.5.24.03.PM.mov

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

…ax/db-connection-ui

codecov · 2021-05-27T19:06:36Z

Codecov Report

Merging #14881 (bd97955) into master (7889564) will decrease coverage by 0.33%.
The diff coverage is 57.01%.

❗ Current head bd97955 differs from pull request most recent head b20882f. Consider uploading reports for the commit b20882f to get more accurate results

@@            Coverage Diff             @@
##           master   #14881      +/-   ##
==========================================
- Coverage   77.16%   76.82%   -0.34%     
==========================================
  Files         975      976       +1     
  Lines       50865    51217     +352     
  Branches     6740     6893     +153     
==========================================
+ Hits        39251    39349      +98     
- Misses      11398    11649     +251     
- Partials      216      219       +3

Flag	Coverage Δ
hive	`?`
javascript	`71.53% <55.76%> (-0.33%)`	⬇️
mysql	`81.65% <71.05%> (+<0.01%)`	⬆️
postgres	`81.67% <71.05%> (+<0.01%)`	⬆️
python	`81.75% <71.05%> (-0.29%)`	⬇️
sqlite	`81.28% <65.78%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rset-frontend/src/components/ErrorMessage/types.ts	`100.00% <ø> (ø)`
...rset-frontend/src/components/InfoTooltip/index.tsx	`100.00% <ø> (ø)`
...RUD/data/database/DatabaseModal/SqlAlchemyForm.tsx	`92.30% <0.00%> (-7.70%)`	⬇️
...set-frontend/src/views/CRUD/data/database/types.ts	`100.00% <ø> (ø)`
...set-frontend/src/views/CRUD/welcome/EmptyState.tsx	`87.17% <ø> (ø)`
...t-frontend/src/views/CRUD/welcome/SavedQueries.tsx	`60.60% <0.00%> (ø)`
superset/config.py	`97.03% <ø> (ø)`
superset/databases/commands/exceptions.py	`94.00% <ø> (ø)`
superset-frontend/src/views/CRUD/hooks.ts	`48.23% <35.00%> (-4.58%)`	⬇️
superset/db_engine_specs/bigquery.py	`87.75% <37.50%> (-2.18%)`	⬇️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7889564...b20882f. Read the comment docs.

* poc picker for db selection * working select * setup is loading for available dbs and step1 view * fix on close * update on fetch * remove unneeded code * add some styls

…ax/db-connection-ui

…uperset into pexdax/db-connection-ui

eschutho · 2021-06-29T22:41:57Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/DatabaseConnectionForm.tsx

+            id="selectedFile"
+            className="input-upload"
+            type="file"
+            onChange={async event => {


small nit, but this is a fairly long function in this event handler. Not necessarily required now, but I think it would read better if we pulled this out into a named function instead.

* chore: simplify errors and issue codes (#15437) * Fix issue number * Fix test

betodealmeida

This looks great! I left a few nits and TODOs that we can tackle after merging.

betodealmeida · 2021-06-30T23:39:16Z

superset/config.py

 PREFERRED_DATABASES: List[str] = [
-    # "PostgreSQL",
-    # "Presto",
-    # "MySQL",
-    # "SQLite",
+    "PostgreSQL",
+    "Presto",
+    "MySQL",
+    "SQLite",
    # etc.


Do we wanna curate this list better? I just wrote these of the top of my head.

I think these are fine for now

@yousoph what do you think?

betodealmeida · 2021-06-30T23:40:10Z

superset/databases/commands/validate.py

@@ -35,6 +35,8 @@
 from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
 from superset.models.core import Database

+BYPASS_VALIDATION_ENGINES = ["bigquery"]


Nit: make this a set

betodealmeida · 2021-06-30T23:41:53Z

superset/databases/schemas.py

+            serialized_encrypted_extra = (
+                data.get("encrypted_extra", "{}")
+                if data.get("encrypted_extra")
+                else "{}"
+            )


Nit:

Suggested change

serialized_encrypted_extra = (

data.get("encrypted_extra", "{}")

if data.get("encrypted_extra")

else "{}"

)

serialized_encrypted_extra = data.get("encrypted_extra") or "{}"

betodealmeida · 2021-06-30T23:44:47Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/ModalHeader.tsx

+          Your database was successfully connected! Here are some optional
+          settings for your database


All the strings in this file need to be translatable, but we can do that later.

betodealmeida · 2021-06-30T23:45:20Z

superset-frontend/src/views/CRUD/data/database/DatabaseModal/ExtraOptions.tsx

@@ -59,20 +60,20 @@ const ExtraOptions = ({
          <div>
            <h4>SQL Lab</h4>
            <p className="helper">
-              Configure how this database will function in SQL Lab.
+              Adjust how this database will interact with SQL Lab.


A few more here that need to be translatable.

…uperset into pexdax/db-connection-ui

* fix: add icons (#15122) * added alerts * revisions * added icon * spinner * added to error map

* fix: add icons (#15122) * added alerts * revisions * added icon * spinner * this will test * fixed test

* send parameters if they are available * fix bigquery sqlalchem issue * fix casting

github-actions · 2021-07-01T21:40:48Z

Ephemeral environment shutdown and build artifacts deleted.

eschutho and others added 13 commits May 20, 2021 15:52

split db modal file

05a1bb7

split db modal file

a8afce9

hook up available databases

b69bf16

add comment

6eae68c

split db modal file

e8cd4f4

hook up available databases

95917e4

use new validation component

e764072

first draft

842f856

use new validation component

7cb4098

first draft

253053b

first draft

a47d52d

merge

8cec32b

Merge branch 'master' of https://github.com/apache/superset into pexd…

5a785d7

…ax/db-connection-ui

pull-request-size bot added the size/L label May 27, 2021

using paxdax feature branch

92d81b9

hughhhh changed the title ~~[WIP] feat: Database Connection UI~~ feat: Database Connection UI May 28, 2021

lyndsiWilliams and others added 7 commits June 1, 2021 08:33

Creating IconButton

6d0c45f

Changed naming: logo is now icon

b0cdaba

Hard-coded inset values for ellipses

8d96434

Removed default SVG

3eea36d

Fixed test

b0ab2da

get tests passing

19a4c3a

Removed logo from test

c8b148e

hughhhh mentioned this pull request Jun 1, 2021

feat(db-connection-ui): Allow users to pick engine #14884

Merged

8 tasks

eschutho and others added 4 commits June 1, 2021 14:02

split db modal file

e7f0dbb

hook up available databases

b57d090

use new validation component

271c6e9

feat(db-connection-ui): Allow users to pick engine (#14884)

ef63e3c

* poc picker for db selection * working select * setup is loading for available dbs and step1 view * fix on close * update on fetch * remove unneeded code * add some styls

hughhhh force-pushed the pexdax/db-connection-ui branch from 2d2dd39 to ef63e3c Compare June 1, 2021 18:03

hughhhh added 2 commits June 29, 2021 14:16

Merge branch 'master' of https://github.com/apache/superset into pexd…

44504f5

…ax/db-connection-ui

Merge branch 'pexdax/db-connection-ui' of https://github.com/apache/s…

1ce8104

…uperset into pexdax/db-connection-ui

eschutho reviewed Jun 29, 2021

View reviewed changes

hughhhh and others added 6 commits June 29, 2021 15:57

fix: Update Invalid Port SIP-40 Reference (#15464)

5bfbfff

Merge branch 'master' into pexdax/db-connection-ui

4cae8cd

fix: Edit Modal not saving properly (#15468)

f8da689

fix: merge conflict (#15479)

ea48784

* chore: simplify errors and issue codes (#15437) * Fix issue number * Fix test

fix print on additional params

ea45822

Merge branch 'master' into pexdax/db-connection-ui

5639c51

betodealmeida approved these changes Jun 30, 2021

View reviewed changes

betodealmeida and others added 9 commits June 30, 2021 16:57

Merge branch 'master' into pexdax/db-connection-ui

7d81622

Merge branch 'pexdax/db-connection-ui' of https://github.com/apache/s…

bcfcdba

…uperset into pexdax/db-connection-ui

address comments

a673ab6

fix merge conflicts

b749d6d

fix pre-commit

0a4ad16

fix linting

46621fc

fix: error mapping in database connection (#15504)

1a16c49

* fix: add icons (#15122) * added alerts * revisions * added icon * spinner * added to error map

fix: RTL for Database Connections (#15513)

4fc5589

* fix: add icons (#15122) * added alerts * revisions * added icon * spinner * this will test * fixed test

fix: Allow for updates on Step 3 (#15512)

b20882f

* send parameters if they are available * fix bigquery sqlalchem issue * fix casting

hughhhh merged commit d4480f5 into master Jul 1, 2021

betodealmeida mentioned this pull request Sep 2, 2021

fix: impersonate user label/tooltip #16573

Merged

9 tasks

cccs-RyanS pushed a commit to CybercentreCanada/superset that referenced this pull request Dec 17, 2021

feat: Database Connection UI (apache#14881)

121f000

QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021

feat: Database Connection UI (apache#14881)

6c772e7

rusackas deleted the pexdax/db-connection-ui branch January 30, 2023 19:22

cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024

feat: Database Connection UI (apache#14881)

65536dd

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Database Connection UI #14881

feat: Database Connection UI #14881

hughhhh commented May 27, 2021 •

edited

Loading

codecov bot commented May 27, 2021 •

edited

Loading

eschutho Jun 29, 2021

betodealmeida left a comment

betodealmeida Jun 30, 2021

hughhhh Jul 1, 2021

betodealmeida Jun 30, 2021

betodealmeida Jun 30, 2021

betodealmeida Jun 30, 2021

betodealmeida Jun 30, 2021

github-actions bot commented Jul 1, 2021

		Your database was successfully connected! Here are some optional
		settings for your database

feat: Database Connection UI #14881

feat: Database Connection UI #14881

Conversation

hughhhh commented May 27, 2021 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented May 27, 2021 • edited Loading

Codecov Report

eschutho Jun 29, 2021

Choose a reason for hiding this comment

betodealmeida left a comment

Choose a reason for hiding this comment

betodealmeida Jun 30, 2021

Choose a reason for hiding this comment

hughhhh Jul 1, 2021

Choose a reason for hiding this comment

betodealmeida Jun 30, 2021

Choose a reason for hiding this comment

betodealmeida Jun 30, 2021

Choose a reason for hiding this comment

betodealmeida Jun 30, 2021

Choose a reason for hiding this comment

betodealmeida Jun 30, 2021

Choose a reason for hiding this comment

github-actions bot commented Jul 1, 2021

hughhhh commented May 27, 2021 •

edited

Loading

codecov bot commented May 27, 2021 •

edited

Loading