Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add endpoint to fetch available DBs #14208

Merged
merged 2 commits into from
Apr 23, 2021

Conversation

betodealmeida
Copy link
Member

@betodealmeida betodealmeida commented Apr 17, 2021

SUMMARY

We're working on redesigning the "Add Database" dialog so that users don't have to type a SQLAlchemy URI or figure out which driver should be used for each database. Instead, in the new flow:

  1. User clicks "Add Database".
  2. User chooses a database from the list of available databases, eg, "PostgreSQL". This list is built based on the installed SQLAlchemy dialects.
  3. User types in the DB-specific information. For PostgreSQL:
  • username
  • password
  • host
  • port
  • database name
  • additional parameters

These parameters are passed to the backend, which builds the SQLAlchemy URI depending on the selected database. Since the logic of building the SQLAlchemy URI is DB-dependent we opted against doing it in the frontend, and store the logic in each DB engine spec.

To make this work, this PR introduces:

  1. A BaseParametersMixin mixin for DB engine specs, which can be progressively added to DB engine specs in order to allow them to be configured via parameters instead of the SQLAlchemy URI. The mixin defines a schema for the parameters (so the frontend knows what do prompt), and methods to convert between the parameters and the SQLAlchemy URI.
  2. A schema mixin, DatabaseParametersSchemaMixin. The schema mixin handles the job of building the SQLAlchemy URI from the parameters before validating a payload representing a database. This way, APIs don't need to be updated to support the new way of configuring databases.
  3. A new endpoint /api/v1/database/available/ that list the available databases. Users can specify in superset/config.py a list of preferred databases, which will then be displayed prominently to the user (with logos, eg).

With this work we hope to simplify the task of adding a new database, removing a lot of the guesswork that is needed today. Users will immediately know which databases are available and what information is needed to configure them. And they will always have the option of falling back to a SQLAlchemy URI if they want or need.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Added unit tests and tested manually that test_connection works with the new payload.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@betodealmeida betodealmeida force-pushed the ch6945 branch 7 times, most recently from c2c4fe1 to 5cf45dc Compare April 20, 2021 00:27
@betodealmeida betodealmeida changed the title Ch6945 feat: add endpoint to fetch available DBs Apr 20, 2021
@betodealmeida betodealmeida marked this pull request as ready for review April 20, 2021 01:54
@codecov
Copy link

codecov bot commented Apr 20, 2021

Codecov Report

Merging #14208 (0d51139) into master (13bf023) will increase coverage by 0.02%.
The diff coverage is 89.69%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #14208      +/-   ##
==========================================
+ Coverage   76.13%   76.16%   +0.02%     
==========================================
  Files         945      945              
  Lines       47946    48030      +84     
  Branches     5950     5950              
==========================================
+ Hits        36503    36580      +77     
- Misses      11237    11244       +7     
  Partials      206      206              
Flag Coverage Δ
hive 80.41% <89.69%> (+0.02%) ⬆️
mysql 80.70% <89.69%> (+0.02%) ⬆️
postgres 80.73% <89.69%> (+0.02%) ⬆️
presto 80.44% <89.69%> (+0.03%) ⬆️
python 81.27% <89.69%> (+0.03%) ⬆️
sqlite 80.34% <89.69%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/__init__.py 65.90% <23.07%> (-15.91%) ⬇️
superset/config.py 91.06% <100.00%> (+0.03%) ⬆️
superset/databases/api.py 92.33% <100.00%> (+0.53%) ⬆️
superset/databases/schemas.py 99.51% <100.00%> (+0.04%) ⬆️
superset/db_engine_specs/base.py 88.38% <100.00%> (+0.88%) ⬆️
superset/db_engine_specs/postgres.py 96.90% <100.00%> (+0.06%) ⬆️
superset/db_engine_specs/presto.py 90.31% <0.00%> (+0.42%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13bf023...0d51139. Read the comment docs.

@@ -839,3 +841,67 @@ def function_names(self, pk: int) -> Response:
if not database:
return self.response_404()
return self.response(200, function_names=database.function_names,)

@expose("/available/", methods=["GET"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My initial thought is that this endpoint refers to a status of databases and not a new resource. Should we make this endpoint more restful and if so, how about something like GET /databases?status=available. On the other hand, I realize that we already have a lot of other existing routes that aren't particularly restful either, so then this route would be an outlier.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong opinion about this... @dpgaspar any thoughts here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the current pattern, i'd rather it be separate endpoint vs. trying to hotwire it into the database endpoint. It is cleaner for us to have endpoint -> command this will also help with error handling in the future.

Would love to hear what @dpgaspar has to say

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like we can leave this as is for consistency.

Copy link
Member

@eschutho eschutho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. I'll try to test it out with a front end POC... if you want to wait, that's cool, or merge and I can give you any real world feedback when I've had a chance to run it.

@@ -839,3 +841,67 @@ def function_names(self, pk: int) -> Response:
if not database:
return self.response_404()
return self.response(200, function_names=database.function_names,)

@expose("/available/", methods=["GET"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the current pattern, i'd rather it be separate endpoint vs. trying to hotwire it into the database endpoint. It is cleaner for us to have endpoint -> command this will also help with error handling in the future.

Would love to hear what @dpgaspar has to say

@betodealmeida betodealmeida merged commit e7ad03d into apache:master Apr 23, 2021
QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021
* feat: add endpoint to fetch available DBs

* Fix lint
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.2.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XL 🚢 1.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants