feat: API endpoint to validate databases using separate parameters #14420

betodealmeida · 2021-04-30T00:11:34Z

SUMMARY

This PR adds a new endpoint at api/v1/database/validate_parameters for validating separate parameters when adding/editing a database. With this flow, users can add databases by passing DB-specific parameters (username, password, host, port, etc.) instead of the full SQLAlchemy URI.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TEST PLAN

Added unit tests for the new command ValidateDatabaseParametersCommand, as well as the new API.

ADDITIONAL INFORMATION

Has associated issue:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

superset/databases/commands/validate.py

superset/databases/api.py

hughhhh · 2021-05-11T01:57:56Z

superset/databases/commands/validate.py

+                "username": url.username,
+                "database": url.database,
+            }
+            errors = database.db_engine_spec.extract_errors(ex, context)


add logger.warning here

No need, raising exceptions in the API get logged by the error handler:

superset/superset/views/base.py

Lines 350 to 361 in 21cf12a

# SIP-40 compatible error responses; make sure APIs raise

# SupersetErrorException or SupersetErrorsException

@superset_app.errorhandler(SupersetErrorException)

def show_superset_error(ex: SupersetErrorException) -> FlaskResponse:

logger.warning(ex)

return json_errors_response(errors=[ex.error], status=ex.status)

@superset_app.errorhandler(SupersetErrorsException)

def show_superset_errors(ex: SupersetErrorsException) -> FlaskResponse:

logger.warning(ex)

return json_errors_response(errors=ex.errors, status=ex.status)

hughhhh

1 comment and 🚢

codecov · 2021-05-11T16:39:31Z

Codecov Report

Merging #14420 (b579f44) into master (09050ae) will decrease coverage by 0.01%.
The diff coverage is 89.55%.

@@            Coverage Diff             @@
##           master   #14420      +/-   ##
==========================================
- Coverage   77.40%   77.38%   -0.02%     
==========================================
  Files         958      959       +1     
  Lines       48326    48450     +124     
  Branches     5677     5680       +3     
==========================================
+ Hits        37405    37493      +88     
- Misses      10721    10757      +36     
  Partials      200      200

Flag	Coverage Δ
hive	`80.96% <89.55%> (+0.06%)`	⬆️
javascript	`72.51% <ø> (+0.04%)`	⬆️
mysql	`81.21% <89.55%> (+0.06%)`	⬆️
postgres	`81.25% <89.55%> (+0.06%)`	⬆️
presto	`?`
python	`81.62% <89.55%> (-0.10%)`	⬇️
sqlite	`80.85% <89.55%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...rset-frontend/src/components/ErrorMessage/types.ts	`100.00% <ø> (ø)`
superset/constants.py	`100.00% <ø> (ø)`
superset/db_engine_specs/postgres.py	`96.93% <ø> (ø)`
superset/databases/commands/validate.py	`76.00% <76.00%> (ø)`
superset/databases/api.py	`92.41% <93.75%> (+0.08%)`	⬆️
superset/db_engine_specs/base.py	`88.42% <96.42%> (+0.33%)`	⬆️
superset/config.py	`91.09% <100.00%> (ø)`
superset/databases/commands/exceptions.py	`94.00% <100.00%> (+0.81%)`	⬆️
superset/databases/schemas.py	`99.54% <100.00%> (+0.01%)`	⬆️
superset/db_engine_specs/bigquery.py	`95.87% <100.00%> (ø)`
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09050ae...b579f44. Read the comment docs.

john-bodley · 2021-07-23T04:26:58Z

superset/config.py

@@ -630,7 +630,7 @@ def _try_json_readsha(  # pylint: disable=unused-argument

 # Default row limit for SQL Lab queries. Is overridden by setting a new limit in
 # the SQL Lab UI
-DEFAULT_SQLLAB_LIMIT = 1000
+DEFAULT_SQLLAB_LIMIT = 10000


@betodealmeida any reason for the change from 1,000 to 10,000? We have had a number of employees mention that 10,000 overloads their local storage use and breaks Chrome and thus I'm unsure if the default of 10,000 makes sense.

I'm not sure, let me change the default back. Sorry for that!

…pache#14420) * feat: new endpoint for validating database parameters * Rebase * Remove broken tests

pull-request-size bot added the size/XL label Apr 30, 2021

betodealmeida force-pushed the ch6759 branch from 7fc7169 to 8a9c7d5 Compare April 30, 2021 00:22

betodealmeida changed the title ~~WIP~~ feat: API endpoint to validate databases using separate parameters Apr 30, 2021

betodealmeida marked this pull request as ready for review April 30, 2021 00:54

betodealmeida requested a review from hughhhh April 30, 2021 17:21

betodealmeida commented Apr 30, 2021

View reviewed changes

superset/databases/commands/validate.py Show resolved Hide resolved

hughhhh reviewed May 10, 2021

View reviewed changes

superset/databases/api.py Show resolved Hide resolved

hughhhh reviewed May 11, 2021

View reviewed changes

hughhhh self-requested a review May 11, 2021 02:00

hughhhh approved these changes May 11, 2021

View reviewed changes

feat: new endpoint for validating database parameters

0a70331

betodealmeida force-pushed the ch6759 branch 2 times, most recently from a5ffd8a to 6b9828f Compare May 11, 2021 16:37

betodealmeida force-pushed the ch6759 branch 7 times, most recently from c5a9205 to db8afac Compare May 12, 2021 04:03

Rebase

2ec1375

betodealmeida force-pushed the ch6759 branch from db8afac to 2ec1375 Compare May 12, 2021 22:33

Remove broken tests

b579f44

betodealmeida merged commit 31f406a into apache:master May 13, 2021

john-bodley reviewed Jul 23, 2021

View reviewed changes

betodealmeida mentioned this pull request Jul 23, 2021

fix: revert DEFAULT_SQLLAB_LIMIT to default #15869

Merged

8 tasks

cccs-RyanS pushed a commit to CybercentreCanada/superset that referenced this pull request Dec 17, 2021

feat: API endpoint to validate databases using separate parameters (a…

1eb66c1

…pache#14420) * feat: new endpoint for validating database parameters * Rebase * Remove broken tests

QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021

feat: API endpoint to validate databases using separate parameters (a…

ae953d4

…pache#14420) * feat: new endpoint for validating database parameters * Rebase * Remove broken tests

cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024

feat: API endpoint to validate databases using separate parameters (a…

7acea63

…pache#14420) * feat: new endpoint for validating database parameters * Rebase * Remove broken tests

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: API endpoint to validate databases using separate parameters #14420

feat: API endpoint to validate databases using separate parameters #14420

betodealmeida commented Apr 30, 2021 •

edited

Loading

hughhhh May 11, 2021

betodealmeida May 11, 2021

hughhhh left a comment

codecov bot commented May 11, 2021 •

edited

Loading

john-bodley Jul 23, 2021 •

edited

Loading

betodealmeida Jul 23, 2021

	# SIP-40 compatible error responses; make sure APIs raise
	# SupersetErrorException or SupersetErrorsException
	@superset_app.errorhandler(SupersetErrorException)
	def show_superset_error(ex: SupersetErrorException) -> FlaskResponse:
	logger.warning(ex)
	return json_errors_response(errors=[ex.error], status=ex.status)


	@superset_app.errorhandler(SupersetErrorsException)
	def show_superset_errors(ex: SupersetErrorsException) -> FlaskResponse:
	logger.warning(ex)
	return json_errors_response(errors=ex.errors, status=ex.status)

feat: API endpoint to validate databases using separate parameters #14420

feat: API endpoint to validate databases using separate parameters #14420

Conversation

betodealmeida commented Apr 30, 2021 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

hughhhh May 11, 2021

Choose a reason for hiding this comment

betodealmeida May 11, 2021

Choose a reason for hiding this comment

hughhhh left a comment

Choose a reason for hiding this comment

codecov bot commented May 11, 2021 • edited Loading

Codecov Report

john-bodley Jul 23, 2021 • edited Loading

Choose a reason for hiding this comment

betodealmeida Jul 23, 2021

Choose a reason for hiding this comment

betodealmeida commented Apr 30, 2021 •

edited

Loading

codecov bot commented May 11, 2021 •

edited

Loading

john-bodley Jul 23, 2021 •

edited

Loading