Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(explore): strip semicolons in virtual table SQL #13801

Merged
merged 11 commits into from
Apr 6, 2021

Conversation

pkdotson
Copy link
Member

SUMMARY

This pr fixes the error when a sql statement in the datasource editor is ended with semicolon. It would cause an error in explore.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Create a virtual datasource query and and save the dataset. Then click on the datasource to land in explore and the chart panel should not give any error.

ADDITIONAL INFORMATION

@codecov
Copy link

codecov bot commented Mar 25, 2021

Codecov Report

Merging #13801 (bc4718b) into master (541d23b) will decrease coverage by 0.14%.
The diff coverage is 75.00%.

❗ Current head bc4718b differs from pull request most recent head 77a4dcd. Consider uploading reports for the commit 77a4dcd to get more accurate results
Impacted file tree graph

@@            Coverage Diff             @@
##           master   #13801      +/-   ##
==========================================
- Coverage   78.35%   78.20%   -0.15%     
==========================================
  Files         934      934              
  Lines       47349    47348       -1     
  Branches     5941     5943       +2     
==========================================
- Hits        37101    37030      -71     
- Misses      10104    10174      +70     
  Partials      144      144              
Flag Coverage Δ
cypress 56.04% <14.28%> (+<0.01%) ⬆️
hive ?
mysql 80.56% <100.00%> (-0.01%) ⬇️
postgres 80.59% <100.00%> (-0.01%) ⬇️
presto 80.28% <100.00%> (-0.01%) ⬇️
python 80.86% <100.00%> (-0.28%) ⬇️
sqlite 80.16% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...et-frontend/src/SqlLab/components/TableElement.jsx 88.60% <ø> (+0.23%) ⬆️
...rontend/src/SqlLab/components/SqlEditorLeftBar.jsx 56.45% <71.42%> (+1.90%) ⬆️
superset/connectors/sqla/models.py 90.52% <100.00%> (-0.26%) ⬇️
superset/db_engines/hive.py 0.00% <0.00%> (-82.15%) ⬇️
superset/db_engine_specs/hive.py 74.42% <0.00%> (-16.42%) ⬇️
superset/utils/core.py 88.83% <0.00%> (-0.13%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 541d23b...77a4dcd. Read the comment docs.

@junlincc junlincc added the explore:dataset Related to the dataset of Explore label Mar 25, 2021
Copy link
Member

@etr2460 etr2460 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one code quality comment

superset/connectors/sqla/models.py Outdated Show resolved Hide resolved
@ktmud
Copy link
Member

ktmud commented Mar 26, 2021

I don't think you need a new method just for this. The fix should be as simple as this diff:

diff --git a/superset/connectors/sqla/models.py b/superset/connectors/sqla/models.py
index 2b61520cd..ea874167b 100644
--- a/superset/connectors/sqla/models.py
+++ b/superset/connectors/sqla/models.py
@@ -833,6 +833,8 @@ class SqlaTable(  # pylint: disable=too-many-public-methods,too-many-instance-at
         """
         Render sql with template engine (Jinja).
         """
+        if not self.sql:
+            return ""
         sql = self.sql
         if template_processor:
             try:
@@ -844,7 +846,7 @@ class SqlaTable(  # pylint: disable=too-many-public-methods,too-many-instance-at
                         msg=ex.message,
                     )
                 )
-        sql = sqlparse.format(sql, strip_comments=True)
+        sql = sqlparse.format(sql.strip("\t\r\n; "), strip_comments=True)
         if not sql:
             raise QueryObjectValidationError(_("Virtual dataset query cannot be empty"))
         if len(sqlparse.split(sql)) > 1:

@pkdotson pkdotson changed the title Fix: add method to strip semicolon fix(explore): add method to strip semicolon Mar 26, 2021
@pkdotson pkdotson requested a review from villebro March 26, 2021 21:25
@ktmud ktmud changed the title fix(explore): add method to strip semicolon fix(explore): strip semicolons in virtual table SQL Mar 26, 2021
@ktmud
Copy link
Member

ktmud commented Mar 26, 2021

Can we also add a unit test somewhere here:

def test_multiple_sql_statements_raises_exception(self):

You can create a new test case with table.sql ending in ; and assert ; is not in the output SQL generated by get_sqla_query.

tests/sqla_models_tests.py Outdated Show resolved Hide resolved
Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
Copy link
Member

@villebro villebro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

tests/sqla_models_tests.py Outdated Show resolved Hide resolved
Copy link
Member

@etr2460 etr2460 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one comment, but lgtm otherwise. and thanks for adding the test!

superset/connectors/sqla/models.py Outdated Show resolved Hide resolved
pkdotson and others added 5 commits March 30, 2021 13:29
@pkdotson pkdotson merged commit 34991f5 into apache:master Apr 6, 2021
lyndsiWilliams pushed a commit to preset-io/superset that referenced this pull request Apr 7, 2021
* add method to strip semicolon

* address comments

* test the test

* Update tests/sqla_models_tests.py

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>

* Update tests/sqla_models_tests.py

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* fix test

* add suggestion

* fix trailing space

* remove logger

* fix unit test

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
allanco91 pushed a commit to allanco91/superset that referenced this pull request May 21, 2021
* add method to strip semicolon

* address comments

* test the test

* Update tests/sqla_models_tests.py

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>

* Update tests/sqla_models_tests.py

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* fix test

* add suggestion

* fix trailing space

* remove logger

* fix unit test

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
QAlexBall pushed a commit to QAlexBall/superset that referenced this pull request Dec 29, 2021
* add method to strip semicolon

* address comments

* test the test

* Update tests/sqla_models_tests.py

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>

* Update tests/sqla_models_tests.py

Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>

* fix test

* add suggestion

* fix trailing space

* remove logger

* fix unit test

Co-authored-by: Jesse Yang <jesse.yang@airbnb.com>
Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.2.0 labels Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels explore:dataset Related to the dataset of Explore preset-io size/S 🚢 1.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants