Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix parsing types and add check to test in future #2652

Merged
merged 8 commits into from
Feb 15, 2022

Conversation

tunetheweb
Copy link
Member

Brief summary of the change made

Fixes #2650 which was caused by statements which did not have a type set, so fell back to base type whereas rule L052 expected them to be statements.

I've also added a change to L052 to raise these errors in future instead of a debugging error that is not checked in CI.

I also added types to all other segments that were missing them.

Finally I added a check to the generate_parse_fixture_yml.py script to check if any segment is set to base or unparsable and alert in this case so we can catch this in future.

Are there any other side effects of this change that we should be aware of?

As per above.

Pull Request checklist

  • Please confirm you have completed any of the necessary steps below.

  • Included test cases to demonstrate any code changes, which may be one or more of the following:

    • .yml rule test cases in test/fixtures/rules/std_rule_cases.
    • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with tox -e generate-fixture-yml).
    • Full autofix test cases in test/fixtures/linter/autofix.
    • Other.
  • Added appropriate documentation for the change.

  • Created GitHub issues for any relevant followup/future enhancements if appropriate.

@codecov
Copy link

codecov bot commented Feb 15, 2022

Codecov Report

Merging #2652 (4ab839e) into main (67116c8) will not change coverage.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##              main     #2652   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          163       163           
  Lines        12000     12014   +14     
=========================================
+ Hits         12000     12014   +14     
Impacted Files Coverage Δ
src/sqlfluff/dialects/dialect_ansi.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_bigquery.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_mysql.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_postgres.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_redshift.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_snowflake.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_spark3.py 100.00% <100.00%> (ø)
src/sqlfluff/dialects/dialect_tsql.py 100.00% <100.00%> (ø)
src/sqlfluff/rules/L052.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 67116c8...4ab839e. Read the comment docs.

@tunetheweb
Copy link
Member Author

tunetheweb commented Feb 15, 2022

This is quite a big change, but most are YML changes, so suggest when reviewing you filter those out:

image

Then check the change, then if comfortable with the code changes, spot check a few YML file changes.

Copy link
Member

@barrywhart barrywhart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doh, tricky bug! Looks good overall. A few nitpicky suggestions about coding style.

src/sqlfluff/rules/L052.py Outdated Show resolved Hide resolved
src/sqlfluff/rules/L052.py Outdated Show resolved Hide resolved
if tree:

# Check we don't have any base types or unparsable sections
types = tree.type_set()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I've been meaning to add an unparsable check for ages. Stops code reviewers having to check for this. Pretty pleased we have this now. And checking for no base segments is an added bonus.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shall not parse!!

tunetheweb and others added 2 commits February 15, 2022 14:41
Co-authored-by: Barry Hart <barrywhart@yahoo.com>
Co-authored-by: Barry Hart <barrywhart@yahoo.com>
Copy link
Member

@barrywhart barrywhart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I have some suggestions about removing "segment" or "grammar" from the segment types -- cosmetic stuff.

src/sqlfluff/dialects/dialect_postgres.py Outdated Show resolved Hide resolved
src/sqlfluff/dialects/dialect_postgres.py Outdated Show resolved Hide resolved
src/sqlfluff/dialects/dialect_snowflake.py Outdated Show resolved Hide resolved
src/sqlfluff/dialects/dialect_tsql.py Outdated Show resolved Hide resolved
@tunetheweb tunetheweb merged commit 669061c into sqlfluff:main Feb 15, 2022
@tunetheweb tunetheweb deleted the fix-parsing-types branch February 15, 2022 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

L052 Error: unable to identify statement terminated by ;
2 participants