All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Bug fix for 1.4.3 which was incorrectly flagging L006 for concat operators (||
) and other two-symbol binary operators.
- Recognise || as an operator to avoid rule L006 flagging it #4168 @tunetheweb
- 🐛 Check verbosity level of pytest run before running certain tests #4167 @pwildenhain
- [snowflake] Add support for snowflake select * exclude/replace syntax #4160 @moreaupascal56
- Rewrote
diff-quality
plugin to run SQLFluff as a subprocess. More reliable, easier to switch betweendiff-quality
and runningsqlfluff lint
directly. - New rule L067 enforces consistent syntax for type casts.
- New rule L068 enforces a consistent number of columns in set queries (e.g. UNION).
- Initial support for Materialize dialect.
- Add flyway variables support via placeholder templater #4026 @srjonemed
- Fix Spark comparison parsing #4066 @ms32035
- Add errors and fails to pytest summary #4076 @alanmcruickshank
- Storage reference segment #4057 @YilangHe
- New rule L069: Consistent syntax for sql type casting #3747 @bolajiwahab
- Postgres: Views and named notations #4073 @davetapley
- Switch reflow buffer from LintFix to LintResult #4083 @alanmcruickshank
- Support parallel linting when many individual files specified #4084 @barrywhart
- Rule: check number of columns in set operations match #4028 @erevear
- Fix syntax in indentation sample code docs #4087 @PBalsdon
- Remove "mystery" binary file added in PR #2923 #4088 @barrywhart
- Fix mypy issue with regex #4097 @barrywhart
- Enable variable reference names to have leading underscore for snowflake dialect #4098 @gavin-tsang
- Fix flake8 issue with segment_predicates.py #4101 @barrywhart
- Fix bug in example rule plugin #4103 @barrywhart
- Fix bug where L034 should ignore INSERT or "CREATE TABLE AS SELECT" with CTE #4108 @barrywhart
- Postgres: Alter type rename value #4100 @greg-finley
- Bug fix: dbt templater ignores .sqlfluff file encoding on Windows #4109 @barrywhart
- Add initial Materialize dialect #4112 @bobbyiliev
- L015: Handle COUNT(DISTINCT(col)) #4110 @barrywhart
- [Snowflake] format type options extensions for copy_into_location #4129 @YilangHe
- Fix tox arguments #4144 @greg-finley
- [DB2] Fix parsing of string identifiers #4134 @borchero
- BigQuery: Allow double quoted literal in export_option_list #4126 @yoichi
- Only upload 3 sets of test results to codecov (possible workaround for hanging builds) #4147 @barrywhart
- SparkSQL: ILIKE #4138 @greg-finley
- SparkSQL: Mark
AS
as optional keyword for CTE & CTS #4127 @ulixius9 - Fix passenv to work with tox 4 #4154 @tunetheweb
- Allow deprecated --disable_progress_bar flag for fix command #4151 @pdebelak
- Implement diff_quality_plugin using command-line rather than Python #4148 @barrywhart
- L037: insert ASC just after column_reference #4149 @yoichi
- @srjonemed made their first contribution in #4026
- @ms32035 made their first contribution in #4066
- @davetapley made their first contribution in #4073
- @PBalsdon made their first contribution in #4087
- @gavin-tsang made their first contribution in #4098
- @bobbyiliev made their first contribution in #4112
- @ulixius9 made their first contribution in #4127
This release is less about internals and much more about some quality of life improvements and dialect changes. The most notable are:
- The introduction of a
sqlfluff render
command to preview the results of templated sql. - Linting errors within templated loops should now only appear once in the linting output.
- Indentation around jinja
{% set %}
statements should now be more consistent. - Linting errors around unparsable code are now more appropriately handled (with more to come soon on that front).
- Error messages when specified files aren't found are now more specific.
We've also got dialect improvements for Redshift, SOQL & SparkSQL.
- Fix type error in
get_rules
hook of plugin example #4060 @Samyak2 - L003: Add missing "pragma: no cover" #4058 @barrywhart
- Fix bug in sparksql SELECT statement termination at UNION #4050 #4052 @anna-azizian
- Deduplicate violations in the source space #4041 @alanmcruickshank
- Use "docker compose", not "docker-compose" #4055 @barrywhart
- Allow warnings for specific rules #4053 @alanmcruickshank
- Better file not found error #1023 #4051 @alanmcruickshank
- Filter out issues in unparsable sections #4032 @alanmcruickshank
- Snowflake: ADD and DROP without COLUMN #4049 @greg-finley
- Make render command #4043 @alanmcruickshank
- Bump after_n_builds to 10 #4046 @greg-finley
- Redshift: allows for parenthesis around FROM content #3962 @adam-tokarski
- Update CI to use Python 3.11 #4038 @greg-finley
- Classify self contained set statements as templated #4034 @alanmcruickshank
- Date and Datetime literals in SOQL #4037 @alanmcruickshank
- mypy edits for 0.990 #4035 @alanmcruickshank
- sparksql: support for create/remove widget clause #4021 @Coola4kov
- Redshift CREATE EXTERNAL FUNCTION statement #4011 @rpr-ableton
- Update Redshift bare functions #4012 @rpr-ableton
- @Coola4kov made their first contribution in #4021
- @anna-azizian made their first contribution in #4052
This is a fix to the configuration migration from 1.4.0. In that release, the configuration of leading/trailing operators would be migrated the wrong way around and precedence between new and old configuration values behaved unexpectedly.
- Config precedence #4007 @alanmcruickshank
- Redshift CREATE/ATTACH/DETACH/DROP RLS POLICY statements #4004 @rpr-ableton
- Redshift: Add support for APPROXIMATE functions #3997 @rpr-ableton
- hotfix to config migration #4005 @alanmcruickshank
This release brings several internal changes, and acts as a prelude to 2.0.0 which will be released fairly soon. In particular, the following config values have changed:
sqlfluff:rules:L007:operator_new_lines
has been changed tosqlfluff:layout:type:binary_operator:line_position
.sqlfluff:rules:comma_style
andsqlfluff:rules:L019:comma_style
have both been consolidated intosqlfluff:layout:type:comma:line_position
.
If any of these values have been set in your config, they will be automatically translated to the new values at runtime, and a warning will be shown. To silence the warning, update your config file to the new values. For more details on configuring layout (including some changes yet to come in future versions) see the docs.
These changes are driven by underlying centralisation in the routines which control layout. While for this release, no breaking changes are expected - you may find slight differences in how SQLFluff handles edge cases. We believe in the majority of cases these are more consistent, but if you find any which are problematic then do post an issue on GitHub as usual.
Other highlights from this release:
- Better dbt supportfor graph nodes and avoiding dependency conflicts.
- Numerous dialect improvements to T-SQL, MySQL, SparkSQL, SQLite, Athena Snowflake, Hive, Postgres & Databricks.
There have also been first time contributions from 10 new contributors! 🎉🎉🎉
- Snowflake partition nonreserved keyword #3972 @YilangHe
- Hive: Add support for EXCHANGE PARTITION in ALTER TABLE #3991 @nahuelverdugo
- Resolve parse error on multiple bracketed statements #3994 @yoichi
- Enable parsing of CLONE keyword in bigquery dialect #3984 @realLyans
- BigQuery: allow nesting of SetExpressionSegment #3990 @yoichi
- feat(clickhouse): Support non-standard CREATE TABLE statement #3986 @tomasfarias
- Fix Windows CI check #3992 @greg-finley
- Snowflake tag reference segment #3985 @WittierDinosaur
- Fix Jinja templater issue where undefined callable threw an exception #3982 @barrywhart
- Reflow Documentation V1 #3970 @alanmcruickshank
- Allow lambda argument columns to be unqualified #3971 @olagjo
- document inline configuration #3981 @alanmcruickshank
- [BUGFIX] Changing cwd temporarily on manifest load as dbt is not using project_dir to read/write target folder #3979 @barrywhart
- Fix type annotation of user_rules in
Linter
#3977 @Samyak2 - Unpin
markupsafe
#3967 @judahrand - Snowflake frame clause variables #3969 @WittierDinosaur
- SparkSQL: added support for : (colon sign) operator (Databricks SQL) #3956 @karabulute
- Athena: Add support for using underscore aliases #3965 @hectcastro
- Snowflake: ALTER TABLE constraint actions #3959 @erevear
- MySQL: Support REPLACE statement #3964 @yoichi
- TSQL: Add support for UPDATE STATISTICS option FULLSCAN #3950 @hectcastro
- ANSI: fixed typos in docstrings and comments #3953 @karabulute
- Postgres: ALTER PROCEDURE #3949 @krokofant
- T-SQL: Allow arbitrary expressions in PARTITION BY clause #3939 @borchero
- Enable dumping of performance information to csv. #3937 @alanmcruickshank
- Consolidate comma style configs #3945 @alanmcruickshank
- Adding missing KeywordSegments for different file types in Athena dialect #3898 @CommonCrisis
- Add templated block uuids #3936 @alanmcruickshank
- Load the full dbt manifest #3926 @davajm
- MySQL: Support optional "IF NOT EXISTS" with CREATE TRIGGER #3943 @yoichi
- T-SQL: Allow to parse SYNONYM statements #3941 @borchero
- Hive: Add support for LATERAL VIEW clause #3935 @hectcastro
- Fix crash in L042 on "UNION" or other "set" queries #3931 @barrywhart
- Refactor Lexer: Split apart elements_to_segments and refine placement of meta segments. #3925 @alanmcruickshank
- Add INSERT INTO <> DEFAULT VALUES to ANSI SQL and T-SQL #3934 @borchero
- Break apart reflow classes and extract methods #3919 @alanmcruickshank
- T-SQL: Fix indendentation of OUTER APPLY #3932 @borchero
- MySQL: Fix create trigger #3928 @adam-tokarski
- SparkSQL: Fixed bug with
QUALIFY
usage withoutWHERE
clause (applies also for Databricks dialect) #3930 @karabulute - T-SQL: Allow specifying join hints #3921 @borchero
- SQLite: Add support for table-level CHECK constraint #3923 @hectcastro
- sparksql: added * EXCEPT for SELECT clause #3922 @adam-tokarski
- Map old configs to new configs #3915 @alanmcruickshank
- [issue_3794] allow to use 'usage' as identifier for postgres #3914 @adam-tokarski
DATABRICKS
: Add Support for Delta Live Tables (DLT) Syntax #3899 @R7L208- Postgres Revoke fix #3912 @greg-finley
- fix: Click output to stderr on errors #3902 @KingMichaelPark
- fix issue with empty enum for postgres #3910 @adam-tokarski
- feat: Optional numerics for postgres arrays #3903 @KingMichaelPark
- fix(test): Return ParseExample namedtuple in get_parse_fixtures #3911 @tomasfarias
- Fix typos #3901 @kianmeng
- provide custom DeprecatedOption #3904 @adam-tokarski
- fix(redshift): Allow keywords in qualified references #3905 @tomasfarias
- Reflow centralisation 2: Rebreak (operators & commas) #3847 @alanmcruickshank
- Bring L008 into reflow work #3908 @alanmcruickshank
- Snowflake: Create network policy ip lists #3888 @greg-finley
- MySQL: Implement (key_part, ...) in index definitions #3887 @yoichi
- Reflow centralisation 1: Scope + Respace #3824 @alanmcruickshank
- Update github badge and add docker badge #3884 @alanmcruickshank
- @kianmeng made their first contribution in #3901
- @KingMichaelPark made their first contribution in #3903
- @hectcastro made their first contribution in #3923
- @karabulute made their first contribution in #3930
- @davajm made their first contribution in #3926
- @CommonCrisis made their first contribution in #3898
- @krokofant made their first contribution in #3949
- @Samyak2 made their first contribution in #3977
- @realLyans made their first contribution in #3984
- @nahuelverdugo made their first contribution in #3991
- @YilangHe made their first contribution in #3972
This is primarily a release for dialect fixes and improvements with additions and changes to TSQL, Snowflake, MySQL & Redshift. The other changes of note are:
- Support for warnings when users set old removed config values. This supports future change work by allowing a mechanism to warn if they are used.
- Improvements to the fix routines for L014 and L042 to handle some trickier cases.
- Add CreateUserSegment for Snowflake dialect #3880 @Gal40n04ek
- raw_segments_with_ancestors #3878 @alanmcruickshank
- Adjust TSQL Operators #3877 @alanmcruickshank
- Refactor path_to #3875 @alanmcruickshank
- Support Removed warning on configs #3874 @alanmcruickshank
- MySQL: Support column-path operator for JSON type #3864 @yoichi
- T-SQL: ALTER FUNCTION/PROCEDURE/VIEW parsing #3867 @differgroup
- MySQL: Support hexadecimal literals and bit value literals #3869 @yoichi
- MySQL: Treat double quotes the same as single quotes #3871 @yoichi
- Snowflake: COMMIT WORK #3862 @rglbr
- Snowflake: AlterShareStatementSegment and CreateDatabaseFromShareStatementSegment #3858 @moreaupascal56
- MySQL: Add CREATE/ALTER VIEW #3859 @wfelipew
- Redshift: CREATE TABLE LIKE enhancements #3853 @greg-finley
- L014 leading underscore capitalization inference fix #3841 @j-svensmark
- MySQL: Add extended DROP TRIGGER functionality #3846 @yoichi
- Allow standalone aliases in L027 #3848 @olagjo
- L042: Enable autofix for some tricky cases #3700 @barrywhart
- Snowflake: CREATE FUNCTION IF NOT EXISTS #3845 @greg-finley
- ignore functions with more than one element ... #3792 @fmms
- MySQL: support remaining constraint parts of CREATE/ALTER TABLE #3827 @yoichi
- @olagjo made their first contribution in #3848
- @j-svensmark made their first contribution in #3841
- @wfelipew made their first contribution in #3859
- @moreaupascal56 made their first contribution in #3858
- @rglbr made their first contribution in #3862
- @differgroup made their first contribution in #3867
- More refactoring of parse structures in preparation for upcoming refactor of formatting/whitespace rules.
- Fixes some bugs in L003 (indentation).
- New config flag
large_file_skip_byte_limit
which applies prior to loading the file.
- Snowflake: Fix syntax errors in tests #3834 @JamesRTaylor
- Add support for additional magic methods on DummyUndefined #3835 @barrywhart
- MySQL: support variable assignments by assignment operator := #3829 @yoichi
- MYSQL: improve lexing for single-quoted strings #3831 @mdahlman
- MySQL: More support for index definition in CREATE TABLE #3826 @yoichi
- Typed matching and ripping out the rest of .name #3819 @alanmcruickshank
- sparksql dialect to support lambda expressions (->) #3821 @juhoautio
- Fixed broken main branch #3825 @alanmcruickshank
- Enable file name logging for multi-files w/ --show-lint-violations flag #3788 @thechopkins
- Take database and schema out of Snowflake reserved keywords list #3818 @NiallRees
- Remove a chunk of name references #3814 @alanmcruickshank
- Fix typo in Snowflake dialect #3813 @Gal40n04ek
- Allow the use of libraries in macro definitions #3803 @bjgbeelen
- Indentation fixes and rule logging improvements #3808 @alanmcruickshank
- Fixes a recursion error in JinjaTemplater handling of undefined values #3809 @barrywhart
- Snowflake: extend
GRANT
syntax #3807 @Gal40n04ek - add warehouse_type in snowflake dialect #3805 @Gal40n04ek
- add Create Notification Integration syntax #3801 @Gal40n04ek
- T-SQL: fix parsing PARTITION BY NULL in window function #3790 @fmms
- SparkSQL: Update L014 rule to not flag Delta Change Data Feed Session & Table Property #3689 @R7L208
- Snowflake: OVER (ORDER BY) clause required for first_value (fixes #3797) #3798 @JamesRTaylor
- add Alter Pipe syntax for snowflake dialect #3796 @Gal40n04ek
- BigQuery: Parse WEEK() in date_part #3787 @yoichi
- Postgres: Support setting user properties using intrinsic ON & OFF values #3793 @chris-codaio
- extend SF dialect for File Format statements #3774 @Gal40n04ek
- Add QUALIFY to SparkSQL dialect #3778 @ThijsKoot
- fix regex for S3Path #3782 @Gal40n04ek
- Snowflake: add Optional parameter ERROR INTEGRATION for PIPE #3785 @Gal40n04ek
- Add a file size check in bytes #3770 @alanmcruickshank
- Require importlib_metadata >=1.0.0 #3769 @alanmcruickshank
- @Gal40n04ek made their first contribution in #3785
- @ThijsKoot made their first contribution in #3778
- @chris-codaio made their first contribution in #3793
- @JamesRTaylor made their first contribution in #3798
- @fmms made their first contribution in #3790
- @bjgbeelen made their first contribution in #3803
- @thechopkins made their first contribution in #3788
This release brings several potentially breaking changes to the underlying parse tree. For users of the cli tool in a linting context you should notice no change. If however your application relies on the structure of the SQLFluff parse tree or the naming of certain elements within the yaml format, then this may not be a drop-in replacement. Specifically:
- The addition of a new
end_of_file
meta segment at the end of the parse structure. - The addition of a
template_loop
meta segment to signify a jump backward in the source file within a loop structure (e.g. a jinja for loop). - Much more specific types on some raw segments, in particular
identifier
andliteral
type segments will now appear in the parse tree with their more specific type (which used to be calledname
) e.g.naked_identifier
,quoted_identifier
,numeric_literal
etc...
If using the python api, the parent type (such as identifier
) will still register if
you call .is_type("identifier")
, as this function checks all inherited types. However the
eventual type returned by .get_type()
will now be (in most cases) what used to be accessible
at .name
. The name
attribute will be deprecated in a future release.
Other highlights:
- New command-line option
--show-lint-violations
to show details on unfixable errors when runningsqlfluff fix
. - Improved consistency of process exit codes.
- Short CLI options for many common options.
- Jinja templater: When
--ignore=templating
is enabled, undefined Jinja variables now take on "reasonable" default values rather than blank string (""
). This can streamline initial rollout of SQLFluff by reducing or eliminating the need to configure templater variables.
There are also a ton of other features and bug fixes in this release, including first-time contributions from 11 new contributors! 🎉
- T-SQL: ALTER TABLE DROP COLUMN #3749 @greg-finley
- Add "# pragma: no cover" to work around sporadic, spurious coverage failure #3767 @barrywhart
- Add end_of_file and template_loop markers #3766 @alanmcruickshank
- Provide usage examples for new users #3765 @sirlark
- SQLite: deferrable in create table statement #3757 @RossOkuno
- When ignore=templating and fix_even_unparsable=True, provide defaults for missing vars #3753 @barrywhart
- BigQuery: Support Materialized Views #3759 @yoichi
- Enhance L062 to ignore blocked words in comments #3754 @tunetheweb
- Fix bug where undefined Jinja variable in macro file crashes linter #3751 @barrywhart
- Migrate analysis, functional and testing to utils #3743 @alanmcruickshank
- Build out rule crawling mechanisms #3717 @alanmcruickshank
- Add current_timestamp to Redshift as a bare function #3741 @RossOkuno
- BigQuery: Fix parsing parameterized data types #3735 @yoichi
- Change MySQL Create Statement Equals Segment to Optional #3730 @keyem4251
- SQLite: add parsing of INSERT statement #3734 @imrehg
- SPARKSQL: Support Delta Lake Drop Column Clause in
ALTER TABLE
#3727 @R7L208 - Add short versions of several cli options #3732 @alanmcruickshank
- Build out type hints in Grammars #3718 @alanmcruickshank
- dbt 1.3.0 compatibility #3708 @edgarrmondragon
- Revise no cover direction and remove unused code. #3723 @alanmcruickshank
- Update broken flattr link #3720 @alanmcruickshank
- BigQuery: remove
key
from unreserved keywords list #3719 @sabrikaragonen - Bigquery reset primary and foreign keys #3714 @sabrikaragonen
- Name Deprecation (Part 1) #3701 @alanmcruickshank
- Teradata: Add two TdTableConstraints #3690 @greg-finley
- Redshift: support expressions in array accessors #3706 @chronitis
- Handle logging issues at teardown #3703 @alanmcruickshank
- L028, L032: Fix bug where fixes were copying templated table names #3699 @barrywhart
- L042: Autofix sometimes results in "fix looping", hitting the linter "loop limit" #3697 @barrywhart
- L042: Address corner cases where fix corrupts the SQL #3694 @barrywhart
- T-SQL: Properly parse collation names #3686 @borchero
- Allow escaping single quotes in single-quoted literal with '' #3682 @pdebelak
- T-SQL: Fix indentation after JOIN/APPLY clauses with no ON statement #3684 @borchero
- T-SQL: Parse
DATEPART
date type as date type instead of column name #3681 @borchero - T-SQL: Allow
COLLATE
clause inJOIN
conditions #3680 @borchero - T-SQL: Fix parsing of CREATE VIEW statements with column name syntax #3669 @borchero
- Fix typo in github issue template #3674 @alanmcruickshank
- Add Athena issue label #3676 @greg-finley
- Set issue dialect labels via Github Actions #3666 @greg-finley
- Allow configuration of processes from config #3662 @alanmcruickshank
- Reposition before recursion in fixes to avoid internal error #3658 @alanmcruickshank
- Use UUIDs for matching #3661 @alanmcruickshank
- Postgres: Add dialect-specific bare functions #3660 @WittierDinosaur
- Postgres: Add
CALL
Support #3659 @WittierDinosaur - ANSI - Add support for
INTERSECT ALL
andEXCEPT ALL
#3657 @WittierDinosaur - Option to show errors on fix #3610 @chaimt
- L042: Fix internal error "Attempted to make a parent marker from multiple files" #3655 @barrywhart
- L026: Add support for
merge_statement
#3654 @barrywhart - Add handling for Redshift
CONVERT
function data type argument #3653 @pdebelak - Deduplicate files before and during templating #3629 @alanmcruickshank
- Rationalise Rule Imports #3631 @alanmcruickshank
- Handle Jinja
{% call ... %}
blocks #3648 @barrywhart - SPARKSQL: Add Delta Lake Constraints syntax to
ALTER TABLE
#3643 @R7L208 - Redshift: syntax for array unnesting with index #3646 @chronitis
- Snowflake -
ALTER TABLE IF EXISTS
andWHEN SYSTEM$STREAM_HAS_DATA()
#3641 @chrisalexeev - L057: In BigQuery, allow hyphens by default #3645 @barrywhart
- Better messages for partial indentation in L003 #3634 @pdebelak
- Add
INTEGER
toPrimitiveTypeSegment
for Sparksql #3624 @ciwassano - Bump version in gettingstarted.rst via the release script #3642 @greg-finley
- Improve handling of BigQuery hyphenated table names #3638 @barrywhart
- update sqlfluff version in gettingstareted.rst #3639 @keyem4251
- L016: Ignore jinja comments if
ignore_comment_clauses=True
#3637 @barrywhart - Add errors for redundant definitions. #3626 @alanmcruickshank
- Object Literals #3620 @alanmcruickshank
- Dialect Crumbs #3625 @alanmcruickshank
- Consistent return codes #3608 @alanmcruickshank
- @keyem4251 made their first contribution in #3639
- @ciwassano made their first contribution in #3624
- @chronitis made their first contribution in #3646
- @chaimt made their first contribution in #3610
- @borchero made their first contribution in #3669
- @sabrikaragonen made their first contribution in #3714
- @edgarrmondragon made their first contribution in #3708
- @imrehg made their first contribution in #3734
- @yoichi made their first contribution in #3735
- @RossOkuno made their first contribution in #3741
- @sirlark made their first contribution in #3765
This is primarily a bugfix release to resolve an issue with the 1.2.0 release
where the new version of sqlfluff-templater-dbt
relied on functionality
from the new version of sqlfluff
but the package configuration had not
been updated. Versions of the two packages are now pinned together.
- Pin sqlfluff-templater-dbt via release script #3613 @greg-finley
- Specifying comma delimited is unnecessary #3616 @alanmcruickshank
- Handle redshift temporary tables with # at the beginning of name #3615 @pdebelak
Major changes include:
- Adding AWS Athena as a dialect.
- A fix routine for L046 (whitespace in jinja tags), and the mechanisms for more source-only fixes in future.
- By default, large files (over 20k characters) are now skipped by sqlfluff. This limit is configurable and disable-able but exists as a sensible default to avoid the performance overhead of linting very large files.
- For the dbt templater, fatal compilation errors now no longer stop linting, and these files are now skipped instead. This enables projects to continue linting beyond the offending file and much better logging information to enable better debugging.
- Improve documentation for custom implemented rules #3604 @Aditya-Tripuraneni
- Add a skip and better logging for fatal dbt issues #3603 @alanmcruickshank
- Add large file check #3600 @alanmcruickshank
- Oracle: extend support for
ALTER TABLE
#3596 @davidfuhr - Immutability fixes #3428 @alanmcruickshank
- Struct type should be a segment #3591 @alanmcruickshank
- Fix Bracketed Struct issue #3590 @alanmcruickshank
- Allow spaces and multiple signs for numeric literals #3581 @tunetheweb
- Add source fixing capability and fix routines for L046 #3578 @alanmcruickshank
- Small grammar cleanup in team rollout docs #3584 @theianrobertson
- Postgres:
CREATE COLLATION
support #3571 @greg-finley - Redshift: Add
TOP X
to select clause modifiers #3582 @pdebelak - Postgres: Small fixes to
COMMENT ON
#3566 @greg-finley - Support MySQL system variables #3576 @qgallet
- Allow no alias for selects in CTEs with a column list #3580 @pdebelak
- New dialect AWS Athena #3551 @cmotta
- Split apart
fix_string()
. #3568 @alanmcruickshank - Fix incorrect L022 with postgres dialect with CTE argument list #3570 @pdebelak
- Simplify lint fixing (prep for source fixes) #3567 @alanmcruickshank
- Exclude .coverage.py from linting #3564 @zidder
- L016:
ignore_comment_clauses
not working for postgres dialect #3549 @barrywhart - Groundwork for a fix routine for L046 #3552 @alanmcruickshank
- Add better handling for SQLFluffUserError when running core cli commands #3431 @alanmcruickshank
- @pdebelak made their first contribution in #3570
- @cmotta made their first contribution in #3551
- @qgallet made their first contribution in #3576
- @theianrobertson made their first contribution in #3584
- @davidfuhr made their first contribution in #3596
- @Aditya-Tripuraneni made their first contribution in #3604
Major changes include:
- L066 - New rule to allow you to set min/max length requirements for aliases to ensure they are meaningful
- L062 - addition of
blocked_regex
as well asblocked_words
- L025 - fix several corner cases where aliases were removed inappropriately
- L059 is now disabled by default for Postgres
- Many more dialect improvements and bug fixes.
- L025: Derived query requires alias -- also handle UNION, etc. #3548 @barrywhart
- L025 should not remove aliases from derived queries #3546 @barrywhart
- T-SQL keyword functions should be treated as keywords #3540 @tunetheweb
- Fix issue where "--nocolor" is ignored [#3536)(sqlfluff#3536) @barrywhart
- Clickhouse: allow
FINAL
modifier #3534 @ThomAub - L018 change to just check for newlines rather than alignment #3499 @zidder
- SparkSQL: Update terminator grammar for
HAVING
,WHERE
,GROUP BY
#3526 @R7L208 - Fix L025 false positive for T-SQL
VALUES
clause #3533 @barrywhart - New rule L066 for enforcing table alias lengths #3384 @f0rk
- Redshift:
CALL
statement #3529 @greg-finley - Core: Compile regexes at init time to avoid overhead #3511 @judahrand
- Disable L059 by default for Postgres #3528 @tunetheweb
- Core: Add
MultiStringParser
to match a collection of strings #3510 @judahrand - SQLite:
PRIMARY KEY AUTOINCREMENT
#3527 @greg-finley - MySQL: Support
LOAD DATA
#3518 @greg-finley - Redshift:
GRANT EXECUTE ON PROCEDURES
#3516 @greg-finley - Allow
DEFAULT
expression in RedshiftALTER TABLE ADD COLUMN
statements #3513 @menzenski - BigQuery: Fix parsing of Array creation from full subquery #3502 @judahrand
- SparkSQL: Allow dateparts as table aliases #3500 @R7L208
- Fix
load_macros_from_path
to actually support multiple paths #3488 @emancu - Allow linter to apply fixes spanning more than 2 slices #3492 @barrywhart
- Fix L022 false positive when the CTE definition has a column list #3490 @barrywhart
- SparkSQL: Support for Delta
RESTORE
statement #3486 @R7L208 - Add values function to
SET
clause #3483 @hgranthorner - SparkSQL: Support for
CONVERT TO DELTA
command #3482 @R7L208 - BigQuery: Remaining procedural statements #3473 @tunetheweb
- Postgres: support grouping sets #3477 @tunetheweb
- SparkSQL: Support for Delta syntax to create manifest files through the
GENERATE
command #3478 @R7L208 - Add config for optionally indenting contents of
ON
blocks #3471 @PeterBalsdon - L026: check standalone aliases as well as table aliases #3470 @tunetheweb
- L045: Add handling for nested queries and CTEs #3468 @barrywhart
- L062: add
blocked_regex
support #3467 @tunetheweb - SparkSQL: Support for the Delta
DESCRIBE DETAIL
command #3465 @R7L208
- @PeterBalsdon made their first contribution in #3471
- @hgranthorner made their first contribution in #3483
- @emancu made their first contribution in #3488
- @judahrand made their first contribution in #3502
- @f0rk made their first contribution in #3384
- @zidder made their first contribution in #3499
- @ThomAub made their first contribution in #3534
This is the first stable release of SQLFluff 🎉🎉🎉.
- Does this mean there are no more bugs? No.
- Does this mean we're going to stop developing new features? No.
- Does this mean that this is a tool that is now broadly usable for many teams? Yes.
We've intentionally chosen to release 1.0.0 at a time of relative stability within SQLFluff and not at a time when new big structural changes are being made. This means that there's a good chance that this release is broadly usable. This also recognises that through the hard work of a huge number of contributors that we've built out this from a fringe tool, to something which gets over 500k downloads a month and over 4k stars on Github.
There's still a lot to do, and some more exciting things on the horizon. If you want to be part of this and join the team of contributors, come and hang out in our slack community or on our twitter account where people can help you get started. If you're a long time user, keep submitting bug reports and inputting on issues on Github.
If you've never used SQLFluff before, or are hesitant about starting to use it in your day to day work, now might be a good time to try it. We have guides on how to get started with the tool, and how to get started with rolling out to a team in our docs.
- Swap to skip file if not found #3464 @alanmcruickshank
- Postgres: treat
GENERATE_SERIES
as a value table function #3463 @tunetheweb - Postgres: Support multiple
CONSTRAINTS
inCREATE DOMAIN
#3460 @tunetheweb - Redshift: Add
ANYELEMENT
support #3458 @tunetheweb - Postgres: Optional select clause elements and better
ON CONFLICT
support #3452 @tunetheweb - Add ClickHouse as a dialect #3448 @tunetheweb
- Postgres: allow keywords in qualified column references #3450 @tunetheweb
- Remove Baron Schwatz Dead Link #3453 @tunetheweb
- Postgres: Finish
ALTER TYPE
#3451 @greg-finley - SparkSQL: Add Delta Syntax for
DESCRIBE HISTORY
statement #3447 @R7L208 - Snowflake: Support Stage data file parameters in
FROM
clauses #3446 @tunetheweb - Redshift: Support Object unpivoting #3441 @tunetheweb
- Python script to automate release #3403 @greg-finley
- Remove Delta Lake Reference in README.md #3444 @R7L208
- Add
databricks
dialect as an alias forsparksql
dialect #3440 @R7L208 - Make all Postgres identifiers quotable #3442 @tunetheweb
- Update JinjaAnalyzer and JinjaTracer to handle
{% block %}
#3436 @barrywhart - SparkSQL: Add support for Delta
VACUUM
statement #3439 @R7L208 - Warning for parsing errors extended to all dialects #3411 @mdahlman
- Handle templater validation errors more gracefully #3433 @alanmcruickshank
- MYSQL: allow for escaped single quotes #3424 @mdahlman
- L027: Fix false positives by reverting the PR for issue #2992: Check table aliases exist #3435 @barrywhart
- Allow
numeric_dollar
templater to have curly braces, updatedollar
+numeric_dollar
templater examples in docs #3432 @menzenski - Allow Redshift
IDENTITY
column(seed, step)
to be optional #3430 @menzenski - L036: Make wildcard behavior configurable #3426 @barrywhart
- L034: Don't autofix if numeric column references #3423 @barrywhart
- L036: Treat wildcard as multiple select targets (i.e. separate line) #3422 @barrywhart
- Snowflake: IDENTIFIER pseudo-function #3409 @mdahlman
- SNOWFLAKE: Fully referenced object names in clone statements #3414 @mdahlman
- Unpin coverage now issue with 6.3 has been resolved #3393 @tunetheweb
- L045: handle
UPDATE
statements with CTEs #3397 @tunetheweb - L027: Add support for
ignore_words
#3398 @dmohns - Postgres: Allow
CREATE FUNCTION
to use Expressions in default values #3408 @tunetheweb - Fix bug in
apply_fixes()
with leading/trailing whitespace #3407 @barrywhart - Redshift: Correct
ALTER TABLE
syntax #3395 @tunetheweb - Postgres: Parse index with column sort #3405 @greg-finley
- MySQL: Improve SET Syntax for Variable Assignment #3394 @mdahlman
- Handle Postgres-style type casts when using the
colon_nospaces
templating style #3383 @benji-york - Capitalization in help message #3385 @mdahlman
- MySQL: Update keywords #3381 @mdahlman
- Teradata: Database statement and Set Session Database #3382 @mdahlman
- @benji-york made their first contribution in #3383
- @menzenski made their first contribution in #3430
Major changes include:
- Fix bug causing L003 to report indentation errors for templated code - sorry we know that one's caused many of you some grief :-(
- Initial support of SOQL (Salesforce Object Query Language).
- Additional Placeholder templating options.
- Start of BigQuery procedural language support (starting simple
FOR
statements andCREATE PROCEDURE
statements). - New rule L065 to put set operators onto new lines.
- Many more dialect improvements and bug fixes.
- All dialects: Allow
RESPECT NULLS
/IGNORE NULLS
in window functions #3376 @tunetheweb - Postgres: correct
RETURNS TABLE
column type #3379 @tunetheweb - L065: Add rule for set operators surrounded by newlines #3330 @dmohns
- L064: Apply preferred quote-style for partially templated quoted literals #3300 @dmohns
- BigQuery: Support Stored Procedures #3369 @tunetheweb
- MySQL extra Boolean operators (
&&
,||
,!
) #3359 @mdahlman - Postgres and Redshift: Support
LOCK [TABLE]
#3350 @tunetheweb - Placeholder updates: Allow optional braces in dollar placeholders, add
colon_nospaces
, and cast to string #3354 @tunetheweb - BigQuery: Basic
FOR..IN..DO...END FOR
support #3340 @tunetheweb - L025: exclude
VALUES
clauses #3358 @tunetheweb - GitHub Actions: Update existing PR on new runs #3367 @greg-finley
- GitHub Actions: Copy draft release notes to CHANGELOG #3360 @greg-finley
- GitHub Action to set version number #3347 @greg-finley
- Postgres and Redshift: support
ALTER SCHEMA
#3346 @mdahlman - MySQL: better
SELECT..INTO
support #3351 @tunetheweb - Postgres: support better function calls in
CREATE TRIGGER
#3349 @tunetheweb - Misc rule doc updates #3352 @tunetheweb
- Snowflake: Move
CASE
keyword to Unreserved list #3353 @tunetheweb - MySQL: Added support for multiple variables in
SET
statement. #3328 @cgkoutzigiannis - SOQL: Support
date_n_literal
#3344 @greg-finley - Update Docs: Getting Started and Index #3339 @mdahlman
- SOQL: Disable L026 rule #3338 @greg-finley
- Fix critical parse error logged after L003 fix #3337 @barrywhart
- SOQL: Disallow non-
SELECT
statements #3329 @greg-finley - ci: bump github actions #3336 @Fdawgs
- Start SOQL dialect #3312 @greg-finley
- Hive: support
CLUSTER
,DISTRIBUTE
,SORT BY
#3304 @barunpuri - Fix typo in Configuration documentation #3319 @mdahlman
- L011: Support
MERGE
statements #3292 @tunetheweb - BigQuery: Add workaround to fix false-positves of L063 #3306 @dmohns
- Snowflake:
REMOVE
statement rework #3308 @jmc-bbk - Snowflake:
PUT
statement #3307 @jmc-bbk - Snowflake:
GET
statement #3305 @jmc-bbk - Snowflake: Support
ALTER EXTERNAL TABLE
#3302 @jmc-bbk - T-SQL: Fix
PIVOT
placement #3298 @jpers36 - Cleanup role references #3287 @tunetheweb
- Adding Typeform and videoask into inthewild.rst #3296 @omonereo-tf
- Snowflake:
LIST
statement enhancement #3295 @jmc-bbk - MySQL: Support
CREATE USER
#3289 @greg-finley - Snowflake: CREATE STAGE grammar enhancement for file formats #3293 @jmc-bbk
- T-SQL: Complete support for
DELETE
statement #3285 @pguyot - MySQL: Support account names #3286 @greg-finley
- L028: In T-SQL dialect, table variables cannot be used to qualify references #3283 @barrywhart
- L007: An operator on a line by itself is okay #3281 @barrywhart
- L046 (spaces around Jinja tags) should check all slices in a segment #3279 @barrywhart
- L003 bug fix: Not ignoring templated newline #3278 @barrywhart
- @omonereo-tf made their first contribution in #3296
- @mdahlman made their first contribution in #3319
- @cgkoutzigiannis made their first contribution in #3328
Major changes include:
- Addition of "rule groups" (currently
core
andall
) to allow ease of turning on and off groups of rules. - Addition of
db2
dialect - PRS errors are now highlighted in red.
- Many bugs fixes and dialect improvements
- Allow optional
AS
inMERGE
statements usingSELECT
#3276 @tunetheweb - Add groups each rule is in to Rules documentation #3272 @tunetheweb
- BigQuery: Enhanced
EXPORT DATA
statement #3267 @tunetheweb - BigQuery:
CREATE TABLE
support forCOPY
andLIKE
#3266 @tunetheweb - Improve error on missing keywords #3268 @greg-finley
- MySQL: Add
FLUSH
support #3269 @greg-finley - Postgres: Add
ALTER TYPE
support #3265 @greg-finley - Bug fix: L036 handle single-column
SELECT
with comment on same line asSELECT
keyword #3259 @barrywhart - Put working example in the README #3261 @greg-finley
- Snowflake: Add
CREATE FILE FORMAT
Support #3104 @jmc-bbk - Bug fix: Disable L059 in snowflake dialect #3260 @barrywhart
- Remove redundant
Anything()
frommatch_grammars
#3258 @tunetheweb - Postgres: Add
DOMAIN
support #3257 @tunetheweb - T-SQL: Allow optional brackets with
EXECUTE
#3255 @pguyot - Add rule groups and a core rules group #3142 @pwildenhain
- MySQL: Better
UNSIGNED
support #3250 @tunetheweb - MySQL (and others): Support
DROP TEMPORARY TABLE
#3251 @tunetheweb - Add Db2 dialect #3231 @ddresslerlegalplans
- BigQuery: Add
CREATE EXTERNAL TABLE
statement #3241 @dmohns - SQLite: Add support for expressions in
CREATE INDEX
columns #3240 @tunetheweb - Fix exception in
check_still_complete
and matching inStartsWith
#3236 @tunetheweb - Snowflake: Add Support for
DROP
Statements #3238 @chrisalexeev - Allow YAML generation script to accept arguments when run through
tox
#3233 @tunetheweb - Bug fix: Cleanly catch and report errors during
load_macros_from_path
#3239 @barrywhart - Indent procedure parameters #3234 @fdw
- Enhance
apply_fixes()
to automatically fix violations ofcan_start_end_non_code
#3232 @barrywhart - T-SQL: Fix
for xml path
identifier #3230 @fdw - SparkSQL: Additional Delta Merge Test Cases #3228 @R7L208
- Fix bug where L018 warns inappropriately if CTE definition includes a column list #3227 @barrywhart
- BigQuery: Better
STRUCT
support #3217 @tunetheweb - Fix bug where L003 and L036 fixes caused a parse error #3221 @barrywhart
- Make
IF EXISTS
work withUNION
selects #3218 @fdw - Fix bug where the
fix_even_unparsable
setting was not being respected in.sqlfluff
#3220 @barrywhart - BigQuery: Better
DELETE
table support #3224 @tunetheweb - Snowflake:
ALTER MATERIALIZED VIEW
statement #3215 @jmc-bbk - BigQuery: recognise
DATE
,DATETIME
andTIME
as a date parts forEXTRACT
#3209 @tunetheweb - Postgres: enhanced
UPDATE
statement support #3203 @tunetheweb - Prevent Date Constructors from being changed to double quotes by L064 #3212 @tunetheweb
- Postgres: Fix
DROP EXTENSION
syntax #3213 @tunetheweb - Snowflake: Handle
FLATTEN()
table function aliases correctly in L025, L027, L028 #3194 @barrywhart - Snowflake: Function
LANGUAGE SQL
#3202 @WittierDinosaur - Postgres: Enhanced
CREATE MATERIALIZED VIEW
#3204 @tunetheweb - T-SQL: Support basic
FOR XML
statements #3193 @fdw - T-SQL: Fix cursor syntax #3192 @fdw
- Snowflake:
REMOVE
statement enhancement #3191 @jmc-bbk - Snowflake: Moved
VIEW
to unreserved keywords #3190 @WittierDinosaur - BigQuery: Support
EXPORT DATA
#3177 @tunetheweb - T-SQL: Fix exception when using variable names in
FROM
clause #3175 @tunetheweb - Fix bug where
encoding
setting in .sqlfluff file was not being respected #3170 @barrywhart - Highlight
PRS
errors in red #3168 @OTooleMichael - Remove unnecessary
StartsWith
and maketerminator
mandatory when using it #3165 @tunetheweb - Postgres: Support Composite Types #3167 @tunetheweb
- T-SQL: Support opening, closing, deallocating and fetching cursors #3166 @fdw
- T-SQL: Add declaration of cursors #3164 @fdw
- Missed #3151 from CHANGELOG #3163 @tunetheweb
- Bug fix: L028 sometimes makes incorrect fix when there are subqueries #3156 @barrywhart
- T-SQL: Support
OUTPUT INTO
#3162 @fdw - T-SQL: Add
CREATE TYPE
statement #3154 @fdw - Hive: Support
TABLESAMPLE
#3159 @barunpuri - Hive: Support back quoted identifier and literal #3158 @barunpuri
- T-SQL: Add table hints to
INSERT
andDELETE
#3155 @fdw
- @ddresslerlegalplans made their first contribution in #3231
- @greg-finley made their first contribution in #3261
Major changes include:
- New Rule (L064) for preferred quotes for quoted literals
- Rule speed improvements and fixing performance regression from 0.12.0
- Add configuration option to disallow hanging indents in L003
- Add
ignore_words_regex
configuration option for rules - New GitHub Annotations option
- Many bug fixes and dialect improvements
- Redshift: Fix CREATE TABLE column constraints and COPY #3151 @tunetheweb
- New Rule L064: Consistent usage of preferred quotes for quoted literals #3118 @dmohns
- L025 bug fix: stop incorrectly flagging on nested inner joins #3145 @tunetheweb
- T-SQL: Add labels, as well as
GRANT
/DENY
/REVOKE
#3149 @tunetheweb - Snowflake: allow bracketless
VALUES
inFROM
clauses #3141 @tunetheweb - T-SQL: Support
TRY_CONVERT
#3143 @fdw - T-SQL: Support
NVARCHAR(MAX)
#3130 @fdw - Allow column-less
INSERT INTO
with bracketedSELECT
in ANSI and BigQuery #3139 @tunetheweb - Hive: Support dynamic partition insert #3126 @barunpuri
- T-SQL -
ALTER TABLE
- add support forWITH CHECK ADD CONSTRAINT
andCHECK CONSTRAINT
#3132 @nevado - TSQL: Support names for transactions #3129 @fdw
- Snowflake:
StartsWith()
inFromExpressionElementSegment
caused performance issues for large queries #3128 @WittierDinosaur - Fix parsing of Compound Statements #3121 @jonyscathe
- SparkSQL: Update to support all valid Literal Types #3102 @R7L208
- TSQL: Exclude non-function-name keywords from function names #3112 @jpers36
- ANSI
AT TIME ZONE
parsing improvements #3115 @tunetheweb - When fixing a file, preserve the input file's permissions #3114 @barrywhart
- Bug: L058 (flatten nested
CASE
) triggers incorrectly (theELSE
contains additional code) #3113 @barrywhart - Bug fix: Handle "lint" human-format file output correctly #3109 @barrywhart
- L003: Add configuration option to disallow hanging indents #3063 @dmohns
- Add native Github-actions output #3107 @dmohns
- Improved signed literal parsing #3108 @tunetheweb
- Don't allow fixes to span template blocks #3105 @barrywhart
- Add
ignore_words_regex
configuration option #3098 @dmohns - Redshift: Better
AT TIME ZONE
support #3087 @tunetheweb - Fix In The Wild typo #3100 @sivaraam
- Snowflake: Add Create Storage Integration grammar. #3075 @jmc-bbk
- ANSI: Allow
indented_using_on
inMERGE
statementsON
#3096 @dmohns - Postgres: Support
COLLATE
in more clauses #3095 @tunetheweb - BigQuery: Support
NORMALIZE
function #3086 @tunetheweb - ANSI (and other dialects): Add
DROP FUNCTION
support #3082 @tunetheweb - Postgres: Support
DROP EXTENSION
#3083 @tunetheweb - Snowflake: Fix bug in Describe Statement #3076 @jmc-bbk
- Update individual rules to take advantage of core rule processing changes #3041 @barrywhart
- L003 forgives non misbalanced Jinja tags #3065 @OTooleMichael
- Fix tsql dialect
EXEC = @Variable StoredProc
Failed Parsing Bug (#3070) #3077 @MartynJones87 - Snowflake Dialect: Add External Function DDL #3071 @chrisalexeev
- SparkSQL: Support for Delta
UPDATE
statement syntax #3073 @R7L208 - SparkSQL: Test cases for Delta
DELETE FROM
syntax #3072 @R7L208 - Postgres: Support quoted
LANGUAGE
params #3068 @tunetheweb - Fix bug handling Jinja set with multiple vars, e.g.:
{% set a, b = 1, 2 %}
#3066 @barrywhart - L007 should ignore templated newlines #3067 @barrywhart
- Allow aliases to pass L028 #3062 @tunetheweb
- Refactor core rule processing for flexibility and speed #3061 @barrywhart
- Add editorconfig and precommit for SQL and YML files #3058 @tunetheweb
- Rule L003 performance: Cache the line number and last newline position #3060 @barrywhart
- Fixed documentation for
sql_file_exts
example #3059 @KulykDmytro - BigQuery: Support
SAFE
functions #3048 @tunetheweb - Postgres: Fix
UNNEST
for L025 #3054 @tunetheweb - Exasol:
CREATE/DROP/ALTER USER/ROLE
clean up for consistency #3045 @tunetheweb - Postgres add
ALTER ROLE
/ALTER USER
support #3043 @mrf - Add CarePay to SQLFluff in the wild #3038 @pvonglehn
- Postgres: Add
ON CONFLICT
Grammar #3027 @jmc-bbk - Add dialect to Docker test #3033 @tunetheweb
- @chrisalexeev made their first contribution in #3071
- @MartynJones87 made their first contribution in #3077
- @sivaraam made their first contribution in #3100
- @jonyscathe made their first contribution in #3121
- @barunpuri made their first contribution in #3126
Major changes include:
- Dialect is now mandatory, either in command line, or in config BREAKING CHANGE
- Rename
spark3
dialect tosparksql
BREAKING CHANGE - L027 now checks tables references exist BREAKING CHANGE
- New rule L063 to allow Datatypes to have a different capitalisation policy from L010. BREAKING CHANGE
- Refactor and performance improvements of Delimited and L003
- Many dialect improvements and fixes
- MySQL: Allow
JOIN
s inUPDATE
expressions #3031 @zapion - Fix bug in patch generation for segments made of templated + literal fixes #3030 @barrywhart
- Formatters code cleanup #3029 @barrywhart
- Postgres better
CREATE USER
/CREATE ROLE
support #3016 @mrf - SparkSQL: Add
MERGE
syntax #3025 @PhilippLange - Remove Delimited workarounds #3024 @tunetheweb
- Add
exclude
option forRef
grammar #3028 @tunetheweb - SparkSQL: Adding support for Delta Lake table schema updates #3013 @R7L208
- L027: Check table aliases exists #2998 @dmohns
- Snowflake: Added support for
REMOVE
statements #3026 @WittierDinosaur - BigQuery: Support
WEEK
function with days of weeks #3021 @tunetheweb - Sparksql quoted identifier in
STRUCT
#3023 @PhilippLange - Force user to specify a dialect #2995 @barrywhart
- BigQuery: Parse
CREATE TABLE
with trailing comma #3018 @dmohns - Snowflake: Add
IS (NOT) DISTINCT FROM
test cases #3014 @kd2718 - BigQuery: Add support for column
OPTIONS
inSTRUCT
definitions #3017 @dmohns - PostgreSQL: added support for
CREATE ROLE
andDROP ROLE
statements #3010 @dnim - Separate slow CI job to it's own workflow #3012 @tunetheweb
- SparkSQL: Test cases for Delta Variation of Writing a table #3009 @R7L208
- Snowflake: Added support for
CLUSTER BY
and otherCREATE TABLE
improvements #3008 @WittierDinosaur - Support
TRIM
function parameters #3007 @tunetheweb - BigQuery: Support
AT TIME ZONE
inEXTRACT
#3004 @tunetheweb - BigQuery: Move some keywords to unreserved #3002 @tunetheweb
- BigQuery: Allow quoted variable names in
DECLARE
#3006 @dmohns - T-SQL: allow optional
AS
keyword in parameters def #3001 @pguyot - T-SQL: add support for
RETURNS @var TABLE
syntax #3000 @pguyot - T-SQL: add support for parenthesized nested joins #2993 @pguyot
- dbt: Read builtins from code #2988 @dmohns
- T-SQL: fix table type in
DECLARE
statements #2999 @pguyot - T-SQL: allow leading
GO
#2997 @pguyot - T-SQL: add support for assignment operators #2996 @pguyot
- BigQuery: Add more
MERGE
statement variants #2989 @dmohns - L041: Fix bug when there is a newline after the select clause modifier #2981 @jmc-bbk
- Rule L045 doesn't recognise CTE usage in a subquery when rule L042 is enabled #2980 @barrywhart
- dbt: Make
is_incremental()
defaults consistent #2985 @dmohns - Rename Grammars for consistency #2986 @tunetheweb
- Added support for MySQL
UPDATE
Statements #2982 @WittierDinosaur - Redshift: Added
CREATE EXTERNAL SCHEMA
, bugfix inPARTITION BY
#2983 @WittierDinosaur - Added
ALTER INDEX
andREINDEX
to Postgres, Some Grammar Cleaning #2979 @WittierDinosaur - T-SQL grammar deduplication #2967 @tunetheweb
- L003 Refactor #2884 @OTooleMichael
- Delimited Refactor #2831 @WittierDinosaur
- SparkSQL: Support for querying snapshots when reading data with Delta Lake #2972 @R7L208
- Fix bug in L063 for BigQuery
STRUCT
params #2975 @tunetheweb - Fix assertion error in dbt templater when file ends with whitespace strip (
-%}
) #2976 @barrywhart - Pass dbt vars to dbt #2923 @tcholewik
- BigQuery: Add support for column
OPTIONS
#2973 @dmohns - BigQuery: Allow expressions in
OPTIONS
clauses #2971 @dmohns - Bump black to 22.3.0 on pre-commit #2969 @pguyot
- T-SQL: Redefine
DatatypeIdentifierSegment
#2959 @alanmcruickshank - T-SQL: Add support for
WAITFOR
statement #2968 @pguyot - T-SQL: Add
WHILE
statement support #2966 @pguyot - T-SQL:
INTO
is optional withinINSERT
statement #2963 @pguyot - Add basic
IS (NOT) DISTINCT FROM
support in most dialects #2962 @tunetheweb - SparkSQL: Create Table Delta Lake Variant #2954 @R7L208
- T-SQL: Add support for
CREATE
/DROP
/DISABLE
TRIGGER
#2957 @tunetheweb - Bug: L042 modifies parse tree even during "lint" #2955 @barrywhart
- Allow multiple post function clauses in Postgres and Redshift #2952 @aviv
- Fix bug in L022 for trailing comments in CTE #2946 @tunetheweb
- More dialect checking, fixes, inheritance cleanup #2942 @barrywhart
- T-SQL: Support
OUTPUT
Params andGOTO
Statements #2949 @tunetheweb - BREAKING CHANGE: change existing dialect name from
spark3
tosparksql
#2924 @R7L208 - Add Symend to SQLFluff In The Wild #2940 @HeyZiko
- Simplify segment creation and inheritance in dialects #2933 @barrywhart
- Snowflake: Add
ALTER STREAM
support #2939 @HeyZiko - T-SQL: Handle multiple nested joins #2938 @tunetheweb
- Snowflake: Add
CREATE STREAM
support #2936 @HeyZiko - T-SQL: Support nested joins #2928 @tunetheweb
- To replace base dialect segment class, must subclass or provide same stuff #2930 @barrywhart
- Add new rule L063 to allow separate capitalisation policy for Datatypes #2931 @tunetheweb
- Adds support for column definitions in table alias expressions #2932 @derickl
- BigQuery: support numeric aliases in
UNPIVOT
clauses #2925 @tunetheweb - T-SQL: Supported nested
MERGE
statements #2926 @tunetheweb
- @HeyZiko made their first contribution in #2936
- @aviv made their first contribution in #2952
- @pguyot made their first contribution in #2963
- @dmohns made their first contribution in #2971
- @tcholewik made their first contribution in #2923
- @jmc-bbk made their first contribution in #2981
- @dnim made their first contribution in #3010
- @kd2718 made their first contribution in #3014
- @mrf made their first contribution in #3016
- @zapion made their first contribution in #3031
- Added Support For Snowflake Inline Comments #2919 @WittierDinosaur
- Spark3: Fix bug which did not allow multiple joins #2917 @tunetheweb
- Added Snowflake Alter View Support #2915 @WittierDinosaur
- Adjust L010 to ignore nulls and booleans covered by L040 #2913 @tunetheweb
- Fix bug where L043 corrupts SQL #2908 @barrywhart
- Jinja: Add support for Block Assignments #2907 @barrywhart
- L042 fix fails with missing function
get_identifier()
on Postgres, Redshift dialects #2899 @barrywhart - BigQuery: Better Set Operators support #2901 @tunetheweb
- Hive: support for complex types in
cast
rowtype
definition #2896 @KulykDmytro - Hive: added
json
type support #2894 @KulykDmytro - Hive: fix incorrect L027 error for rowtype attribute name #2893 @KulykDmytro
- Hive: Add
ARRAY
support #2891 @tunetheweb - Implemented
PIVOT
/UNPIVOT
Redshift + Fixed Snowflake Bug + StandardizedPIVOT
/UNPIVOT
Parsing #2888 @PLBMR - Fix AssertionError in dbt templater when file ends with multiple newlines #2887 @barrywhart
- Hive: Row typecasting in
cast
function #2889 @KulykDmytro - dbt templater should gracefully skip macro files #2886 @barrywhart
- Disable L031 on BigQuery due to complex backtick / project name behavior #2882 @barrywhart
- Documentation: Update dbt templater docs with more detail about pros and cons #2885 @barrywhart
- BigQuery: Better
STRUCT
Array Support #2881 @tunetheweb - L042: Detect violations when column is templated #2879 @barrywhart
- Improve parsing of
BETWEEN
statements #2878 @MarcAntoineSchmidtQC
Major changes include:
- A number of changes to
fix
code to make these more robust - Improvements to templating blocks
generate_parse_fixture_yml
options to allow quicker, partial regeneration of YML files- Numerous rule fixes including adding auto fix to L042
- Numerous grammar changes
- Spark3: Support for
SHOW
statements #2864 @R7L208 - Add Markerr to list of organizations using SQLFluff in the wild #2874 @kdw2126
- Refactor JinjaTracer: Split into two classes, break up
_slice_template()
function #2870 @barrywhart - BigQuery: support Parameterized Numeric Literals #2872 @tunetheweb
- L042 autofix #2860 @OTooleMichael
- Redshift: transaction statement #2852 @rpr-ableton
- JinjaTracer fix for endif/endfor inside "set" or "macro" blocks #2868 @barrywhart
- L009: Handle adding newline after
{% endif %}
at end of file #2862 @barrywhart - Redshift: Add support for
AT TIME ZONE
#2863 @tunetheweb - L032 bug fix and fix improvement #2859 @OTooleMichael
- Refactor JinjaTracer; store lex output as individual strings where possible #2856 @barrywhart
- Add ability to regenerate subsets of fixture YAMLs (by dialect, or new only) #2850 @OTooleMichael
- Fix bug with Jinja and dbt
{% set %}
blocks #2849 @barrywhart - Bug fix:
ValueError: Position Not Found for lint/parse/fix
in JinjaTracer #2846 @barrywhart - Reduce unnecessary setting run ci #2847 @zhongjiajie
- Spark3: statements to
SET
andRESET
spark runtime configurations #2839 @R7L208 - BigQuery - prevent L006 flagging hyphenated table references #2842 @tunetheweb
- T-SQL fix
CONVERT
function definition #2843 @tunetheweb - Change rule test script from bash to python #2840 @OTooleMichael
- Spark3: Support
DESCRIBE
statement #2837 @R7L208 - Spark3: Refactor
REFRESH
statements into one class #2838 @R7L208 - Prevent rules incorrectly returning conflicting fixes to same position #2830 @barrywhart
- Redshift and BigQuery: Update dateparts values and functions #2829 @rpr-ableton
- MySQL add
NOW
support #2825 @tunetheweb - MySQL
DELETE FROM
support #2823 @tunetheweb - Rule L059 bug with
IF
#2824 @tunetheweb - Prevent exceptions when running
fix
on dialect fixtures #2818 @tunetheweb - Spark3: Support to handle
CACHE
ANDUNCACHE
auxiliary statements #2814 @R7L208 - Fix L036 error on
CREATE VIEW AS SELECT
#2816 @tunetheweb - Fixes for the new post-fix parse check #2813 @barrywhart
- Add initial
MERGE
syntax to most dialects #2807 @PhilippLange - Automated tests should fail if a lint fix introduces a parse error #2809 @barrywhart
Major changes include:
- Changes rule L030 to use
extended_capitalisation_policy
to support PascalCase BREAKING CHANGE - Fixes dbt error on ephemeral models
- Log warnings for fixes that seem to corrupt the parse SQL as may cause incorrect fixes in other rules.
- Bug fix to rule L011 for
implicit
aliases - Bug fix to rule L019 for commas besides templated code
- Rule L051 can now optionally be applied to
LEFT
/RIGHT
/OUTER JOIN
s - Improvements to Test Suite
- Many dialect improvements
- Exasol: Fix
INTERVAL
literals / expression #2804 @sti0 - Exasol: Add
IDLE_TIMEOUT
andSNAPSHOT_MODE
#2805 @sti0 - Exasol: Support value range clause within
INSERT
statements (7.1+) #2802 @sti0 - Exasol: Add lua adapter scripts (7.1+) #2801 @sti0
- Exasol: Add openid support for create/alter user (7.1+) #2800 @sti0
- Exasol: New consumer group params and unreserved keywords (7.1+) #2799 @sti0
- Snowflake: Complete
INSERT
grammar #2798 @jpy-git - Fix Postgres
VALUES
, make Spark3VALUES
consistent #2797 @jpy-git - Postgres:
INSERT DEFAULT
value #2796 @jpy-git - Postgres: Make
AS
optional in PostgresDELETE
#2794 @jpy-git - BigQuery support
UNEST
aliases #2793 @tunetheweb - Postgres: Add all range operators #2789 @jpy-git
- Snowflake: Complete
DELETE FROM
grammar #2792 @jpy-git - Postgres: Complete
DELETE FROM
grammar #2791 @jpy-git - Postgres: Add
RETURNING
grammar toINSERT INTO
statement #2790 @jpy-git - Snowflake: Complete
PATTERN
grammar #2788 @jpy-git - Redshift: add
[ALTER/CREATE/DROP] PROCEDURE
segments #2774 @rpr-ableton - Spark3: Support for
ANALYZE TABLE
statement #2780 @R7L208 - Snowflake: Add
MATCH_RECOGNIZE
clause #2781 @jpy-git - Snowflake: Complete
LIMIT
grammar #2784 @jpy-git - Rough autofix for L028 #2757 @OTooleMichael
- Spark3 bug: Create with complex data types (#2761) #2782 @PhilippLange
- Snowflake: Complete
LIKE
grammar #2779 @jpy-git - Spark3: Auxiliary
FILE
andJAR
statements #2778 @R7L208 - Snowflake: Refine
SET
/UNSET
MASKING POLICY
grammar #2775 @jpy-git - L049 bug: correct over zealous
=
-->IS
#2760 @OTooleMichael - Make extension case insensitive #2773 @tunetheweb
- Snowflake: Add dollar quoted string literal #2770 @jpy-git
- Bug fix: L036 corrupts
SELECT DISTINCT id
query #2768 @barrywhart - Snowflake: Add
CHANGES
clause #2764 @jpy-git - Spark3: Support for
EXPLAIN
statement #2767 @R7L208 - Snowflake: Add
CONNECT BY
clause #2763 @jpy-git - Spark3: Support for
TRANSFORM
clause #2762 @R7L208 - Snowflake: Fix
GROUP BY {CUBE|ROLLUP|GROUPING SETS}
parsing #2759 @jpy-git - BigQuery: allow identifiers starting with dash #2756 @tunetheweb
- Add
ignore_words
options to L057 and L059 #2753 @tunetheweb - L012 bug fix for T-SQL alternative alias types #2750 @tunetheweb
- Spark3: Support for
PIVOT
clause #2752 @R7L208 - Update Redshift reserved keywords list #2751 @rpr-ableton
- L007 autofix #2735 @OTooleMichael
- L032 fixable in easy cases #2737 @OTooleMichael
- Fix dbt templater runtime error in
inject_ctes_into_sql()
#2748 @barrywhart - L059: Exasol: Allow quotes around passwords in
CREATE USER
#2744 @sti0 - Improve docs for
load_macros_from_path
#2743 @barrywhart - Make L045 (Query defines a CTE but does not use it) case insensitive #2746 @barrywhart
- Add L049 test for T-SQL alternate alias syntax (
=
) #2745 @barrywhart BaseSegment.pos_marker
is typed as non optional but sometimes set toNone
#2741 @barrywhart- Support Pascal case for L030 #2739 @tunetheweb
- Postgres, Redshift: Support
SIMILAR TO
pattern matching expressions #2732 @PLBMR - Forgive shorthand cast only / bracket only expressions from L013 #2729 @OTooleMichael
- L052: Refactor
_eval()
into individual functions to improve readability #2733 @barrywhart - L018: Move closing parenthesis to next line #2734 @barrywhart
- Improve rule yaml tests: assert that
fix_str
passes the rule #2624 @juhoautio - Extend rule L051 to
LEFT
/RIGHT
/OUTER
JOIN
s #2719 @rpr-ableton - T-SQL: Allow aliases with
=
#2727 @fdw - T-SQL: Support table variables #2728 @fdw
- Support for checking violations in YAML rule tests #2718 @juhoautio
- Roll back PR #2610 #2726 @barrywhart
- Redshift: Allow whitespace around cast operators #2721 @PLBMR
- Support database links in Oracle #2725 @tunetheweb
- Rule L019: Ignore comma placement violations if the adjacent code is templated #2717 @barrywhart
- T-SQL: Add drop constraint syntax #2724 @fdw
- ANSI: Support optionally bracketed CTE #2716 @OTooleMichael
- Spark3: Test cases for
CASE
clause #2714 @R7L208 - Spark3: Support for
WINDOW
functions #2711 @R7L208 - T-SQL: Add variables as options for
RAISERROR
parameters #2709 @jpers36 - T-SQL: Add
OPTION
clause toUPDATE
#2707 @jpers36 - Spark3: Test cases for
WHERE
clause #2704 @R7L208 - Spark3: test cases for Table-Valued Functions #2703 @R7L208
- T-SQL: Allow for optionally bracketed
PARTITION BY
elements #2702 @jpers36 - T-SQL: Fix
SET TRANSACTION ISOLATION LEVEL
parsing #2701 @jpers36 - Migrate tricky L004 tests to python #2681 @juhoautio
- Core linter enhancement: Check for successful parse after applying fixes #2657 @barrywhart
- Spark3: Support for
LATERAL VIEW
clause #2687 @R7L208 - Document python requirement for tox/mypy & remove basepython from conf #2644 @juhoautio
- Fix rule L011 for implicit aliases #2683 @tunetheweb
- Pin markupsafe to prevent CI failures #2685 @tunetheweb
- Exasol: Allow
CROSS
joins #2680 @sti0 - Exasol: Improve function formatting #2678 @sti0
- T-SQL: Add indentation for
CREATE
INDEX
/STATISTICS
#2679 @jpers36 - Spark3: Support for
TABLESAMPLE
clause #2674 @R7L208 - T-SQL: Improve
RAISERROR
functionality #2672 @jpers36 - Snowflake dialect update for
MERGE INTO
predicates #2670 @The-Loud - Assert that fix_str is set #2663 @juhoautio
- @The-Loud made their first contribution in #2670
- @OTooleMichael made their first contribution in #2716
- @PhilippLange made their first contribution in #2782
Major changes include:
- Improvements to rules L023, L045, L048, L052, L059 to make them more accurate.
- If
sqlfluff fix
cannot find a stable fix afterrunaway_limit
iterations (default 10) then no fixes will be applied. - Addition of
--write-output
config to command line so prevent errors corrupting output. - Various dialect improvements
- Redshift: Support DATETIME as a valid datatype #2665 @PLBMR
- Support L033 for RedShift #2661 @tunetheweb
- Fix parsing types and add check to test in future #2652 @tunetheweb
- Spark3: Support for
SORT BY
Clause #2651 @R7L208 - Migrate issue template from markdown to yaml #2626 @zhongjiajie
- L048 - handle more statements and exclude casting operators #2642 @tunetheweb
- MySQL support
CURRENT_TIMESTAMP()
inCREATE TABLE
#2648 @tunetheweb - Postgres enhanced
DELETE FROM
syntax #2643 @tunetheweb - Bug fix: L025 should consider BigQuery
QUALIFY
clause #2647 @barrywhart - Bug fix: L025 overlooking
JOIN ON
clause if join expression in parentheses #2645 @barrywhart - L045 not reporting unused CTEs if the query uses templating #2641 @barrywhart
- Fix IndexError in L001 #2640 @barrywhart
- L052: If require_final_semicolon is set, ensure semicolon after ALL statements #2610 @barrywhart
- L023 to also fix extra newlines in CTE #2623 @juhoautio
- Spark3: Enhancements for Set Operators #2622 @R7L208
- Doc a better choice for default env #2630 @juhoautio
- Ensure ordering of fix compatible and config in rules docs #2620 @zhongjiajie
- Pin python version for tox -e mypy #2629 @juhoautio
- Hitting the linter loop limit should be treated as an error #2628 @barrywhart
- Allow file output directly from cli #2625 @alanmcruickshank
- BigQuery
UNPIVOT
andPIVOT
fixes #2619 @tunetheweb - L059 quoted identifiers bug #2614 @tunetheweb
- Snowflake dialect: Adjust snowflake array access #2621 @alanmcruickshank
- Spark3: Test Cases for
ORDER BY
inSELECT
#2618 @R7L208 - Fix typos in 0.10.0 changelog #2605 @tunetheweb
- T-SQL: Indent
IF
clause expression segments #2615 @jpers36 - Spark3: Enhancements for
LIMIT
Clause #2612 @R7L208 - Allow Bare Functions in column constraints #2607 @tunetheweb
- Add Oracle at and double at sign (execution symbol) #2608 @r0fls
- Spark3: Enhancements to
LIKE
clause #2604 @R7L208
Major changes include:
- Dropping support of DBT < 0.20 BREAKING CHANGE
sqlfluff fix
no will no longer fix SQL containing parsing or templating errors BREAKING CHANGE- New rule L062 to allow blocking of list of configurable words (e.g. syntax, or schemas, or tables you do not want people to use)
- Lots and lots of docs improvements
- Looser requirements for
click
python package
- L046: Detect Jinja spacing issues where segment begins with literal content #2603 @barrywhart
- MySQL Add BINARY support #2602 @tunetheweb
- Support indenting WINDOWS clauses and (optionally) CTEs #2601 @tunetheweb
- Postgres: Support expressions in arrays #2599 @tunetheweb
- BigQuery support Array of Structs #2598 @tunetheweb
- Support wildcards in triggers #2597 @tunetheweb
- Support CTEs in CREATE VIEW statements #2596 @tunetheweb
- SQLite Support more CREATE TRIGGER options #2594 @tunetheweb
- Snowflake: Support Column Comments in Alter Table statements #2593 @tunetheweb
- Redshift: Add DATETIME as reserved keyword #2591 @tunetheweb
- Support LIMIT and ORDER BY clauses in Values clauses #2590 @tunetheweb
- L016: New option "ignore_comment_clause" to ignore column COMMENTs, etc. #2589 @barrywhart
- Bug fix: L016 ("Line is too long") should consider length of prior fixes #2587 @barrywhart
- Add mysql INSERT ON DUPLICATE KEY #2494 @rpr-ableton
- Snowflake ALTER TABLE: Add multiple columns #2578 @erevear
- MySQL: UNIQUE KEY in CREATE TABLE #2525 @jpy-git
- Spark3: JOIN clause enhancements #2570 @R7L208
- Bug fix: L003 should fix indentation for templated code #2580 @barrywhart
- Exasol: Improve
COMMENT
andWITH [NO] DATA
clause usage. #2583 @sti0 - Exasol: Allow multiple
LOCAL
keywords inWHERE
clause #2582 @sti0 - Exasol: Allow
LOCAL
keyword withinPREFERRING
clause #2579 @sti0 - Add/Improve docs for config settings: "ignore", "ignore_templated_areas" #2574 @barrywhart
- Look for .sqlfluffignore in current directory #2573 @barrywhart
- Snowflake: L054 should ignore "WITHIN GROUP" clauses #2571 @barrywhart
- Redshift: Support Redshift SUPER Data Types #2564 @PLBMR
- Capitalization rules (L010, L014, L030, L040) should ignore templated code #2566 @barrywhart
- T-SQL: Add Frame clause unreserved keywords #2562 @jpers36
- Simple API: Fix bug where omitted parameters still override .sqlfluff #2563 @barrywhart
- Spark3: Add Direct File Query #2553 @R7L208
- Redshift dialect: replace AnyNumberOf with AnySetOf where it makes sense #2561 @rpr-ableton
- jinja and dbt templaters: More robust handling of whitespace control #2559 @barrywhart
- Improve how "sqlfluff fix" handles templating and parse errors #2546 @barrywhart
- Jinja and dbt templater: Fix "list index out of range" error #2555 @barrywhart
- Fix typo in sqlfluffignore docs #2551 @tunetheweb
- Correct parsing for BigQuery
SELECT REPLACE
clauses. #2550 @elyobo - Rules documentation improvements #2542 @tunetheweb
- Remove requirement for Click>=8 #2547 @tunetheweb
- Allow L059 to be configured to always prefer quoted identifiers #2537 @niconoe-
- Adds new rule L062 to allow blocking of certain words #2540 @tunetheweb
- Update to latest Black, drop support for dbt < 0.20 #2536 @barrywhart
- dbt templater: Fix bug where profile wasn't found if DBT_PROFILES_DIR contained uppercase letters #2539 @barrywhart
- Spark3: Added segments & grammar needed for hints #2528 @R7L208
- Spark3: parse some VALUES clauses #2245 @mcannamela
- T-SQL: Allow multiple params in SET statements #2535 @tunetheweb
- T-SQL: Add indentation for SET statement #2531 @jpers36
- Add additional documentation on dbt-adapter in pre-commit #2530 @robertdefilippi
- T-SQL: Add indentation for UPDATE statement #2532 @jpers36
- Fix Snowflake Unordered Select Clause #2529 @tunetheweb
- Fix Quoted Literals for Postgres and Redshift affecting rule L039 #2526 @tunetheweb
- Postgres specific CTEDefinitionSegment #2524 @jpy-git
- @robertdefilippi made their first contribution in #2530
- @niconoe- made their first contribution in #2537
- @elyobo made their first contribution in #2550
- @erevear made their first contribution in #2578
Major changes include:
- dbt performance improvements
- Fix
click
dependency error. - Better datepart versus identifier parsing.
- Fix some Jinja errors.
- Various grammar fixes and improvements
- Spark3: test cases for HAVING clause in SELECT statement #2518 @R7L208
- Update click version requirement in setup.cfg to match that in requirements.txt #2518 @barrywhart
- Postgres: Implement DO Statements + Refactored Language Clause #2511 @PLBMR
- Spark3: Support for Grouping Sets,
CUBE
andROLLUP
inGROUP BY
clause ofSELECT
statement #2505 @R7L208 - Refactor date part functions #2510 @tunetheweb
- Postgres: EXPLAIN ANALYSE allows British spelling #2507 @jpy-git
- "noqa": Add support for ignoring template (TMP) and parse (PRS) errors #2509 @barrywhart
- Freeze Black due to incompatibility between 22.1 and flake8-black #2513 @tunetheweb
- Support NATURAL JOINS #2506 @tunetheweb
- dbt Docker environment: Mount the test profiles.yml at ~/.dbt #2502 @barrywhart
- Add dbt_artifacts package to in the wild docs #2504 @NiallRees
- Spark3: Support
DISTRIBUTE BY
clause inSELECT
statement #2503 @R7L208 - dbt templater: For performance reasons, cache the database connection across models #2498 @barrywhart
- Bug fix: Defining and using Jinja macro in the same file causes runtime error #2499 @barrywhart
- Spark3: Support
CLUSTER BY
clause inSELECT
statement #2491 @R7L208 - Grammar: Adds support for COPY statement for Postgres dialect #2481 @derickl
- Add raiserror for T-SQL #2490 @fdw
- Enforce parentheses for function definitions in T-SQL #2489 @fdw
- Add guards to prevent rule crashes #2488 @barrywhart
Major changes include:
- Add
ignore_words
option for rules L010, L014, L029, L030, L040 - Fix some issues in 0.9.2 preventing some queries linting
- Prevent L031 throwing exception on unparsable code #2486 @tunetheweb
- Add linting of fixtures SQL for critical rules errors to tox #2473 @tunetheweb
- Fix L039 for T-SQL comparison operator using space #2485 @tunetheweb
- Fix bug in get_alias causing rule Critical errors for T-SQL #2479 @tunetheweb
- Tweak GitHub templates #2471 @tunetheweb
- Small speed improvement to L054 #2476 @tunetheweb
- L003: Revisit recent change to improve speed #2474 @barrywhart
- Fix select_crawler issue with some Exasol statements #2470 @tunetheweb
- Cleanup date logic by removing DatePartClause and using DatetimeUnitSegment instead #2464 @tunetheweb
- Fix L044 exception when final statement has no SELECT #2468 @tunetheweb
- Support T-SQL system variables (e.g. @@rowcount) #2463 @tunetheweb
- Add base rule to developing rules page #2462 @tunetheweb
- L003: Ignore indentation of lines that only exist in templated space #2460 @barrywhart
- Ignore words for various rules #2459 @tunetheweb
- Support Foreign Key options for MySQL #2461 @tunetheweb
- Exclude WINDOW clauses from L054 #2455 @tunetheweb
- Fix bug with L026 for simple deletes #2458 @tunetheweb
- Spark3: test cases for Common Table Expressions #2454 @R7L208
- Fix T-SQL's IDENTITY_INSERT syntax #2452 @fdw
- T-SQL: Support stored procedures in insert statements #2451 @fdw
- Spark3: Support for
LOAD DATA
statements #2450 @R7L208
We are pleased to include 110 improvements and fixes in this release, and welcome 7 new contributors to the code.
Major changes include:
- Initial Oracle support (note: SQL, but not PL/SQL)
- Fix more dbt 1.0.0 connection issues
- Improved configuration documentation
- New rule (L059) to flag unnecessary quoted identifiers
- New rule (L060) to prefer
COALESCE
instead ofIFNULL
orNVL
- New rule (L061) to prefer
!=
over<>
- Many rule fixes
- Many dialect improvements
- Add Postgres DROP PROCEDURE support #2446 @rpr-ableton
- MySQL Alter table ADD/DROP/RENAME INDEX support #2443 @tunetheweb
- Add basic CREATE PROCEDURE support to Postgres #2441 @tunetheweb
- Indent T-SQL DECLARE and EXEC statements #2439 @tunetheweb
- Hive alternative types: INTEGER, DEC, NUMERIC #2438 @tunetheweb
- Implement Snowflake Dateparts #2437 @tunetheweb
- Fix rule L028 for T-SQL for params #2442 @tunetheweb
- Support CREATE UNIQUE INDEX #2440 @tunetheweb
- Make BigQuery typeless STRUCTs Expressions #2435 @tunetheweb
- T-SQL support default params and no RETURN value #2434 @tunetheweb
- "sqlfluff fix" should report any parse errors found #2423 @barrywhart
- Redshift VACUUM support #2433 @rpr-ableton
- Add Oracle PROMPT statement #2413 @r0fls
- Spark3: Support for
INSERT OVERWRITE DIRECTORY
with Hive Format #2389 @R7L208 - Exasol: Fix escaped identifiers #2431 @sti0
- Exasol: Fix
LOCAL.ALIAS
Syntax #2430 @sti0 - Exasol: Allow quoted identifier for various statements. #2428 @sti0
- Misc grammar improvements for Snowflake #2421 @chwiese
- New rule L061 to use != over <> #2409 @sti0
- Correct TRANS to TRAN #2425 @fdw
- Remove the "heuristic" slicer, as it was replaced by JinjaTracer #2422 @barrywhart
- L060: More specific description #2419 @jpy-git
- Fix code formatting in Rule docs #2418 @tunetheweb
- Allow UPDATE SET statements in RedShift #2417 @tunetheweb
- Add Redshift cursor DECLARE, FETCH & CLOSE support #2414 @rpr-ableton
- Add Redshift ANALYZE COMPRESSION support #2412 @rpr-ableton
- ANSI Values statement fixes #2404 @jpy-git
- Exasol: Overhaul drop statements #2407 @sti0
- L044, L045: Handle Exasol VALUES clause #2400 @barrywhart
- L060: Use COALESCE instead of IFNULL or NVL. #2405 @jpy-git
- Postgres: Fix Values alias regression #2401 @jpy-git
- Align line length in Python code to 88 characters #2264 @chwiese
- Jinja templater: Allow "load_macros_from_path" to be a comma-separated list of paths #2387 @barrywhart
- Add "TRANS" keyword for T-SQL #2399 @fdw
- Docstrings: Replace double backtics with single quote for lint results. #2386 @jpy-git
- Spark3: Support for
INSERT OVERWRITE DIRECTORY
statements #2385 @R7L208 - Fix unnecessary white underline in doc site #2383 @tunetheweb
- Rolls back some code cleanup that caused coverage report to show gaps #2384 @barrywhart
- Fix "connection already closed" issue with dbt 1.0 and dbt_utils #2382 @barrywhart
- Spark3: Support for
INSERT [TABLE]
data manipulation statements #2290 @R7L208 - Comment out line in bug report template #2378 @jpy-git
- Postgres: EXPLAIN statement updates #2374 @jpy-git
- Make TABLE a non-reserved word in Postgres #2377 @tunetheweb
- Snowflake COLUMN is not a reserved word #2376 @tunetheweb
- T-SQL: Complete ASA Table Index Clause functionality #2373 @jpers36
- Add support for Jinja import and include #2355 @barrywhart
- Add Redshift INTERVAL datatype support #2366 @rpr-ableton
- Whitespace concatenated string literals for MySQL, Postgres and Redshift #2356 @jpy-git
- Fix L026 false positive on "SELECT INTO" statement #2371 @barrywhart
- Exclude EMIT clauses from rule L013 #2364 @tunetheweb
- Functional API: Segments.recursive_crawl #2369 @jpy-git
- Complete Redshift CREATE EXTERNAL TABLE support #2354 @rpr-ableton
- L041: Fix duplicate DISTINCT corruption #2365 @jpy-git
- Bigquery Create View with Options #2359 @tunetheweb
- L026: Handle DML statements and multiple levels of nesting #2336 @barrywhart
- Postgres & MySQL: cleanup AliasExpressionSegment #2353 @jpy-git
- Redefine MySQL Interval segment #2351 @rpr-ableton
- Postgres: INSERT INTO table alias #2349 @jpy-git
- L043: Remove redundant CASE statement replacing NULLS with NULLS #2346 @jpy-git
- Add RedShift DATASHARE support #2350 @rpr-ableton
- Various documentation updates #2347 @tunetheweb
- Snowflake ALTER TABLE: Drop multiple columns #2348 @jpy-git
- Configuration doc: add rule configuration section #2291 @juhoautio
- Redshift: create model, show model & data types #2338 @rpr-ableton
- L059: Unnecessary quoted identifier #2341 @jpy-git
- L043: Use simple replace to apply fixes #2343 @jpy-git
- T-SQL: Add functionality to PARTITION BY clause #2335 @jpers36
- L039 casting operator postgres fix #2334 @jpy-git
AnySetOf
grammar #2326 @jpy-git- Redshift: update CREATE TABLE AS match_grammar #2333 @rpr-ableton
- Redshift CREATE EXTERNAL TABLE: TABLE PROPERTIES #2330 @jpy-git
- Snowflake: Flush out
ALTER TABLE
'stableColumnAction
grammar #2332 @wong-codaio - Snowflake ALTER TABLE: Add clusteringAction #2329 @jpy-git
- Snowflake ALTER TABLE: Add searchOptimizationAction #2328 @jpy-git
- Fix numeric literal grammar for Postgres/MySQL/Exasol #2324 @jpy-git
- L039: Remove spaces between comparison operators (T-SQL) #2325 @jpy-git
- Enable setting a target of a dbt profile #2236 @yu-iskw
- Snowflake: Add support for column rename #2327 @wong-codaio
- Snowflake: Added
AlterTableStatement
specific for Snowflake #2267 @wong-codaio - Full REFERENCES grammar for CREATE TABLE statement #2315 @jpy-git
- Fix Spark numeric literals #2317 @jpy-git
- Change type of Snowflake stage paths to fix issues with L044 #2320 @chwiese
- Add Bytes Quoted Literals to Spark dialect #2312 @jpy-git
- Fix L044 assertion failure with delete stmt & cte #2321 @barrywhart
- L003 should consider only literal leading whitespace (ignore templated) #2304 @barrywhart
- Redshift: update reserved keywords #2318 @rpr-ableton
- docs: Document how to run SQLFluff with local changes to test them #2316 @kayman-mk
- Update redshift unreserved keywords #2310 @jpy-git
- Fix spark and hive quoted literals #2311 @jpy-git
- Oracle Dialect #2293 @r0fls
- Redshift dialect: add COPY and UNLOAD statements #2307 @rpr-ableton
- L052: Fix case where no preceding segments and mulitline #2279 @jpy-git
- Update rule L049 to handle EXEC assignments #2308 @tunetheweb
- Remove DATE, DATETIME and TIME from BigQuery DatePart #2283 @tunetheweb
- Fix #1292: nocolor and verbose can work in config files #2300 @cympfh
- Allow pyproject.toml as extra_config_path #2305 @jpy-git
- L009: Handle adding newline after trailing templated code #2298 @barrywhart
- added missing "t" in doc for Rule_L020 #2294 @Xilorole
- docs: Document configuration keyword for rule L054 #2288 @tomasfarias
- Update L009 to operate in raw, not templated space #2285 @barrywhart
- Redshift CREATE LIBRARY statements #2277 @rpr-ableton
- L025 with 'bigquery' dialect: Correctly interpret calling functions with a table as a parameter #2278 @barrywhart
- Spark3: Coverage for
REFRESH
auxiliary statements #2282 @R7L208 - Spark3: Coverage for
USE DATABASE
statement. #2276 @R7L208 - Fix link for editing 'In The Wild' page with new base branch,
main
#2280 @barnett - Optionally allow additional configurable characters in L057 #2274 @tunetheweb
- L025 should look at subqueries #2273 @barrywhart
- Add coverage for
TRUNCATE
statement in Spark3 dialect #2272 @R7L208 - Upgrade
click
version to 8.0+ to supportclick.shell_completion
#2271 @wong-codaio - Improve release checklist to make releases easier #2263 @tunetheweb
- @barnett made their first contribution in #2280
- @tomasfarias made their first contribution in #2288
- @Xilorole made their first contribution in #2294
- @cympfh made their first contribution in #2300
- @r0fls made their first contribution in #2293
- @yu-iskw made their first contribution in #2236
- @fdw made their first contribution in #2399
- Fix dbt 1.0.0 connection issue
- Fix some SQL corruption issues with templated code
- New components to simplify creating rules
- Remove support for Python 3.6
- Fix delimited identifier parsing for spark3 #2111 @mcannamela
- Stop numeric literal from splitting valid naked identifiers. #2114 @jpy-git
- Grammar: Add CREATE USER/GROUP statement to Redshift dialect #2115 @jpy-git
- Fix mypy type raise in L003 #2127 @barrywhart
- Add ability to parse multiple GO/semicolon delimiters #2124 @jpy-git
- Allowed array/struct values in
default
definition ofdeclare
#2120 @KulykDmytro - Normalise input newlines #2128 @jpy-git
- Clean up all files using the pre-commit hook #2123 @kayman-mk
- Refined LintFix API #2133 @jpy-git
- Hotfix for LintFix comparisons #2138 @jpy-git
- Lint spaces in qualified names #2130 @jpers36
- Remove support for Python 3.6 (it's "end of life" December 23, 2021) #2141 @barrywhart
- Fully remove python3.6 references #2142 @jpy-git
- Fix L022 to not flag CTE column definitions #2139 @jpy-git
- docs: set
dbt_modules
todbt_packages
#2143 @ciklista - Hive: add INTERVAL syntax #2144 @juhoautio
- Fix mypy error on python 3.7 #2147 @juhoautio
- Update PR template to reference tox generate-fixture-yml command #2148 @jpy-git
- Update index.rst notable changes with 0.9.0 details #2132 @jpy-git
- Add ALTER USER and ALTER GROUP to redshift dialect #2131 @jpy-git
- Add complete DESCRIBE grammar to Snowflake dialect #2149 @jpy-git
- Fix bug with BigQuery UNPIVOT #2156 @tunetheweb
- Make L057 compatible with BigQuery #2151 @tunetheweb
- T-SQL: Proper Indentation of ELSE IF #2157 @jpers36
- Linter Test Name Duplication #2158 @jpers36
- Add struct support for
hive
andredshift
(L026, L028) #2154 @KulykDmytro - Postgres - Support functions prepended with _ and containing $ #2159 @jpy-git
- T-SQL: function parsing/linting #2155 @jpers36
- T-SQL: Add THROW statement #2163 @jpers36
- Add yamllint to project #2162 @tunetheweb
- Fix outdated docstring in dialects_test #2166 @juhoautio
- Minor comment fixes #2179 @juhoautio
- L010 to apply to date_part (capitalization policy for time units) #2167 @juhoautio
- ALTER GROUP fix to accommodate quoted objects #2188 @tdstark
- Lexer: add non-breaking spaces to whitespace #2189 @jpers36
- Grammar: Add COMMENT statement to Snowflake #2173 @jpy-git
- Grammar: Add DISCARD statement to Postgres dialect #2175 @jpy-git
- Grammar: Add UNDROP statement to Snowflake dialect #2177 @jpy-git
- Grammar: Add UNSET statement to Snowflake dialect #2181 @jpy-git
- Grammar: Add RESET statement to Postgres dialect #2182 @jpy-git
- Grammar: Add LOAD statement to Postgres dialect #2183 @jpy-git
- Grammar: Fix TRUNCATE statement in Snowflake dialect #2184 @jpy-git
- Grammar: Add HELP statement to MySQL dialect #2191 @jpy-git
- Grammar: Add PURGE BINARY LOGS statement to MySQL dialect #2193 @jpy-git
- Grammar: Add RESET MASTER statement to MySQL dialect #2194 @jpy-git
- Grammar: Add RENAME TABLE statement to MySQL dialect #2195 @jpy-git
- Grammar: Tidy up transaction statements in Snowflake dialect #2196 @jpy-git
- Modifying Redshift USER/GROUP Statements To Use
ObjectReferenceSegment
#2190 @tdstark - Grammar: Fix TRUNCATE statement in Postgres dialect #2185 @jpy-git
- Grammar: Add LISTEN, NOTIFY, and UNLISTEN statements to Postgres dialect #2174 @jpy-git
- Grammar: Tidy up Snowflake/MySQL/HIVE USE statements #2187 @jpy-git
- Make Snowflake keywords unreserved: account, organization, pivot #2172 @jpy-git
- Grammar: Add SET sql_log_bin statement to MySQL dialect #2192 @jpy-git
- Grammar: Add CALL statement to Snowflake dialect #2176 @jpy-git
- L027 Fix self referring column alias edge case #2171 @jpy-git
- T-SQL: Remove dependency on ANSI keyword lists #2170 @jpers36
- Grammar: Add Table Maintenance Statements to MySQL dialect #2198 @jpy-git
- Adding CREATE TABLE AS to Redshift #2205 @tdstark
- T-SQL: Add support for ALTER TABLE ALTER COLUMN #2208 @jpers36
- Remove oyaml in favour of pyyaml #2210 @jpy-git
- Support Spark
CREATE TABLE LIKE
syntax #2207 @R7L208 - Add override for linguist to include SQL in language statistics #2214 @jpy-git
- Add type stubs for appdirs and colorama to improve mypy coverage #2211 @jpy-git
- Remove cached-property in favour of stdlib functools implementation #2212 @jpy-git
- Restructure CASE segment (extract WHEN and ELSE into their own segment types) #2213 @barrywhart
- Add types-regex package for type checking #2216 @jpy-git
- Snowflake: Split out
CREATE VIEW
into its own segment #2217 @wong-codaio - Grammar: Fix multi-character comparison operators #2197 @jpy-git
- Snowflake: Support TOP N select clause modifier #2222 @wong-codaio
- Fix CLI arguments to allow for autocompletion #2218 @jpy-git
- Simplify rule creation by adding a functional API to RuleContext #2126 @barrywhart
- Simplify nested cases #2223 @barrywhart
- Reword lint message for L058 per review #2226 @barrywhart
- Update BaseRule.discard_unsafe_fixes() to avoid touching templated code #2220 @barrywhart
- Add L059 - Capitalization on Data Types #2227 @tdstark
- T-SQL: Table valued functions #2233 @jpers36
- Don't allow fixes to COPY code from templated regions #2231 @barrywhart
- Fix several small issues with rule docs #2234 @barrywhart
- postgres: Add datatypes #2121 @kayman-mk
- Combine L059 and L010 #2238 @tdstark
- Fix L044 assertion failure: "SELECT *" with no "FROM" clause #2239 @barrywhart
- Docs: Make Specific Rules docstring more user friendly #2241 @jpy-git
- Fix a bug handling Jinja "{% set %}" blocks with a templated block inside #2240 @barrywhart
- Redshift lint create external table statements #2229 @tinder-albertyue
- Update tox.ini for best practices #2243 @jpy-git
- Docs: Make code blocks consistent #2242 @jpy-git
- Add support for nested Jinja macros #2246 @barrywhart
- Support
DROP
DDL statements for Spark3 #2215 @R7L208 - Docker Compose environment for SQLFluff developers #2254 @barrywhart
- T-SQL: Add OFFSET unreserved keyword #2258 @jpers36
- Fix connection issue in dbt 1.0.0 #2230 @NiallRees
- Redshift CREATE SCHEMA statements #2252 @rpr-ableton
- Enhance Snowflake COPY INTO #2250 @chwiese
- Coverage for 'REPAIR' Statements for Hive & Spark3 dialect #2256 @R7L208
- @mcannamela made their first contribution in #2111
- @ciklista made their first contribution in #2143
- @juhoautio made their first contribution in #2144
- @tinder-albertyue made their first contribution in #2229
- @rpr-ableton made their first contribution in #2252
This release brings about several great new additions including:
- dbt 1.0.0 compatibility.
- CLI and Simple API parameters to provide custom paths to config files.
- Refinement to Simple API to return parse output in JSON format rather than as an internal SQLFluff object (BREAKING CHANGE).
- An Official SQLFluff Docker Image.
- Grammar improvements across various dialects.
- A new rule (L057) to check for non-alphanumeric values in identifiers.
There have also been many bug fixes and improvements to the CI and development processes.
- T-SQL: Reserved Keyword cleanup #2100 @jpers36
- Add wiki links to CONTRIBUTING.md #2106 @tunetheweb
- Add snowflake create stage and alter stage statements + RegexParser case fix #2098 @chwiese
- Allow for more value types in ALTER TABLE ALTER COLUMN SET DEFAULT statement #2101 @derickl
- Grammar: Adds support for ALTER VIEW statement for Postgres dialect #2096 @derickl
- Add example for using JSON output of Simple API parse function #2099 @jpy-git
- Allow optional keywords in create table unique constraints #2077 @kayman-mk
- Grammar: Adds support for ALTER FUNCTION statement for Postgres dialect #2090 @derickl
- Grammar: adds support for CREATE/ALTER/DROP DATABASE for Postgres dialect #2081 @derickl
- Update parse method of Simple API to output JSON parse tree #2082 @jpy-git
- T-SQL dialect: add parsing for MERGE statement #2057 @tkachenkomaria244
- Simple API config path #2080 @jpy-git
- dbt 1.0.0 compatibility #2079 @alanmcruickshank
- Parse
on delete
andon update
clause for create table constraints #2076 @kayman-mk - Pre-commit: Add hook for doc8 #2074 @jpy-git
- Grammar: Fix typo in Alter Table parser in Postgres dialect #2072 @derickl
- Grammar: Adds support for materialized views for postgres dialect #2041 @derickl
- Add basic pre-commit config #2067 @jpy-git
- CLI: Add --ignore-local-config flag #2061 @jpy-git
- T-SQL: INSERT INTO #2054 @jpers36
- Add --disable-noqa option to CLI and config #2043 @jpy-git
- T-SQL: TRY/CATCH #2044 @jpers36
- enabled arrays support in
declare
andset
statements forbigquery
dialect #2038 @KulykDmytro - L008 refactor #2004 @jpy-git
- Support init.py for library_path #1976 @Tonkonozhenko
- L052: Redefine semi-colon newline to multiline newline #2022 @jpy-git
- Grammar: Remove hash inline comment from Postgres #2035 @jpy-git
noqa
enhancement: Enable glob rule matching for inline comments #2002 @jpy-git- T-SQL (ASA): Allow for table identifier in DELETE clause #2031 @jpers36
- T-SQL (ASA): Fix CTAS with WITH statement #2028 @jpers36
- Grammar: Parse multiple grants #2023 @jpy-git
- Add tsql nested block comment support and add regex package dependency #2027 @jpy-git
- Grammar: Add complete Snowflake datetime units #2026 @jpy-git
- Grammar: Add DROP POLICY statement to postgres dialect #2024 @jpy-git
- Grammar: Add complete datetime units to postgres dialect #2025 @jpy-git
- Grammar: Postgres CREATE POLICY #2021 @jpy-git
- Speed up CI #1957 @pwildenhain
- Add support for Snowflake create/alter SQL and js UDF #1993 @chwiese
- Add encoding CLI argument #1994 @jpy-git
- T-SQL: Spaces allowed in comparison operators #1965 @jpers36
- Add Snowflake schema options #1950 @chwiese
- CLI/
.sqlfluff
enhancement: Rule globs #1972 @jpy-git - Add config CLI argument to lint, fix, and parse #1986 @jpy-git
- Add type hints to simple API #1951 @jpy-git
- New rule to flag special characters in identifiers #1958 @jpers36
- Allow column references in IN statement #1971 @tunetheweb
- Remove config.ini in favor of setup.cfg #1966 @jpy-git
- Convert sqlfluff-templater-dbt setup.py to setup.cfg #1963 @jpy-git
- Official Docker image: Dockerfile and Github Actions workflow #1945 @jpy-git
- Move package metadata to setup.cfg #1960 @jpy-git
- Fix tsql block comment close #2095 @jpy-git
- Fix PlaceholderTemplater slice_type for templated code (substitutions) #2085 @barrywhart
- Exasol: Fix UDF script syntax #2083 @sti0
- Fix issues with placeholder templating docs #2078 @jpy-git
- Update dbt templater docs to clarify that the profiles_dir setting is optional #2070 @barrywhart
- Bug fix of L054 for Snowflake and Exasol #2069 @tunetheweb
- Fix L043 issue when trying to autofix functions #2059 @jpy-git
- Add request for users dbt version in bug_report issue template #2058 @jpy-git
- Fix parameters for Snowflake create tasks statement #2037 @chwiese
- Linguist: Include test/** in language statistics to better reflect use of SQL #2034 @jpy-git
- L044 should handle nested CTEs #1991 @barrywhart
- Add dbt adapter install advice to configuration documentation #2011 @jpy-git
- Update pre-commit dbt instructions to reference separate dbt package #2005 @jpy-git
- Fix config.get for iterable sections #2020 @jpy-git
- Fix inline comment interactions with L052 #2019 @jpy-git
- Make Snowflake tags DRY #1992 @chwiese
- Rename whitelist/blacklist to allowlist/denylist #1989 @jpy-git
- Fix issue with inline ignores not respecting comment lines #1985 @jpy-git
- Fix L009 FileSegment child + new create_before/create_after edit types #1979 @jpy-git
- Adds extra check to L054 to avoid weird error messages #1988 @tunetheweb
- BigQuery: Allow keywords in column reference components #1987 @tunetheweb
- L027: Remove unnessary crawl in get_select_statement_info #1974 @jpy-git
- Add all attributes to init.py files to resolve F401 #1949 @jpy-git
- Fix incorrect comment on L055 #1967 @jpy-git
- Docs: fix docker hub link to public URL #1964 @kevinmarsh
- Fix issue releasing dbt package: tox commands run relative to repo root #1962 @jpy-git
One of the biggest new features in this release is the support for SQLAlchemy and other "placeholder" templating within SQL queries. Check out the documentation on how to set it up.
This release also adds seven new rules. Get some help with your leading whitespace, semi-colon placement, inconsistent column references in GROUP BY/ORDER BY
, and getting rid of RIGHT JOIN
's among other useful lints with our new rules! See our rules documentation for more details.
On top of those, we have made loads of grammar improvements across many dialects, improvements to the dbt templater (including issues where sqlfluff fix
would corrupt the code 😱), more fix routines, and lots more improvements.
- [many dialects] Implement generic placeholder templating #1887 @jacopofar
- [many dialects] Add support for SQLAlchemy templating #1878 @jacopofar
- Add DROP PROCEDURE statement to T-SQL #1921 @jpy-git
- T-SQL dialect: fix index/tables creation options #1955 @tkachenkomaria244
- Add DROP TYPE statement to ANSI dialect #1919 @jpy-git
- Add INSERT INTO statements to Redshift Dialect #1896 @tdstark
- Added TABLESAMPLE support to Bigquery #1897 @WittierDinosaur
- Add [LEFT] ANTI and [LEFT] SEMI joins to the Spark3 dialect #1942 @jpy-git
- Parse UPDATE/INSERT within WITH clause #1889 @jpy-git
- Add OVERRIDING SYSTEM/USER VALUE to insert statement in postgres dialect #1869 @jpy-git
- Add support for DROP SCHEMA [IF EXISTS] name [ CASCADE | RESTRICT ] #1865 @gimmyxd
- Add CREATE TABLE Statement To Redshift #1855 @tdstark
- Add DROP TYPE statement in postgres dialect #1870 @jpy-git
- Add SEQUENCE NAME to postgres sequence options #1866 @jpy-git
- Added SET Statement to Postgres #1877 @WittierDinosaur
- Postgres: Allow use of quoted identifiers to ALTER TABLE OWNER TO #1856 @markpolyak
- Updates to COPY INTO grammar in Snowflake #1884 @WittierDinosaur
- Postgres & T-SQL: Drop Function #1924 @WittierDinosaur
- Add Expressions to SET syntax #1852 @tunetheweb
- Update DbtTemplater to use JinjaTracer #1788 @barrywhart
- L043 refinement: Add autofix for common use of CASE to fill NULL values. #1923 @jpy-git
- New Rule L050: No leading whitespace #1840 @jpy-git
- L050: updating to target jinja templates #1885 @jpy-git
- New rule L051 to forbid lone JOIN #1879 @jpy-git
- New Rule L052: Semi colon alignment #1902 @jpy-git
- New Rule L053: Remove outer brackets from top-level statements. #1916 @jpy-git
- New Rule L054: Inconsistent column references in GROUP BY/ORDER BY clauses. #1917 @jpy-git
- New Rule L055: Use LEFT JOIN instead of RIGHT JOIN. #1931 @jpy-git
- New Rule L056: 'SP_' prefix should not be used for user-defined stored procedures #1930 @jpy-git
- Tsql partition by multiple columns #1906 @jpers36
- Added bare functions to values clause #1876 @WittierDinosaur
- Remove unnecessary context section, from code and the docs #1905 @jacopofar
- L036 docstring refinements #1903 @jpy-git
- Add
exclude_rules
option for the Simple API #1850 @tunetheweb - Tox improvements: Streamline development/testing environments. #1860 @jpy-git
- Add Tox publish commands #1853 @jpy-git
- Documentation: Change inheritance dialect example to Redshift #1900 @chwiese
- Remove failing requires.io badge #1898 @jpy-git
- [Snowflake] Allow naked AUTOINCREMENT #1883 @gordonhart
- Add support for curly brackets in SnowSQL ampersand variables #1901 @chwiese
- Add short form help option (-h) #1947 @jpy-git
- Remove plaintext API key from benchmark utility #1863 @jpy-git
- Add
skip_install
to static analysis sections of tox.ini #1851 @jpy-git - Move typing_extensions from
requirements_dev.txt
torequirements.txt
#1956 @jpy-git
- Fix bug where "sqlfluff fix" deletes dbt "{% snapshot %}" line #1907 @barrywhart
- Fix subquery bug in L026 #1948 @jpy-git
- Fix bug where L041 was confused by L016's placement of newlines in the parse tree #1904 @barrywhart
- Fix progressbar artifacts within linter errors #1873 @adam-tokarski
- Correct Snowflake warehouse sizes #1872 @jpy-git
- Fixed Delimited() logic, added T-SQL grammar #1894 @WittierDinosaur
- L036 refinement - FROM clause interaction #1893 @jpy-git
- Add missing chardet install in setup.py #1928 @jpy-git
- Fix misplaced TableAliasInfo in L031 documentation #1946 @jpy-git
- Fix broken link to external SQL style guide #1918 @kevinmarsh
Fixes missing dependency issue with 0.8.0 for tqdm
, plus add a test to ensure this does not happen again.
- Fix: add tqdm to setup.py installation requirements #1842 @skykasko
- Add test to ensure pip install works #1843 @tunetheweb
This release brings an improvement to the performance of the parser, a rebuild of the Jinja Templater, and a progress bar for the CLI. Lots of dialect improvements have also been done. Full list of changes below:
- Updated L009 logic to only allow a single trailing newline. #1838 @jpy-git
- Progressbar utility #1609 @adam-tokarski
- Teradata dialect: Add support for SEL form of SELECT #1776 @samlader
- Added trigger support in ANSI - and extended it in Postgres #1818 @WittierDinosaur
- Exasol: Make references more strict #1829 @sti0
- Hive: INSERT statement support #1828 @mifercre
- ANSI: Add TABLESAMPLE support #1811 @CrossNox
- T-SQL: Support trailing commas in CREATE TABLE #1817 @tommydb
- Spark3: Add CREATE VIEW support #1813 @DipeshCS
- BigQuery: Support PIVOT and UNPIVOT #1794 @tunetheweb
- L029: Optionally check quoted identifiers in addition to naked identifiers #1775 @jpers36
- Add sysdate to Redshift as a bare function #1789 @tdstark
- Robust Jinja raw/template mapping #1678 @barrywhart
- Add CREATE TABLE AS to Postgres and Redshift #1785 @tdstark
- Improve Parser Performance By Caching Values #1744 @WittierDinosaur
- templater-dbt: Change dbt dependency to dbt-core #1786 @amardeep
- T-SQL: Create Schema definition #1773 @jpers36
- T-SQL: allow optional brackets for column default constraints #1760 @nevado
- Postgres: Support parameters and identifiers prepended with _ and containing $ #1765 @WittierDinosaur
- Postgres: Added support for double precision #1764 @WittierDinosaur
- "sqlfluff fix": Write to a temporary .sql file first #1763 @barrywhart
- Update older dbt dependency #1756 @alanmcruickshank
- T-SQL: add IDENTITY column constraint #1757 @nevado
- Update CI to run under Python 3.10 #1739 @rooterkyberian
- MySQL: Add drop index support #1738 @fatelei
- Snowflake dialect improvements #1737 @tunetheweb
- Add missing test case #1735 @tunetheweb
- Fix: Add missing init file to sqlfluff.core.templaters.slicers #1826 @CrossNox
- Hive: Fix order of CREATE TEMPORARY EXTERNAL TABLE #1825 @mifercre
- T-SQL: add AS keyword as optional in PIVOT-UNPIVOT #1807 @tkachenkomaria244
- Prevent L019 plus L034 corrupting SQL #1803 @barrywhart
- L028 fix - Allow SELECT column alias in WHERE clauses for certain dialects #1796 @tunetheweb
- Comment out instructions in GitHub templates #1792 @tunetheweb
- Fix internal error in L016 when template/whitespace-only line too long #1795 @barrywhart
- Fix L049 to allow = NULL in SET clauses #1791 @tunetheweb
- Hive: Fix bug in CREATE TABLE WITH syntax #1790 @iajoiner
- Fixed encoding error when linting to file #1787 @WittierDinosaur
- Fix L012 documentation #1782 @jpers36
- T-SQL: fix quote alias #1766 @jpers36
- Fix incorrect indentation issue #1733 @tunetheweb
- T-SQL: Fix OVER functionality for functions #1731 @jpers36
Highlights of this release contains a lot of T-SQL dialect improvements (shout out to @jpers36 for most of these!). We also added Spark3 as a new dialect thanks to @R7L208. The complete list of changes are shown below.
- T-SQL: Add rank functions #1725 @jpers36
- Spark3 Dialect Support #1706 @R7L208
- Postgres Array Support #1722 @WittierDinosaur
- Hive: Add LEFT SEMI JOIN support #1718 @fatelei
- MySQL: Change and drop column in alter table #1670 @MontealegreLuis
- Added type hints to some rule files #1616 @ttomasz
- Added Redshift to README #1720 @WittierDinosaur
- Exasol: Fix create table statement #1700 @sti0
- T-SQL: Add optional delimiter to SET #1717 @jpers36
- T-SQL: Escaped quotes #1715 @jpers36
- T-SQL: SELECT INTO #1714 @jpers36
- Postgres: Added support for psql variables #1709 @WittierDinosaur
- T-SQL: split location clause out from index clause #1711 @jpers36
- T-SQL: Override ANSI HAVING #1707 @jpers36
- T-SQL: Add UPDATE STATISTICS #1703 @jpers36
- T-SQL: CTAS Option Clause #1705 @jpers36
- T-SQL: DECLARE has optional AS #1704 @jpers36
- T-SQL: DROP STATISTICS and INDEX #1698 @jpers36
- T-SQL: CTAS select can be optionally bracketed #1697 @jpers36
- Exasol: Make function_script_terminator more strict #1696 @sti0
- T-SQL distribution index location #1695 @jpers36
- T-SQL: allow for non-alphanumeric initial characters in delimited identifiers #1693 @jpers36
- T-SQL: allow for semi-colon after BEGIN in a BEGIN/END block #1694 @jpers36
- Exasol: Fix adapter script syntax #1692 @sti0
- T-SQL: Basic EXECUTE functionality #1691 @jpers36
- T-SQL: Add #, @ to valid identifier characters #1690 @jpers36
- T-SQL - add support for Filegroups in table create #1689 @nevado
- Exclude Exasol scripts from rule L003 #1684 @tunetheweb
- Added PostGIS keyword data types to Postgres #1686 @WittierDinosaur
- Indent LIMIT values if on separate line #1683 @tunetheweb
- Postgres: Added support for SELECT INTO statements #1676 @WittierDinosaur
- Allow :: casting of CASE statements #1657 @tunetheweb
- Add more keywords to Redhift and BigQuery to avoid errors #1671 @tunetheweb
- T-SQL begin end delimiter #1664 @jpers36
- Teradata: Added date as bare function for #1663 @anzelpwj
- T-SQL: CREATE STATISTICS #1662 @jpers36
- T-SQL table and query hints #1661 @jpers36
- T-SQL: Allow spaces in qualified names #1654 @jpers36
- EXASOL: Fix typo in alter_table_statement #1726 @sti0
- Fix markdown links in production.rst #1721 @asottile
- Correct contributing testing information #1702 @adam-tokarski
- More ORDER BY clarifications #1681 @tunetheweb
- Fix T-SQL L025 linter exception #1677 @tunetheweb
- Improve Jinja whitespace handling in rules #1647 @barrywhart
BREAKING CHANGE
This release extracts the dbt templater to a separately installable plugin
sqlfluff-templater-dbt.
For users who take advantage of the dbt templater see the
updated docs on how to migrate.
It also adds the redshift
dialect and removes the exasol_fs
dialect which has been merged
into the exasol
dialect.
- src/sqlfluff/core/linter: Improve ignore file processing #1650 @CyberShadow
- Misc documentation updates #1644 @tunetheweb
- Segregate dbt plugin tests #1610 @alanmcruickshank
- Add initial Redshift support #1641 @tunetheweb
- Update docs for dbt templater, improve error messages when not installed. #1583 @alanmcruickshank
- Make templaters pluggable and move the dbt templater into a plugin #1264 @alanmcruickshank
- T-SQL: CTAS delimiter #1652 @jpers36
- T-SQL: Allow for multiple variables DECLAREd in the same statement #1651 @jpers36
- T-SQL: Allow DECLARE/SET statements to parse using ExpressionStatement #1649 @jpers36
- T-SQL PRINT statement parsing #1648 @jpers36
- Better date function for tsql #1636 @tunetheweb
- T-SQL: Allow for multiple statements in a procedure #1637 @jpers36
- T-SQL: Allow for !>, !< operators #1640 @jpers36
- T-SQL: Fix GROUP BY delimiter #1635 @jpers36
- T-SQL: Fix DROP delimiter #1633 @jpers36
- T-SQL: +RENAME statement for Azure Synapse Analytics #1631 @jpers36
- T-SQL: Fix CASTing variables #1627 @jpers36
- Snowflake: Add implementation for CREATE TASK statement #1597 #1603 @JoeHut
- Allow global config for rule testcases #1580 @sti0
- Snowflake dollar sign literals #1591 @myschkyna
- Rename test/fixtures/parser directory to test/fixtures/dialects #1585 @tunetheweb
- Rename keyword files #1584 @tunetheweb
- Add some more unreserved keywords to BigQuery #1588 @tunetheweb
- Increase minimum runs before coverage report is issued #1596 @tunetheweb
- Snowflake: Support CURRENT_TIMESTAMP as a column default value #1578 @wong-codaio
- T-SQL temp tables #1574 @jpers36
- Fix NoneType exception in L031 #1643 @tunetheweb
- Stop rule L048 complaining if literal is followed by a semicolon #1638 @tunetheweb
- L031 desc updated to cover both 'from' and 'join' #1625 @nevado
- Snowflake auto increments fixes #1620 @myschkyna
- Fix DECLARE Delimitation #1615 @jpers36
- Snowflake drop column fixes #1618 @myschkyna
- T-SQL: fix statement delimitation #1612 @jpers36
- Snowflake: Fixed data type casting not working in
SET
statement #1604 @wong-codaio - Postgres dialect: Fix parse error for "on delete", "on update" clauses in column constraints #1586 @samlader
- Fix AttributeError: 'NoneType' object has no attribute 'get_child' error with rule L031 #1595 @barrywhart
- Fix zero length templated file bug. #1577 @alanmcruickshank
- Fully remove exasol_fs dialect and bump version #1573 @alanmcruickshank
Another dbt bugfix from 0.6.7 and 0.6.8, plus a host of dialect and syntax improvements.
- Correct and expand Snowflake CREATE TABLE syntax [#1567] @tunetheweb
- Support brackets in Postgres Meta commands #1548 @tunetheweb
- added type hints to rule files #1515 @ttomasz
- Update Rule L028 to handle T-SQL PIVOT columns #1545 @tunetheweb
- T-SQL IF/ELSE #1564 @jpers36
- Enums for format types and colors added #1558 @adam-tokarski
- Add dbt 0.21.0 to the test suite #1566 @alanmcruickshank
- Merge EXASOL_FS dialect into EXASOL dialect #1498 @sti0
- T-SQL - BEGIN/END blocks #1553 @jpers36
- Small refactor with type hints and string formatting #1525 @adam-tokarski
- Add Github Preview Image #1557 @alanmcruickshank
- Support SETOF in Postgres #1522 @tunetheweb
- Support Double Precision in ANSI #1524 @tunetheweb
- Support LATERAL joins in Postgres #1519 @adam-tokarski
- Add a rule to warn on "= NULL" or "<> NULL" comparisons #1527 @barrywhart
- Support Group and Groups as table names #1546 @tunetheweb
- Support more complex IN (...) expressions #1550 @tunetheweb
- Support CROSS APPLY and OUTER APPLY and TOP in T-SQL #1551 @tunetheweb
- Add support for WITHOUT ROWID to SQLite #1531 @tunetheweb
- Postgres: add
CONCURRENTLY
andFINALIZE
keywords toDETACH PARTITION
#1529 @kevinmarsh - Better support of MySQL CREATE TABLE TIMESTAMP/DATESTAMP #1530 @tunetheweb
- "Found unparsable section" instead of stack trace when multiple semicolons provided #1517 @adam-tokarski
- Fix test coverage #1569 @tunetheweb
- Remove lint_templated_tokens as no longer does anything #1570 @tunetheweb
- Fix broken block comments in exasol #1565 @sti0
- Rethink sequence_files in dbt templater. #1563 @alanmcruickshank
- T-SQL: fix STRING_AGG() WITHIN GROUP clause #1559 @jpers36
- fix spelling: occurrence>occurrence #1507 @jpers36
Fixed a DBT bug introduced in 0.6.7 - apologies!
SQLFluff can't find dbt models #1513 @barrywhart T-SQL: Support for unicode literals #1511 @adam-tokarski
Lots of fixes to our rules (particularly when running sqlfluff fix
, and particularly for Jinja and DBT templates). We also have good improvements to Exasol, Snowflake, and T-SQL dialects amongst others. Plus we added Hive and SQLite as supported dialects!
- Snowflake better WAREHOUSE and CREATE (EXTERNAL) TABLES support #1508 @tunetheweb
- Exasol: Fix typo in
REORGANIZE
statement #1509 @sti0 - Fix bug that can prevent linting ephemeral dbt models #1496 @barrywhart
- Disable rules L026 and L028 for BigQuery by default, with option to re-enable #1504 @tunetheweb
- BigQuery keywords #1506 @tunetheweb
- Inline --noqa not always honoured by "sqlfluff fix" #1502 @barrywhart
- Snowflake - fix parsing of UNPIVOT #1505 @michael-the1
- Better parsing of DATEADD function #1486 @jpers36
- Fix handling of ISNULL and NOTNULL keywords #1483 @leamingrad
- Improved test cases names #1501 @ttomasz
- Exasol: Fix CREATE TABLE in-/outline constraint / Adjusted DISTRIBUTE/PARTITION clause #1491 @sti0
- Add support for SnowSQL variables #1497 @samlader
- Ignore erroneous newline segments in L016 (e.g. Jinja for loops) #1494 @tunetheweb
- Indentation error on Jinja templated test case #1444 @barrywhart
- Improve EXASOL dialect #1484 @sti0
- T-SQL dialect - +support for CONVERT() special function #1489 @jpers36
- Allow Postgres column references to use
AT TIME ZONE
#1485 @leamingrad - T-SQL dialect - provide alternate ASA PR incorporating ASA into T-SQL #1478 @jpers36
- Modest parser performance improvement #1475 @NathanHowell
- Disable rule L033 for dialects that do not support it (e.g. Exasol, Postgres) #1482 @tunetheweb
- Adding a new BaseFileSegment class for FileSegments to inherit from #1473 @sti0
- EXASOL_FS: Fix adapter script type #1480 @sti0
- Dialect/tsql update - added pivot / unpivot, view support, sequence support on table creation #1469 @ericmuijs
- Correct typo in SQLFluff name #1470 @tunetheweb
- Stop L008 from adding spaces for simple SELECTs #1461 @CyberShadow
- Add SQLite dialect #1453 @tunetheweb
- Fix Windows Clause for Exasol #1463 @tunetheweb
- Add CHECK constraint syntax to ANSI SQL #1451 @tunetheweb
- Move Exasol test statements fixtures from Python to SQL files #1449 @tunetheweb
- fix spelling of "preceding" #1455 @jpers36
- Add NORMALIZE to Teradata dialect #1448 @tunetheweb
- Add @ and $ symbols to Exasol to avoid lexing errors #1447 @tunetheweb
- Stop fix adding then removing whitespace #1443 @barrywhart
- Stop exception in L016 for long Jinja comments #1440 @tunetheweb
- Fix some issues where the SQL file is corrupted by lint "fixes" in or near Jinja loops #1431 @barrywhart
- T-SQL: Remove Limit and NamedWindow segments as not supported in T-SQL #1420 @jpers36
- Fix runtime error (IndexError ) when linting file with jinja "if" #1430 @barrywhart
- Add Hive dialect (#985) @satish-ravi
- Further fix for L036 #1428 @tunetheweb
- Add default parameter to dbt "var" macro stub #1426 @CyberShadow
Fixed some of our autofix rules where running fix
sometimes made unintended changes. Added config to rules L011 and L012 to allow preferring implicit aliasing. Also further improved our Postgres support and documentation.
- Rule L036 bug fixes #1427 @tunetheweb
- Added support for psql meta commands to Postgres #1423 @WittierDinosaur
- Remaining line endings #1415 @tunetheweb
- T-SQL: Remove match possibilities for segments with no T-SQL equivalent #1416 @jpers36
- Fix generate error on test file with just a comment #1413 @tunetheweb
- Misc fixes to workflow files #1412 @tunetheweb
- Added support for escape character strings to Postgres #1409 @WittierDinosaur
- Issue 845: L016 should compute line length prior to template expansion #1411 @barrywhart
- Add .editorconfig config and enforce style rules #1410 @tunetheweb
- Allow optional enforcing of implicit aliasing of tables (L011) and columns (L012) #1402 @tunetheweb
- Better error messages on error #1407 @tunetheweb
- Add README on how to generate docs #1403 @tunetheweb
- Fix extra underscores in case rules (L010 and L014) #1396 @tunetheweb
- Remove unused deps in tox test docbuild #1406 @zhongjiajie
- Prevent CodeCov commenting on coverage differences too early #1404 @tunetheweb
- Fix "sqlfluff fix compatible" rules indenting to much in documentation #1405 @tunetheweb
- Fix documentation SQL highlight error #1393 @zhongjiajie
- Support TIMESTAMPTZ in TIME ZONE queries for Postgres #1398 @tunetheweb
- Improve datatypes: CHARACTER VARYING for ANSI, and Postgres and also TIMESTAMP AT TIME ZONE for Postgres #1378 @WittierDinosaur
- Improve rules L003 and L019 by processing multi-line fixes in one pass. #1391 @barrywhart
- Correct codecov badge for Docs website #1390 @tunetheweb
- Change fix to use non-zero exit code if unfixable #1389 @tunetheweb
- Bugfix, frame clauses in window functions were not working #1381 @WittierDinosaur
- Handle template and unfixable errors when fixing stdin #1385 @nolanbconaway
- CREATE, ALTER, DROP SEQUENCE support, with Postgres extensions #1380 @WittierDinosaur
- Postgres analyze #1377 @WittierDinosaur
- L016: "sqlfluff fix" adds too many newlines #1382 @barrywhart
- L003 fix mixes hanging and clean indents #1383 @barrywhart
- L034 should not fix inside "INSERT" or "CREATE TABLE AS SELECT" #1384 @barrywhart
This release includes initial support of Transact-SQL (T-SQL), much better Postgres and Snowflake support, improvements to our documentation, 100% coverage for Python code (with a small number of accepted exceptions), along with numerous other bug fixes and improvements.
Many thanks to all the contributors helping to improve SQLFluff!
- Simplify rule L030 and fix recursion bug (#1376) (@tunetheweb
- Move from CircleCI to GitHub Actions for Continuous Integration (#1361) (@tunetheweb
- Postgres enhance create index (#1375) (@WittierDinosaur
- Initial support for Transact-SQL (T-SQL) dialect (#1313) (@ericmuijs
- Handle initial whitespace lines in rule L001 (#1372) (@tunetheweb
- Postgres Improved DEFAULT column constraint support (#1373) (@WittierDinosaur
- Minor grammar, spelling, and readability fixes (#1370) (@WittierDinosaur
- Issues 854, 1321: Handle Jinja leading whitespace-only lines (#1364) (@barrywhart
- Enhanced the Postgres grammar for create table (#1369) (@WittierDinosaur
- Added ability to Grant and Revoke Grant to multiple users (#1367) (@WittierDinosaur
- Add BigQuery Parameter Lexing and Parsing (#1363) (@rileyrunnoe
- Rule L030 bugfix (#1360) (@WittierDinosaur
- Add Postgres dialect for COMMENT ON (#1358) (@miketheman
- Allow ORDER BY and LIMIT after QUALIFY in BigQuery (#1362) (@tunetheweb
- Correct CircleCI badge reference (#1359) @miketheman
- Minor grammar corrections to documentation (#1355) @miketheman
- Pytest coverage exceptions to get us to 100% coverage! (#1346) @tunetheweb
- Greatly improved Snowflake syntax support (#1353) @tunetheweb
- Postgres keyword support (#1347) @WittierDinosaur
- Added full support for postgres's ALTER DEFAULT PRIVILEGES. (#1350) @creste
- Show all LintResult in Rule_L020 (#1348) @zhongjiajie
- Enhance error message L010 base on configure (#1351) @zhongjiajie
- Remove unused variable insert_str (#1352) @zhongjiajie
- Pytest coverage exceptions for Core code - part 1 (#1343) @tunetheweb
- BigQuery: Allow Qualify Clause for UnorderedSelectStatements (#1341) @tunetheweb
- Postgres "ALTER TABLE" enhancement, and timestamp bug fix (#1338) @WittierDinosaur
- Improve pytest coverage for non-core code (#1319) @tunetheweb
- Support additional GRANTs in Postgres (#1339) @creste
- Allow optional alias for BigQuery WITH OFFSET (#1330) @tunetheweb
- Improve function support in Postgres dialect (#1336) @WittierDinosaur
- Using github star instead of watch in docs (#1337) @zhongjiajie
- Add unittest for rules docstring (#1335) @zhongjiajie
- Bugfix PR, fixes issue #1333, wherein test___main___help() defaults to your default Python installation (#1334) @WittierDinosaur
- Improve wording of L007 now the before/after is configurable (#1325) @tunetheweb
- Fix a couple of small issues with CI jobs (#1326) @tunetheweb
- Add updated sqlfluff graphics and source. (#1315) @alanmcruickshank
- Issue 1277: Enforce that YML test files are computer generated and not edited (#1279) @barrywhart
- Fix typo in README (#1320) @tunetheweb
- Fix link in README (#1316) @jmks
- Update documentation to make the project more discoverable (#1311) @tunetheweb
- Show latest version number on unsupported Python error message (#1307) @zhongjiajie
- Fix typo in github PR template (#1308) @zhongjiajie
- Added support for empty WINDOWS specifications (#1293) @matthieucan
- Added auto release drafter (#1287) @tunetheweb
- Fix typo in the in the wild page (#1285) @tunetheweb
- Fix spacing issue for BigQuery UNNEST statement for rules L003 and L025 (#1303) @tunetheweb
- Update GitHub templates (#1297) @tunetheweb
- Allow BigQuery UDF with triple quoted bodies to pass rule L048 (#1300) @tunetheweb
- Add Parameterless Functions and more function names support to BigQuery (#1299) @tunetheweb
- Add release drafter (#1295) @tunetheweb
- Support empty OVER() clause in Window Specification (#1294) @tunetheweb
- Fix typo on the In the Wild page (#1285) @tunetheweb
- Support for primary index name, collect stats improvement, COMMENT statement for teradata dialect #1232
- Support config for L007 to prefer end of line operators #1261
- Support for DETERMINISTIC user defined functions in BigQuery dialect #1251
- Support more identifiers in BigQuery dialect #1253
- Support function member field references in BigQuery dialect #1255
- Support alternative indentation for USING and ON clauses #1250
- Support COUNT(0) preference over COUNT(*) or COUNT(1) #1260
- Support for BigQuery "CREATE table OPTIONS ( description = 'desc' )" #1205
- Support wildcard member field references in BigQuery dialect #1269
- Support ARRAYS of STRUCTs in BigQuery dialect #1271
- Support fields of field references in BigQuery dialect #1276
- Support OFFSET and ORDINAL clauses of Array Functions in BigQuery dialect #1171
- Added check for generated YML files #1277
- Support QUALIFY to BigQuery dialect #1242
- Fix comma removed by L019 #939
- Update L019 (leading/trailng comma rule) so it doesn't run on unparsable code.
- The
--nocolor
command-line option should suppress emoji output #1246 - Added HTTP Archive to the In The Wild page
- Support for looping statements (loop, while, repeat) and supporting statements to mysql dialect #1180
- Added dbt 0.20.* to the default test suite.
- Updated manifest loading in dbt 0.20.* to use the new
ManifestLoader
#1220 - Handle newlines in rule list configuration in .sqlfluff #1215
- Fix looping interaction between L008 and L030 #1207
- Linting output now supports GitHub Actions #1190
- Support for QUALIFY syntax specific to teradata dialect #1184
- Support for TRUNCATE statement #1194
- Support for prepared statement syntaxes specific to mysql dialect #1147
- Support for GET DIAGNOSTICS statement syntax specific to mysql dialect #1148
- Support for cursor syntax specific to mysql dialect #1145
- Support sequential shorthand casts #1178
- Support for select statement syntax specific to mysql dialect #1175
- Support for the CALL statement for the mysql dialect #1144
- Support for OVERLAPS predicate #1091
- Support for the CREATE/DROP PROCEDURE statement for the mysql dialect #901
- Specific allowed/required syntaxes for CREATE/DROP FUNCTION within the mysql dialect #901
- Support for DECLARE statement for the mysql dialect #1140
- Support for the IF-THEN-ELSEIF-ELSE syntax for the mysql dialect #1140
- Support for the DEFINER syntax for the mysql dialect #1131
- Preserve existing file encoding in the "fix" command. Partially addresses #654
- Support for DECLARE and SET variable syntax for the BigQuery dialect #1127
- Support for ALTER TASK statement on Snowflake #1211
- Fix runtime error in diff-cover plugin caused by new diff-cover release 6.1.0 #1195
- Resolved an issue with the snowflake dialect where backslash escaped single quoted strings led to fatal lexing errors #1200
- [@GitHub-Username](Link to GitHub profile) ([#PR-Number](Link to PR))
- @dflss (#1154)
- @barrywhart (#1177, #1195)
- @niallrees (#1178)
- @barnabyshearer (#1194)
- @silverbullettruck2001 (#1141, #1159, #1161, #1176, #1179, #1181, #1193, #1203)
- Respect XDG base dirs on Mac OS (#889).
- Added support for additional delimiters by creating a new DelimiterSegment in the ANSI dialect which defaults to the semicolon, but allows it to be more intuitive when overridden in a specific child dialect (mysql) #901)
- Added support for the DELIMITER statement in the mysql dialect #901)
- Added support for additional delimiters by creating a new DelimiterSegment in the ANSI dialect which defaults to the semicolon, but allows it to be more intuitive when overridden in a specific child dialect (mysql) #901)
- Added support for function as a default column value #849.
- Add an
--include-metas
option for parse output to show the meta segments in the parse tree. - Allow CREATE FUNCTION syntax without arguments @bolajiwahab #1063.
- Added support for the CREATE/DROP PROCEDURE statement for the mysql dialect #901)
- Added specific allowed/required syntaxes for CREATE/DROP FUNCTION within the mysql dialect #901)
- Now possible to run sqlfluff commands outside the project root when using the dbt templater.
- Renamed --parallel CLI argument to --processes to be more accurate.
- L034 now ignores select statements which contain macros.
- L034 now ignores select statements part of a set expression, most commonly a union.
- Fix bug #1082, adding
support for BigQuery
select as struct '1' as bb, 2 as aa
syntax. - Rationalisation of the placement of meta segments within templated queries to support more reliable indentation. This includes prioritising longer invariant sections first and then dropping any shorter ones which then are relatively out of place.
- Additional logging within the lexer and templater engines.
- Allow meta segments to parse within
Delimited
grammars which otherwise don't allow gaps. This is facilitated through an optional argument totrim_non_code_segments
. - Fix bug #1079, addressing
issues with L025 and L026 with BigQuery column references involving
STRUCT
. - #1080 Add SET SCHEMA and DROP SCHEMA support to ANSI dialect.
- Better exception handling for the simple parsing API (
sqlfluff.parse
) which now raises an exception which holds all potential parsing issues and prints nicely with more than one issue. - Fix bug #1037, in which fix logging had been sent to stdout when reading data from stdin.
- Add a little bit of fun on CLI exit 🎉!
- Disabled models in the dbt templater are now skipped entirely rather than returning an untemplated file.
- Add a changelog check to SQLFluff continuous integration.
- Fix bug #1083, adding support for BigQuery named function arguments, used with functions such as ST_GEOGFROMGEOJSON()
- Update documentation links to sqlfluff-online.
- Lint and fix parallelism using
--parallel
CLI argument - Fix 1051, adding support
for bitwise operators
&
,|
,^
,<<
,>>
- Bugfix release for an issue in
L016
introduced in0.5.4
. - Fix for
L016
issue whereDISTINCT
keywords were mangled during fixing #1024.
- Bugfix release for an off-by-one error introduced in L016 as part of
0.5.4
.
- Parsing of Postgres dollar quoted literals.
- Parsing of Postgres filter grammar.
- Parsing of "ALTER DEFAULT PRIVILEGES" Postgres statement.
- Parsing of Postgres non-explicit role granting and function execution.
- Early failing on fatal dbt templater fails.
- Big rewrite of the lexer, segments and position markers for simplicity and to support future parallelism work.
- Fix to L036 which previously mangled whitespace.
L009
can now be enforced whentemplater = dbt
.- Parsing of
EXPLAIN
,USE
statements. - Parsing of
ALTER TABLE x RENAME TO y
syntax. - Parsing of
ALTER SESSION
in snowflake. - Parsing of numeric literals with exponents.
- Added rule codes to diff_cover output.
- Fix
templater = dbt
L009 bug #861 where:sqlfluff lint
would incorrectly always returnL009 | Files must end with a trailing newline.
sqlfluff fix
would remove trailing newlines whenexclude_rules = L009
.
- Fix bug with BigQuery comparison operators.
- Fix recursion bug with L045.
- Fix tuple index bug with L016.
- Fix mange coalecse bug with L043.
- Fix Jinja templating error with UnboundLocalError.
- Improve array parsing.
- Simplify bracket parsing.
- Speed up L010 with caching capitalisation policy.
- Output of
sqlfluff dialects
is now sorted. - Handle disabled
dbt
models.
- Fix false positive in L045 when CTE used in WHERE clause (#944)
- Logging and readout now includes more detail and a notification of dbt compilation.
- Fix bug in L048 which flagged adjoining commas as failures.
- Fix bug in L019 with inline comments.
- Fix bug in L036 with multiple newlines.
- Skip disabled dbt models. (#931).
- Support "USE" statement in ANSI (#902).
- Parse explain statement (#893).
- Parsing improvements around optional brackets.
- Better parsing of set operators (like
UNION
) and how they interact withORDER BY
clauses. - Support for comparison operators like
~
. - Fix parsing of snowflake
SAMPLE
syntax. - Fix recursion issues in L044.
SPACE
keyword now has no special meaning in the postgres dialect.
pascal
(PascalCase)capitalisation_policy
option for L014 (unquoted identifiers)only_aliases
configuration option for L014 (unquoted identifiers)- Dialects now have more advanced dependency options to allow less repetition
between related dialects. The methods
get_segment
andget_grammar
can be used on unexpanded grammars to access elements of the parent grammars. Thecopy
method on grammars can be used to copy with alterations. - Rule L046 to line whitespace within jinja tags.
- Enable and Disable syntax for ignoring violations from ranges of lines.
- Renamed the BaseCrawler class to BaseRule. This is the base class for all rules. This is a breaking change for any custom rules that have been added via plugins or by forking the SQLFluff repo.
- Renamed
sqlfluff.rules()
tosqlfluff.list_rules()
andsqlfluff.dialects()
tosqlfluff.list_dialects()
due to naming conflicts with the now separatesqlfluff.dialects
module. - Extracted dialect definitions from the
sqlfluff.core
module so that each dialect is better isolated from each other. This also allows more focused testing and the potential for dialect plugins in future. Dialects are now only imported as needed at runtime. All dialects should now be accessed using the selector methods insqlfluff.core.dialects
rather than importing fromsqlfluff.dialects
directly. - Add support for
ALTER USER
commands in Snowflake dialect. - Added describe statement to ANSI dialect
- Renamed
capitalisation_policy
toextended_capitalisation_policy
for L014 to reflect the fact that it now accepts more options (pascal
) than regularcapitalisation_policy
still used by L010 and others. - Replaced
only_aliases
config withunquoted_identifiers_policy
and added it to rule L014 in addition to L029. - Parse structure of
FROM
clauses to better represent nested joins and table functions. - Parse structure of expressions to avoid unnecessary nesting and overly recursive method calls.
- Initial architecture for rule plugins to allow custom rules. This initial release should be considered beta until the release of 0.5.0.
- Add tests for dbt 0.19.0.
- General increased parsing coverage.
- Added some missing Postgres syntax elements.
- Added some basic introspection API elements to output what dialects and rules are available for use within the API.
- Fix several Snowflake parsing bugs.
- Refactor from clause to handle flattens after joins.
- Fix .get_table_references() in Snowflake dialect.
- Macros defined within the .sqlfluff config will take precedence over the macros defined in the
path that is defined with config value
sqlfluff:templater:jinja:load_macros_from_path
. - Fix Snowflake indent parsing.
- Fixed incorrect parsing of syntax-like elements in comments.
- Altered parsing of
NULL
keywords, so parse as Literals where appropriate. - Fixed bug in expression parsing leading to recursion errors.
- Public API to enable people to import
sqlfluff
as a python module and callparse
,lint
andfix
within their own projects. See the docs for more information. (#501) - The ability to use
dbt
as a templating engine directly allowing richer and more accurate linting arounddbt
macros (and packages related todbt
). For more info see the docs. (#508) - Support for modulo (
%
) operator. (#447) - A limit in the internal fix routines to catch any infinite loops. (#494)
- Added the
.is_type()
method on segments to more intelligently deal with type matching in rules when inheritance is at play. - Added the ability for the user to add their own rules when interacting
with the
Linter
directly usinguser_rules
. - Added L034 'Fields should be stated before aggregates / window functions' per dbt coding convenventions (#495)
- Templating tags, such as
{{ variables }}
,{# comments #}
and{% loops %}
(in jinja) now have placeholders in the parsed structure. Rule L003 (indentation), also now respects these placeholders so that their indentation is linted accordingly. For loop or block tags, they also generate anIndent
andDedent
tag accordingly (which can be enabled or disabled) with a configuration value so that indentation around these functions can be linted accordingly. (#541) - MyPy type linting into a large proportion of the core library. (#526, #580)
- Config values specific to a file can now be defined using a comment
line starting with
-- sqlfluff:
. (#541) - Added documentation for
--noqa:
use in rules. (#552) - Added
pre-commit
hooks forlint
andfix
. (#576) - Added a fix routine for Rule L019 (comma placement). (#575)
- Added Rule L031 to enforce "avoid using alias in the
FROM
/JOIN
clauses" from thedbt
coding conventions. (#473, #479) - Added Rule L032 to enforce "do not use
USING
" from thedbt
coding conventions. (#487) - Added Rule L033 to enforce "prefer
UNION ALL
toUNION *
" from thedbt
coding conventions. (#489) - Added Rule L034 to enforce "fields should be stated before aggregate/window functions" from the
dbt
coding conventions. (#495) - Added Rule L038 to forbid (or require) trailing commas in select clauses. (#362)
- Added Rule L039 to lint unnecessary whitespace between elements. (#502)
- Added a fix routine for L015. (#732)
- Added a fix routine for L025. (#404)
- Adopted the
black
coding style. (#485) - Added validation and documentation for rule configuration options. (#462)
- Added documentation for which rules are fixable. (#594)
- Added
EPOCH
keyword for postgres dialect. (#522) - Added column index identifier in snowflake dialect. (#458)
- Added
USE
statement to the snowflake dialect. (#537) - Added
CODE_OF_CONDUCT
to the project. (#471) - Added
ISNULL
andNOTNULL
keywords to ansi dialect. (#441) - Added support for python 3.9. (#482)
- Added
requirements_dev.txt
for local testing/linting. (#500) - Added CLI option
--disregard-sqlfluffignores
to allow direct linting of files in the.sqlfluffignore
. (#486) - Added
dbt
incremental
macro. (#363) - Added links to cockroachlabs expression grammars in ansi dialect. (#592)
- Added favicon to the docs website. (#589)
- Added
CREATE FUNCTION
syntax for postgres and for bigquery. (#325) - Added
CREATE INDEX
andDROP INDEX
for mysql. (#740) - Added
IGNORE NULLS
,RESPECT NULLS
,GENERATE_DATE_ARRAY
andGENERATE_TIMESTAMP_ARRAY
for bigquery. ( #667, #527) - Added
CREATE
andCREATE ... CLONE
for snowflake. (#539) - Added support for EXASOL. (#684)
- Fixed parsing of semi-structured objects in the snowflake of dialects with whitespace gaps. #634
- Handle internal errors elegantly, reporting the stacktrace and the error-surfacing file. #632
- Improve message for when an automatic fix is not available for L004. #633
- Linting errors raised on templated sections are now ignored by default and added a configuration value to show them. (#713)
- Big refactor of logging internally.
Linter
is now decoupled from logging so that it can be imported directly by subprojects without needing to worry about weird output or without the log handing getting in the way of your project. (#460) - Linting errors in the final file are now reported with their position in the source file rather than in the templated file. This means when using sqlfluff as a plugabble library within an IDE, the references match the file which is being edited. (#541)
- Created new Github Organisation (https://github.com/sqlfluff) and migrated from https://github.com/alanmcruickshank/sqlfluff to https://github.com/sqlfluff/sqlfluff. (#444)
- Changed the handling of
*
anda.b.*
expressions to have their own expressions. Any dependencies on this structure downstream will be broken. This also fixes the linting of both kinds of expressions with regard to L013 and L025. (#454) - Refactor of L022 to handle poorly formatted CTEs better. (#494)
- Restriction of L017 to only fix when it would delete whitespace or newlines. (#598)
- Added a configuration value to L016 to optionally ignore lines containing only comments. (#299)
- Internally added an
EphemeralSegment
to aid with parsing efficiency without altering the end structure of the query. (#491) - Split
ObjectReference
intoColumnReference
andTableReference
for more useful API access to the underlying structure. (#504) KeywordSegment
and the newSymbolSegment
both now inherit from_ProtoKeywordSegment
which allows symbols to match in a very similar way to keywords without later appearing with thetype
ofkeyword
. (#504)- Introduced the
Parser
class to parse a lexed query rather than relying on users to instantiate aFileSegment
directly. As a result theFileSegment
has been moved from the core parser directly into the dialects. Users can refer to it via theget_root_segment()
method of a dialect. (#510) - Several performance improvements through removing unused functionality, sensible caching and optimising loops within functions. (#526)
- Split up rule tests into separate
yml
files. (#553) - Allow escaped quotes in strings. (#557)
- Fixed
ESCAPE
parsing inLIKE
clause. (#566) - Fixed parsing of complex
BETWEEN
statements. (#498) - Fixed BigQuery
EXCEPT
clause parsing. (#472) - Fixed Rule L022 to respect leading comma configuration. (#455)
- Improved instructions on adding a virtual environment in the
README
. (#457) - Improved documentation for passing CLI defaults in
.sqlfluff
. (#452) - Fix bug with templated blocks +
capitalisation_policy = lower
. (#477) - Fix array accessors in snowflake dialect. (#442)
- Color
logging
warnings red. (#497) - Allow whitespace before a shorthand cast. (#544)
- Silenced warnings when fixing from stdin. (#522)
- Allow an underscore as the first char in a semi structured element key. (#596)
- Fix PostFunctionGrammar in the Snowflake dialect which was causing strange behaviour in L012. (#619)
Bracketed
segment now obtains its brackets directly from the dialect using a set namedbracket_pairs
. This now enables better configuration of brackets between dialects. (#325)
- Dropped support for python 3.5. (#482)
- From the CLI, the
--no-safety
option has been removed, the default is now that all enabled rules will be fixed. (#583) - Removed
BaseSegment.grammar
,BaseSegment._match_grammar()
andBaseSegment._parse_grammar()
instead preferring references directly toBaseSegment.match_grammar
andBaseSegment.parse_grammar
. (#509) - Removed
EmptySegmentGrammar
and replaced with better non-code handling in theFileSegment
itself. (#509) - Remove the
ContainsOnly
grammar as it remained only as an anti-pattern. (#509) - Removed the
expected_string()
functionality from grammars and segments (#509) as it was poorly supported. - Removed
BaseSegment.as_optional()
as now this functionality happens mostly in grammars (includingRef
). (#509) - Removed
ColumnExpressionSegment
in favour ofColumnReference
. (#512) - Removed the
LambdaSegment
feature, instead replacing with an internal to the grammar module calledNonCodeMatcher
. (#512) - Case sensitivity as a feature for segment matching has been removed as not required for existing dialects. (#517)
- Dependency on
difflib
orcdifflib
, by relying on source mapping instead to apply fixes. (#541)
sqlfluff dialects
command to get a readout of available dialects [+ associated docs].- More helpful error messages when trying to run in Python2.
- Window functions now parse with
IGNORE
/RESPECT
NULLS
. - Parsing of
current_timestamp
and similar functions. Thanks @dmateusp. - Snowflake
QUALIFY
clause.
- Respect user config directories. Thanks @sethwoodworth.
- Fix incorrect reporting of L013 with
*
. Thanks @dmateusp. - Fix incorrect reporting of L027 with column aliases. Thanks @pwildenhain.
- Simplification of application of fixes and correction of a case where fixes could be depleted. Thanks @NiallRees.
- Fix functions with a similar structure to
SUBSTRING
. - Refactor BigQuery
REPLACE
andEXCEPT
clauses. - Bigquery date parts corrected.
- Snowflake array accessors.
- Psotgres
NOTNULL
andISNULL
. - Bugfix in snowflake for keywords used in semistructured queries.
- Nested
WITH
statements now parse. - Performance improvements in the
fix
command. - Numeric literals starting with a decimal now parse.
- Refactor the jinja templater.
- Patterns and Anti-patterns in documentation. Thanks @flpezet.
- Functions in
GROUP BY
. Thanks @flpezet.
- Deep bugfixes in the parser to handle simple matching better for a few edge cases. Also added some logging deeper in the parser.
- Added in the
SelectableGrammar
and some related segments to make it easier to refer to select-like things in other grammars. - Fixes to
CASE
statement parsing. Thanks @azhard. - Fix to snowflake
SAMPLE
implementation. Thanks @rkm3. - Numerous docs fixes. Thanks @SimonStJG, @flpezet, @s-pace, @nolanbconaway.
- Implementation of the bigquery
CREATE MODEL
syntax. Thanks @barrywhart. - Bugfixes for:
- Edge cases for L006
- False alarms on L025
ORDER BY x NULLS FIRST|LAST
FOR
keyword in bigquerySYSTEM_TIME
syntax.
- Added the
--nofail
option toparse
andlint
commands to assist rollout. - Added the
--version
option to complement theversion
option already available on the cli. - Parsing for
ALTER TABLE
. - Warning for unset dialects when getting parsing errors.
- Configurable line lengths for output.
- Support for the Teradata dialect. Thanks @Katzmann1983!
- A much more detailed getting started guide in the docs.
- For the
parse
command, added the--profiler
and--bench
options to help debugging performance issues. - Support for the
do
command in the jinja templater. - Proper parsing of the concatenate operator (
||
). - Proper indent handling of closing brackets.
- Logging and benchmarking of parse performance as part of the CI pipeline.
- Parsing of object references with defaults like
my_db..my_table
. - Support for the
INTERVAL '4 days'
style interval expression. - Configurable trailing or leading comma linting.
- Configurable indentation for
JOIN
clauses. - Rules now have their own logging interface to improve debugging ability.
- Snowflake and Postgres dialects.
- Support for a
.sqlfluffignore
file to ignore certain paths. - More generic interfaces for managing keywords in dialects, including
set
interfaces for managing and creating keywords and theRef.keyword()
method to refer to them, and the ability to refer directly to keyword names in most grammars using strings directly. IncludesSegmentGenerator
objects to bind dialect objects at runtime from sets. Thanks @Katzmann1983! - Rule
L029
for using unreserved keywords as variable names. - The jinja templater now allows macros loaded from files, and the
hydration of variables ending in
_path
in the config files. - JSON operators and the
DISTINCT ON ()
syntax for the postgres dialect.
- Refactor of whitespace and non-code handling so that segments are less greedy and default to not holding whitespace on ends. This allows more consistent linting rule application.
- Change config file reading to case-sensitive to support case sensitivity in jinja templating.
- Non-string values (including lists) now function in the python and jinja templating libraries.
- Validation of the match results of grammars has been reduced. In production cases the validation will still be done, but only on parse and not on match.
- At low verbosities, python level logging is also reduced.
- Some matcher rules in the parser can now be classified as simple which allows them to shortcut some of the matching routines.
- Yaml output now double quotes values with newlines or tab characters.
- Better handling on hanging and closing indents when linting rule L003.
- More capable handline of multi-line comments so that indentation and line length parsing works. This involves some deep changes to the lexer.
- Getting violations from the linter now automatically takes into account of ignore rules and filters.
- Several bugfixes, including catching potential infinite regress during fixing of files, if one fix would re-introduce a problem with another.
- Behaviour of the
Bracketed
grammar has been changed to treat its content as aSequence
rather than aOneOf
. - Move to
SandboxedEnvironment
rather thanEnvironment
for jinja templating for security. - Improve reporting of templating issues, especially for the jinja templater so that missing variables are rendered as blanks, but still reported as templating violations.
- Support for
a.b.*
on top ofa.*
in select target expressions.
- Deprecated python 2.7 and python 3.4 which are now both past their maintenance horizon. The 0.2.x branch will remain available for continued development for these versions.
- Rule L003 is now significantly smarter in linting indentation with support for hanging indents and comparison to the most recent line which doesn't have an error. The old (more simple) functionality of directly checking whether an indent was a multiple of a preset value has been removed.
- Fixed the "inconsistent" bug in L010. Thanks @nolanbconaway.
- Updated logging of parsing and lexing errors to have more useful error codes.
- Changed parsing of expressions to favour functions over identifiers to fix the expression bug.
- Fixed the "inconsistent" bug in L010. Thanks @nolanbconaway.
- Moved where the
SELECT
keyword is parsed within a select statement, so that it belongs as part of the newly renamedselect_clause
(renamed from previouslyselect_target_group
). - Clarified handling of the
type
andname
properties of the BaseSegment class and its children.name
should be specific to a particular kind of segment, andtype
should express a wider group. Handling of thenewline
,whitespace
andcomma
segments has been updated so that we use thetype
property for most use cases rather thanname
.
- Meta segments for indicating where things can be present in the parsed
tree. This is mostly illustrated using the
Indent
andDedent
segments used for indicating the position of theoretical indents in the structure. Several helper functions have been added across the codebase to handle this increase in the kinds of segments which might be encountered by various grammars. - Rule L016 has been added to lint long lines. In the
fix
phase of this rule, there is enough logic to try and reconstruct a sensible place for line breaks as re-flow the query. This will likely need further work and may still encounter places where it doesn't fix all errors but should be able to deal with the majority of simple cases. - BigQuery dialect, initially just for appropriate quoting.
- Added parsing of DDL statements such as
COMMIT
,DROP
,GRANT
,REVOKE
andROLLBACK
. Thanks @barrywhart. --format
option to theparse
command that allows a yaml output. This is mostly to make test writing easier in the development process but might also be useful for other things.- Parsing of set operations like
UNION
. - Support for the
diff-cover
tool. Thanks @barrywhart. - Enabled the
fix
command while usingstdin
. Thanks @nolanbconaway. - Rule to detect incorrect use of
DISTINCT
. Thanks @barrywhart. - Security fixes from DeepCover. Thanks @sanketsaurav.
- Automatic fix testing, to help support the newer more complicated rules.
- Interval literals
- Support for the
source
macro from dbt. Thanks @Dandandan - Support for functions with spaces between the function name and the brackets
and a linting rule
L017
to catch this. - Efficiency cache for faster pruning of the parse tree.
- Parsing of array notation as using in BigQuery and Postgres.
- Enable the
ignore
parameter on linting and fixing commands to ignore particular kinds of violations.
- A
--code-only
option to theparse
command to spit out a more simplified output with only the code elements. - Rules can now optionally override the description of the violation
and pass that back via the
LintingResult
.
- Bugfix, correct missing files in
setup.py
install_requires
section. - Better parsing of the not equal operator.
- Added more exclusions to identifier reserved words to fix cross joins.
- At verbosity levels 2 or above, the root config is printed and then any diffs to that for specific files are also printed.
- Linting and parsing of directories now reports files in alphabetical order. Thanks @barrywhart.
- Better python 2.7 stability. Thanks @barrywhart.
- Fixing parsing of
IN
/NOT IN
andIS
/IS NOT
.
- Bugfix, default config not included.
- Tweek rule L005 to report more sensibly with newlines.
- Rework testing of rules to be more modular.
- Fix a config file bug if no root config file was present for some values. Thanks @barrywhart.
- Lexing rules are now part of the dialect rather than a global so that they can be overridden by other dialects when we get to that stage.
- Templating support (jinja2, python or raw).
- Variables + Macros.
- The
fix
command is also sensitive to fixing over templates and will skip certain fixes if it feels that it's conflicted.
- Config file support, including specifying context for the templater.
- Documentation via Sphinx and readthedocs.
- Including a guide on the role of SQL in the real world. Assisted by @barrywhart.
- Documentation LINTING (given we're a linting project) introduced in CI.
- Reimplemented L006 & L007 which lint whitespace around operators.
- Ability to configure rule behaviour directly from the config file.
- Implemented L010 to lint capitalisation of keywords.
- Allow casting in the parser using the
::
operator. - Implemented
GROUP BY
andLIMIT
. - Added
ORDER BY
using indexes and expressions. - Added parsing of
CASE
statements. - Support for window/aggregate functions.
- Added linting and parsing of alias expressions.
- Fixed a bug which could cause potential infinite recursion in configuration
- Changed how negative literals are handled, so that they're now a compound segment rather than being identified at the lexing stage. This is to allow the parser to resolve the potential ambiguity.
- Restructure of rule definitions to be more streamlined and also enable
autodocumentation. This includes a more complete
RuleSet
class which now holds the filtering code. - Corrected logging in fix mode not to duplicate the reporting of errors.
- Now allows insert statements with a nested
with
clause. - Fixed verbose logging during parsing.
- Allow the
Bracketed
grammar to optionally match empty brackets using the optional keyword.
- Python 3.8 Support!
- Moved some of the responsibility for formatted logging into the linter to mean that we can log progressively in large directories.
- Fixed a bug in the grammar where one of the return values was messed up.
- Added a
--exclude-rules
argument to most of the commands to allow rule users to exclude specific subset of rules, by @sumitkumar1209 - Added lexing for
!=
,~
and::
. - Added a new common segment:
LambdaSegment
which allows matching based on arbitrary functions which can be applied to segments. - Recursive Expressions for both arithmetic and functions, based heavily off the grammar provided by the guys at CockroachDB.
- An
Anything
grammar, useful in matching rather than in parsing to match anything.
- Complete rewrite of the bracket counting functions, using some centralised class methods
on the
BaseGrammar
class to support common matching features across multiple grammars. In particular this affects theDelimited
grammar which is now much simpler but does also require slightly more liberal use of terminators to match effectively. - Rather than passing around multiple variables during parsing and matching, there is now
a
ParseContext
object which contains things like the dialect and various depths. This simplifies the parsing and matching code significantly. - Bracket referencing is now done from the dialect directly, rather than in individual
Grammars (except the
Bracketed
grammar, which still implements it directly). This takes out some originally duplicated code. - Corrected the parsing of ordering keywords in and
ORDER BY
clause.
- Removed the
bracket_sensitive_forward_match
method from theBaseGrammar
. It was ugly and not flexible enough. It's been replaced by a suite of methods as described above.
- Tweak to the L001 rule so that it doesn't crash the whole thing.
- Fixed the errors raised by the lexer.
- Fixed which modules from sqlfluff are installed in the setup.py. This affects
the
version
command.
- Big Rewrite - some loss in functionality might be apparent compared to pre-0.1.0. Please submit any major problems as issues on github
- Changed unicode handling for better escape codes in python 2. Thanks @mrshu
- BIG rewrite of the parser, completely new architecture. This introduces
breaking changes and some loss of functionality while we catch up.
- In particular, matches now return partial matches to speed up parsing.
- The
Delimited
matcher has had a significant re-write with a major speedup and broken the dependency onSequence
. - Rewrite of
StartsWith
andSequence
to use partial matches properly. - Different treatment of numeric literals.
- Both
Bracketed
andDelimited
respect bracket counting. - MASSIVE rewrite of
Bracketed
.
- Grammars now have timers.
- Joins properly parsing,
- Rewrite of logging to selectively output commands at different levels
of verbosity. This uses the
verbosity_logger
method. - Added a command line
sqlfluff parse
option which runs just the parsing step of the process to better understand how a file is being parsed. This also has options to configure how deep we recurse. - Complete Re-write of the rules section, implementing new
crawlers
which implement the linting rules. Now with inbuilt fixers in them. - Old rules removed and re implemented so we now have parity with the old rule sets.
- Moved to using Ref mostly within the core grammar so that we can have recursion.
- Used recursion to do a first implementation of arithmetic parsing. Including a test for it.
- Moved the main grammar into a separate dialect and renamed source and test files accordingly.
- Moved to file-based tests for the ansi dialect to make it easier to test using the tool directly.
- As part of file tests - expected outcomes are now encoded in yaml to make it easier to write new tests.
- Vastly improved readability and debugging potential of the _match logging.
- Added support for windows line endings in the lexer.
- Added a
sqlfluff fix
as a command to implement auto-fixing of linting errors. For now onlyL001
is implemented as a rule that can fix things. - Added a
rules
command to introspect the available rules. - Updated the cli table function to use the
testwrap
library and also deal a lot better with longer values. - Added a
--rules
argument to most of the commands to allow rule users to focus their search on a specific subset of rules.
- Refactor the cli tests to use the click CliRunner. Much faster
- Number matching
- Fixed operator parsing and linting (including allowing the exception of
(*)
)
- Much better documentation including the DOCS.md
- Fixed comma parsing and linting
- Added operator regexes
- Added a priority for matchers to resolve some ambiguity
- Added tests for operator regexes
- Added ability to initialise the memory in rules
- Refactor of rules to allow rules with memory
- Adding comma linting rules (correcting the single character matchers)
- Adding mixed indentation linting rules
- Integration with CircleCI, CodeCov and lots of badges
- Changed import of version information to fix bug with importing config.ini
- Added basic violations/file reporting for some verbosities
- Refactor of rules to simplify definition
- Refactor of color cli output to make it more reusable
- Longer project description
- Proper exit codes
- colorama for colored output
- Significant CLI changes
- Much improved output from CLI
- Initial Commit! - VERY ALPHA
- Restructure into package layout
- Adding Tox and Pytest so that they work