Simplify rule creation by adding a functional API to RuleContext

[barrywhart]

Brief summary of the change made

Makes progress on #1808

Are there any other side effects of this change that we should be aware of?

• Updates many existing rules to use the new classes (see changes to rules.rst for the current list).
• Updates L041 to use awesome-pattern-matching package

These changes generally make the rule code:

• Shorter
• Less complex
• More declarative & functional

This style of code should be easier for heavy SQL users and mathematicians to author. 👍🏽💪🏽

Reviewers may also wish to look at this branch, which implements a new rule (suggested in #1799) using the new API. I thought it'd be a good test case to create a new rule, not just rewrite existing ones. Note that it's a potentially pretty tricky rule, but it lacks some of the typical "code smells" of past tricky rules, e.g. complex loops, lots of if statements, heavy use of segment indices.

Pull Request checklist

• Please confirm you have completed any of the necessary steps below.

• Included test cases to demonstrate any code changes, which may be one or more of the following:

  • .yml rule test cases in test/fixtures/rules/std_rule_cases.
  • .sql/.yml parser test cases in test/fixtures/dialects (note YML files can be auto generated with python test/generate_parse_fixture_yml.py or by running tox locally).
  • Full autofix test cases in test/fixtures/linter/autofix.
  • Other.

• Added appropriate documentation for the change.

• Created GitHub issues for any relevant followup/future enhancements if appropriate.

[codecov]

Codecov Report

Merging #2126 (3880142) into main (f33be99) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##              main     #2126    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          150       155     +5     
  Lines        10886     10988   +102     
==========================================
+ Hits         10886     10988   +102     

Impacted Files	Coverage Δ
src/sqlfluff/core/parser/segments/base.py	100.00% <100.00%> (ø)
src/sqlfluff/core/rules/base.py	100.00% <100.00%> (ø)
src/sqlfluff/core/rules/functional/__init__.py	100.00% <100.00%> (ø)
...core/rules/functional/raw_file_slice_predicates.py	100.00% <100.00%> (ø)
.../sqlfluff/core/rules/functional/raw_file_slices.py	100.00% <100.00%> (ø)
...lfluff/core/rules/functional/segment_predicates.py	100.00% <100.00%> (ø)
src/sqlfluff/core/rules/functional/segments.py	100.00% <100.00%> (ø)
src/sqlfluff/rules/L001.py	100.00% <100.00%> (ø)
src/sqlfluff/rules/L004.py	100.00% <100.00%> (ø)
src/sqlfluff/rules/L008.py	100.00% <100.00%> (ø)
... and 15 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f33be99...3880142. Read the comment docs.

[barrywhart]

@jpy-git, @tunetheweb: Please take a preliminary look at this. I'm not looking for a typical "pre-merge" review. This is more of a design review:

• Does this look like a good start towards making it easier to write rules?
• Do you see any areas where the API might be improved?
• Are there other rules we should look at updating to use this new API, as a kind of "beta test"? I can update them, or I'd be happy for you to update them instead.

Note that some of the functions on the new classes are just stubs (they raise NotImplementedError). If/when we get to a more proper review, I can implement these functions or remove them for now (i.e. add them later when needed).

[jpy-git]

@barrywhart Like it a lot! 😎 Added my initial thoughts/questions. See if there's anything you think we should change off the back of that and then I can maybe try applying this to L009 to get a better idea of the user experience 👍

[jpy-git]

A couple more random thoughts to add, shaping up nicely!

Will have a more in depth look at the edited rules tomorrow 😄

[alanmcruickshank]

Code looks great - docs aren't building properly at the moment but otherwise love it 👍

[alanmcruickshank]

I'm not sure it makes sense to keep this list given how many you've converted. It will just end up out of date. If we want to still give some measure of scale, but without having to maintain the list, it might be better to say something like: ...at 2021-12-30, 17 rules use this method.

[barrywhart]

Will do.

[alanmcruickshank]

These autodoc commands don't seem to render properly at the moment:

[barrywhart]

Fixes incoming...

[alanmcruickshank]

@barrywhart - can you push your most recent changes? I can't see them on the branch. I'll give them a spin when they're up.

[barrywhart]

@alanmcruickshank: Thanks for checking the docs! I hadn't built and previewed those yet. Should be good now.

[barrywhart]

@alanmcruickshank: Okay, the fixes are up now. (Earlier, I had accidentally pushed them to the wrong branch.)

[jpy-git]

@barrywhart I'm conscious that things are still getting added to this PR and we still haven't reviewed the awesome-pattern-matching dependency.

Looking at the repo it seems like quite a small project?

Only 700 downloads in the past month: https://pypistats.org/packages/awesome-pattern-matching

This issue here mentions there may be a lot of overlap with the matching introduced in Python 3.10
(relevant PEP: https://www.python.org/dev/peps/pep-0622/)
So maybe there's value in seeing if there's an available backport for that instead?

I feel like maybe it's best to introduce the functional API and it's rule changes/documentation in this PR, as there are no concerns there and it's been pretty thoroughly reviewed, and separate the pattern matching stuff into it's own PR afterwards. They're not really related, other than both touching rules, and would be good to have any discussion on apm separated for future reference. I can see that the functional API has been linked to several issues so we should get that merged to unblock them.

[barrywhart]

@jpy-git: That's definitely valid feedback. I'm happy to remove the pattern matching stuff for now.

I did take a look at the new built-in pattern matching in Python 3.10, and there is a backport (although the backport looks new and kind of unofficial). From my evaluation, the built-in pattern matching is less powerful, e.g. it doesn't support patterns with both leading and trailing "don't care" sections. It's also unclear if it supports matching across multiple levels of a tree structure.

OTOH:

• awesome-pattern-matching is definitely immature, e.g. current lack of support for nested multi-segment patterns (I asked about it here: Question: Is it possible to match a subsequence multiple times? scravy/awesome-pattern-matching#8 (comment))
• It's also unclear how often we'd need the additional power of awesome-pattern-matching. Until it matures a bit, I often find try more of a distraction than a boon. If you get something working, the code looks nice, but you usually encounter some odd behavior (bugs) along the way.

TL;DR I'll remove it from this PR. I may keep tinkering with it, possibly contributing to it. Among the packages out there, it comes closest to my dream of "regex-like power applied to arbitrary objects, plus support for patterns that span levels of a tree". (AFAICT, most of them aren't even trying to do one or both of these.) Part of the appeal is that the patterns closely resemble SQLFluff dialect definitions, so could be very approachable and intuitive for contributors.

[jpy-git]

Makes sense, I def like the concept so it's worth investigating further afterwards 😄

[barrywhart]

@jpy-git: Ok, I've removed all references to and usage of awesome-pattern-matching. Ready for a closer review?

I created an issue, #2221, to revisit the package in the future, and for reference, that issue also includes the rewritten rules and documentation that was removed from this PR.

[jpy-git]

small mypy cleanup otherwise good 👍

[jpy-git]

Looks good! 👍 Think we can get this merged if no one else has any objections? 🚀

Docs issue has been fixed :)

[barrywhart]

I went ahead and merged. I hope that was okay? 😬 @alanmcruickshank