Discuss of sqlparser-rs project status / maintenance Feb 2023

[alamb]

Update of #643

Current Status

The good!

At the time of writing, this project has achieved "platinum" status on crates.io (over 1 million downloads) -- https://crates.io/crates/sqlparser 🎉

There are also many forks (337 on github alone at the time of writing, and there are several "copy into a new repo" type forks I know of as well).

There are releases every month or so https://crates.io/crates/sqlparser/versions

The crate is easy to contribute to: there is a long history of accepting relatively small and targeted PRs from new contributors that add SQL syntax and functionality as described in https://github.com/sqlparser-rs/sqlparser-rs#contributing.

The API of this crate is very stable. While new releases do sometimes require changes in downstream crates, as new fields or enum variants are added, the overhead of updating has been quite low.

The not so good 😢

Limited maintenance bandwidth

I have merged almost all PRs over the last year, though @AugustoFKL has helped recently with reviews (thank you!), I am not sure he will be able to continue to do so.

Unfortunately no one else has (yet) to my knowledge consistently reviewed PRs from the community checking that they have adequate tests, follow existing code conventions, are documented, etc.

This leads to, among other things:

1. PRs often sit for weeks without a review which is a bad experience for contributors
2. Issues filed in this repo go unanswered

No way to make major breaking API changes

There is also a long history of people proposing significant changes and additions to this crate. For example, recently, @ankrgyl proposal to add location information #790

These tend to get "stuck" due to two factors:

1. Limited maintenace bandwidth, as mentioned above
2. Potential major downstream code churn

There are, unsurprisingly, different opinions on the benefit vs cost tradeoff for major downstream code churn, but we don't have any way to rationalize this nor any outlet for people who want features vs a stable API

Interestingly, the specific features cited on #643 were eventually merged (after a lot of back and forth to minimize ongoing maintenance and API churn)

• @lovasoa 's proposal to add a visitor in Add an AST visitor #601
• @46bit 's proposal to avoid stack overflows in Prevent stack overflows by limiting recursion #501

Call to Action

First of all, I would love additional help maintaining this crate (especially from contributors who know the code well). Being able to review a PR where someone else has already ensured there is sufficient testing, documentation, and the code follows the existing structures would help me a LOT. Also I view consistent PR review as the path to granting someone write access to this repo.

There are also other possible things we could work on as a community:

• We work on an "ecosystem" of crates so that additional add on crates can add functionality that is not in the core sqlparser crate (and add appropriate extension points as needed)
• Users who want new features can fork the project (it is all apache 2 licensed and there are no restrictions on doing so)
• Someone can come up with a way to have a stable branch and a "new feature" branch with different APIs

Background / People with Write Access

This project was initially started by @andygrove

At various times, @benesch @maxcountryman @Dandandan @nickolay and myself have helped maintain the code. This is also the list of people who have write access to the github repo and crates.io https://crates.io/crates/sqlparser

[sivchari]

I'm being helped by this project, so I would continuous contribution for it.

[alamb]

I'm being helped by this project, so I would continuous contribution for it.

Thanks @sivchari -- I think what would be most helpful is to help review PRs as they come in (watch the repo or something) and then ping me when you think they are ready for review and merge.

@ankrgyl has been helping do this recently and it really helps

[sivchari]

Thank you for reply, @alamb .

I think what would be most helpful is to help review PRs as they come in (watch the repo or something)

I think so too. I review PRs, thanks 👍

[alamb]

Here is another proposal: #1139

[alamb]

sqlparser is now part of DataFusion. see #1294 for details

@iffyio has become a committer of DataFusion (and thus also of sqlparser-rs) now as well 🎉