-
Notifications
You must be signed in to change notification settings - Fork 14.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add StarRocks support #23209
feat: Add StarRocks support #23209
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks super solid, thanks for the contribution! ❤️ Tagging a few more committers for reviews who I believe have experience with Starrocks.
Codecov Report
@@ Coverage Diff @@
## master #23209 +/- ##
==========================================
- Coverage 68.31% 68.29% -0.03%
==========================================
Files 1951 1956 +5
Lines 75385 75589 +204
Branches 8185 8223 +38
==========================================
+ Hits 51496 51620 +124
- Misses 21786 21861 +75
- Partials 2103 2108 +5
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Thanks @miomiocat for the contribution. We (Airbnb) are using StarRocks within Superset, but given StarRocks is compatible with the MySQL protocol we've had good mileage by simply extending the class StarRocksEngineSpec(MySQLEngineSpec):
"""
The StarRocks engine specification.
"""
engine = "starrocks"
engine_name = "StarRocks" I was wondering whether you considered this option as this adheres to the DRY principle. |
Thanks for your reply. Actually, StarRocks is mostly compatible with mysql but not all. for example, you can reproduce this issue with following steps:
The reason for this exception is that mysql does not handle the catalog in StarRocks so I resolve this issue from We are considering changing the format of the current URI and other things. For future extension, we need a separate StarRocks engine spec. |
@etr2460 @john-bodley Could you help review this PR? |
Thanks @miomiocat for clarifying. I've left some comments. |
Thanks @john-bodley |
e0150cc
to
fb6db49
Compare
Closing and re-opening to kick-start CI (not sure why it's stuck in some eternal waiting state). |
I've kicked CI a couple of times, but the DB tests still seem to be failing, so those need some 👀. @villebro do you know if this is victim to a flaky test, or if this looks legit? |
62b9eb4
to
7bf8d24
Compare
Fixed field type issues because mysql does not have STRUCT/MAP/ARRAY types. |
PR was failing due to some pre-commit hook linting errors. Apologies for the churn as I try to get those ironed out ;) |
Now getting an error from mypy:
|
c1b1761
to
f9b4908
Compare
"starrocks://user:password@host:port/catalog.db[?key=value&key=value...]" | ||
) | ||
|
||
_default_column_type_mappings = ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be
_default_column_type_mappings = ( | |
column_type_mappings = ( |
Also note, that you don't need to reimplement all types - Any types that are already covered by _default_column_type_mappings
can be omitted. In addition, I would suggest adding unit tests to ensure that the main types are identified correctly (see tests/unit_tests/db_engine_specs
for other examples of how this is done).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks:)
added unit tests and updated codes, please review it again.
Signed-off-by: miomiocat <284487410@qq.com>
@rusackas I've fixed this issue and passed all check except one error in
|
@miomiocat if you check the |
Looks like we're good! Thanks so much, @miomiocat!!! |
SUMMARY
Add StarRocks support
Starrocks database is based on mysql protocal but introduce the concept of catalog and other differences, so there will be incompatibility when directly using mysqldb engine spec and it is necessary to introduce a separate Starrocks engine spec to fix various issues and use starrocks python client directly
main differences:
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
ADDITIONAL INFORMATION