-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CrateDB connector #3
base: cratedb
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Excellent stuff, thanks a stack. I've diverted the PR to be targeted towards the cratedb
branch, so you can merge what works, and submit later fixes on behalf of subsequent PRs.
Other than this, I added a few quick suggestions on a few spots of particular interest. I did not run a thorough review yet, but this can also happen on a final review run before upstreaming all changes.
'cratedb': [ | ||
'psycopg2==2.9.3', | ||
'psycopg2-binary==2.9.3', | ||
'sshtunnel==0.4.0', | ||
], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we have the choice, I'd go for psycopg3 right away (package name is psycopg[binary]
, IIRC), but sure it can happen on behalf of a later iteration, depending on your urgency to submit the patch to upstream.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few more comments while quickly skimming the patch once more.
``` | ||
cratedb-client==1.6.9 | ||
sentence-transformers==2.2.2 | ||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe those are not needed at all?
``` | |
cratedb-client==1.6.9 | |
sentence-transformers==2.2.2 | |
``` |
dict( | ||
block_type=BlockType.DATA_LOADER, | ||
groups=[GROUP_DATABASES], | ||
language=BlockLanguage.PYTHON, | ||
name='CrateDB', | ||
path='data_loaders/cratedb.py', | ||
), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah wow. The framework allows database adapters to be written in other languages than Python?
""" | ||
Template for exporting data to a PostgreSQL database. | ||
Specify your configuration settings in 'io_config.yaml'. | ||
|
||
Docs: https://docs.mage.ai/design/data-loading#postgresql | ||
""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docstrings also need a few more adjustments, but sure all that can happen after merging to cratedb
, just any time before submitting to upstream. It will probably take another while. I can also support on such details.
Description
How Has This Been Tested?
Checklist
docs/mint.json
cc: