This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New Source: Weaviate #20012
Labels
area/connectors
Connector related issues
autoteam
community
help-welcome
Accepting external contributions
new-connector
team/connectors-python
team/destinations
Destinations team's backlog
Comments
samos123
added a commit
to samos123/airbyte
that referenced
this issue
Dec 6, 2022
sherifnada
added
team/connectors-python
team/extensibility
team/destinations
sajarin
pushed a commit
that referenced
this issue
Jan 12, 2023
* Add Weaviate Destination #20012 * Fix formatting and standards * Fix flake issue * Fix unused client variable * Add support for int based ID fields * Ensure stream name meets Weaviate class reqs * add integration test for using pokemon as source * handle nested objects by converting to json string * create schema for transforming data to weaviate * Add docs for weaviate destination * Remove pokemon-schema external dependency * Remove pikachu integration test external dep * Add large batch test case * add test for second sync * Fix issue with fields starting with uppercase * add more checks to line_break test * Update README for Weaviate * Make batch_size configurable with 100 as default * Add support for providing vectors * Update docs * Add test for existing Weaviate class * Add trying to create schema in check connection * Add support for mongodb _id fields * Add support for providing custom ID * remove unused file * fix flow of is_ready() check * Move standalone functions to utils.py * Support overwrite mode * Add regex based stream_name_class_name conversion * remove unneeded print statement * Add "airbyte_secret" : true to password config * add support for array of arrays * remove unneeded variable declaration * change to MutableMapping since we use del * change name from queued_write to buffered_write * add retry on partial batch error * Fix partial batch retry and add tests * fix ID generation * Clean up recursive retry logic * fix flake tests * ran flake reformat * add definitions Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com> Co-authored-by: itaseski <itaseski@debian-BULLSEYE-live-builder-AMD64>
jbfbell
pushed a commit
that referenced
this issue
Jan 13, 2023
* Add Weaviate Destination #20012 * Fix formatting and standards * Fix flake issue * Fix unused client variable * Add support for int based ID fields * Ensure stream name meets Weaviate class reqs * add integration test for using pokemon as source * handle nested objects by converting to json string * create schema for transforming data to weaviate * Add docs for weaviate destination * Remove pokemon-schema external dependency * Remove pikachu integration test external dep * Add large batch test case * add test for second sync * Fix issue with fields starting with uppercase * add more checks to line_break test * Update README for Weaviate * Make batch_size configurable with 100 as default * Add support for providing vectors * Update docs * Add test for existing Weaviate class * Add trying to create schema in check connection * Add support for mongodb _id fields * Add support for providing custom ID * remove unused file * fix flow of is_ready() check * Move standalone functions to utils.py * Support overwrite mode * Add regex based stream_name_class_name conversion * remove unneeded print statement * Add "airbyte_secret" : true to password config * add support for array of arrays * remove unneeded variable declaration * change to MutableMapping since we use del * change name from queued_write to buffered_write * add retry on partial batch error * Fix partial batch retry and add tests * fix ID generation * Clean up recursive retry logic * fix flake tests * ran flake reformat * add definitions Co-authored-by: Ivica Taseski <ivica.taseski94@gmail.com> Co-authored-by: itaseski <itaseski@debian-BULLSEYE-live-builder-AMD64>
|
@samos123 looks like you added the weaviate destination, thank you! Should this issue be closed? Your description makes it look like the destination (not source) is what you were after. |
github-project-automation
bot
moved this from Todo
to Done
in Community Contributions Connectors
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Tell us about the new connector you’d like to have
Which source and which destination?
Weaviate source and destination
Do you need a specific version of the underlying data source e.g: you specifically need support for an older version of the API or DB?
Describe the context around this new connector
Why do you need this integration? How does your team intend to use the data? This helps us understand the use case.
Allow loading data from Mongo and snowflake into Weaviate for doing semantic search with Weaviate
How often do you want to run syncs?
Ideally stay in sync through streaming
If this is an API source connector, which entities/endpoints do you need supported?
Weaviate is more like a database
If the connector is for a paid service, can we name you as a mutual user when we subscribe for an account? Which company should we name?
Not a paid service, it's OSS that's also available through a managed service.
Describe the alternative you are considering or using
What are you considering doing if you don’t have this integration through Airbyte?
Building Airflow DAG for moving data between Weaviate and other databases
Are you willing to submit a PR?
Yes
The text was updated successfully, but these errors were encountered: