Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cross database transfers - data movements between databases [Sqlite -Snowflake and vice-versa] #1730

Closed
6 tasks
sunank200 opened this issue Feb 7, 2023 · 0 comments · Fixed by #1829
Closed
6 tasks

Cross database transfers - data movements between databases [Sqlite -Snowflake and vice-versa] #1730

sunank200 opened this issue Feb 7, 2023 · 0 comments · Fixed by #1829
Assignees
@utkarsharma2
Labels
feature New feature or request universal_transfer_operator
Milestone
Universal transfe...

Comments

@sunank200
Copy link
Contributor

sunank200 commented Feb 7, 2023 •
edited
Loading

Please describe the feature you'd like to see

  • Add a transfer workflow for Sqlite to Snowflake
  • Add a transfer workflow for Snowflake to sqlite
  • Add example DAG for all the scenarios

Please note that other cross-database transfers will be added as part of data provider implementation. For example, when big query support is added, it will contain the cross-database transfer support.

Acceptance Criteria

  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
@sunank200 sunank200 added the feature New feature or request label Feb 7, 2023
@sunank200 sunank200 changed the title Cross database transfers - data movements between databases Cross database transfers - data movements between databases [Sqlite -Snowflake and vice-versa] Feb 7, 2023
@utkarsharma2 utkarsharma2 self-assigned this Feb 22, 2023
@utkarsharma2 utkarsharma2 mentioned this issue Mar 9, 2023
Merged
9 tasks
sunank200 pushed a commit to astronomer/apache-airflow-providers-transfers that referenced this issue Mar 24, 2023
@rajaths010494 @pre-commit-ci
@utkarsharma2 @phanikumv
Implement transfer for BigQuery - read/write (#1829)
32afcef 
**Please describe the feature you'd like to see**
- Add `DataProvider` for Bigquery - read/write methods
- Add non-native transfer implementation for GCS to BigQuery
-  Add non-native transfer implementation for S3 to BigQuery
- Add non-native transfer example DAG for BigQuery to Sqlite 
- Add non-native transfer example DAG for BigQuery to Snowflake
- Add example DAG
- Add tests with 90% coverage

**Acceptance Criteria**

- [ ] All checks and tests in the CI should pass
- [ ] Unit tests (90% code coverage or more, [once
available](astronomer/astro-sdk#191))
- [ ] Integration tests (if the feature relates to a new database or
external service)
- [ ] Example DAG
- [ ] Docstrings in
[reStructuredText](https://peps.python.org/pep-0287/) for each of
methods, classes, functions and module-level attributes (including
Example DAG on how it should be used)
- [ ] Exception handling in case of errors
- [ ] Logging (are we exposing useful information to the user? e.g.
source and destination)
- [ ] Improve the documentation (README, Sphinx, and any other relevant)
- [ ] How to use Guide for the feature
([example](https://airflow.apache.org/docs/apache-airflow-providers-postgres/stable/operators/postgres_operator_howto_guide.html))


closes: #1732 
closes: astronomer/astro-sdk#1785
closes: astronomer/astro-sdk#1730

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Utkarsh Sharma <utkarsharma2@gmail.com>
Co-authored-by: Phani Kumar <94376113+phanikumv@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@utkarsharma2 utkarsharma2

Labels
feature New feature or request universal_transfer_operator
Projects
None yet
Milestone
Universal transfer operator - phase 2
Development

Successfully merging a pull request may close this issue.

Implement transfer for BigQuery - read/write astronomer/astro-sdk
2 participants
@sunank200 @utkarsharma2