Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Sync mode #3

Merged
merged 23 commits into from
Jun 29, 2022
Merged

Implement Sync mode #3

merged 23 commits into from
Jun 29, 2022

Conversation

Phanatic
Copy link
Contributor

@Phanatic Phanatic commented Jun 23, 2022

This PR implements the Sync mode for a Singer Tap connecting to PlanetScale databases.

  1. Sync mode works by accepting a source_config and a catalog file which contain the connection details for the database and the schema we should sync from PlanetScale, respectively.
  2. Once it has the selected catalog, it will then iterate over the Streams within the Catalog and filter out any streams that are not selected, along with any properties that are not selected.
  3. Once it has all these pieces of information, it will then go retrieve the shards in the database and syncs the selected tables across all shards

Review Notes
All the code except sync.go is copied from the Airbyte Source.

Example Run

go run cmd/singer-tap/main.go --config sources/demo/source.json  --catalog sources/demo/departments.json 2> /dev/null
{"type":"SCHEMA","stream":"departments","tap_stream_id":"import-on-scaler:departments","schema":{"type":["null","object"],"additionalProperties":false,"properties":{"dept_name":{"type":["null","string"]},"dept_no":{"type":["null","string"]}}},"table-name":"departments","metadata":[{"metadata":{"selected":true,"inclusion":"available","valid-replication-keys":["dept_no"],"table-key-properties":["dept_no"],"breadcrumb":[]}},{"metadata":{"selected":true,"inclusion":"available","breadcrumb":["properties","dept_no"]}},{"metadata":{"selected":true,"inclusion":"available","breadcrumb":["properties","dept_name"]}}],"key_properties":["dept_no"],"bookmark_properties":["dept_no"]}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.200956-05:00","record":{"dept_name":"Marketing","dept_no":"d001"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.2011-05:00","record":{"dept_name":"Finance","dept_no":"d002"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201103-05:00","record":{"dept_name":"Human Resources","dept_no":"d003"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201104-05:00","record":{"dept_name":"Production","dept_no":"d004"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201106-05:00","record":{"dept_name":"Development","dept_no":"d005"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201108-05:00","record":{"dept_name":"Quality Management","dept_no":"d006"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201113-05:00","record":{"dept_name":"Sales","dept_no":"d007"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201114-05:00","record":{"dept_name":"Research","dept_no":"d008"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201116-05:00","record":{"dept_name":"Customer Service","dept_no":"d009"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201118-05:00","record":{"dept_name":"insert test","dept_no":"d010"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201119-05:00","record":{"dept_name":"Primary mode","dept_no":"d011"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201121-05:00","record":{"dept_name":"Replica mode","dept_no":"d012"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201123-05:00","record":{"dept_name":"Incremental Sync 8","dept_no":"d092"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201128-05:00","record":{"dept_name":"Incremental Sync 7","dept_no":"d093"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201129-05:00","record":{"dept_name":"Incremental Sync 6","dept_no":"d094"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201131-05:00","record":{"dept_name":"Incremental Sync 5","dept_no":"d095"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201133-05:00","record":{"dept_name":"Incremental Sync 4","dept_no":"d096"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201134-05:00","record":{"dept_name":"Incremental Sync 3","dept_no":"d097"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201138-05:00","record":{"dept_name":"Incremental Sync 2","dept_no":"d098"}}
{"type":"RECORD","stream":"departments","time_extracted":"2022-06-23T13:06:01.201139-05:00","record":{"dept_name":"Incremental Sync","dept_no":"d099"}}

@Phanatic Phanatic changed the base branch from http-tap to main June 23, 2022 19:36
@Phanatic Phanatic requested a review from mdlayher June 24, 2022 18:57
cmd/internal/planetscale_edge_database.go Outdated Show resolved Hide resolved
cmd/internal/planetscale_edge_database.go Show resolved Hide resolved
cmd/internal/sync.go Outdated Show resolved Hide resolved
@Phanatic Phanatic merged commit 91bcecb into main Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants