configuration documentation #5

cgardens · 2020-07-30T22:18:04Z

Doc that tries to enumerate the requirements for the source side of our product. Proposes what configuration needs to be taken in. Still rough, but hopefully enough to push the conversation forwards.

docs/configuration_data_model.md

sherifnada · 2020-07-31T00:01:52Z

docs/configuration_data_model.md

+
+Fivetran has a similar feature, where at configuration time, it detects the scheme of the data source and allows a user to select a subset of the columns discovered.
+
+## Source


Can we add a section about supported sources/destinations? here are some imo good MVP candidates:
Sources:

Postgres

S3 CSV

MySQL

Destinations:

BigQuery

RedShift

Postgres

MySQL

We should also have one SaaS source

the goal here is to describe how configuration works, can we keep this conversation in the reqs doc: https://docs.google.com/document/d/1X6M3qhbg9E9adykdI8KmO3xV7mr0XK7O3jLbN5Z7ydw/edit#?

docs/configuration_data_model.md

sherifnada

Left some comments, might have some more tomorrow when I've had some time to think about it

sherifnada · 2020-08-04T07:08:19Z

docs/configuration_data_model.md

+                    },
+                    "dataType": {
+                      "type": "string",
+                      "enum": ["string", "number", "uuid", "boolean"]


this enum seems overly restrictive since we could custom data types e.g: postgres dates or json blobs. What is the value of making this an enum vs. having it be a string? Or are we pooling anything that isn't a uuid/number/boolean into string? in that case, should uuid also be pooled into string?

good question. the value of having these be an enum in the future is that we can make sure we are matching up columns in different tables that are compatible. like recognize if we are piping a uuid into a string column is maybe not what we want to do. for right now it doesn't matter probably?

i was being lazy when i wrote this enum, just putting in values as an example; i would definitely include more types.

i think defining our own types will probably be valuable at some point, but kinda ambivalent as to whether we need to do it MVP or do what you said and just use strings. what would your preference be @sherifnada ?

sherifnada · 2020-08-04T07:09:14Z

docs/configuration_data_model.md

+* Receive feedback on whether Dataline was able to reach the source with the given credentials.
+* Insert credentials for a destination.
+* Receive feedback on whether Dataline was able to reach the destination with the given credentials.
+* Show intent to connect source to destination.


What does this point mean?

lol. yeah awkward phrasing. all steps so far have just been source or destination. this step is where you say i want to connect source X to destination Y.

sherifnada · 2020-08-04T07:10:09Z

docs/configuration_data_model.md

+* Show intent to connect source to destination.
+* Receives schema of the source.
+* Selects which part of the schema will be synced.
+* Triggers a manual sync or inputs schedule on which syncs should take place.


in the case the user triggers a manual sync, is this saying the line would be a one-time transient transfer?

it would just attempt to run a sync (using whatever existing configuration is, full_refresh or append).

docs/configuration_data_model.md

sherifnada · 2020-08-04T07:24:59Z

docs/configuration_data_model.md

+
+### Conduit Types
+
+#### StandardConduitConfiguration


Why the Standard prefix?

as in you don't think this should be the same across all taps. what parts do you think can't be standardized? i think the goal here is to try to get the conduit configuration to be the mode (which is standard) and the part of the schema that we want to sync (which is standardized in tables/columns).

sherifnada · 2020-08-04T07:26:58Z

docs/configuration_data_model.md

+Configuration that is the SAME for all tap / target combinations. Describes the sync mode (full refresh or append) as well what part of the schema will be synced.
+
+```json
+{


Should this also have a handle to source and destination?

yeah... i actually all of these input configs are missing sourceId or destinationId. do you have an opinion about putting these in the configs versus just adding them to interface of the methods. (e.g. sync(sourceId, destinationId, etc...)). i'm a little inclined to take this approach since ConnectionConfiguration is not standard. if we did add the id to the configs, then we would maybe need to add a StandardConnectionConfiguration as well.

talked offline. i will add the source destination id into the objects.

docs/configuration_data_model.md

michel-tricot · 2020-08-04T01:48:07Z

docs/configuration_data_model.md

+
+1. Add a source _without_ needing to write HTML. They should be responsible for only 2 things:
+   1. Define Configuration: define a json object which describes which properties need to be collected by a user. Then the UI figures out how to render it.
+   1. Implement: `testConnection`, `discoverSchema`, and `sync`. These functions should only rely on the configurations defined in the json and should return objects that match the interfaces that are described below.


I wonder if we should are more granularity in the sync

can you say more? you think the configuration should be more granular? or are you talking about splitting up the steps in the sync step more? if the latter, we can figure that out in sherif's work doc.

michel-tricot · 2020-08-04T01:49:55Z

docs/configuration_data_model.md

+
+Fivetran has a similar feature, where at configuration time, it detects the scheme of the data source and allows a user to select a subset of the columns discovered.
+
+## Source


We should also have one SaaS source

michel-tricot · 2020-08-04T01:51:36Z

docs/configuration_data_model.md

+}
+```
+
+#### StandardConnectionStatus


Why is it prefixed with Standard?

so the configuration for a connection is not standard, but i was imagining that when you implement a test connection check you need to return something against a standard interface (i.e. it connected or it didn't). is there something you feel is non standard here?

or is the question just what standard means? it means it's a configuration / interface that is the same for all taps or targets.

docs/configuration_data_model.md

Pr feedback

#6503) * 🎉 Destination MSSQL: Added support for connection via SSH tunnels (#5970) * 🎉 Destination MSSQL: bumb image version * 🎉 Destination MSSQL: update mssql.md file

…-dbt Bump Redshift No DBT version

Feat/source shipstation

* Replaced sidebar menu with headlessui * Removed unnecessary tag li * Removed unnecessary Sidebar Popout * Changed export type * Added keyboard support * Removed styled-components * Removed commented code * Disabled ESLint rule css-modules/no-undef-class * Review fixes * Fixed shorthand property * Review fixes * Review fixes #2 * Review fixes #3 * Review fixes #4 * Review fixes #5 * Review fixes #6 * Review fixes #7 * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com> * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com> Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com>

* Replaced sidebar menu with headlessui * Removed unnecessary tag li * Removed unnecessary Sidebar Popout * Changed export type * Added keyboard support * Removed styled-components * Removed commented code * Disabled ESLint rule css-modules/no-undef-class * Review fixes * Fixed shorthand property * Review fixes * Review fixes airbytehq#2 * Review fixes airbytehq#3 * Review fixes airbytehq#4 * Review fixes airbytehq#5 * Review fixes airbytehq#6 * Review fixes airbytehq#7 * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com> * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com> Co-authored-by: Vladimir <volodymyr.s.petrov@globallogic.com>

initial files for source walmart seller

* https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector * bump minor version * bump minor version * Edit connector doc

…Merge) #26658 (#26942) * https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector (#1) * Ssdp 165/enable schema evolution in the airbyte databricks connector (#4) * https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector * bump minor version * bump minor version * Ssdp 165/add doc (#5) * https://prima-assicurazioni-spa.myjetbrains.com/youtrack/issue/SSDP-165/Enable-schema-evolution-in-the-Airbyte-Databricks-connector * bump minor version * bump minor version * Edit connector doc

cgardens force-pushed the charles/source_configuration_docs branch 2 times, most recently from ac1269f to 19108df Compare July 30, 2020 22:21

jrhizor reviewed Jul 30, 2020

View reviewed changes

sherifnada reviewed Jul 31, 2020

View reviewed changes

cgardens commented Aug 4, 2020

View reviewed changes

docs/configuration_data_model.md Outdated Show resolved Hide resolved

cgardens changed the title ~~source / connected source requirements and configuration documentation~~ configuration documentation Aug 4, 2020

sherifnada reviewed Aug 4, 2020

View reviewed changes

michel-tricot reviewed Aug 5, 2020

View reviewed changes

cgardens force-pushed the charles/source_configuration_docs branch 2 times, most recently from 8fc2613 to da5d407 Compare August 5, 2020 21:32

configuration docs

01c3ec4

cgardens force-pushed the charles/source_configuration_docs branch from da5d407 to 01c3ec4 Compare August 5, 2020 21:45

cgardens added 5 commits August 5, 2020 16:49

add more states to the status enum

ae8fb31

add minvalue

cb13f3e

typo

eb7475a

remove wrong description

f6ed430

update sync output struct

8107100

cgardens mentioned this pull request Aug 7, 2020

add java codegen from json schema #14

Merged

cgardens marked this pull request as ready for review August 10, 2020 20:20

cgardens added 5 commits August 10, 2020 13:31

test link

06fcc28

link test

aea7153

reference json files

f1bcd66

hard commit

af6c9d9

clean up

77f29ab

cgardens merged commit 0589033 into master Aug 10, 2020

cgardens deleted the charles/source_configuration_docs branch August 10, 2020 22:13

davinchia added a commit that referenced this pull request May 3, 2021

Merge pull request #5 from davinchia/PR-feedback

06eadea

Pr feedback

This was referenced Sep 29, 2021

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6532

Closed

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6539

Closed

jrhizor mentioned this pull request Sep 29, 2021

Bump Airbyte version from 0.29.22-alpha to 0.30.0-alpha #6548

Merged

mohamagdy added a commit to mohamagdy/airbyte that referenced this pull request May 1, 2022

Merge pull request airbytehq#5 from mohamagdy/destination-redshift-no…

1b95872

…-dbt Bump Redshift No DBT version

ahmed-buksh added a commit to ahmed-buksh/airbyte that referenced this pull request May 25, 2022

Merge pull request airbytehq#5 from tydo/feat/source-shipstation

fe67e54

Feat/source shipstation

sbjorn mentioned this pull request Aug 29, 2022

🎉 New Source: Adjust #16051

Merged

18 tasks

matter-q added a commit that referenced this pull request Sep 16, 2022

Review fixes #5

1cf690f

This was referenced Oct 28, 2022

🎉 New source: Pagar.me [python cdk] #18622

Closed

🎉 New source: Breezometer [low-code cdk] #18650

Merged

🎉 New source: Breezometer Air Quality [low-code cdk] #18652

Closed

sarafonseca-123 mentioned this pull request Nov 1, 2022

🐛 Source RD Station Marketing: Fix stream analytics conversions #18826

Merged

14 tasks

danilo-dti mentioned this pull request Nov 12, 2022

🎉 New Source: Microsoft Dataverse [python cdk] #18646

Merged

37 tasks

ganpatagarwal added a commit to ganpatagarwal/airbyte that referenced this pull request May 18, 2023

Merge pull request airbytehq#5 from ganpatagarwal/walmart-seller

f07d2d7

initial files for source walmart seller

tolik0 mentioned this pull request May 23, 2023

🐛Source GitLab: Fix syncing branches for the project with Repository feature disabled #26422

Merged

avirajsingh7 mentioned this pull request Aug 3, 2023

🎉 New Source : Avni #28141

Closed

4 tasks

maxi297 mentioned this pull request Aug 25, 2023

File CDK: migrate S3 configs #29435

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

configuration documentation #5

configuration documentation #5

cgardens commented Jul 30, 2020

sherifnada Jul 31, 2020

michel-tricot Aug 4, 2020

cgardens Aug 5, 2020

sherifnada left a comment

sherifnada Aug 4, 2020

cgardens Aug 6, 2020

sherifnada Aug 4, 2020

cgardens Aug 5, 2020

sherifnada Aug 4, 2020

cgardens Aug 5, 2020

sherifnada Aug 4, 2020

cgardens Aug 5, 2020

sherifnada Aug 4, 2020

cgardens Aug 6, 2020

cgardens Aug 6, 2020

michel-tricot Aug 4, 2020

cgardens Aug 6, 2020

michel-tricot Aug 4, 2020

michel-tricot Aug 4, 2020

cgardens Aug 6, 2020

cgardens Aug 6, 2020


		Fivetran has a similar feature, where at configuration time, it detects the scheme of the data source and allows a user to select a subset of the columns discovered.

		## Source

configuration documentation #5

configuration documentation #5

Conversation

cgardens commented Jul 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment