Skip to content

Commit

Permalink
Added auto-detect schema info to docs (#21501)
Browse files Browse the repository at this point in the history
* added auto-detect schema info to docs

* small edits

* added info

* edits

* edits
  • Loading branch information
sophia-wiley authored Jan 25, 2023
1 parent f7524f0 commit 0cb8388
Showing 1 changed file with 80 additions and 8 deletions.
88 changes: 80 additions & 8 deletions docs/cloud/managing-airbyte-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ To view the sync summary:
3. To view the full sync log, click the sync summary dropdown.

### Sync summary

| Data | Description |
|--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| x GB (also measured in KB, MB) | Amount of data moved during the sync. If basic normalization is on, the amount of data would not change since normalization occurs in the destination. |
Expand All @@ -186,17 +187,18 @@ In a successful sync, the number of emitted records and committed records should

## Edit stream configuration

1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** and then click a connection in the list you want to change.
1. On the [Airbyte Cloud](http://cloud.airbyte.io) dashboard, click **Connections** and then click the connection you want to change.

2. Click the **Replication** tab.

The **Transfer** and **Streams** settings include the following parameters:

| Parameter | Description |
|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| Replication Frequency | How often the data syncs |
| [Destination Namespace](https://docs.airbyte.com/understanding-airbyte/namespaces/) | Where the replicated data is written |
| Destination Stream Prefix | Helps you identify streams from different connectors |
| Parameter | Description |
|--------------------------------------|-------------------------------------------------------------------------------------|
| Replication frequency | How often the data syncs |
| [Non-breaking schema updates](#review-non-breaking-schema-changes) detected | How Airbyte handles syncs when it detects non-breaking schema changes in the source |
| Destination Namespace | Where the replicated data is written |
| Destination Stream Prefix | Helps you identify streams from different connectors |

:::note

Expand Down Expand Up @@ -274,9 +276,79 @@ Airbyte recommends that you reset streams. A reset will delete data in the desti
To refresh the source schema:
1. Click **Refresh source schema** to fetch the schema of your data source.

2. If there are changes to the schema, the **Refreshed source schema** dialog displays them.
2. If the schema has changed, the **Refreshed source schema** dialog displays them.

## Manage schema changes

Once every 24 hours, Airbyte checks for changes in your source schema and allows you to review the changes and fix breaking changes.

:::note

Schema changes are flagged in your connection but are not propagated to your destination.

:::

### Review non-breaking schema changes

To review non-breaking schema changes:
1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection with non-breaking changes (indicated by a **yellow exclamation mark** icon).

2. Click **Review changes**.

3. The **Refreshed source schema** dialog displays the changes.

4. Review the changes and click **OK** to close the dialog.

5. Scroll to the bottom of the page and click **Save changes**.

:::note

By default, Airbyte ignores non-breaking changes and continues syncing. You can configure how Airbyte handles syncs when it detects non-breaking changes by [editing the stream configuration](#edit-stream-configuration).

:::

### Fix breaking schema changes

:::note

Breaking changes can only occur in the **Cursor** or **Primary key** fields.

:::

To review and fix breaking schema changes:
1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection with breaking changes (indicated by a **red exclamation mark** icon).

2. Click **Review changes**.

3. The **Refreshed source schema** dialog displays the changes.

4. Review the changes and click **OK** to close the dialog.

5. In the streams table, the stream with a breaking change is highlighted.

6. Fix the breaking change by selecting a new **Cursor** or **Primary key**.

7. Scroll to the bottom of the page and click **Save changes**.

:::note

If a connection’s source schema has breaking changes, it will stop syncing. You must review and fix the changes before editing the connection or resuming syncs.

:::

### Enable schema update notifications

To get notified when your source schema changes:
1. Make sure you have [webhook notifications](https://docs.airbyte.com/cloud/managing-airbyte-cloud#manage-airbyte-cloud-notifications) set up.

2. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click **Connections** and select the connection you want to receive notifications for.

3. Click the **Settings** tab on the Connection page.

4. Toggle **Schema update notifications**.

## Display Connection State

**Connection State** provides additional information about incremental syncs. It includes the most recent values for the global or stream-level cursors, which can aid in debugging or determining which data will be included in the next syncs.

To display **Connection State**:
Expand All @@ -291,7 +363,7 @@ To display **Connection State**:

5. Click the **Settings** tab on the Connection page.

The **Connection State** displays.
The **Connection State** displays.

## Choose the data residency for a connection
You can choose the data residency for your connection in the connection settings. You can also choose data residency when creating a [new connection](https://docs.airbyte.com/cloud/getting-started-with-airbyte-cloud#set-up-a-connection), or you can set the [default data residency](#choose-your-default-data-residency) for your workspace.
Expand Down

0 comments on commit 0cb8388

Please sign in to comment.