Skip to content

Commit

Permalink
Update public documentation to include restrictions for backwards inc…
Browse files Browse the repository at this point in the history
…ompatible changes (#14731)

* Update public documentation to include restrictions for backwards incompatible schema changes

* move a section down a bit

* Update docs/connector-development/best-practices.md

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
  • Loading branch information
brianjlai and sherifnada authored Jul 18, 2022
1 parent 2dc4759 commit 92a45d9
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 0 deletions.
15 changes: 15 additions & 0 deletions docs/connector-development/best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,18 @@ When reviewing connectors, we'll use the following "checklist" to verify whether

Most APIs enforce rate limits. Your connector should gracefully handle those \(i.e: without failing the connector process\). The most common way to handle rate limits is to implement backoff.

## Maintaining connectors

Once a connector has been published for use within Airbyte, we must take special care to account for the customer impact of updates to the connector.

### Schema Breaking Changes

For connectors that are GA certified or highly used by customers, we should not introduce backwards incompatible changes into a stream's schema that impact how existing data is replicated and represented in the downstream destination. The schema serves as a contract with customers to define how data is synchronized. Subtractive changes can COMPLETELY BREAK a customer's workflows built on top of Airbyte, sometimes in a silent way.

The following types of changes are to be considered to be breaking:

* Removing a property field from a schema
* Renaming a property field in a schema
* Changing a property's data type

Exceptions can be made on a case by case basis, but if your updates require a subtractive change to the schema, you should consider whether the change is necessary or if there is an alternative that will not break backwards compatibility.
3 changes: 3 additions & 0 deletions docs/connector-development/cdk-python/schemas.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ If you are implementing a connector to pull data from an API which publishes an

We also provide a tool for generating schemas using a connector's `read` command output. Detailed information can be found [here](https://github.com/airbytehq/airbyte/tree/master/tools/schema_generator/).

### Backwards Compatibility

Because statically defined schemas explicitly define how data is represented in a destination, updates to a schema must be backwards compatible with prior versions. More information about breaking changes can be found [here](../best-practices.md#schema-breaking-changes)

## Dynamic schemas

Expand Down

0 comments on commit 92a45d9

Please sign in to comment.