Tutorial and documentation for config-based connectors

airbyte-cdk/python/airbyte_cdk/sources/declarative/auth/token.py

@@ -70,7 +70,7 @@ def token(self) -> str:
 
 class BasicHttpAuthenticator(AbstractHeaderAuthenticator):
     """
-    Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using bas64
+    Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64
     https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme
 
     The header is of the form

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/class_types_registry.py

@@ -4,6 +4,7 @@
 
 from typing import Mapping, Type
 
+from airbyte_cdk.sources.declarative.auth.oauth import DeclarativeOauth2Authenticator
 from airbyte_cdk.sources.declarative.auth.token import ApiKeyAuthenticator, BasicHttpAuthenticator, BearerAuthenticator
 from airbyte_cdk.sources.declarative.datetime.min_max_datetime import MinMaxDatetime
 from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream
@@ -56,6 +57,7 @@
     "ListStreamSlicer": ListStreamSlicer,
     "MinMaxDatetime": MinMaxDatetime,
     "NoPagination": NoPagination,
+    "OAuthAuthenticator": DeclarativeOauth2Authenticator,
     "OffsetIncrement": OffsetIncrement,
     "RecordSelector": RecordSelector,
     "RemoveFields": RemoveFields,

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/yaml_parser.py

@@ -15,7 +15,7 @@ class YamlParser(ConnectionDefinitionParser):
     """
     Parses a Yaml string to a ConnectionDefinition
 
-    In addition to standard Yaml parsing, the input_string can contain refererences to values previously defined.
+    In addition to standard Yaml parsing, the input_string can contain references to values previously defined.
     This parser will dereference these values to produce a complete ConnectionDefinition.
 
     References can be defined using a *ref(<arg>) string.

airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/limit_paginator.py

@@ -24,7 +24,7 @@ class LimitPaginator(Paginator):
         * updates the request path with  "{{ response._metadata.next }}"
           paginator:
             type: "LimitPaginator"
-            limit_value: 10
+            page_size: 10
             limit_option:
               option_type: request_parameter
               field_name: page_size
@@ -41,7 +41,7 @@ class LimitPaginator(Paginator):
         `
           paginator:
             type: "LimitPaginator"
-            limit_value: 5
+            page_size: 5
             limit_option:
               option_type: header
               field_name: page_size
@@ -58,7 +58,7 @@ class LimitPaginator(Paginator):
         `
           paginator:
             type: "LimitPaginator"
-            limit_value: 5
+            page_size: 5
             limit_option:
               option_type: request_parameter
               field_name: page_size

docs/connector-development/config-based/authentication.md

@@ -0,0 +1,70 @@
+# Authentication
+
+The `Authenticator` defines how to configure outgoing HTTP requests to authenticate on the API source.
+
+## Authenticators
+
+### ApiKeyAuthenticator
+
+The `ApiKeyAuthenticator` sets an HTTP header on outgoing requests.
+The following definition will set the header "Authorization" with a value "Bearer hello":
+
+```
+authenticator:
+  type: "ApiKeyAuthenticator"
+  header: "Authorization"
+  token: "Bearer hello"
+```
+
+### BearerAuthenticator
+
+The `BearerAuthenticator` is a specialized `ApiKeyAuthenticator` that always sets the header "Authorization" with the value "Bearer {token}".
+The following definition will set the header "Authorization" with a value "Bearer hello"
+
+```
+authenticator:
+  type: "BearerAuthenticator"
+  token: "hello"
+```
+
+More information on bearer authentication can be found [here](https://swagger.io/docs/specification/authentication/bearer-authentication/)
+
+### BasicHttpAuthenticator
+
+The `BasicHttpAuthenticator` set the "Authorization" header with a (USER ID/password) pair, encoded using base64 as per [RFC 7617](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme).
+The following definition will set the header "Authorization" with a value "Basic <encoded credentials>"
+
+The encoding scheme is:
+
+1. concatenate the username and the password with `":"` in between
+2. Encode the resulting string in base 64
+3. Decode the result in utf8
+
+```
+authenticator:
+  type: "BasicHttpAuthenticator"
+  username: "hello"
+  password: "world"
+```
+
+The password is optional. Authenticating with APIs using Basic HTTP and a single API key can be done as:
+
+```
+authenticator:
+  type: "BasicHttpAuthenticator"
+  username: "hello"
+```
+
+### OAuth
+
+OAuth authentication is supported through the `OAuthAuthenticator`, which requires the following parameters:
+
+- token_refresh_endpoint: The endpoint to refresh the access token
+- client_id: The client id
+- client_secret: Client secret
+- refresh_token: The token used to refresh the access token
+- scopes: The scopes to request
+- token_expiry_date: The access token expiration date
+- access_token_name: THe field to extract access token from in the response
+- expires_in_name:The field to extract expires_in from in the response
+- refresh_request_body: The request body to send in the refresh request

docs/connector-development/config-based/error-handling.md

@@ -0,0 +1,173 @@
+# Error handling
+
+By default, only retry server errors (HTTP 5XX) and too many requests (HTTP 429) will be retried up to 5 times with exponential backoff.
+Other HTTP errors will result in a failed read.
+
+Other behaviors can be configured through the `Requester`'s `error_handler` field.
+
+## Defining errors
+
+Response filters can be used to define how to handle requests resulting in responses with a specific HTTP status code.
+For instance, this example will configure the handler to also retry responses with 404 error:
+
+```
+requester:
+  <...>
+  error_handler:
+    response_filters:
+      - http_codes: [404]
+        action: RETRY
+```
+
+Response filters can be used to specify HTTP errors to ignore instead of retrying.
+For instance, this example will configure the handler to ignore responses with 404 error:
+
+```
+requester:
+  <...>
+  error_handler:
+    response_filters:
+      - http_codes: [404]
+        action: IGNORE
+```
+
+Errors can also be defined by parsing the error message.
+For instance, this error handler will ignores responses if the error message contains the string "ignorethisresponse"
+
+```
+requester:
+  <...>
+  error_handler:
+    response_filters:
+      - error_message_contain: "ignorethisresponse"
+        action: IGNORE
+```
+
+This can also be done through a more generic string interpolation strategy with the following parameters:
+
+- response:
+
+This example ignores errors where the response contains a "code" field:
+
+```
+requester:
+  <...>
+  error_handler:
+    response_filters:
+      - predicate: "{{ 'code' in response }}"
+        action: IGNORE
+```
+
+The error handler can have multiple response filters.
+The following example is configured to ignore 404 errors, and retry 429 errors:
+
+```
+requester:
+  <...>
+  error_handler:
+    response_filters:
+      - http_codes: [404]
+        action: IGNORE
+      - http_codes: [429]
+        action: RETRY
+```
+
+## Backoff Strategies
+
+The error handle supports a few backoff strategies, which are described in the following sections.
+
+### Exponential backoff
+
+This is the default backoff strategy. The requester will backoff with an exponential backoff interval
+
+### Constant Backoff
+
+When using the `ConstantBackoffStrategy`, the requester will backoff with a constant interval.
+
+### Wait time defined in header
+
+When using the `WaitTimeFromHeaderBackoffStrategy`, the requester will backoff by an interval specified in the response header.
+In this example, the requester will backoff by the response's "wait_time" header value:
+
+```
+requester:
+  <...>
+  error_handler:
+    <...>
+    backoff_strategies:
+      - type: "WaitTimeFromHeaderBackoffStrategy"
+        header: "wait_time"
+```
+
+Optionally, a regex can be configured to extract the wait time from the header value.
+
+```
+requester:
+  <...>
+  error_handler:
+    <...>
+    backoff_strategies:
+      - type: "WaitTimeFromHeaderBackoffStrategy"
+        header: "wait_time"
+        regex: "[-+]?\d+"
+```
+
+### Wait until time defined in header
+
+When using the `WaitUntilTimeFromHeaderBackoffStrategy`, the requester will backoff until the time specified in the response header.
+In this example, the requester will wait until the time specified in the "wait_until" header value:
+
+```
+requester:
+  <...>
+  error_handler:
+    <...>
+    backoff_strategies:
+      - type: "WaitUntilTimeFromHeaderBackoffStrategy"
+        header: "wait_until"
+        regex: "[-+]?\d+"
+        min_wait: 5
+```
+
+The strategy accepts an optional regex to extract the time from the header value, and a minimum time to wait.
+
+## Advanced error handling
+
+The error handler can have multiple backoff strategies, allowing it to fallback if a strategy cannot be evaluated.
+For instance, the following defines an error handler that will read the backoff time from a header, and default to a constant backoff if the wait time could not be extracted from the response:
+
+```
+requester:
+  <...>
+  error_handler:
+    <...>
+    backoff_strategies:
+      - type: "WaitTimeFromHeaderBackoffStrategy"
+        header: "wait_time"
+      - type: "ConstantBackoffStrategy"
+        backoff_time_in_seconds: 5
+
+```
+
+The `requester` can be configured to use a `CompositeErrorHandler`, which sequentially iterates over a list of error handlers, enabling different retry mechanisms for different types of errors.
+
+In this example, a constant backoff of 5 seconds, will be applied if the response contains a "code" field, and an exponential backoff will be applied if the error code is 403:
+
+```
+requester:
+  <...>
+  error_handler:
+    type: "CompositeErrorHandler"
+    error_handlers:
+      - response_filters:
+          - predicate: "{{ 'code' in response }}"
+            action: RETRY
+        backoff_strategies:
+          - type: "ConstantBackoffStrategy"
+            backoff_time_in_seconds: 5
+      - response_filters:
+          - http_codes: [ 403 ]
+            action: RETRY
+        backoff_strategies:
+          - type: "ExponentialBackoffStrategy"
+```

docs/connector-development/config-based/overview.md

@@ -0,0 +1,90 @@
+# Config-based connectors overview
+
+The goal of this document is to give enough technical specifics to understand how config-based connectors work.
+When you're ready to start building a connector, you can start with [the tutorial](../../../config-based/tutorial/0-getting-started.md) or dive into the [reference documentation](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.html)
+
+## Overview
+
+Config-based connectors work by parsing a YAML configuration describing the Source, then running the configured connector using a Python backend.
+
+The process then submits HTTP requests to the API endpoint, and extracts records out of the response.
+
+## Source
+
+Config-based connectors are a declarative way to define HTTP API sources.
+
+A source is defined by 2 components:
+
+1. The source's `Stream`s, which define the data to read
+2. A `ConnectionChecker`, which describes how to run the `check` operation to test the connection to the API source
+
+## Stream
+
+Streams define the schema of the data of interest, as well as how to read it from the underlying API source.
+A stream generally corresponds to a resource within the API. They are analogous to tables for a RDMS source.
+
+A stream is defined by:
+
+1. Its name
+2. A primary key: used to uniquely identify records, enabling deduplication
+3. A schema: describes the data to sync
+4. A data retriever: describes how to retrieve the data from the API
+5. A cursor field: used to identify the stream's state from a record
+6. A set of transformations to be applied on the records read from the source before emitting them to the destination
+7. A checkpoint interval: defines when to checkpoint syncs.
+
+More details on streams and sources can be found in the [basic concepts section](../cdk-python/basic-concepts.md).
+More details on cursor fields, and checkpointing can be found in the [incremental-stream section](../cdk-python/incremental-stream.md)
+
+## Data retriever
+
+The data retriever defines how to read the data from an API source, and acts as an orchestrator for the data retrieval flow.
+The is currently only one implementation, the `SimpleRetriever`, which is defined by
+
+1. Requester: describes how to submit requests to the API source
+2. Paginator[^1]: describes how to navigate through the API's pages
+3. Record selector: describes how to select records from an HTTP response
+4. Stream Slicer: describes how to partition the stream, enabling incremental syncs and checkpointing
+
+Each of those components (and their subcomponents) are defined by an explicit interface and one or many implementations.
+The developer can choose and configure the implementation they need depending on specifications of the integrations they are building against.
+
+### Data flow
+
+The retriever acts as a coordinator, moving the data between its components before emitting `AirbyteMessage`s that can be read by the platform.
+The `SimpleRetriever`'s data flow can be described as follows:
+
+1. Given the connection config and the current stream state, the `StreamSlicer` computes the stream slices to read.
+2. Iterate over all the stream slices defined by the stream slicer.
+3. For each stream slice,
+    1. Submit a request as defined by the requester
+    2. Select the records from the response
+    3. Repeat for as long as the paginator points to a next page
+
+More details on the paginator can be found in the [pagination section](pagination.md)
+More details on the record selector can be found in the [record selector section](record-selector.md)
+More details on the stream slicers can be found in the [stream slicers section](stream-slicers.md)
+
+## Requester
+
+The `Requester` defines how to prepare HTTP requests to send to the source API [^2].
+There currently is only one implementation, the `HttpRequester`, which is defined by
+
+1. A base url: the root of the API source
+2. A path: the specific endpoint to fetch data from for a resource
+3. The HTTP method: the HTTP method to use (GET or POST)
+4. A request options provider: defines the request parameters and headers to set on outgoing HTTP requests
+5. An authenticator: defines how to authenticate to the source
+6. An error handler: defines how to handle errors
+
+More details on authentication can be found in the [authentication section](authentication.md).
+More details on error handling can be found in the [error handling section](error-handling.md)
+
+## Connection Checker
+
+The `ConnectionChecker` defines how to test the connection to the integration.
+
+The only implementation as of now is `CheckStream`, which tries to read a record from a specified list of streams and fails if no records could be read.
+
+[^1] The paginator is conceptually more related to the requester than the data retriever, but is part of the `SimpleRetriever` because it inherits from `HttpStream` to increase code reusability.
+[^2] As of today, the requester acts as a config object and is not directly responsible for preparing the HTTP requests. This is done in the `SimpleRetriever`'s parent class `HttpStream`.