Tutorial and documentation for config-based connectors

airbyte-cdk/python/airbyte_cdk/sources/declarative/auth/token.py

@@ -78,7 +78,7 @@ def token(self) -> str:
 
 class BasicHttpAuthenticator(AbstractHeaderAuthenticator):
     """
-    Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using bas64
+    Builds auth based off the basic authentication scheme as defined by RFC 7617, which transmits credentials as USER ID/password pairs, encoded using base64
     https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme
 
     The header is of the form

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/class_types_registry.py

@@ -4,6 +4,7 @@
 
 from typing import Mapping, Type
 
+from airbyte_cdk.sources.declarative.auth.oauth import DeclarativeOauth2Authenticator
 from airbyte_cdk.sources.declarative.auth.token import ApiKeyAuthenticator, BasicHttpAuthenticator, BearerAuthenticator
 from airbyte_cdk.sources.declarative.datetime.min_max_datetime import MinMaxDatetime
 from airbyte_cdk.sources.declarative.declarative_stream import DeclarativeStream
@@ -56,6 +57,7 @@
     "ListStreamSlicer": ListStreamSlicer,
     "MinMaxDatetime": MinMaxDatetime,
     "NoPagination": NoPagination,
+    "OAuthAuthenticator": DeclarativeOauth2Authenticator,
     "OffsetIncrement": OffsetIncrement,
     "RecordSelector": RecordSelector,
     "RemoveFields": RemoveFields,

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/yaml_parser.py

@@ -15,7 +15,7 @@ class YamlParser(ConnectionDefinitionParser):
     """
     Parses a Yaml string to a ConnectionDefinition
 
-    In addition to standard Yaml parsing, the input_string can contain refererences to values previously defined.
+    In addition to standard Yaml parsing, the input_string can contain references to values previously defined.
     This parser will dereference these values to produce a complete ConnectionDefinition.
 
     References can be defined using a *ref(<arg>) string.

airbyte-cdk/python/airbyte_cdk/sources/declarative/requesters/paginators/limit_paginator.py

@@ -24,7 +24,7 @@ class LimitPaginator(Paginator):
         * updates the request path with  "{{ response._metadata.next }}"
           paginator:
             type: "LimitPaginator"
-            limit_value: 10
+            page_size: 10
             limit_option:
               option_type: request_parameter
               field_name: page_size
@@ -41,7 +41,7 @@ class LimitPaginator(Paginator):
         `
           paginator:
             type: "LimitPaginator"
-            limit_value: 5
+            page_size: 5
             limit_option:
               option_type: header
               field_name: page_size
@@ -58,7 +58,7 @@ class LimitPaginator(Paginator):
         `
           paginator:
             type: "LimitPaginator"
-            limit_value: 5
+            page_size: 5
             limit_option:
               option_type: request_parameter
               field_name: page_size

docs/connector-development/config-based/authentication.md

@@ -0,0 +1,70 @@
+# Authentication
+
+The `Authenticator` defines how to configure outgoing HTTP requests to authenticate on the API source.
+
+## Authenticators
+
+### ApiKeyAuthenticator
+
+The `ApiKeyAuthenticator` sets an HTTP header on outgoing requests.
+The following definition will set the header "Authorization" with a value "Bearer hello":
+
+```
+authenticator:
+  type: "ApiKeyAuthenticator"
+  header: "Authorization"
+  token: "Bearer hello"
+```
+
+### BearerAuthenticator
+
+The `BearerAuthenticator` is a specialized `ApiKeyAuthenticator` that always sets the header "Authorization" with the value "Bearer {token}".
+The following definition will set the header "Authorization" with a value "Bearer hello"
+
+```
+authenticator:
+  type: "BearerAuthenticator"
+  token: "hello"
+```
+
+More information on bearer authentication can be found [here](https://swagger.io/docs/specification/authentication/bearer-authentication/)
+
+### BasicHttpAuthenticator
+
+The `BasicHttpAuthenticator` set the "Authorization" header with a (USER ID/password) pair, encoded using base64 as per [RFC 7617](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#basic_authentication_scheme).
+The following definition will set the header "Authorization" with a value "Basic <encoded credentials>"
+
+The encoding scheme is:
+
+1. concatenate the username and the password with `":"` in between
+2. Encode the resulting string in base 64
+3. Decode the result in utf8
+
+```
+authenticator:
+  type: "BasicHttpAuthenticator"
+  username: "hello"
+  password: "world"
+```
+
+The password is optional. Authenticating with APIs using Basic HTTP and a single API key can be done as:
+
+```
+authenticator:
+  type: "BasicHttpAuthenticator"
+  username: "hello"
+```
+
+### OAuth
+
+OAuth authentication is supported through the `OAuthAuthenticator`, which requires the following parameters:
+
+- token_refresh_endpoint: The endpoint to refresh the access token
+- client_id: The client id
+- client_secret: Client secret
+- refresh_token: The token used to refresh the access token
+- scopes: The scopes to request
+- token_expiry_date: The access token expiration date
+- access_token_name: THe field to extract access token from in the response
+- expires_in_name:The field to extract expires_in from in the response
+- refresh_request_body: The request body to send in the refresh request

docs/connector-development/config-based/connector-definition.md

@@ -0,0 +1,243 @@
+# Connector definition
+
+Connectors are defined as a yaml configuration describing the connector's Source.
+
+2 top-level fields are required:
+
+1. `streams`: list of streams that are part of the source
+2. `check`: component describing how to check the connection.
+
+The configuration will be validated against this JSON Schema, which defines the set of valid properties.
+
+We recommend using the `Configuration Based Source` template from the template generator in `airbyte-integrations/connector-templates/generator` to generate the basic file structure.
+
+See the [tutorial for a complete connector definition](tutorial/6-testing.md)
+
+## Object instantiation
+
+This section describes the object that are to be instantiated from the YAML definition.
+
+If the component is a literal, then it is returned as is:
+
+```
+3
+```
+
+will result in
+
+```
+3
+```
+
+If the component is a mapping with a "class_name" field,
+an object of type "class_name" will be instantiated by passing the mapping's other fields to the constructor
+
+```
+{
+  "class_name": "fully_qualified.class_name",
+  "a_parameter: 3,
+  "another_parameter: "hello"
+}
+```
+
+will result in
+
+```
+fully_qualified.class_name(a_parameter=3, another_parameter="helo"
+```
+
+If the component definition is a mapping with a "type" field,
+the factory will lookup the [CLASS_TYPES_REGISTRY](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/class_types_registry.py) and replace the "type" field by "class_name" -> CLASS_TYPES_REGISTRY[type]
+and instantiate the object from the resulting mapping
+
+If the component definition is a mapping with neither a "class_name" nor a "type" field,
+the factory will do a best-effort attempt at inferring the component type by looking up the parent object's constructor type hints.
+If the type hint is an interface present in [DEFAULT_IMPLEMENTATIONS_REGISTRY](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/default_implementation_registry.py,
+then the factory will create an object of its default implementation.
+
+If the component definition is a list, then the factory will iterate over the elements of the list,
+instantiate its subcomponents, and return a list of instantiated objects.
+
+If the component has subcomponents, the factory will create the subcomponents before instantiating the top level object
+
+```
+{
+  "type": TopLevel
+  "param":
+    {
+      "type": "ParamType"
+      "k": "v"
+    }
+}
+```
+
+will result in
+
+```
+TopLevel(param=ParamType(k="v"))
+```
+
+More details on object instantiation can be found [here](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.parsers.html?highlight=factory#airbyte_cdk.sources.declarative.parsers.factory.DeclarativeComponentFactory).
+
+### $options
+
+Parameters can be passed down from a parent component to its subcomponents using the $options key.
+This can be used to avoid repetitions.
+
+```
+outer:
+  $options:
+    MyKey: MyValue
+  inner:
+   k2: v2
+```
+
+This the example above, if both outer and inner are types with a "MyKey" field, both of them will evaluate to "MyValue".
+
+The value can also be used for string interpolation:
+
+```
+outer:
+  $options:
+    MyKey: MyValue
+  inner:
+   k2: "MyKey is {{ options.MyKey }}"
+```
+
+In this example, outer.inner.k2 will evaluate to "MyValue"
+
+## References
+
+Strings can contain references to values previously defined.
+The parser will dereference these values to produce a complete ConnectionDefinition
+
+References can be defined using a *ref(<arg>) string.
+
+```
+key: 1234
+reference: "*ref(key)"
+```
+
+will produce the following definition:
+
+```
+key: 1234
+reference: 1234
+```
+
+This also works with objects:
+
+```
+key_value_pairs:
+  k1: v1
+  k2: v2
+same_key_value_pairs: "*ref(key_value_pairs)"
+```
+
+will produce the following definition:
+
+```
+key_value_pairs:
+  k1: v1
+  k2: v2
+same_key_value_pairs:
+  k1: v1
+  k2: v2
+```
+
+The $ref keyword can be used to refer to an object and enhance it with addition key-value pairs
+
+```
+key_value_pairs:
+  k1: v1
+  k2: v2
+same_key_value_pairs:
+  $ref: "*ref(key_value_pairs)"
+  k3: v3
+```
+
+will produce the following definition:
+
+```
+key_value_pairs:
+  k1: v1
+  k2: v2
+same_key_value_pairs:
+  k1: v1
+  k2: v2
+  k3: v3
+```
+
+References can also point to nested values.
+Nested references are ambiguous because one could define a key containing with `.`
+in this example, we want to refer to the limit key in the dict object:
+
+```
+dict:
+    limit: 50
+limit_ref: "*ref(dict.limit)"
+```
+
+will produce the following definition:
+
+```
+dict
+    limit: 50
+limit-ref: 50
+```
+
+whereas here we want to access the `nested.path` value.
+
+```
+nested:
+    path: "first one"
+nested.path: "uh oh"
+value: "ref(nested.path)
+```
+
+will produce the following definition:
+
+```
+nested:
+    path: "first one"
+nested.path: "uh oh"
+value: "uh oh"
+```
+
+to resolve the ambiguity, we try looking for the reference key at the top level, and then traverse the structs downward
+until we find a key with the given path, or until there is nothing to traverse.
+
+More details on referencing values can be found [here](https://airbyte-cdk.readthedocs.io/en/latest/api/airbyte_cdk.sources.declarative.parsers.html?highlight=yamlparser#airbyte_cdk.sources.declarative.parsers.yaml_parser.YamlParser).
+
+## String interpolation
+
+String values can be evaluated as Jinja2 templates.
+
+Interpolation strategy using the Jinja2 template engine.
+
+If the input string is a raw string, the interpolated string will be the same.
+`"hello world" -> "hello world"`
+
+The engine will evaluate the content passed within {{}}, interpolating the keys from context-specific arguments.
+the "options" keyword [see ($options)](connector-definition.md#object-instantiation) can be referenced.
+
+For example, inner_object.key will evaluate to "Hello airbyte" at runtime.
+
+```
+some_object:
+  $options:
+    name: "airbyte"
+  inner_object:
+    key: "Hello {{ options.name }}"
+```
+
+Some components also pass in additional arguments to the context.
+This is the case for the [record selector](record-selector.md), which passes in an additional `response` argument.
+
+In additional to passing additional values through the kwargs argument, macros can be called from within the string interpolation.
+For example,
+`"{{ max(2, 3) }}" -> 3`
+
+The macros available can be found [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/python/airbyte_cdk/sources/declarative/interpolation/macros.py).
+
+Additional information on jinja templating can be found at https://jinja.palletsprojects.com/en/3.1.x/templates/#