From cae63965bd9b736b7dacce28e6a7dcf20544e243 Mon Sep 17 00:00:00 2001 From: Amruta Ranade <11484018+Amruta-Ranade@users.noreply.github.com> Date: Tue, 3 Jan 2023 19:18:35 +0530 Subject: [PATCH] Deployment docs and sidebar cleanup (#20965) --- docs/api-documentation.md | 15 ++-- docs/deploying-airbyte/on-aws-ec2.md | 54 ++++++------- .../on-azure-vm-cloud-shell.md | 13 +-- .../on-digitalocean-droplet.md | 6 +- .../on-gcp-compute-engine.md | 9 +-- .../on-kubernetes-via-helm.md | 59 ++++++++------ docs/deploying-airbyte/on-kubernetes.md | 81 ++++++++++--------- docs/deploying-airbyte/on-oci-vm.md | 18 ++--- docs/deploying-airbyte/on-plural.md | 35 ++++---- docs/deploying-airbyte/on-restack.md | 34 ++++---- docs/integrations/destinations/csv.md | 2 +- docs/integrations/destinations/pubsub.md | 2 +- .../sources/{hellobaton.md => baton.md} | 0 docs/integrations/sources/faker.md | 2 +- docs/integrations/sources/http-request.md | 2 +- docs/integrations/sources/mailerlite.md | 2 +- docs/integrations/sources/omnisend.md | 2 +- docs/integrations/sources/s3.md | 2 +- docs/integrations/sources/xero.md | 2 +- docs/operator-guides/security.md | 41 +++++++++- docusaurus/sidebars.js | 9 +-- 21 files changed, 209 insertions(+), 181 deletions(-) rename docs/integrations/sources/{hellobaton.md => baton.md} (100%) diff --git a/docs/api-documentation.md b/docs/api-documentation.md index 1267ef1eea51..78175265edc5 100644 --- a/docs/api-documentation.md +++ b/docs/api-documentation.md @@ -2,24 +2,21 @@ ## Open-Source API -:::caution - -For Airbyte Open-Source you don't need the API Token for Authentication! -All endpoints are possible to access using the API without it. +Airbyte Open Source doesn't require an API Token for Authentication. +All endpoints are acccessible without the token. -::: +:::caution Our Open-Source API (configuration API) is still in an alpha state and might change. You won’t lose any functionality, but you may need to update your code to catch up to any backwards incompatible changes in the API. +::: **Check out the** [**Open-Source API documentation**](https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html)**.** -Don't hesitate to ask us on [Slack](https://slack.airbyte.io), if you have any questions about it. +Contact us on [Slack](https://slack.airbyte.io) if you have any questions about it. --- ## Airbyte Cloud API -:::info - -Airbyte Cloud will also support API access in 2023. See [Airbyte roadmap](https://app.harvestr.io/roadmap/view/pQU6gdCyc/airbyte-roadmap) for more details. +Airbyte Cloud will support API access in 2023. See [Airbyte roadmap](https://app.harvestr.io/roadmap/view/pQU6gdCyc/airbyte-roadmap) for more details. diff --git a/docs/deploying-airbyte/on-aws-ec2.md b/docs/deploying-airbyte/on-aws-ec2.md index 889da1549a9b..7c982a3b4c8a 100644 --- a/docs/deploying-airbyte/on-aws-ec2.md +++ b/docs/deploying-airbyte/on-aws-ec2.md @@ -1,35 +1,34 @@ # Deploy Airbyte on AWS (Amazon EC2) -This page guides you through deploying Airbyte Open Source on an Amazon EC2 instance by setting up the deployment environment, installing and starting Airbyte, and connecting it to the Amazon EC2 instance. +This page guides you through deploying Airbyte Open Source on an [Amazon EC2 instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html) by setting up the deployment environment, installing and starting Airbyte, and connecting it to the Amazon EC2 instance. -> INFO -> -> The instructions have been tested on Amazon Linux 2 AMI (HVM) +:::info + +The instructions have been tested on Amazon Linux 2 AMI (HVM). + +::: ## Requirements - To test Airbyte, we recommend a `t2.medium` instance - To deploy Airbyte in a production environment, we recommend a `t2.large` instance - Make sure your Docker Desktop app is running to ensure all services run smoothly - -[Create and download an SSH key to connect to the instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html) +- [Create and download an SSH key to connect to the instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/create-key-pairs.html) ## Set up the environment -1. Create an Amazon EC2 instance. - -To connect to your instance, run the following command on your local terminal: +1. To connect to your instance, run the following command on your local terminal: -``` +``` bash SSH_KEY=~/Downloads/dataline-key-airbyte.pem # the file path you downloaded the key -INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP # you can find your IP address in the EC2 console under the Instances tab +INSTANCE_IP=REPLACE_WITH_YOUR_INSTANCE_IP # find your IP address in the EC2 console under the Instances tab chmod 400 $SSH_KEY # or ssh will complain that the key has the wrong permissions ssh -i $SSH_KEY ec2-user@$INSTANCE_IP # connect to the aws ec2 instance AMI and the your private IP address ``` 2. To install Docker, run the following command in your SSH session on the instance terminal: -``` +``` bash sudo yum update -y sudo yum install -y docker sudo service docker start @@ -38,7 +37,7 @@ sudo usermod -a -G docker $USER 3. To install `docker-compose`, run the following command in your ssh session on the instance terminal: -``` +``` bash sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose docker-compose --version @@ -46,7 +45,7 @@ docker-compose --version 4. To close the SSH connection, run the following command in your SSH session on the instance terminal: -``` +``` bash logout ``` @@ -54,15 +53,15 @@ logout In your local terminal, run the following commands: -1. Connect to your instance +1. Connect to your instance: -``` +``` bash ssh -i $SSH_KEY ec2-user@$INSTANCE_IP ``` -2. Install Airbyte +2. Install Airbyte: -``` +``` bash mkdir airbyte && cd airbyte wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml} docker-compose up -d # run the Docker container @@ -70,18 +69,19 @@ docker-compose up -d # run the Docker container ## Connect to Airbyte -> DANGER -> -> We strongly recommend not exposing Airbyte on Internet available ports for security reasons. +:::caution -1. Create an SSH tunnel for port 8000 +For security reasons, we strongly recommend not exposing Airbyte on Internet available ports. -> INFO -> -> If you want to use different ports, modify `API_URL` in your .env file and restart Airbyte. -> Run the following commands in your workstation terminal from the downloaded key folder: +::: -``` +1. Create an SSH tunnel for port 8000: + +If you want to use different ports, modify `API_URL` in your .env file and restart Airbyte. +Run the following commands in your workstation terminal from the downloaded key folder: + + +``` bash # In your workstation terminal SSH_KEY=~/Downloads/dataline-key-airbyte.pem ssh -i $SSH_KEY -L 8000:localhost:8000 -N -f ec2-user@$INSTANCE_IP diff --git a/docs/deploying-airbyte/on-azure-vm-cloud-shell.md b/docs/deploying-airbyte/on-azure-vm-cloud-shell.md index c51301403ed3..0b6ac3a3ad9e 100644 --- a/docs/deploying-airbyte/on-azure-vm-cloud-shell.md +++ b/docs/deploying-airbyte/on-azure-vm-cloud-shell.md @@ -1,6 +1,6 @@ # Deploy Airbyte on Azure (Cloud Shell) -This page guides you through deploying Airbyte Open Source on a Microsoft Azure VM by setting up the deployment environment, installing and starting Airbyte, and connecting it to the VM. +This page guides you through deploying Airbyte Open Source on a [Microsoft Azure VM](https://learn.microsoft.com/en-us/azure/virtual-machines/) by setting up the deployment environment, installing and starting Airbyte, and connecting it to the VM. :::info @@ -81,13 +81,11 @@ Download Airbyte and deploy it in the VM using Docker Compose: wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml} ``` -4. Start Airbyte by running the following command: +4. To start Airbyte, run the following command: ```bash sudo docker-compose up -d ``` - This step takes about two minutes to complete. When done, you will see the cursor prompt. - ## Connect to Airbyte @@ -100,14 +98,11 @@ Test a remote connection to your VM locally and verify that Airbyte is up and ru ```bash ssh -N -L 8000:localhost:8000 -i @ ``` - As a result, nothing happens. The cursor prompt keeps blinking. 4. Open a web browser and navigate to `http://localhost:8000`. You will see Airbyte’s landing page. -:::danger - -For security reasons, it is strongly recommended not to expose Airbyte on Internet available ports. Future versions will add support for SSL and authentication. - +:::caution +For security reasons, we strongly recommend not exposing Airbyte on Internet available ports. ::: ## Troubleshooting diff --git a/docs/deploying-airbyte/on-digitalocean-droplet.md b/docs/deploying-airbyte/on-digitalocean-droplet.md index 73f072b4e32f..a14ead2affc6 100644 --- a/docs/deploying-airbyte/on-digitalocean-droplet.md +++ b/docs/deploying-airbyte/on-digitalocean-droplet.md @@ -1,13 +1,13 @@ # Deploy Airbyte on DigitalOcean -This page guides you through deploying Airbyte Open Source on a DigitalOcean droplet by setting up the deployment environment, and installing and starting Airbyte. +This page guides you through deploying Airbyte Open Source on a [DigitalOcean droplet](https://docs.digitalocean.com/products/droplets/how-to/create/) by setting up the deployment environment, and installing and starting Airbyte. Alternatively, you can deploy Airbyte on DigitalOcean in one click using their [marketplace](https://cloud.digitalocean.com/droplets/new?onboarding_origin=marketplace&appId=95451155&image=airbyte&utm_source=deploying-airbyte_on-digitalocean-droplet). ## Requirements -- To test Airbyte, we recommend a $20/month droplet. -- To deploy Airbyte in a production environment, we recommend a $40/month instance. +- To test Airbyte, we recommend a $20/month droplet +- To deploy Airbyte in a production environment, we recommend a $40/month instance ## Set up the Environment diff --git a/docs/deploying-airbyte/on-gcp-compute-engine.md b/docs/deploying-airbyte/on-gcp-compute-engine.md index 09e70d999909..a3eac4e37d0f 100644 --- a/docs/deploying-airbyte/on-gcp-compute-engine.md +++ b/docs/deploying-airbyte/on-gcp-compute-engine.md @@ -1,6 +1,3 @@ -import Tabs from '@theme/Tabs'; -import TabItem from '@theme/TabItem'; - # Deploy Airbyte on GCP (Compute Engine) This page guides you through deploying Airbyte Open Source on a [Google Cloud Platform (GCP) Compute Engine instance](https://cloud.google.com/compute/docs/instances) by setting up the deployment environment, installing and starting Airbyte, and connecting it to the GCP instance. @@ -13,8 +10,8 @@ The instructions have been tested on a `Debian GNU/Linux 10` VM instance. ## Requirements -- To test Airbyte, we recommend an `e2.medium` instance and provision at least 30GBs of disk per node. -- To deploy Airbyte in a production environment, we recommend a `n1-standard-2` instance. +- To test Airbyte, we recommend an `e2.medium` instance and provision at least 30GBs of disk per node +- To deploy Airbyte in a production environment, we recommend a `n1-standard-2` instance ## Set up the environment @@ -95,7 +92,7 @@ docker-compose up -d ## Connect to Airbyte :::caution -Warning: For security reasons, we strongly recommended not exposing Airbyte publicly. +For security reasons, we strongly recommended not exposing Airbyte publicly. ::: 1. In your local terminal, create an SSH tunnel to connect the GCP instance to Airbyte: diff --git a/docs/deploying-airbyte/on-kubernetes-via-helm.md b/docs/deploying-airbyte/on-kubernetes-via-helm.md index 2775d5e22b8b..2fa4eb0617b3 100644 --- a/docs/deploying-airbyte/on-kubernetes-via-helm.md +++ b/docs/deploying-airbyte/on-kubernetes-via-helm.md @@ -6,7 +6,8 @@ Airbyte allows scaling sync workloads horizontally using Kubernetes. The core co ## Quickstart -If you don't want to configure your own Kubernetes cluster and Airbyte instance, you can use the free, open-source project [Plural](https://www.plural.sh/) to bring up a Kubernetes cluster and Airbyte for you. Use [this guide](on-plural.md) to get started.
+If you don't want to configure your own Kubernetes cluster and Airbyte instance, you can use the free, open-source project [Plural](https://www.plural.sh/) to bring up a Kubernetes cluster and Airbyte for you. Use [this guide](on-plural.md) to get started. + Alternatively, you can deploy Airbyte on [Restack](https://www.restack.io) to provision your Kubernetes cluster on AWS. Follow [this guide](on-restack.md) to get started. ## Getting Started @@ -34,33 +35,39 @@ If you do not already have the CLI tool `kubectl` installed, please follow [thes Configure `kubectl` to connect to your cluster by using `kubectl use-context my-cluster-name`. -* For GKE - * Configure `gcloud` with `gcloud auth login`. - * On the Google Cloud Console, the cluster page will have a `Connect` button, which will give a command to run locally that looks like +For GKE: + +1. Configure `gcloud` with `gcloud auth login`. +2. On the Google Cloud Console, the cluster page will have a `Connect` button, which will give a command to run locally that looks like `gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE_NAME --project $PROJECT_NAME`. - * Use `kubectl config get-contexts` to show the contexts available. - * Run `kubectl config use-context $GKE_CONTEXT` to access the cluster from `kubectl`. -* For EKS - * [Configure your AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project. - * Install [eksctl](https://eksctl.io/introduction/) - * Run `eksctl utils write-kubeconfig --cluster=` to make the context available to `kubectl` - * Use `kubectl config get-contexts` to show the contexts available. - * Run `kubectl config use-context ` to access the cluster with `kubectl`. +3. Use `kubectl config get-contexts` to show the contexts available. +4. Run `kubectl config use-context $GKE_CONTEXT` to access the cluster from `kubectl`. + +For EKS: + +1. [Configure your AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project. +2. Install [eksctl](https://eksctl.io/introduction/) +3. Run `eksctl utils write-kubeconfig --cluster=` to make the context available to `kubectl` +4. Use `kubectl config get-contexts` to show the contexts available. +5. Run `kubectl config use-context ` to access the cluster with `kubectl`. ### Install helm To install helm simply run: -* For MacOS - * `brew install helm` -* For Linux - * Download installer script `curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3` - * Assign required premissions `chmod 700 get_helm.sh` - * Run script `./get_helm.sh` +For MacOS: + + `brew install helm` -### Adding Helm Repository +For Linux: + +1. Download installer script `curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3` +2. Assign required premissions `chmod 700 get_helm.sh` +3. Run script `./get_helm.sh` + +### Add Helm Repository From now charts are stored in helm-repo thus there're no need to clone the repo each time you need to deploy the chart. @@ -85,7 +92,8 @@ airbyte-oss/webapp 0.30.23 0.39.37-alpha Helm chart to de airbyte-oss/worker 0.30.23 0.39.37-alpha Helm chart to deploy airbyte-worker ``` -## Deploying airbyte +## Deploy Airbyte + ### Default deployment If you don't intend to customise your deployment, you can deploy airbyte as is with default values. @@ -95,20 +103,19 @@ In order to do so, run the command: helm install %release_name% airbyte/airbyte ``` -It'll deploy airbyte with default configuration +### Custom deployment -### Configuring deployment +In order to customize your deployment, you need to create `values.yaml` file in the local folder and populate it with default configuration override values. -In order to configure deployment, you need to create values.yaml file in local folder and populate it with default configuration override values. +`values.yaml` example can be located in [charts/airbyte](https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/values.yaml) folder of the Airbyte repository. -values.yaml example can be located in [charts/airbyte](https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/values.yaml) folder of the airbyte repository. +After specifying your own configuration, run the following command: -After specifying your own configuration, proceed with chart deployment by running ```text helm install --values path/to/values.yaml %release_name% airbyte/airbyte ``` -## Migration from old chart to new ones +## Migrate from old charts to new ones Starting from `0.39.37-alpha` we've revisited helm charts structure and separated all components of airbyte into their own independent charts, thus by allowing our developers to test single component without deploying airbyte as a whole and by upgrading single component at a time. diff --git a/docs/deploying-airbyte/on-kubernetes.md b/docs/deploying-airbyte/on-kubernetes.md index a17246c948eb..ea0a3a4e8879 100644 --- a/docs/deploying-airbyte/on-kubernetes.md +++ b/docs/deploying-airbyte/on-kubernetes.md @@ -11,31 +11,37 @@ To test locally, you can use one of the following: * [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) -To test on Google Kubernetes Engine(GKE) create a standard zonal cluster. +To test on Google Kubernetes Engine(GKE), create a standard zonal cluster. -To test on Amazon Elastic Kubernetes Service (Amazon EKS), install eksctl and then create a cluster. +To test on Amazon Elastic Kubernetes Service (Amazon EKS), install eksctl and create a cluster. -**_Note:_** Airbyte deployment is tested on GKE and EKS with version v1.19 and above. If you run into problems, reach out on the `#airbyte-help` channel in our Slack or create an issue on GitHub. +:::info +Airbyte deployment is tested on GKE and EKS with version v1.19 and above. If you run into problems, reach out on the `#airbyte-help` channel in our Slack or create an issue on GitHub. +::: ## Install and configure `kubectl ` Install `kubectl` and run the following command to configure it and connect to your cluster: -``` + +```bash kubectl use-context ``` -* To configure `kubectl` in `GKE`: - * Initialize the `gcloud` cli. - * To view cluster details, go to the `cluster` page in the Google Cloud Console and click `connect`. Run the following command to test cluster details: + +To configure `kubectl` in `GKE`: + +1. Initialize the `gcloud` cli. +2. To view cluster details, go to the `cluster` page in the Google Cloud Console and click `connect`. Run the following command to test cluster details: `gcloud container clusters get-credentials --zone --project `. - * To view contexts, run: `kubectl config get-contexts`. - * To access the cluster from `kubectl` run : `kubectl config use-context `. +3. To view contexts, run: `kubectl config get-contexts`. +4. To access the cluster from `kubectl` run : `kubectl config use-context `. + +To configure `kubectl` in `EKS`: -* To configure `kubectl` in `EKS`: - * [Configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project. - * Install [`eksctl`](https://eksctl.io/introduction/). - * To Make contexts available to `kubectl`, run `eksctl utils write-kubeconfig --cluster=` - * To view available contexts, run `kubectl config get-contexts`. - * To Access the cluster run `kubectl config use-context `. +1. [Configure AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) to connect to your project. +2. Install [`eksctl`](https://eksctl.io/introduction/). +3. To Make contexts available to `kubectl`, run `eksctl utils write-kubeconfig --cluster=` +4. To view available contexts, run `kubectl config get-contexts`. +5. To access the cluster, run `kubectl config use-context `. ## Configure Logs @@ -53,7 +59,7 @@ Airbyte supports logging to the `Minio` layer, `S3` bucket, and `GCS` bucket. To write to a custom location, update the following `.env` variable in the `kube/overlays/stable` directory (you will find this directory at the location you launched Airbyte) -``` +``` bash S3_LOG_BUCKET= AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= @@ -69,7 +75,7 @@ For the `S3` log location, create an S3 bucket with your AWS credentials. To write to a custom location, update the following `.env` variable in the `kube/overlays/stable` directory (you can find this directory at the location you launched Airbyte) -``` +``` bash S3_LOG_BUCKET= S3_LOG_BUCKET_REGION= # Set this to empty. @@ -78,12 +84,12 @@ S3_MINIO_ENDPOINT= S3_PATH_STYLE_ACCESS= ``` Replace the following variable in `.secrets` file in the `kube/overlays/stable` directory: -``` + +```bash AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= ``` - ### Configure the Custom GCS Log Location​ Create a GCS bucket and GCP credentials if you haven’t already. Make sure your GCS log bucket has read/write permission. @@ -91,13 +97,14 @@ Create a GCS bucket and GCP credentials if you haven’t already. Make sure you To configure the custom log location: Base encode the GCP JSON secret with the following command: -``` + +```bash # The output of this command will be a Base64 string. $ cat gcp.json | base64 ``` To populate the `gcs-log-creds` secrets with the Base64-encoded credential, take the encoded GCP JSON secret from the previous step and add it to `secret-gcs-log-creds.yaml` file as the value for `gcp.json` key. -``` +```bash apiVersion: v1 kind: Secret metadata: @@ -109,13 +116,13 @@ data: In the `kube/overlays/stable` directory, update the `GCS_LOG_BUCKET` with your GCS log bucket credentials: -``` +```bash GCS_LOG_BUCKET= ``` Modify `GOOGLE_APPLICATION_CREDENTIALS` to the path to `gcp.json` in the `.secrets` file at `kube/overlays/stable` directory. -``` +```bash # The path the GCS creds are written to. Unless you know what you are doing, use the below default value. GOOGLE_APPLICATION_CREDENTIALS=/secrets/gcs-log-creds/gcp.json @@ -126,14 +133,12 @@ GOOGLE_APPLICATION_CREDENTIALS=/secrets/gcs-log-creds/gcp.json The following commands will help you launch Airbyte: -``` +```bash git clone https://github.com/airbytehq/airbyte.git cd airbyte kubectl apply -k kube/overlays/stable ``` -It might take 2 to 5 minutes for the setup depending on your network and system configuration. - To check the pod status, run `kubectl get pods | grep airbyte`. If you are on Windows, run `kubectl get pods` to the list of pods. @@ -141,9 +146,9 @@ If you are on Windows, run `kubectl get pods` to the list of pods. Run `kubectl port-forward svc/airbyte-webapp-svc 8000:80` to allow access to the UI/API. Navigate to http://localhost:8000 in your browser to verify the deployment. -## Deploying Airbyte on Kubernetes in production +## Deploy Airbyte on Kubernetes in production -### Setting resource limits +### Set resource limits * Core container pods @@ -160,7 +165,7 @@ Navigate to http://localhost:8000 in your browser to verify the deployment. * To specify different volume sizes for the persistent volume backing Airbyte, modify `kube/resources/volume-*` files. -### Increasing job parallelism +### Increase job parallelism The ability to run parallel jobs like getting specs, checking connections, discovering schemas and performing syncs is limited by a few factors. `Airbyte-worker-pods` picks and executes the job. Increasing the number of workers will allow more jobs to be processed. @@ -181,13 +186,16 @@ Airbyte publishes logs every minute, so it’s normal to have minute-long log de Each log file is uncompressed and named `{yyyyMMddHH24mmss}_{podname}_{UUID}`. To view logs, navigate to the relevant folder and download the file for the time period you want. -### Using external databases +### Use external databases + You can configure a custom database instead of a simple `postgres` container in Kubernetes. This separate instance (AWS RDS or Google Cloud SQL) should be easier and safer to maintain than Postgres on your cluster. -## Customizing Airbytes Manifests +## Customize Airbytes Manifests + We use Kustomize to allow configuration for different environments. Our shared resources are in the `kube/resources` directory. We recommend defining overlays for each environment and creating your own overlay to customize your deployments. The overlay can live in your own version control system. An example of `kustomization.yaml` file: -``` + +```bash apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization @@ -215,7 +223,7 @@ You can view real-time logs in `kubectl logs deployments/airbyte-server` directo All logs can be accessed by viewing the scheduler logs. As for connector container logs, use Airbyte UI or Airbyte API to isolate them for a specific job attempt and for easier understanding. Connector pods launched by Airbyte will not relay logs directly to Kubernetes logging. You must access these logs through Airbyte. -### Resizing Volumes +### Resize Volumes To resize a volume, change the `.spec.resources.requests.storage` value. After re-applying, extend the mount(if that operation is supported for your mount type). For a production deployment, track the usage of volumes to ensure they don't run out of space. @@ -223,13 +231,13 @@ To resize a volume, change the `.spec.resources.requests.storage` value. After r To copy files, use the [`cp` command in kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#cp). -### Listing Files +### List Files To list files, run: `kubectl exec -it airbyte-server-6b5747df5c-bj4fx ls /tmp/workspace/8` -### Reading Files +### Read Files To read files, run: @@ -241,7 +249,7 @@ Running Airbyte on a GKE regional cluster requires enabling persistent regional Sample `volume-configs.yaml` file: -``` +```bash apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -256,6 +264,7 @@ spec: storage: 500Mi storageClassName: standard-rwo ``` + ## Troubleshooting If you encounter any issues, reach out to our community on [Slack](https://slack.airbyte.com/). diff --git a/docs/deploying-airbyte/on-oci-vm.md b/docs/deploying-airbyte/on-oci-vm.md index f3ccd6cb0605..8b431d4d0364 100644 --- a/docs/deploying-airbyte/on-oci-vm.md +++ b/docs/deploying-airbyte/on-oci-vm.md @@ -1,6 +1,6 @@ # Deploy Airbyte on Oracle Cloud -This page guides you through deploying Airbyte Open Source on an Oracle Cloud Infrastructure (OCI) Virtual Machine (VM) Instance. +This page guides you through deploying Airbyte Open Source on an [Oracle Cloud Infrastructure (OCI) Virtual Machine (VM) Instance](https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/launchinginstance.htm#Creating_an_Instance). :::info @@ -12,9 +12,9 @@ These instructions have been tested on an Oracle Linux 7 instance. To deploy Airbyte Open Source on Oracle cloud: -* Create an [OCI VM compute instance](https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/launchinginstance.htm#Creating_an_Instance). -* Allowlist a port for a CIDR range in the [security list of your OCI VM Instance subnet](https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/securitylists.htm). -* Connect to the instance using a [bastion port forwarding session](https://docs.oracle.com/en-us/iaas/Content/Bastion/Tasks/connectingtosessions.htm#connect-port-forwarding). +* Create an [OCI VM compute instance](https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/launchinginstance.htm#Creating_an_Instance) +* Allowlist a port for a CIDR range in the [security list of your OCI VM Instance subnet](https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/securitylists.htm) +* Connect to the instance using a [bastion port forwarding session](https://docs.oracle.com/en-us/iaas/Content/Bastion/Tasks/connectingtosessions.htm#connect-port-forwarding) :::caution @@ -26,7 +26,7 @@ For security reasons, we strongly recommend not having a Public IP for the Insta Install Docker and Docker Compose on the VM: -### 1. Install Docker +1. Install Docker In the terminal connected to your OCI Instance for Airbyte, run the following commands: @@ -40,7 +40,7 @@ sudo service docker start sudo usermod -a -G docker $USER ``` -### 2. Install Docker Compose +2. Install Docker Compose In the terminal connected to your OCI Instance for Airbyte, run the following commands: @@ -89,12 +89,6 @@ Alternatively, you can get Airbyte running on your OCI VM instance using a diffe 2. On your browser, visit port 8000 [port 8000](http://localhost:8000/) -:::info - -Please note that Airbyte currently does not support SSL/TLS certificates. - -::: - ## Troubleshooting If you encounter any issues, reach out to our community on [Slack](https://slack.airbyte.com/). diff --git a/docs/deploying-airbyte/on-plural.md b/docs/deploying-airbyte/on-plural.md index cad6ee0ccd7a..88a61fe12fe1 100644 --- a/docs/deploying-airbyte/on-plural.md +++ b/docs/deploying-airbyte/on-plural.md @@ -1,30 +1,25 @@ # Deploy Airbyte on Plural -## Overview - -[Plural](https://www.plural.sh/) is a free, open-source, unified application deployment platform that makes it easy to run open-source software on Kubernetes. It aims to make applications as portable as possible, without sacrificing the ability for the users to own the applications they desire to use. - -## Getting Started - :::tip If you'd prefer to follow along with a video, check out the Plural Airbyte deployment guide video [here](https://youtu.be/suvTJyJ6PzI) ::: -First, create an account on [https://app.plural.sh](https://app.plural.sh). This is simply to track your installations and allow for the delivery of automated upgrades, you will not be asked to provide any infrastructure credentials or sensitive information. +## Getting started -Then, install the Plural CLI by following steps 1, 2, and 3 of the instructions [here](https://docs.plural.sh/getting-started). Through this, you will also configure your cloud provider and the domain name under which your application will be deployed to. +1. Create an account on [https://app.plural.sh](https://app.plural.sh). +2. Install the Plural CLI by following steps 1, 2, and 3 of the instructions [here](https://docs.plural.sh/getting-started). Through this, you will also configure your cloud provider and the domain name under which your application will be deployed to. We now need a Git repository to store your Plural configuration in. This will also contain the Helm and Terraform files that Plural will autogenerate for you. -You now have two options: +You have two options: - Run `plural init` in any directory to let Plural initiate an OAuth workflow to create a Git repo for you. -- Create a Git repo manually, clone it down and run `plural init` inside it. +- Create a Git repo manually, clone it, and run `plural init` inside it. -Running `plural init` will configure your installation and cloud provider for the repo. You're now ready to install Airbyte on your Plural repo! +Running `plural init` will configure your installation and cloud provider for the repo. ## Installing Airbyte -To install Airbyte on your Plural repo, simply run: +To install Airbyte on your Plural repo, run: ```bash plural bundle install airbyte $CONSOLE_BUNDLE_NAME @@ -42,9 +37,9 @@ For example, this is what it looks like for AWS: plural bundle install airbyte airbyte-aws ``` -Plural's Airbyte distribution currently has support for AWS, GCP and Azure set up and ready to go, so feel free to pick whichever best fits your infrastructure. +Plural's Airbyte distribution currently has support for AWS, GCP and Azure. Select the Cloud that best fits your infrastructure. -The CLI will prompt you to choose whether or not you want to use Plural OIDC. [OIDC](https://openid.net/connect/) allows you to login to the applications you host on Plural with your login to [app.plural.sh](https://app.plural.sh), acting as an SSO provider. +The CLI prompts you to choose whether or not you want to use Plural OIDC. [OIDC](https://openid.net/connect/) allows you to login to the applications you host on Plural with your login to [app.plural.sh](https://app.plural.sh), acting as an SSO provider. After this, run: @@ -55,7 +50,7 @@ plural deploy --commit "deploying airbyte" ## Adding the Plural Console -To make management of your installation as simple as possible, we recommend installing the Plural Console. The console provides tools to manage resource scaling, receiving automated upgrades, dashboards tailored to your Airbyte installation, and log aggregation. This can be done using the exact same process as above: +To make management of your installation as simple as possible, we recommend installing the Plural Console. The console provides tools to manage resource scaling, receiving automated upgrades, dashboards tailored to your Airbyte installation, and log aggregation. Run: ```bash plural bundle install console console-aws @@ -65,11 +60,11 @@ plural deploy --commit "deploying the console too" ## Accessing your Airbyte Installation -Now, just head over to `airbyte.SUBDOMAIN_NAME.onplural.sh` to access the Airbyte UI. +Navigate to `airbyte.SUBDOMAIN_NAME.onplural.sh` to access the Airbyte UI. ## Accessing your Console Installation -To monitor and manage your Airbyte installation, head over to the Plural Console at `console.YOUR_ORGANIZATION.onplural.sh` (or whichever subdomain you chose). +To monitor and manage your Airbyte installation, navigate to the Plural Console at `console.YOUR_ORGANIZATION.onplural.sh` (or whichever subdomain you chose). ## Advanced Use Cases @@ -85,10 +80,10 @@ https://username:password@airbytedomain ## Troubleshooting -If you have any issues with installing Airbyte on Plural, feel free to join our [Discord Community](https://discord.gg/bEBAMXV64s) and we can help you out. +If you have any issues with installing Airbyte on Plural, join Plural's [Discord Community](https://discord.gg/bEBAMXV64s). -If you'd like to request any new features for our Airbyte installation, feel free to open an issue or PR at https://github.com/pluralsh/plural-artifacts. +If you'd like to request any new features for our Airbyte installation, open an issue or PR at https://github.com/pluralsh/plural-artifacts. ## Further Reading -To learn more about what you can do with Plural and more advanced uses of the platform, feel free to dive deeper into our docs [here.](https://docs.plural.sh) +To learn more about Plural, refer to the [Plural documentation](https://docs.plural.sh) diff --git a/docs/deploying-airbyte/on-restack.md b/docs/deploying-airbyte/on-restack.md index 3e219e97bafd..fbb3f11f26d6 100644 --- a/docs/deploying-airbyte/on-restack.md +++ b/docs/deploying-airbyte/on-restack.md @@ -14,7 +14,7 @@ To deploy Airbyte with Restack: ## Sign up for a Restack account To Sign up for a Restack account, visit [www.restack.io/signup](https://www.restack.io/signup). You can sign up with your corporate email address or your GitHub profile. You do not need a credit card to sign up. -If you already have an account, go ahead and login to Restack at [www.restack.io/login](https://www.restack.io/login). +If you already have an account, login to Restack at [www.restack.io/login](https://www.restack.io/login). ## Add AWS credentials with AdministratorAccess @@ -22,43 +22,45 @@ To deploy Airbyte in your own AWS infrastructure with Restack, you will need to Make sure that this account has *AdministratorAccess*. This is how Restack can ensure an end-to-end cluster creation and cluster management process. -1. Navigate to *Clusters* in the left-hand navigation menu. -2. Select the *Credentials* tab. -3. Click *Add credential*. +1. Navigate to **Clusters** in the left-hand navigation menu. +2. Select the **Credentials** tab. +3. Click **Add credential**. 4. Give a suitable title to your credentials for managing them later. -5. Enter your *AWS Access Key ID* and *AWS Secret Access key*. -6. Click *Add credential*. - ->[How to get your AWS Access key ID and AWS Secret Access Key](https://docs.aws.amazon.com/accounts/latest/reference/root-user-access-key.html) +5. Enter your [**AWS Access Key ID** and **AWS Secret Access key**.](https://docs.aws.amazon.com/accounts/latest/reference/root-user-access-key.html) +6. Click **Add credential**. ## One-click cluster creation with Restack :::tip -Why do I need a cluster?
Running your application on a Kubernetes cluster lets you deploy, scale and monitor the application reliably. ::: -Once you have added your credentials, -1. Navigate to the *Clusters* tab on the same page and click on *Create cluster*. + +Once you have added your credentials: +1. Navigate to the **Clusters** tab on the same page and click on **Create cluster**. 2. Give a suitable name to your cluster. 3. Select the region you want to deploy the cluster in. 4. Select the AWS credentials you added in the previous step. The cluster creation process will start automatically. Once the cluster is ready, you will get an email on the email id connected with your account. -
Creating a cluster is a one-time process. From here you can add other open source tools or multiple instances of Airbyte in the same cluster. + +Creating a cluster is a one-time process. From here you can add other open source tools or multiple instances of Airbyte in the same cluster. Any application you deploy in your cluster will be accessible via a free **restack domain**. -
Contact the Restack team via chat to set a custom domain for your Airbyte instances. +Contact the Restack team via chat to set a custom domain for your Airbyte instances. ## Deploy Airbyte on Restack -1. Click *Add application* from the Cluster description or go to the Applications tab in the left hand side navigation. -2. Click *Airbyte*. + +1. Click **Add application** from the Cluster description or go to the Applications tab in the left hand side navigation. +2. Click **Airbyte**. 3. Select the cluster you have already provisioned. -4. Click *Add application*. +4. Click **Add application**. ## Start using Airbyte + Airbyte will be deployed on your cluster and you can access it using the link under the *URL* tab. You can also check the workloads and volumes that are deployed within Airbyte. ## Deploy multiple instances of Airbyte + Restack makes it easier to deploy multiple instances of Airbyte on the same or multiple clusters.
So you can test the latest version before upgrading or have a dedicated instance for development and for production. \ No newline at end of file diff --git a/docs/integrations/destinations/csv.md b/docs/integrations/destinations/csv.md index fed3d3242dc8..21fc48c5a35c 100644 --- a/docs/integrations/destinations/csv.md +++ b/docs/integrations/destinations/csv.md @@ -1,4 +1,4 @@ -# CSV Destination +# CSV The Airbyte Destination for CSV files. diff --git a/docs/integrations/destinations/pubsub.md b/docs/integrations/destinations/pubsub.md index e8b8150a6028..2888af138c0f 100644 --- a/docs/integrations/destinations/pubsub.md +++ b/docs/integrations/destinations/pubsub.md @@ -4,7 +4,7 @@ description: >- Provider. --- -# Google PubSub +# PubSub ## Overview diff --git a/docs/integrations/sources/hellobaton.md b/docs/integrations/sources/baton.md similarity index 100% rename from docs/integrations/sources/hellobaton.md rename to docs/integrations/sources/baton.md diff --git a/docs/integrations/sources/faker.md b/docs/integrations/sources/faker.md index ef69a14377cb..5671be77cdd8 100644 --- a/docs/integrations/sources/faker.md +++ b/docs/integrations/sources/faker.md @@ -1,4 +1,4 @@ -# Sample Data (Faker) +# Faker ## Sync overview diff --git a/docs/integrations/sources/http-request.md b/docs/integrations/sources/http-request.md index 716aedf4b191..7d76e2a20c3b 100644 --- a/docs/integrations/sources/http-request.md +++ b/docs/integrations/sources/http-request.md @@ -1,4 +1,4 @@ -# HTTP Request (Graveyarded) +# HTTP Request :::caution diff --git a/docs/integrations/sources/mailerlite.md b/docs/integrations/sources/mailerlite.md index 2e192db53de6..d5e87337bf3d 100644 --- a/docs/integrations/sources/mailerlite.md +++ b/docs/integrations/sources/mailerlite.md @@ -1,4 +1,4 @@ -# ActiveCampaign +# MailerLite ## Sync overview diff --git a/docs/integrations/sources/omnisend.md b/docs/integrations/sources/omnisend.md index 84082611fd9f..613b9c54b8a2 100644 --- a/docs/integrations/sources/omnisend.md +++ b/docs/integrations/sources/omnisend.md @@ -1,4 +1,4 @@ -# ConvertKit +# Omnisend ## Sync overview diff --git a/docs/integrations/sources/s3.md b/docs/integrations/sources/s3.md index 5b039fdd4b9e..23bb3f120d1b 100644 --- a/docs/integrations/sources/s3.md +++ b/docs/integrations/sources/s3.md @@ -1,4 +1,4 @@ -# Amazon S3 +# S3 This page contains the setup guide and reference information for the Amazon S3 source connector. diff --git a/docs/integrations/sources/xero.md b/docs/integrations/sources/xero.md index 7fd6946db95e..4d9b3279d6d8 100644 --- a/docs/integrations/sources/xero.md +++ b/docs/integrations/sources/xero.md @@ -1,4 +1,4 @@ -# Airbyte Source Connector for Xero +# Xero This is a setup guide for the Xero source connector which ingests data from the Accounting API. diff --git a/docs/operator-guides/security.md b/docs/operator-guides/security.md index baa70aed94d3..2eb0ea2b8d01 100644 --- a/docs/operator-guides/security.md +++ b/docs/operator-guides/security.md @@ -86,10 +86,43 @@ Only certain Airbyte staff can access Airbyte infrastructure and technical logs ### Network security -- You may need to allowlist one of our IP addresses to enable access to Airbyte: - - 34.106.109.131 - - 34.106.196.165 - - 34.106.60.246 +Depending on your [data residency](https://docs.airbyte.com/cloud/managing-airbyte-cloud#choose-your-default-data-residency) location, you may need to allowlist the following IP addresses to enable access to Airbyte: + +#### United States and Airbyte Default + +GCP region: us-west3 +* 34.106.109.131 +* 34.106.196.165 +* 34.106.60.246 +* 34.106.229.69 +* 34.106.127.139 +* 34.106.218.58 +* 34.106.115.240 +* 34.106.225.141 + +#### European Union + +:::note + +Some workflows still run in the US, even when the data residency is in the EU. If you use the EU as a data residency, you must allowlist the following IP addresses from both GCP us-west3 and AWS eu-west-3. + +::: + +GCP region: us-west3 +* 34.106.109.131 +* 34.106.196.165 +* 34.106.60.246 +* 34.106.229.69 +* 34.106.127.139 +* 34.106.218.58 +* 34.106.115.240 +* 34.106.225.141 + +AWS region: eu-west-3 +* 13.37.4.46 +* 13.37.142.60 +* 35.181.124.238 + ### Credential management diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index b04abbd14d7b..ea6c76746f7a 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -31,12 +31,11 @@ module.exports = { { type: 'category', label: 'Connector Catalog', + link: { + type: 'doc', + id: 'integrations/README', + }, items: [ - { - type: 'doc', - label: 'Overview', - id: 'integrations/README', - }, { type: 'category', label: 'Sources',