Add dependencies to docker-compose (#19321)

* Revert "Revert "Add dependencies to docker-compose (#19257)" (#19306)"

This reverts commit b16f28f.

* Bump docker-compose version

* test

* Revert "test"

This reverts commit c3b28c9.

* Bump docker compose version to 3.8

* test

* test

* Use a newer docker for platform build

* test

* Use docker compose v2

* Update docker for frontend e2e

* Use docker compose v2 for e2e tests

* Update tools to use docker compose v2

* Update more docker compose references to v2

* Update docker compose usage in docs

* Update deploying airbyte docs

* Fix permission

* Case insensistive check

* Add extra mention for docker compose in upgrading airbyte

* Improve upgrade docker note

.github/actions/build-and-push-branch/action.yml

@@ -13,6 +13,16 @@ inputs:
 runs:
   using: "composite"
   steps:
+    - name: Delete default old docker and replace it with a new one
+      shell: bash
+      run: |
+        sudo rm /var/lib/dpkg/lock
+        sudo rm /var/lib/dpkg/lock-frontend
+        sudo apt-get remove docker.io || sudo apt-get remove docker
+        curl -fsSL https://get.docker.com | bash -
+        sudo rm -f /var/lib/dpkg/lock
+        sudo rm -f /var/lib/dpkg/lock-frontend
+
     - name: Build
       id: build
       uses: ./.github/actions/build-branch
@@ -29,5 +39,5 @@ runs:
       run: |
         GIT_REVISION=$(git rev-parse HEAD)
         [ [ -z "$GIT_REVISION" ] ] && echo "Couldn't get the git revision..." && exit 1
-        VERSION=${{ steps.build.outputs.branch_version_tag }} GIT_REVISION=$GIT_REVISION docker-compose -f docker-compose-cloud.buildx.yaml push
+        VERSION=${{ steps.build.outputs.branch_version_tag }} GIT_REVISION=$GIT_REVISION docker compose -f docker-compose-cloud.buildx.yaml push
       shell: bash

.github/workflows/commands-for-testing-tool.yml

@@ -110,7 +110,7 @@ jobs:
           mkdir secrets
           mkdir result
       - name: Run Airbyte
-        run: docker-compose up -d
+        run: docker compose up -d
       - name: Connect to secret manager
         uses: jsdaniell/create-json@1.1.2
         with:

.github/workflows/gradle.yml

@@ -467,6 +467,16 @@ jobs:
         with:
           python-version: "3.9"
 
+      - name: Delete default old docker and replace it with a new one
+        shell: bash
+        run: |
+          sudo rm /var/lib/dpkg/lock
+          sudo rm /var/lib/dpkg/lock-frontend
+          sudo apt-get remove docker.io || sudo apt-get remove docker
+          curl -fsSL https://get.docker.com | bash -
+          sudo rm -f /var/lib/dpkg/lock
+          sudo rm -f /var/lib/dpkg/lock-frontend
+
       - name: Set up CI Gradle Properties
         run: |
           mkdir -p ~/.gradle/
@@ -592,6 +602,16 @@ jobs:
         with:
           node-version: "lts/*"
 
+      - name: Delete default old docker and replace it with a new one
+        shell: bash
+        run: |
+          sudo rm /var/lib/dpkg/lock
+          sudo rm /var/lib/dpkg/lock-frontend
+          sudo apt-get remove docker.io || sudo apt-get remove docker
+          curl -fsSL https://get.docker.com | bash -
+          sudo rm -f /var/lib/dpkg/lock
+          sudo rm -f /var/lib/dpkg/lock-frontend
+
       - name: Set up CI Gradle Properties
         run: |
           mkdir -p ~/.gradle/

README.md

@@ -49,7 +49,7 @@ You can run Airbyte locally with Docker.
 ```bash
 git clone https://github.com/airbytehq/airbyte.git
 cd airbyte
-docker-compose up
+docker compose up
 ```
 
 Login to the web app at [http://localhost:8000](http://localhost:8000) by entering the default credentials found in your .env file.

airbyte-integrations/connectors/source-zuora/README.md

@@ -130,7 +130,7 @@ To run acceptance and custom integration tests:
 # Run Airbyte using docker-compose
 Under Airbyte's root directory:
 ```
-docker-compose up -d
+docker compose up -d
 ```
 Open the web-page of Airbyte:
 ```

airbyte-webapp-e2e-tests/README.md

@@ -10,7 +10,7 @@ Steps:
 2) Build the OSS backend for the current commit with `SUB_BUILD=PLATFORM ../gradlew clean build`.
 3) Create the test database: `npm run createdbsource` and `npm run createdbdestination`.
 4) When running the connector builder tests, start the dummy API server: `npm run createdummyapi`
-5) Start the OSS backend: `BASIC_AUTH_USERNAME="" BASIC_AUTH_PASSWORD="" VERSION=dev docker-compose --file ../docker-compose.yaml up`. If you want, follow this with `docker-compose stop webapp` to turn off the dockerized frontend build; interactive cypress sessions don't use it.
+5) Start the OSS backend: `BASIC_AUTH_USERNAME="" BASIC_AUTH_PASSWORD="" VERSION=dev docker compose --file ../docker-compose.yaml up`. If you want, follow this with `docker compose stop webapp` to turn off the dockerized frontend build; interactive cypress sessions don't use it.
 6) The following two commands will start a separate long-running server, so open another terminal window. In it, `cd` into the `airbyte-webapp/` directory.
 7) If you have not already done so, run `npm install` to install the frontend app's dependencies.
 8) Start the frontend development server with `npm start`.
@@ -26,7 +26,7 @@ Steps:
 2) Build the OSS backend for the current commit with `SUB_BUILD=PLATFORM ../gradlew clean build`.
 3) Create the test database: `npm run createdbsource` and `npm run createdbdestination`.
 4) When running the connector builder tests, start the dummy API server: `npm run createdummyapi`
-5) Start the OSS backend: `BASIC_AUTH_USERNAME="" BASIC_AUTH_PASSWORD="" VERSION=dev docker-compose --file ../docker-compose.yaml up`.
+5) Start the OSS backend: `BASIC_AUTH_USERNAME="" BASIC_AUTH_PASSWORD="" VERSION=dev docker compose --file ../docker-compose.yaml up`.
 6) Start the cypress test run with `npm run cypress:ci` or `npm run cypress:ci:record`.
 
 ## Test setup
@@ -35,4 +35,4 @@ When the tests are run as described above, the platform under test is started vi
 
 For testing the connector builder UI, a dummy api server based on a node script is started (`createdummyapi`). It is providing a simple http API with bearer authentication returning a few records of hardcoded data. By running it in the internal airbyte network, the connector builder server can access it under its container name.
 
-The tests in here are instrumenting a Chrome instance to test the full functionality of Airbyte from the frontend, so other components of the platform (scheduler, worker, connector builder server) are also tested in a rudimentary way.
+The tests in here are instrumenting a Chrome instance to test the full functionality of Airbyte from the frontend, so other components of the platform (scheduler, worker, connector builder server) are also tested in a rudimentary way.

docker-compose-cloud.buildx.yaml

@@ -1,7 +1,7 @@
 # Defines the minimum set of images needed to run Cloud.
 # Used to push OSS images that Cloud depends on.
 
-version: "3.7"
+version: "3.8"
 
 services:
   bootloader:

docker-compose.acceptance-test.yaml

@@ -1,7 +1,7 @@
 # Adds ports to the db and access to the temporal UI for debugging purposes.
 # Expected to be used like this:
-# VERSION=dev docker-compose -f docker-compose.yaml -f docker-compose.debug.yaml up
-version: "3.7"
+# VERSION=dev docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
+version: "3.8"
 x-logging: &default-logging
   options:
     max-size: "100m"

docker-compose.build.yaml

@@ -1,4 +1,4 @@
-version: "3.7"
+version: "3.8"
 
 services:
   init:

docker-compose.datadog.yaml

@@ -5,7 +5,7 @@
 # 1. create an API Key in datadog
 # 2. wget -O dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'
 # 3. DD_API_KEY=[datadog api key] VERSION=dev docker-compose -f docker-compose.yaml -f docker-compose.datadog.yaml up -d
-version: "3.7"
+version: "3.8"
 
 x-datadog: &datadogged
   volumes:

docker-compose.debug.yaml

@@ -1,7 +1,7 @@
 # Adds ports to the db and access to the temporal UI for debugging purposes.
 # Expected to be used like this:
-# VERSION=dev docker-compose -f docker-compose.yaml -f docker-compose.debug.yaml up
-version: "3.7"
+# VERSION=dev docker compose -f docker-compose.yaml -f docker-compose.debug.yaml up
+version: "3.8"
 x-logging: &default-logging
   options:
     max-size: "100m"

docker-compose.yaml

@@ -1,4 +1,4 @@
-version: "3.7"
+version: "3.8"
 #https://github.com/compose-spec/compose-spec/blob/master/spec.md#using-extensions-as-fragments
 x-logging: &default-logging
   options:
@@ -30,6 +30,9 @@ services:
       - LOG_LEVEL=${LOG_LEVEL}
     networks:
       - airbyte_internal
+    depends_on:
+      init:
+        condition: service_completed_successfully
   db:
     image: airbyte/db:${VERSION}
     logging: *default-logging
@@ -114,6 +117,9 @@ services:
       - 9000
     networks:
       - airbyte_internal
+    depends_on:
+      bootloader:
+        condition: service_completed_successfully
   server:
     image: airbyte/server:${VERSION}
     logging: *default-logging
@@ -156,6 +162,9 @@ services:
       - ${LOCAL_ROOT}:${LOCAL_ROOT}
     networks:
       - airbyte_internal
+    depends_on:
+      bootloader:
+        condition: service_completed_successfully
   webapp:
     image: airbyte/webapp:${VERSION}
     logging: *default-logging
@@ -175,6 +184,9 @@ services:
       - TRACKING_STRATEGY=${TRACKING_STRATEGY}
     networks:
       - airbyte_internal
+    depends_on:
+      bootloader:
+        condition: service_completed_successfully
   airbyte-temporal:
     image: airbyte/temporal:${VERSION}
     logging: *default-logging
@@ -214,6 +226,9 @@ services:
       - workspace:${WORKSPACE_ROOT}
     networks:
       - airbyte_internal
+    depends_on:
+      bootloader:
+        condition: service_completed_successfully
   airbyte-connector-builder-server:
     image: airbyte/connector-builder-server:${VERSION}
     logging: *default-logging
@@ -225,6 +240,9 @@ services:
       - AIRBYTE_VERSION=${VERSION}
     networks:
       - airbyte_internal
+    depends_on:
+      bootloader:
+        condition: service_completed_successfully
   airbyte-proxy:
     image: airbyte/proxy:${VERSION}
     container_name: airbyte-proxy

docs/connector-development/tutorials/build-a-connector-the-hard-way.md

@@ -1078,7 +1078,7 @@ $  docker images | head
 If the Airbyte server isn't already running, start it by running **from the Airbyte repository root**:
 
 ```bash
-docker-compose up
+docker compose up
 ```
 
 When Airbyte server is done starting up, it prints the following banner in the log output \(it can take 10-20 seconds for the server to start\):
@@ -1098,7 +1098,7 @@ airbyte-server      | Version: dev
 airbyte-server      | 
 ```
 
-After you see the above banner printed out in the terminal window where you are running `docker-compose up`, visit [http://localhost:8000](http://localhost:8000) in your browser and log in with the default credentials: username `airbyte` and password `password`. 
+After you see the above banner printed out in the terminal window where you are running `docker compose up`, visit [http://localhost:8000](http://localhost:8000) in your browser and log in with the default credentials: username `airbyte` and password `password`.
 
 If this is the first time using the Airbyte UI, then you will be prompted to go through a first-time wizard. To skip it, click the "Skip Onboarding" button.
 

docs/contributing-to-airbyte/developing-locally.md

@@ -73,7 +73,7 @@ These instructions explain how to run a version of Airbyte that you are developi
 
 ```bash
 SUB_BUILD=PLATFORM ./gradlew build
-VERSION=dev docker-compose up
+VERSION=dev docker compose up
 ```
 
 The build will take a few minutes. Once it completes, Airbyte compiled at current git revision will be running in `dev` mode in your environment.
@@ -88,7 +88,7 @@ These instructions explain how to run a version of an Airbyte connector that you
 
 ```bash
 SUB_BUILD=PLATFORM ./gradlew build
-VERSION=dev docker-compose up
+VERSION=dev docker compose up
 ```
 
 - Then, build the connector image:
@@ -169,7 +169,7 @@ Note: If you are contributing a Python file without imports or function definiti
 - Spin up Airbyte locally so the UI can make requests against the local API.
 
 ```bash
-BASIC_AUTH_USERNAME="" BASIC_AUTH_PASSWORD="" docker-compose up
+BASIC_AUTH_USERNAME="" BASIC_AUTH_PASSWORD="" docker compose up
 ```
 
 Note: [basic auth](https://docs.airbyte.com/operator-guides/security#network-security) must be disabled by setting `BASIC_AUTH_USERNAME` and `BASIC_AUTH_PASSWORD` to empty values, otherwise requests from the development server will fail against the local API.
@@ -192,8 +192,8 @@ The Configuration API caches connector specifications. This is done to avoid nee
 2. Restart the server by running the following commands:
 
 ```bash
-VERSION=dev docker-compose down -v
-VERSION=dev docker-compose up
+VERSION=dev docker compose down -v
+VERSION=dev docker compose up
 ```
 
 ### Resetting the Airbyte developer environment
@@ -203,7 +203,7 @@ Sometimes you'll want to reset the data in your local environment. One common ca
 - Delete the datastore volumes in docker
 
   ```bash
-    VERSION=dev docker-compose down -v
+    VERSION=dev docker compose down -v
   ```
 
 - Remove the data on disk
@@ -217,7 +217,7 @@ Sometimes you'll want to reset the data in your local environment. One common ca
 
   ```bash
    SUB_BUILD=PLATFORM ./gradlew clean build
-   VERSION=dev docker-compose up -V
+   VERSION=dev docker compose up -V
   ```
 
 While not as common as the above steps, you may also get into a position where want to erase all of the data on your local docker server. This is useful if you've been modifying image tags while developing.
@@ -257,4 +257,4 @@ env JAVA_HOME=/usr/lib/jvm/java-14-openjdk ./gradlew  :airbyte-integrations:conn
 
 ### Inspecting the messages passed between connectors
 
-You can enable `LOG_CONNECTOR_MESSAGES=true` to log the messages the Airbyte platform receives from the source and destination when debugging locally. e.g. `LOG_CONNECTOR_MESSAGES=true VERSION=dev docker-compose up`
+You can enable `LOG_CONNECTOR_MESSAGES=true` to log the messages the Airbyte platform receives from the source and destination when debugging locally. e.g. `LOG_CONNECTOR_MESSAGES=true VERSION=dev docker compose up`

docs/deploying-airbyte/local-deployment.md

@@ -13,7 +13,7 @@ These instructions have been tested on MacOS, Windows 10 and Ubuntu 20.04.
 ```bash
 git clone https://github.com/airbytehq/airbyte.git
 cd airbyte
-docker-compose up
+docker compose up
 ```
 
 - In your browser, just visit [http://localhost:8000](http://localhost:8000)
@@ -53,7 +53,7 @@ Make sure to select the options:
 ```bash
 git clone https://github.com/airbytehq/airbyte.git
 cd airbyte
-docker-compose up
+docker compose up
 ```
 
 - In your browser, just visit [http://localhost:8000](http://localhost:8000)

docs/deploying-airbyte/on-aws-ec2.md

@@ -38,9 +38,8 @@ sudo usermod -a -G docker $USER
 3. To install `docker-compose`, run the following command in your ssh session on the instance terminal:
 
 ``` bash
-sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
-sudo chmod +x /usr/local/bin/docker-compose
-docker-compose --version
+sudo yum install -y docker-compose-plugin
+docker compose version
 ```
 
 4. To close the SSH connection, run the following command in your SSH session on the instance terminal:
@@ -64,7 +63,7 @@ ssh -i $SSH_KEY ec2-user@$INSTANCE_IP
 ``` bash
 mkdir airbyte && cd airbyte
 wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
-docker-compose up -d # run the Docker container
+docker compose up -d # run the Docker container
 ```
 
 ## Connect to Airbyte

docs/deploying-airbyte/on-azure-vm-cloud-shell.md

@@ -36,14 +36,13 @@ Install Docker and Docker Compose in the VM:
 4. To install Docker Compose, run the following command:
 
     ```bash
-    sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
-    sudo chmod +x /usr/local/bin/docker-compose
+    sudo apt-get install docker-compose-plugin -y
     ```
 
 5. Check Docker Compose version:
 
     ```bash
-    docker-compose --version
+    docker compose version
     ```
 
 6. Close the SSH connection to ensure that the group modification is considered:
@@ -84,7 +83,7 @@ Download Airbyte and deploy it in the VM using Docker Compose:
 4. To start Airbyte, run the following command:
 
     ```bash
-    sudo docker-compose up -d
+    sudo docker compose up -d
     ```
 
 ## Connect to Airbyte

docs/deploying-airbyte/on-digitalocean-droplet.md

@@ -31,9 +31,8 @@ To deploy Airbyte Open Source on DigitalOcean:
 4. To install Docker-Compose, run the following command:
 
 ```bash
-    sudo wget https://github.com/docker/compose/releases/download/1.26.2/docker-compose-$(uname -s)-$(uname -m) -O /usr/local/bin/docker-compose
-    sudo chmod +x /usr/local/bin/docker-compose
-    docker-compose --version
+    sudo apt install docker-compose-plugin
+    docker compose version
 ```
 
 ## Install Airbyte
@@ -45,7 +44,7 @@ To install and start Airbyte :
 ```bash
   mkdir airbyte && cd airbyte
   wget https://raw.githubusercontent.com/airbytehq/airbyte/master/{.env,docker-compose.yaml}
-  docker-compose up -d
+  docker compose up -d
 ```
 
 2. Verify the connection by visiting [http://localhost:8000](http://localhost:8000) in your browser.