From 92071cba473aaa30523556e658ae59a9691c6a12 Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Mon, 22 Jul 2024 09:40:50 -0300 Subject: [PATCH 1/3] Update server 4.3+ docs on DLC requiring S3 bucket versioning --- .../v4.3/installation/phase-1-prerequisites.adoc | 12 +++++++++++- .../server/v4.3/installation/upgrade-server.adoc | 1 + .../v4.4/installation/phase-1-prerequisites.adoc | 12 +++++++++++- .../v4.5/installation/phase-1-prerequisites.adoc | 13 +++++++++++-- 4 files changed, 34 insertions(+), 4 deletions(-) diff --git a/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc index 8de87b524a9..58e827f55ce 100644 --- a/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc @@ -495,9 +495,19 @@ aws s3api create-bucket \ --create-bucket-configuration LocationConstraint= ---- +==== b. Enable bucket versioning + +To use the https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to do that on the bucket created in the previous step: +[source,shell] +---- +aws s3api put-bucket-versioning \ + --bucket \ + --region \ + --versioning-configuration Status=Enabled +---- [#set-up-authentication-aws] -==== b. Set up authentication +==== c. Set up authentication There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. diff --git a/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc b/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc index c45ec742c4c..d280f565716 100644 --- a/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc +++ b/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc @@ -36,6 +36,7 @@ Please note the following significant changes in server v4.3. * Old DLC volumes will not be carried over and need to be wiped manually * Projects that stop using DLC will not delete final DLC cache without manual intervention * DLC now runs through S3 and GCS instead of SSD Volumes +* If AWS S3 is used for https://circleci.com/docs/server/v4.3/installation/phase-1-prerequisites/#s3-storage[object storage], DLC now requires https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html[versioning to be enabled] on the bucket. [#machine] === Machine jobs diff --git a/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc index dbebef805f8..aba7efe7bc6 100644 --- a/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc @@ -508,9 +508,19 @@ aws s3api create-bucket \ --create-bucket-configuration LocationConstraint= ---- +==== b. Enable bucket versioning + +To use the https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to do that on the bucket created in the previous step: +[source,shell] +---- +aws s3api put-bucket-versioning \ + --bucket \ + --region \ + --versioning-configuration Status=Enabled +---- [#set-up-authentication-aws] -==== b. Set up authentication +==== c. Set up authentication There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. diff --git a/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc index f863bebbb56..c67433fa882 100644 --- a/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc @@ -511,10 +511,19 @@ aws s3api create-bucket \ --create-bucket-configuration LocationConstraint= ---- +==== b. Enable bucket versioning -[#set-up-authentication-aws] -==== b. Set up authentication +To use the https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to do that on the bucket created in the previous step: +[source,shell] +---- +aws s3api put-bucket-versioning \ + --bucket \ + --region \ + --versioning-configuration Status=Enabled +---- +[#set-up-authentication-aws] +==== c. Set up authentication There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. [.tab.authentication.IRSA] From 3c587d54a9072bab82eb1aa5f8fbebc7901024c6 Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Mon, 22 Jul 2024 09:58:09 -0300 Subject: [PATCH 2/3] Address linting errors --- .../v4.3/installation/phase-1-prerequisites.adoc | 12 ++++++------ .../server/v4.3/installation/upgrade-server.adoc | 10 +++++----- .../v4.4/installation/phase-1-prerequisites.adoc | 10 +++++----- .../v4.5/installation/phase-1-prerequisites.adoc | 4 ++-- 4 files changed, 18 insertions(+), 18 deletions(-) diff --git a/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc index 58e827f55ce..eb3c836b0a5 100644 --- a/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc @@ -272,7 +272,7 @@ CAUTION: If GitHub Enterprise and CircleCI server are not on the same domain, th Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both GitHub.com and GitHub Enterprise. -. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and click the **New OAuth App** button. +. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and select the **New OAuth App** button. + .New GitHub OAuth App image::github-oauth-new.png[Screenshot showing setting up a new OAuth app] @@ -302,7 +302,7 @@ ifndef::env-gcp[] === GCP - Reserve a static external IP address The link:https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#external-ip[Google Cloud docs] provide information on how reserve an IP address. -Make note of the returned IPv4 address for use later in the values.yaml file. +Make note of the returned IPv4 address for use later in the `values.yaml` file. endif::env-gcp[] @@ -327,7 +327,7 @@ aws ec2 allocate-address #} ---- -Make note of each of the returned `AllocationId` values from the CLI for use in the values.yaml file. +Make note of each of the returned `AllocationId` values from the CLI for use in the `values.yaml` file. endif::env-aws[] @@ -482,7 +482,7 @@ NOTE: If you are installing behind a proxy, object storage should be behind this ifndef::env-gcp[] [#s3-storage] -=== S3 Storage +=== S3 storage [#create-aws-s3-bucket] ==== a. Create AWS S3 Bucket @@ -509,7 +509,7 @@ aws s3api put-bucket-versioning \ [#set-up-authentication-aws] ==== c. Set up authentication -There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. +Two ways to authenticate CircleCI with S3 are: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. [.tab.authentication.IRSA] -- @@ -517,7 +517,7 @@ There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accou The following is a summary of link:https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html[AWS's Documentation on IRSA] that is sufficient for installing CircleCI. -. Create an IAM OIDC identity provider for your EKS Cluster. +. Create an IAM OIDC Identity Provider for your EKS Cluster. + [source,shell] ---- diff --git a/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc b/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc index d280f565716..eeae3e8cb3d 100644 --- a/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc +++ b/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc @@ -28,10 +28,10 @@ To see some common upgrade path options, see link:https://support.circleci.com/h [#notes] == Notes on changes in v4.3 -Please note the following significant changes in server v4.3. +Note the following significant changes in server v4.3. [#dlc] -=== Docker layer caching (DLC) +=== Docker layer caching (or DLC) * Old DLC volumes will not be carried over and need to be wiped manually * Projects that stop using DLC will not delete final DLC cache without manual intervention @@ -65,7 +65,7 @@ Please note the following significant changes in server v4.3. [#workflows-conductor] === Workflows conductor -* There is a `workflows-conductor` data migration that will run as a job in the background after your upgrade. The migration processes projects in batches of 1000 and sleeps for 60 seconds before starting another. Processing time for a batch depends on the MongoDB and PostgreSQL data stores, but in our production environment, a batch of 1000 projects took around 6 seconds. +* A `workflows-conductor` data migration will run as a job in the background after your upgrade. The migration processes projects in batches of 1000 and sleeps for 60 seconds before starting another. Processing time for a batch depends on the MongoDB and PostgreSQL data stores, but in our production environment, a batch of 1000 projects took around 6 seconds. * Ensure that your CircleCI server v4.3 installation is left running until after the migration has completed. You can confirm that the migration has completed by checking the logs of `workflows-conductor-event-consumer`. There will be a `starting next-build-seq-migration` log message with `project_count=0` [#vault] @@ -85,9 +85,9 @@ Please note the following significant changes in server v4.3. [#upgrade-steps] == Upgrade steps -NOTE: This upgrade is a disruptive process and downtime is expected. Please do not attempt to run jobs during this upgrade. +NOTE: This upgrade is a disruptive process and downtime is expected. Do not attempt to run jobs during this upgrade. -. Ensure your cluster is running a compatible kubernetes version for this release (1.26 - 1.29). +. Ensure your cluster is running a compatible Kubernetes version for this release (1.26 - 1.29). . Check the link:https://circleci.com/server/changelog/[changelog] and make sure there are no actions you need to take before deploying a new version. diff --git a/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc index aba7efe7bc6..c7e2404556f 100644 --- a/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc @@ -285,7 +285,7 @@ CAUTION: If GitHub Enterprise and CircleCI server are not on the same domain, th Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both GitHub.com and GitHub Enterprise. -. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and click the **New OAuth App** button. +. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and select the **New OAuth App** button. + .New GitHub OAuth App image::github-oauth-new.png[Screenshot showing setting up a new OAuth app] @@ -315,7 +315,7 @@ ifndef::env-gcp[] === GCP: Reserve a static external IP address The link:https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#external-ip[Google Cloud docs] provide information on how reserve an IP address. -Make note of the returned IPv4 address for use later in the values.yaml file. +Make note of the returned IPv4 address for use later in the `values.yaml` file. endif::env-gcp[] @@ -340,7 +340,7 @@ aws ec2 allocate-address #} ---- -Make note of each of the returned `AllocationId` values from the CLI for use in the values.yaml file. +Make note of each of the returned `AllocationId` values from the CLI for use in the `values.yaml` file. endif::env-aws[] @@ -522,7 +522,7 @@ aws s3api put-bucket-versioning \ [#set-up-authentication-aws] ==== c. Set up authentication -There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. +Two ways to authenticate CircleCI with S3 are: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. [.tab.authentication.IRSA] -- @@ -530,7 +530,7 @@ There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accou The following is a summary of link:https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html[AWS's Documentation on IRSA] that is sufficient for installing CircleCI. -. Create an IAM OIDC identity provider for your EKS Cluster. +. Create an IAM OIDC Identity Provider for your EKS Cluster. + [source,shell] ---- diff --git a/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc index c67433fa882..0d615b8d2a0 100644 --- a/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc @@ -274,7 +274,7 @@ CAUTION: If GitHub Enterprise and CircleCI server are not on the same domain, th Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both GitHub.com and GitHub Enterprise. -. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and click the **New OAuth App** button. +. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and select the **New OAuth App** button. + .New GitHub OAuth App image::github-oauth-new.png[Screenshot showing setting up a new OAuth app] @@ -524,7 +524,7 @@ aws s3api put-bucket-versioning \ [#set-up-authentication-aws] ==== c. Set up authentication -There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. +Two ways to authenticate CircleCI with S3 are: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method. [.tab.authentication.IRSA] -- From 8c5b91d57d660c6889b7ab0e22693433d926d926 Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Tue, 23 Jul 2024 07:16:53 -0300 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Rosie Yohannan --- .../_cci2/server/v4.3/installation/phase-1-prerequisites.adoc | 2 +- jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc | 2 +- .../_cci2/server/v4.4/installation/phase-1-prerequisites.adoc | 2 +- .../_cci2/server/v4.5/installation/phase-1-prerequisites.adoc | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc index eb3c836b0a5..61d27fa20ab 100644 --- a/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.3/installation/phase-1-prerequisites.adoc @@ -497,7 +497,7 @@ aws s3api create-bucket \ ==== b. Enable bucket versioning -To use the https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to do that on the bucket created in the previous step: +To use the link:https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step: [source,shell] ---- aws s3api put-bucket-versioning \ diff --git a/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc b/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc index eeae3e8cb3d..a40d4157c73 100644 --- a/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc +++ b/jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc @@ -36,7 +36,7 @@ Note the following significant changes in server v4.3. * Old DLC volumes will not be carried over and need to be wiped manually * Projects that stop using DLC will not delete final DLC cache without manual intervention * DLC now runs through S3 and GCS instead of SSD Volumes -* If AWS S3 is used for https://circleci.com/docs/server/v4.3/installation/phase-1-prerequisites/#s3-storage[object storage], DLC now requires https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html[versioning to be enabled] on the bucket. +* If AWS S3 is used for xref:phase-1-prerequisites#s3-storage[object storage], the use of Docker layer caching (DLC) requires link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html[versioning to be enabled] on the bucket. [#machine] === Machine jobs diff --git a/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc index c7e2404556f..2dae8c5b32c 100644 --- a/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.4/installation/phase-1-prerequisites.adoc @@ -510,7 +510,7 @@ aws s3api create-bucket \ ==== b. Enable bucket versioning -To use the https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to do that on the bucket created in the previous step: +To use the link:https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step: [source,shell] ---- aws s3api put-bucket-versioning \ diff --git a/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc b/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc index 0d615b8d2a0..aeabdbb0224 100644 --- a/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc +++ b/jekyll/_cci2/server/v4.5/installation/phase-1-prerequisites.adoc @@ -513,7 +513,7 @@ aws s3api create-bucket \ ==== b. Enable bucket versioning -To use the https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to do that on the bucket created in the previous step: +To use the link:https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step: [source,shell] ---- aws s3api put-bucket-versioning \