Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update server 4.3+ docs on DLC requiring S3 bucket versioning [ONPREM-1361] #8870

Merged
merged 4 commits into from
Jul 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ CAUTION: If GitHub Enterprise and CircleCI server are not on the same domain, th

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both github.com and GitHub Enterprise.

. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and click the **New OAuth App** button.
. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and select the **New OAuth App** button.
+
.New GitHub OAuth App
image::github-oauth-new.png[Screenshot showing setting up a new OAuth app]
Expand Down Expand Up @@ -302,7 +302,7 @@ ifndef::env-gcp[]
=== GCP - Reserve a static external IP address
The link:https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#external-ip[Google Cloud docs] provide information on how reserve an IP address.

Make note of the returned IPv4 address for use later in the values.yaml file.
Make note of the returned IPv4 address for use later in the `values.yaml` file.

endif::env-gcp[]

Expand All @@ -327,7 +327,7 @@ aws ec2 allocate-address
#}
----

Make note of each of the returned `AllocationId` values from the CLI for use in the values.yaml file.
Make note of each of the returned `AllocationId` values from the CLI for use in the `values.yaml` file.

endif::env-aws[]

Expand Down Expand Up @@ -482,7 +482,7 @@ NOTE: If you are installing behind a proxy, object storage should be behind this
ifndef::env-gcp[]

[#s3-storage]
=== S3 Storage
=== S3 storage

[#create-aws-s3-bucket]
==== a. Create AWS S3 Bucket
Expand All @@ -495,19 +495,29 @@ aws s3api create-bucket \
--create-bucket-configuration LocationConstraint=<YOUR_REGION>
----

==== b. Enable bucket versioning

To use the link:https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step:
[source,shell]
----
aws s3api put-bucket-versioning \
--bucket <YOUR_BUCKET_NAME> \
--region <YOUR_REGION> \
--versioning-configuration Status=Enabled
----

[#set-up-authentication-aws]
==== b. Set up authentication
==== c. Set up authentication

There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method.
Two ways to authenticate CircleCI with S3 are: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method.

[.tab.authentication.IRSA]
--
**Option 1:** IRSA

The following is a summary of link:https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html[AWS's Documentation on IRSA] that is sufficient for installing CircleCI.

. Create an IAM OIDC identity provider for your EKS Cluster.
. Create an IAM OIDC Identity Provider for your EKS Cluster.
+
[source,shell]
----
Expand Down
11 changes: 6 additions & 5 deletions jekyll/_cci2/server/v4.3/installation/upgrade-server.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,15 @@ To see some common upgrade path options, see link:https://support.circleci.com/h
[#notes]
== Notes on changes in v4.3

Please note the following significant changes in server v4.3.
Note the following significant changes in server v4.3.

[#dlc]
=== Docker layer caching (DLC)
=== Docker layer caching (or DLC)

* Old DLC volumes will not be carried over and need to be wiped manually
* Projects that stop using DLC will not delete final DLC cache without manual intervention
* DLC now runs through S3 and GCS instead of SSD Volumes
* If AWS S3 is used for xref:phase-1-prerequisites#s3-storage[object storage], the use of Docker layer caching (DLC) requires link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/manage-versioning-examples.html[versioning to be enabled] on the bucket.

[#machine]
=== Machine jobs
Expand Down Expand Up @@ -64,7 +65,7 @@ Please note the following significant changes in server v4.3.
[#workflows-conductor]
=== Workflows conductor

* There is a `workflows-conductor` data migration that will run as a job in the background after your upgrade. The migration processes projects in batches of 1000 and sleeps for 60 seconds before starting another. Processing time for a batch depends on the MongoDB and PostgreSQL data stores, but in our production environment, a batch of 1000 projects took around 6 seconds.
* A `workflows-conductor` data migration will run as a job in the background after your upgrade. The migration processes projects in batches of 1000 and sleeps for 60 seconds before starting another. Processing time for a batch depends on the MongoDB and PostgreSQL data stores, but in our production environment, a batch of 1000 projects took around 6 seconds.
* Ensure that your CircleCI server v4.3 installation is left running until after the migration has completed. You can confirm that the migration has completed by checking the logs of `workflows-conductor-event-consumer`. There will be a `starting next-build-seq-migration` log message with `project_count=0`

[#vault]
Expand All @@ -84,9 +85,9 @@ Please note the following significant changes in server v4.3.
[#upgrade-steps]
== Upgrade steps

NOTE: This upgrade is a disruptive process and downtime is expected. Please do not attempt to run jobs during this upgrade.
NOTE: This upgrade is a disruptive process and downtime is expected. Do not attempt to run jobs during this upgrade.

. Ensure your cluster is running a compatible kubernetes version for this release (1.26 - 1.29).
. Ensure your cluster is running a compatible Kubernetes version for this release (1.26 - 1.29).

. Check the link:https://circleci.com/server/changelog/[changelog] and make sure there are no actions you need to take before deploying a new version.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ CAUTION: If GitHub Enterprise and CircleCI server are not on the same domain, th

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both github.com and GitHub Enterprise.

. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and click the **New OAuth App** button.
. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and select the **New OAuth App** button.
+
.New GitHub OAuth App
image::github-oauth-new.png[Screenshot showing setting up a new OAuth app]
Expand Down Expand Up @@ -315,7 +315,7 @@ ifndef::env-gcp[]
=== GCP: Reserve a static external IP address
The link:https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#external-ip[Google Cloud docs] provide information on how reserve an IP address.

Make note of the returned IPv4 address for use later in the values.yaml file.
Make note of the returned IPv4 address for use later in the `values.yaml` file.

endif::env-gcp[]

Expand All @@ -340,7 +340,7 @@ aws ec2 allocate-address
#}
----

Make note of each of the returned `AllocationId` values from the CLI for use in the values.yaml file.
Make note of each of the returned `AllocationId` values from the CLI for use in the `values.yaml` file.

endif::env-aws[]

Expand Down Expand Up @@ -508,19 +508,29 @@ aws s3api create-bucket \
--create-bucket-configuration LocationConstraint=<YOUR_REGION>
----

==== b. Enable bucket versioning

To use the link:https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step:
[source,shell]
----
aws s3api put-bucket-versioning \
--bucket <YOUR_BUCKET_NAME> \
--region <YOUR_REGION> \
--versioning-configuration Status=Enabled
----

[#set-up-authentication-aws]
==== b. Set up authentication
==== c. Set up authentication

There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method.
Two ways to authenticate CircleCI with S3 are: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method.

[.tab.authentication.IRSA]
--
**Option 1:** IRSA

The following is a summary of link:https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html[AWS's Documentation on IRSA] that is sufficient for installing CircleCI.

. Create an IAM OIDC identity provider for your EKS Cluster.
. Create an IAM OIDC Identity Provider for your EKS Cluster.
+
[source,shell]
----
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ CAUTION: If GitHub Enterprise and CircleCI server are not on the same domain, th

Registering and setting up a new GitHub OAuth app for CircleCI server allows for authorization control to your server installation using GitHub OAuth and for updates to GitHub projects/repos using build status information. The following steps apply for both github.com and GitHub Enterprise.

. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and click the **New OAuth App** button.
. In your browser, navigate to **your GitHub instance** > **User Settings** > **Developer Settings** > **OAuth Apps** and select the **New OAuth App** button.
+
.New GitHub OAuth App
image::github-oauth-new.png[Screenshot showing setting up a new OAuth app]
Expand Down Expand Up @@ -511,11 +511,20 @@ aws s3api create-bucket \
--create-bucket-configuration LocationConstraint=<YOUR_REGION>
----

==== b. Enable bucket versioning

[#set-up-authentication-aws]
==== b. Set up authentication
To use the link:https://circleci.com/docs/docker-layer-caching/[Docker layer caching] (DLC) feature in CircleCI, link:https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html[bucket versioning] needs to be enabled. Run the following command to enable bucket versioning on the bucket created in the previous step:
[source,shell]
----
aws s3api put-bucket-versioning \
--bucket <YOUR_BUCKET_NAME> \
--region <YOUR_REGION> \
--versioning-configuration Status=Enabled
----

There are two ways to authenticate CircleCI with S3: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method.
[#set-up-authentication-aws]
==== c. Set up authentication
Two ways to authenticate CircleCI with S3 are: IAM Roles for Service Accounts (IRSA), and IAM access keys. IRSA is the recommended method.

[.tab.authentication.IRSA]
--
Expand Down