Skip to content

Commit

Permalink
Add docs for multi repo pipelines (#1142)
Browse files Browse the repository at this point in the history
* Add docs for multi repo pipelines

* Fix typos

* rename section

* address PR comments

* Add detail on when multi-repo might be useful

* more pr comments

* fix yml file example
  • Loading branch information
Andrew Ellison authored Sep 29, 2023
1 parent 0eb05c5 commit 90671c4
Show file tree
Hide file tree
Showing 15 changed files with 227 additions and 112 deletions.
39 changes: 39 additions & 0 deletions _docs-sources/pipelines/hello-world/github-enterprise.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Allowing Pipelines Actions in GitHub Actions

Gruntwork Pipelines uses a set of Gruntwork built re-usable Github Actions, which are available in the GitHub Actions marketplace. Gruntwork is a Verified Creator of GitHub Actions.

### GitHub Enterprise

Gruntwork recommends explicitly allowing Actions required by Gruntwork Pipelines to run in your GitHub organization. See [Allowing select actions and reusable workflows to run](https://docs.github.com/en/enterprise-cloud@latest/admin/policies/enforcing-policies-for-your-enterprise/enforcing-policies-for-github-actions-in-your-enterprise#allowing-select-actions-and-reusable-workflows-to-run) to learn more.

You will need to allow the following GitHub Actions for Gruntwork Pipelines to run successfully:
- [gruntwork-io/pipelines-dispatch](https://github.com/gruntwork-io/pipelines-dispatch)
- [gruntwork-io/pipelines-orchestrate](https://github.com/gruntwork-io/pipelines-orchestrate)
- [gruntwork-io/pipelines-execute](https://github.com/gruntwork-io/pipelines-execute)
- [gruntwork-io/terragrunt-action](https://github.com/gruntwork-io/terragrunt-action)
- [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials)
- [Codex-/return-dispatch](https://github.com/Codex-/return-dispatch)
- [Codex-/await-remote-run](https://github.com/Codex-/await-remote-run)
- [dsaltares/fetch-gh-release-asset](https://github.com/dsaltares/fetch-gh-release-asset)
- [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request)

Gruntwork recommends using a tagged version of the `gruntwork-io` actions and a wildcard for the non-Gruntwork actions, as the non-Gruntwork actions are wrapped in the Gruntwork actions. This will require less administrative work to keep policies up to date with versions being used by Gruntwork actions. An example policy can be found below.

```txt
gruntwork-io/pipelines-dispatch@<newest release>,
gruntwork-io/pipelines-orchestrate@<newest release>,
gruntwork-io/pipelines-execute@<newest release>,
gruntwork-io/terragrunt-action@<newest release>,
aws-actions/configure-aws-credentials@*,
Codex-/return-dispatch@*,
Codex-/await-remote-run@*,
dsaltares/fetch-gh-release-asset@*,
peter-evans/create-pull-request@*
```

Navigate to each Gruntwork repository to retrieve the latest tagged release for each action.

### GitHub Team and Pro

Currently GitHub Actions does not support selecting specific repos outside of your own GitHub organization for the team and pro tier. To use Gruntwork Pipelines you must select the **Allow all actions and reusable workflows** option in the Actions general settings at the Organization level.

4 changes: 2 additions & 2 deletions _docs-sources/pipelines/hello-world/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ For a simple proof of concept, the default repo configuration will suffice.
Before using these repositories in a production environment, we recommend setting up a
[branch protection rule](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule)
for your `main` branch. At a minimum, we recommend enabling requiring a pull request before merging with at least one reviewer required.
See [Using Pipelines](/pipelines/using-pipelines#recommended-settings) for recommended settings.
See [Using Pipelines](../security/branch-protection#recommended-settings) for recommended settings.
:::

### Setting up secrets
Expand All @@ -67,7 +67,7 @@ First, navigate to the `infrastructure-live` repository. Select the `Settings` t
Next, Navigate to the `infrastructure-pipelines` repository. Select the `Settings` tab, select the `Secrets and variables` drop down on the left side panel, then select `Actions`. Create two secrets named `INFRA_LIVE_ACCESS_TOKEN` and `GRUNTWORK_CODE_ACCESS_TOKEN`. Use your GitHub PAT as the value for both secrets.

:::warning
Using a single token with broad access is sufficient for a POC or demo environments. In a production environment, we recommend using a mix of fine-grained and classic PATs to apply the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to all tokens used in Pipelines workflows. See [machine users](../using-pipelines/machine-users.md) for more information.
Using a single token with broad access is sufficient for a POC or demo environments. In a production environment, we recommend using a mix of fine-grained and classic PATs to apply the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to all tokens used in Pipelines workflows. See [machine users](../security/machine-users.md) for more information.
:::

## Generating code
Expand Down
3 changes: 0 additions & 3 deletions _docs-sources/pipelines/multi-account/index.md

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Using Pipelines
# Branch Protection

Gruntwork Pipelines is designed to be used with a PR based workflow.
This means an approval on a PR is an approval to deploy infrastructure, making the configuration of repo settings and branch protection especially important.
Expand Down Expand Up @@ -43,42 +43,3 @@ The following is an example of the recommended settings for branch protection:
1. Gruntwork Pipelines runs `apply` on any changes from the PR
- On Success, a comment is placed on the PR indicating success
- On Failure, a new GitHub issue is created describing the failure. A new PR must be created to resolve any failures.

## Allowing Pipelines Actions in GitHub Actions

Gruntwork Pipelines uses a set of Gruntwork built re-usable Github Actions, which are available in the GitHub Actions marketplace. Gruntwork is a Verified Creator of GitHub Actions.

### GitHub Enterprise

Gruntwork recommends explicitly allowing Actions required by Gruntwork Pipelines to run in your GitHub organization. See [Allowing select actions and reusable workflows to run](https://docs.github.com/en/enterprise-cloud@latest/admin/policies/enforcing-policies-for-your-enterprise/enforcing-policies-for-github-actions-in-your-enterprise#allowing-select-actions-and-reusable-workflows-to-run) to learn more.

You will need to allow the following GitHub Actions for Gruntwork Pipelines to run successfully:
- [gruntwork-io/pipelines-dispatch](https://github.com/gruntwork-io/pipelines-dispatch)
- [gruntwork-io/pipelines-orchestrate](https://github.com/gruntwork-io/pipelines-orchestrate)
- [gruntwork-io/pipelines-execute](https://github.com/gruntwork-io/pipelines-execute)
- [gruntwork-io/terragrunt-action](https://github.com/gruntwork-io/terragrunt-action)
- [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials)
- [Codex-/return-dispatch](https://github.com/Codex-/return-dispatch)
- [Codex-/await-remote-run](https://github.com/Codex-/await-remote-run)
- [dsaltares/fetch-gh-release-asset](https://github.com/dsaltares/fetch-gh-release-asset)
- [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request)

Gruntwork recommends using a tagged version of the `gruntwork-io` actions and a wildcard for the non-Gruntwork actions, as the non-Gruntwork actions are wrapped in the Gruntwork actions. This will require less administrative work to keep policies up to date with versions being used by Gruntwork actions. An example policy can be found below.

```txt
gruntwork-io/pipelines-dispatch@<newest release>,
gruntwork-io/pipelines-orchestrate@<newest release>,
gruntwork-io/pipelines-execute@<newest release>,
gruntwork-io/terragrunt-action@<newest release>,
aws-actions/configure-aws-credentials@*,
Codex-/return-dispatch@*,
Codex-/await-remote-run@*,
dsaltares/fetch-gh-release-asset@*,
peter-evans/create-pull-request@*
```

Navigate to each Gruntwork repository to retrieve the latest tagged release for each action.

### GitHub Team and Pro

Currently GitHub Actions does not support selecting specific repos outside of your own GitHub organization for the team and pro tier. To use Gruntwork Pipelines you must select the **Allow all actions and reusable workflows** option in the Actions general settings at the Organization level.
4 changes: 2 additions & 2 deletions _docs-sources/pipelines/security/controls.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Pipelines takes a defense in depth approach to securing workflows. This document

## Dual-repository approach

Pipelines dual-repository approach separates infrastructure definitions from infrastructure deployment mechanisms. Pipelines requires two repositories —`infrastructure-pipelines`, where deployment workflows are defined and `infrastructure-live`, where infrastructure is defined as code. Each repository should have branch protection rules to prevent un-reviewed code from being deployed. Refer to [Recommended Settings](../using-pipelines#recommended-settings) in [using pipelines](../using-pipelines) to learn more.
Pipelines dual-repository approach separates infrastructure definitions from infrastructure deployment mechanisms. Pipelines requires two repositories —`infrastructure-pipelines`, where deployment workflows are defined and `infrastructure-live`, where infrastructure is defined as code. Each repository should have branch protection rules to prevent un-reviewed code from being deployed. Refer to [Recommended Settings](branch-protection#recommended-settings) in [Branch Protection](branch-protection) to learn more.

To control access to these repositories we recommend creating GitHub teams. Write access to the `infrastructure-pipelines` repository should be limited to individuals that already have administrative access to your AWS accounts (see [accessing AWS resources](#accessing-aws-resources)). Read and write access to the `infrastructure-live` repository should be granted to any individual who needs to define infrastructure as code. See [repository access](repository-access.md) for more details.

Expand All @@ -20,7 +20,7 @@ Gruntwork Pipelines uses a series of GitHub Personal Access Tokens (PAT) to allo
- `INFRA_LIVE_ACCESS_TOKEN` - A [fine-grained PAT](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) that has read and write access to your `infrastructure-live` repository. This token is used in Pipelines to create pull requests after code generation and add pull request comments.
- `PIPELINES_DISPATCH_TOKEN` - A fine-grained PAT that can run Workflows in your `infrastructure-pipelines` repository.

Steps to create a PAT can be found in the official documentation. Refer to [creating a personal access token classic](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) and [creating a fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token), respectively. We recommend using [machine users](../using-pipelines/machine-users.md) for this use case.
Steps to create a PAT can be found in the official documentation. Refer to [creating a personal access token classic](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) and [creating a fine-grained personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token), respectively. We recommend using [machine users](../security/machine-users.md) for this use case.

To learn more about GitHub PATs, refer to their documentation on [managing personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens).

Expand Down
54 changes: 54 additions & 0 deletions _docs-sources/pipelines/security/multi-account.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Multiple Infrastructure-Live Repos

We recommend using a single `infrastructure-live` git repository for managing your organization's infrastructure.
Sometimes, this isn't possible due to team structure, security requirements, or other limitations.
You may choose to use multiple `infrastructure-live` repos to:

1. Facilitate more granular access controls
1. Separate concerns that do not require shared configuration
1. Ease the burden of high traffic repos (reducing the likelihood of feature branches becoming out-of-date relative to `main`)

Note that when using multiple repositories, it is more difficult to share a infrastructure configuration across environments,
so think carefully about your specific use case before making the decision.


## Create Additional Repos

New `infrastructure-live` repositories can be created using the same process described in the
[Hello World](../hello-world#setting-up-the-repositories) documentation.

:::info
Once the repository is created, you'll need to set up machine user access using either the existing machine user and `PIPELINES_DISPATCH` PAT token,
or one created specifically for this purpose. See [Machine Users](machine-users) for more information.
:::

No special configuration is required for the new `infrastructure-live` repository,
the Pipelines Dispatch job will identify the source repository and pass that information
to the shared `infrastructure-pipelines` repository.

## Enable Additional Repos

:::warning
Once a repository is enabled for pipelines, any code pushed to the `main` branch of that repository will be eligible to access your
AWS account using OIDC. Ensure you have the [recommended settings](branch-protection) for branch protection configured before adding the new
repository to the allowlist.
:::

To ensure no unauthorized access is granted to your `infrastructure-pipelines` repository,
an allowlist of `infrastructure-live` repositories exists in the `.gruntwork/config.yml` file in the `infrastructure-pipelines` repository.
To allow resources to be deployed by your new repository,
add the repository to the `repo-allow-list` section of `.gruntwork/config.yml`.

The new resource should match the name of your repository **exactly** in the format
`github-org/infrastructure-live-repo-name` with a single repository per line. See the example file below:

```yml title=infrastructure-pipelines/.gruntwork/config.yml
# The git repos that have permissions to invoke Pipelines jobs
repo-allow-list:
- acme/team-1-infrastructure-live
- acme/team-2-infrastructure-live
```
:::info
The `INFRA_LIVE_ACCESS_TOKEN` available to the `infrastructure-pipelines` repository must have content read & write access to all repositories in the allowlist.
:::
47 changes: 47 additions & 0 deletions docs/pipelines/hello-world/github-enterprise.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Allowing Pipelines Actions in GitHub Actions

Gruntwork Pipelines uses a set of Gruntwork built re-usable Github Actions, which are available in the GitHub Actions marketplace. Gruntwork is a Verified Creator of GitHub Actions.

### GitHub Enterprise

Gruntwork recommends explicitly allowing Actions required by Gruntwork Pipelines to run in your GitHub organization. See [Allowing select actions and reusable workflows to run](https://docs.github.com/en/enterprise-cloud@latest/admin/policies/enforcing-policies-for-your-enterprise/enforcing-policies-for-github-actions-in-your-enterprise#allowing-select-actions-and-reusable-workflows-to-run) to learn more.

You will need to allow the following GitHub Actions for Gruntwork Pipelines to run successfully:
- [gruntwork-io/pipelines-dispatch](https://github.com/gruntwork-io/pipelines-dispatch)
- [gruntwork-io/pipelines-orchestrate](https://github.com/gruntwork-io/pipelines-orchestrate)
- [gruntwork-io/pipelines-execute](https://github.com/gruntwork-io/pipelines-execute)
- [gruntwork-io/terragrunt-action](https://github.com/gruntwork-io/terragrunt-action)
- [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials)
- [Codex-/return-dispatch](https://github.com/Codex-/return-dispatch)
- [Codex-/await-remote-run](https://github.com/Codex-/await-remote-run)
- [dsaltares/fetch-gh-release-asset](https://github.com/dsaltares/fetch-gh-release-asset)
- [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request)

Gruntwork recommends using a tagged version of the `gruntwork-io` actions and a wildcard for the non-Gruntwork actions, as the non-Gruntwork actions are wrapped in the Gruntwork actions. This will require less administrative work to keep policies up to date with versions being used by Gruntwork actions. An example policy can be found below.

```txt
gruntwork-io/pipelines-dispatch@<newest release>,
gruntwork-io/pipelines-orchestrate@<newest release>,
gruntwork-io/pipelines-execute@<newest release>,
gruntwork-io/terragrunt-action@<newest release>,
aws-actions/configure-aws-credentials@*,
Codex-/return-dispatch@*,
Codex-/await-remote-run@*,
dsaltares/fetch-gh-release-asset@*,
peter-evans/create-pull-request@*
```

Navigate to each Gruntwork repository to retrieve the latest tagged release for each action.

### GitHub Team and Pro

Currently GitHub Actions does not support selecting specific repos outside of your own GitHub organization for the team and pro tier. To use Gruntwork Pipelines you must select the **Allow all actions and reusable workflows** option in the Actions general settings at the Organization level.



<!-- ##DOCS-SOURCER-START
{
"sourcePlugin": "local-copier",
"hash": "cc71dbd996bb0349ed2c57e4d726d656"
}
##DOCS-SOURCER-END -->
6 changes: 3 additions & 3 deletions docs/pipelines/hello-world/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ For a simple proof of concept, the default repo configuration will suffice.
Before using these repositories in a production environment, we recommend setting up a
[branch protection rule](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule)
for your `main` branch. At a minimum, we recommend enabling requiring a pull request before merging with at least one reviewer required.
See [Using Pipelines](/pipelines/using-pipelines#recommended-settings) for recommended settings.
See [Using Pipelines](../security/branch-protection#recommended-settings) for recommended settings.
:::

### Setting up secrets
Expand All @@ -67,7 +67,7 @@ First, navigate to the `infrastructure-live` repository. Select the `Settings` t
Next, Navigate to the `infrastructure-pipelines` repository. Select the `Settings` tab, select the `Secrets and variables` drop down on the left side panel, then select `Actions`. Create two secrets named `INFRA_LIVE_ACCESS_TOKEN` and `GRUNTWORK_CODE_ACCESS_TOKEN`. Use your GitHub PAT as the value for both secrets.

:::warning
Using a single token with broad access is sufficient for a POC or demo environments. In a production environment, we recommend using a mix of fine-grained and classic PATs to apply the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to all tokens used in Pipelines workflows. See [machine users](../using-pipelines/machine-users.md) for more information.
Using a single token with broad access is sufficient for a POC or demo environments. In a production environment, we recommend using a mix of fine-grained and classic PATs to apply the [principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege) to all tokens used in Pipelines workflows. See [machine users](../security/machine-users.md) for more information.
:::

## Generating code
Expand Down Expand Up @@ -200,6 +200,6 @@ If you are not going to continue using Pipelines after this tutorial, clean up t
<!-- ##DOCS-SOURCER-START
{
"sourcePlugin": "local-copier",
"hash": "ad5384f641a95bb77b231135c7fc36ab"
"hash": "2e016e548a17af8040cc2d800a5fe6dd"
}
##DOCS-SOURCER-END -->
11 changes: 0 additions & 11 deletions docs/pipelines/multi-account/index.md

This file was deleted.

Loading

0 comments on commit 90671c4

Please sign in to comment.