Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backstage pattern for Backstage add-on #81

Merged
merged 32 commits into from
Jun 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
88e4e69
first commit
freschri May 11, 2023
dc58633
removed redundant reference to clusterInfo
freschri May 12, 2023
41654e5
refactored following refactor from upstream
freschri May 20, 2023
cdaa745
Merge branch 'main' into backstage
freschri May 20, 2023
7589874
added architecture diagram
freschri May 21, 2023
f4f9ddd
reduced architecture diagram image size
freschri May 21, 2023
0bdeb49
set image size in md
freschri May 21, 2023
df65963
updated image with all services used
freschri May 22, 2023
906b68c
added missing link in the diagram
freschri May 22, 2023
c057114
db position correction in diagram
freschri May 23, 2023
126f8c5
moved arrow positions in diagram to make flow clearer
freschri May 23, 2023
ba346fd
minor change to comment
freschri May 24, 2023
633ea07
corrected readme and aligned pattern name to others
freschri May 24, 2023
3bba28a
Merge branch 'aws-samples:main' into backstage
freschri Jun 7, 2023
31f7bbc
updated label to avoid build fail on stack name reg expression check
freschri Jun 7, 2023
824dd40
replaced sample values to fix build failure in github
freschri Jun 7, 2023
0d4b818
updated doc with xcompile suggestion
freschri Jun 8, 2023
981ff7b
addressing PR review comments from elamaran11
freschri Jun 13, 2023
09b7335
addressing variables comment from elamaran11
freschri Jun 18, 2023
1a47d33
Merge branch 'main' into backstage
freschri Jun 20, 2023
26dd141
running make lint-fix
freschri Jun 20, 2023
cc98d01
Improving documentation for publishing on https://aws-samples.github.…
freschri Jun 21, 2023
d239fa0
amended destroy command in doc
freschri Jun 21, 2023
25759a9
spelling fixes in doc
freschri Jun 21, 2023
88f42d7
prepended parameter names with "backstage."
freschri Jun 21, 2023
5a8a1cb
improved documentation
freschri Jun 23, 2023
742400b
further updates to the documentation
freschri Jun 23, 2023
48d797f
updated documentation
freschri Jun 25, 2023
d7a355a
more updates to documentation
freschri Jun 25, 2023
3c3363e
minor
freschri Jun 25, 2023
5d5102a
missing file extension in doc
freschri Jun 25, 2023
f1c09da
remove image resize
freschri Jun 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions bin/backstage.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import { BackstageConstruct } from '../lib/backstage-construct';
import { configureApp } from '../lib/common/construct-utils';

const app = configureApp();

new BackstageConstruct(app, 'backstage-stack');
199 changes: 199 additions & 0 deletions docs/patterns/backstage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
# Backstage on EKS

## Objective

[Backstage](https://backstage.io/) is an application that aims to facilitate introduction and maintenance of standards and best practices, across the organization, tying all infrastructure tooling, resources, owners, contributors, and administrators together in one place.

The base functionality is provided by the Core component, which is assembled together with Plugins into an Application. Plugins extend the Core with additional functionalities that can be open source, or proprietary to a company.

The objective of this pattern is to illustrate how to deploy a Backstage pre-built Docker image, using the [Amazon EKS Blueprints Backstage add-on](https://github.com/aws-quickstart/cdk-eks-blueprints/blob/main/docs/addons/backstage.md).

## Architecture

<img src="./images/backstage-diagram.png" width="720">

## Approach

This blueprint will include the following:

- A new Well-Architected VPC with both Public and Private subnets
- A new Well-Architected EKS cluster in the region and account you specify
- An Application Load Balancer (ALB), implementing the Backstage Ingress rules
- An Amazon RDS for PostgreSQL instance
- A certificate, assigned to the ALB
- A Secret in AWS Secrets Manager, storing the database credentials, imported into the cluster via [ExternalsSecretsAddOn](https://aws-quickstart.github.io/cdk-eks-blueprints/addons/external-secrets/)
- Other popular add-ons

## Prerequisites

Ensure that you have installed the following tools on your machine:

- [aws cli](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) (also ensure it is [configured](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html#getting-started-quickstart-new))
- [cdk](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_install)
- [npm](https://docs.npmjs.com/cli/v8/commands/npm-install)
- [tsc](https://www.typescriptlang.org/download)
- [make](https://www.gnu.org/software/make/)
- [Docker](https://docs.docker.com/get-docker/)

Let’s start by setting the account and region environment variables:

```sh
ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
AWS_REGION=$(aws configure get region)
```

Create the [Backstage application](https://backstage.io/docs/getting-started/create-an-app), command reported here for your convenience:

```sh
npx @backstage/create-app@latest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cd to application directory is missing

```

Build the corresponding [Docker image](https://backstage.io/docs/deployment/docker), commands reported here for your convenience:

```sh
cd ./backstage
yarn install --frozen-lockfile
yarn tsc
yarn build:backend --config app-config.yaml
```

Note: if the above command throws an error caused by app-config.yaml not found, you can explicitly set the path to the file:

```sh
yarn build:backend --config $(pwd)/app-config.yaml
```
Then you can progress with the docker image build:

```sh
docker image build . -f packages/backend/Dockerfile --tag backstage
```

Note: consider the platform you are building on, and the target platform the image will run on, you might want to use the [--platform option](https://docs.docker.com/engine/reference/commandline/buildx_build/), e.g.:

```sh
docker buildx build ... --platform=...
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is the docker build and push required to be shown, how does this relate to the solution. Is this not something generic.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is, but the building of an image as an off-band task and dependency is not frequent across patterns; I could see that other colleagues testing the pattern hit a failure later on because the image was built for another platform. I think being a bit more verbose helps and does no harm?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No harm. But please make it as part of your complete document and add this to pre req.

```

Note: If you are running a version of Docker Engine version earlier than 23.0, you might need to enable BuildKit manually, like explained in the [Getting Started section](https://docs.docker.com/build/buildkit/#getting-started) of the BuildKit webpage.

(Optional) to show examples on the UI, add to Docker file:

```sh
COPY --chown=node:node examples /examples
```

Create an Amazon Elastic Container Registry (ECR) repository, named _backstage_:

```sh
aws ecr create-repository --repository-name backstage
```

```sh
DOCKER_IMAGE_ID=... #see output of image id from above image creation
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
docker tag $DOCKER_IMAGE_ID $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/backstage:latest
docker push $ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/backstage:latest
```

Setup a Hosted Zone in Route 53, with your parent domain. The pattern will create a new subdomain with format _{backstage subdomain label}.{parent domain}_. The default value for _{backstage subdomain label}_ is _backstage_ (see parameters below).

## Deployment

Clone the repository:

```sh
git clone https://github.com/aws-samples/cdk-eks-blueprints-patterns.git
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cd cdk-eks-blueprints-patterns is missing

cd cdk-eks-blueprints-patterns
```

Set the pattern's parameters in the CDK context by overriding the _cdk.json_ file (edit _PARENT_DOMAIN_NAME_ as it fits):

```sh
PARENT_DOMAIN_NAME=example.com
HOSTED_ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name $PARENT_DOMAIN_NAME --query "HostedZones[].Id" --output text | xargs basename)
cat << EOF > cdk.json
{
"app": "npx ts-node dist/lib/common/default-main.js",
"context": {
"backstage.image.registry.name": "${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com",
"backstage.parent.domain.name":"${PARENT_DOMAIN_NAME}",
"backstage.hosted.zone.id": "${HOSTED_ZONE_ID}"
}
}
EOF
```

(Optional) The full list of parameters you can set in the _context_ is:

```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not helpful at all. It took me so much time to create the file, populate field by field and also get it done. You should use a easier approach for people to use [this](https://github.com/aws-samples/cdk-eks-blueprints-patterns/blob/main/docs/patterns/secureingresscognito.md#:~:text=The%20CDK%20code%20expects%20the%20allowed%20domain%20and%20subdomain%20names%20in%20the%20CDK%20context%20file%20(cdk.json). on how we have simplified this for the user. For fields you can fetch from AWS like hosted zone, etc etc you should try to use an AWS commadn to populate that variable than use it to create the cdk.json

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is unclear; in a comment above you recommended "Please move these variables to cdk.json context. Please follow other patterns on how to read context."
so I am using the file now. what are the param which should be fetched and not set by the user? hosted zones can be multiple, so I need that from the user.. etc etc means? thanks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check the example i shared, We are using a way like below which creates the file with all required context values with east. Currently what you have is very difficult. It takes time, people will lose interest.

PARENT_HOSTED_ZONE=mycompany.a2z.com
DEV_SUBZONE_NAME=dev.mycompany.a2z.com
cat << EOF > cdk.json
{
    "app": "npx ts-node dist/lib/common/default-main.js",
    "context": {
        "parent.hostedzone.name": "${PARENT_HOSTED_ZONE}",
        "dev.subzone.name": "${DEV_SUBZONE_NAME}"
      }
}
EOF

Do the above for all variables. For variables which needs to get stuff like hosted zones using aws route53 commands. Specifically if you have multiple hz, you can query by the domain name

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will have a look, but "populate field by field.. taking much time".. it is 2 variables, the rest are defaults...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

"context": {
"backstage.namespace.name": ...,
"backstage.image.registry.name": ...,
"backstage.image.repository.name": ...,
"backstage.image.tag.name": ...,
"backstage.parent.domain.name": ...,
"backstage.subdomain.label": ...,
"backstage.hosted.zone.id": ...,
"backstage.certificate.resource.name": ...,
"backstage.database.resource.name": ...,
"backstage.database.instance.port": ...,
"backstage.database.secret.resource.name": ...,
"backstage.database.username": ...,
"backstage.database.secret.target.name": ...,
}
```

You can assign values to the above keys according to the following criteria (values are required where you don't see _default_ mentioned):

- "backstage.namespace.name": Backstage's namespace, the default is "backstage"
- "backstage.image.registry.name": the image registry for the Backstage Helm chart in Amazon ECR, a value similar to "youraccount.dkr.ecr.yourregion.amazonaws.com"
- "backstage.image.repository.name": the image repository for the Backstage Helm chart, the default is "backstage"
- "backstage.image.tag.name": the image tag, the default is "latest"
- "backstage.parent.domain.name": the parent domain in your Hosted Zone
- "backstage.subdomain.label": to be used as _{"subdomain.label"}.{"parent.domain.name"}_, the default is "backstage"
- "backstage.hosted.zone.id": the Hosted zone ID (format: 20x chars/numbers)
- "backstage.certificate.resource.name": resource name of the certificate, registered by the resource provider, the default is "backstage-certificate"
- "backstage.database.resource.name": resource name of the database, registered by the resource provider, the default is "backstage-database"
- "backstage.database.instance.port": the port the database will use, the default is 5432
- "backstage.database.secret.resource.name": resource name of the database's Secret, registered by the resource provider, the default is "backstage-database-credentials"
- "backstage.database.username": the username for the database's credentials, the default is "postgres"
- "backstage.database.secret.target.name": the name to be used when creating the Secret, the default is "backstage-database-secret"

If you haven't done it before, [bootstrap your cdk account and region](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html).

Run the following commands:

```sh
make deps
make build
make pattern backstage deploy
```
When deployment completes, the output will be similar to the following:

![Backstage deployment output](./images/backstage-console-output.png)

Navigate to the URL indicated by the first line in the output (_backstage-blueprint.BackstagebaseURL = https://backstage..._), you should see the screen below:

![Backstage console](./images/backstage-screen.png)

To see the deployed resources within the cluster, please run:

```sh
kubectl get pod,svc,secrets,ingress -A
```

A sample output is shown below:

![Backstage kubectl output](./images/backstage-kubectl-output.png)

## Next steps

You can go the [AWS Blog](https://aws.amazon.com/blogs/) to explore how to use Backstage e.g., [as an API Developer Portal for Amazon API Gateway](https://aws.amazon.com/blogs/opensource/how-traveloka-uses-backstage-as-an-api-developer-portal-for-amazon-api-gateway/) or [to provision infrastructure using AWS Proton](https://aws.amazon.com/blogs/containers/provisioning-infrastructure-using-the-aws-proton-open-source-backstage-plugin/). On the Backstage website you can also see other examples of [how to use and expand Backstage](https://backstage.io/demos/).

## Cleanup

To clean up your EKS Blueprints, run the following commands:

```sh
make pattern backstage destroy
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot of backstage is good. Can you also share some k8s resources that needs to be verified and also show the output of the deploy command, how to get k8s context etc etc like other docs

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also things like how to get the backstage URL etc etc is also missing which first time user will find difficulty to grab.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

above the picture I am saying "Navigate to {"subdomain.label"}.{"parent.domain.name"}, " that is the url...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya URL piece is good but what about resources deployed. I would atleast show k8s. resources created by backstage.
Also if you can point a link on what you can do next with backstage environment like aws docs, links, blogs will make user move to next step with backstage.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/patterns/images/backstage-diagram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/patterns/images/backstage-kubectl-output.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/patterns/images/backstage-screen.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
113 changes: 113 additions & 0 deletions lib/backstage-construct/backstage-secret-addon.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
import * as blueprints from '@aws-quickstart/eks-blueprints';
import * as eks from "aws-cdk-lib/aws-eks";
import { Construct } from 'constructs';
import { dependable } from "@aws-quickstart/eks-blueprints/dist/utils";
import { ISecret } from 'aws-cdk-lib/aws-secretsmanager';

export interface BackstageSecretAddOnProps {
/**
* Backstage Namespace
*/
namespace: string,

/**
* The name of the Secret
*/
databaseSecretTargetName: string,

/**
* The name of the Secret from the Resource Provider
*/
databaseSecretResourceName: string
}

export class BackstageSecretAddOn implements blueprints.ClusterAddOn {
readonly props: BackstageSecretAddOnProps;

constructor(props: BackstageSecretAddOnProps) {
this.props = props;
}

@dependable(blueprints.addons.ExternalsSecretsAddOn.name)
deploy(clusterInfo: blueprints.ClusterInfo): void | Promise<Construct> {
const cluster = clusterInfo.cluster;

const secretStoreName = "secret-manager-store";
const secretStore = new eks.KubernetesManifest(cluster.stack, "ClusterSecretStore", {
cluster: cluster,
manifest: [
{
apiVersion: "external-secrets.io/v1beta1",
kind: "ClusterSecretStore",
metadata: {
name: secretStoreName,
namespace: this.props.namespace
},
spec: {
provider: {
aws: {
service: "SecretsManager",
region: cluster.stack.region,
auth: {
jwt: {
serviceAccountRef: {
name: "external-secrets-sa",
namespace: "external-secrets",
},
},
},
},
},
},
},
],
});

const databaseCredentialsSecret: ISecret | undefined = clusterInfo.getResource(this.props.databaseSecretResourceName);
if (databaseCredentialsSecret === undefined) {
throw new Error("Database Secret not found in context");
}
const databaseInstanceCredentialsSecretName = databaseCredentialsSecret.secretName;
const externalSecret = new eks.KubernetesManifest(cluster.stack, "BackstageDatabaseExternalSecret", {
cluster: cluster,
manifest: [
{
apiVersion: "external-secrets.io/v1beta1",
kind: "ExternalSecret",
metadata: {
name: "external-backstage-db-secret",
namespace: this.props.namespace
},
spec: {
secretStoreRef: {
name: secretStoreName,
kind: "ClusterSecretStore",
},
target: {
name: this.props.databaseSecretTargetName,
},
data: [
{
secretKey: "POSTGRES_PASSWORD",
remoteRef: {
key: databaseInstanceCredentialsSecretName,
property: "password"
}
},
{
secretKey: "POSTGRES_USER",
remoteRef: {
key: databaseInstanceCredentialsSecretName,
property: "username"
}
},
],
},
},
],
});

externalSecret.node.addDependency(secretStore);
return Promise.resolve(secretStore);
}
}
30 changes: 30 additions & 0 deletions lib/backstage-construct/database-credentials.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
import { Secret,ISecret } from 'aws-cdk-lib/aws-secretsmanager';
import { ResourceContext, ResourceProvider } from '@aws-quickstart/eks-blueprints';

export interface DatabaseInstanceCredentialsProviderProps {
/**
* The username for the database secret
*/
username: string,
}

export class DatabaseInstanceCredentialsProvider implements ResourceProvider<ISecret> {
readonly props: DatabaseInstanceCredentialsProviderProps;

constructor(props: DatabaseInstanceCredentialsProviderProps) {
this.props = props;
}

provide(context: ResourceContext): ISecret {
return new Secret(context.scope, "database-secret", {
generateSecretString: {
secretStringTemplate: JSON.stringify({
username: this.props.username,
}),
excludePunctuation: true,
includeSpace: false,
generateStringKey: "password"
}
});
}
}
Loading