-
Notifications
You must be signed in to change notification settings - Fork 15
Self‐hosting the portal
The Code PushUp UI and API are both distributed as private Docker images. They're hosted by Code PushUp in GCP's Artifact Registry and use version tags.
To authorize Docker to pull these images, refer to GCP's docs in Configure authentication to Artifact Registry for Docker. A new IAM principal should be created by Code PushUp for each customer and given the Artifact Registry Reader role. This principal should usually be a service account. However, if the customer will be hosting the portal using Cloud Run in their own GCP project, then refer to docs on Deploying images from other Google Cloud projects instead.
IAM roles can be managed in the IAM page in the GCP console. Enable the Include Google-provided role grants checkbox to include Cloud Run service agents. Go to the Service accounts page to create service accounts and manage their keys.
For demo and testing purposes, you can run a fully isolated instance of the portal on any machine which has Docker Engine and Docker Compose installed. You should also follow the Configure authentication to Artifact Registry for Docker guide described in the Distribution section above, otherwise Docker won't be authorized to pull private images from Code PushUp.
You will need to configure environment variables for the portal. For convenience, store them in a .env
file. Then create a docker-compose.yml
file with the following content:
services:
# front-end - single-page app
ui:
image: europe-docker.pkg.dev/code-pushup/portal/portal-ui:latest
environment:
- API_URL=http://localhost:4000/graphql
ports:
- 8000:80
depends_on:
- api
# back-end - GraphQL API
api:
image: europe-docker.pkg.dev/code-pushup/portal/portal-api:latest
env_file:
- .env
environment:
- PORTAL_URL=http://localhost:8000
- MONGODB_URI=mongodb://db:27017
- MONGODB_IS_REPLICA_SET=false
- PORT=4000
ports:
- 4000:4000
restart: always
depends_on:
- db
# back-end - MongoDB database
db:
image: mongo:latest
env_file:
- .env
ports:
- 27017:27017
restart: always
volumes:
- db-data:/data/db
volumes:
db-data:
The official MongoDB Docker image is used in this example. The database will be empty initially, so you should create a first organization and project. There are two options:
-
Run scripts described in Adding organization and projects section.
-
Before running the container for the first time, add
./mongo-init.js:/docker-entrypoint-initdb.d/mongo-init.js:ro
tovolumes
array indb
service. Create amongo-init.js
script with the following content and add the following environment variables to your.env
file:mongo-init.js
db = db.getSiblingDB('qmdb'); const collection = db.organizations; console.log('Parsing environment variables...'); const data = parseVariables(); const exists = collection.countDocuments({ slug: data.slug }) > 0; if (exists) { console.log( `Organization with slug '${data.slug}' already exists, skipping document creation.`, ); } else { console.log('Inserting document into organizations collection...'); console.log(collection.insertOne(data)); } console.log('Organizations in database:'); console.log(collection.find({})); console.log('Setup complete.'); /************************ HELPER FUNCTIONS ************************/ /** * Validates environment variables and converts to organization document data. */ function parseVariables() { const { CP_ORGANIZATION_SLUG, CP_ORGANIZATION_FRIENDLY_NAME, CP_ORGANIZATION_ALLOWED_EMAILS, CP_PROJECT_SLUG, CP_PROJECT_FRIENDLY_NAME, CP_PROJECT_REPOSITORY_TYPE, CP_PROJECT_REPOSITORY_OWNER, CP_PROJECT_REPOSITORY_REPO, } = process.env; const slugRegex = /^[a-z0-9-]+$/; // inspired by https://www.regular-expressions.info/ const allowedEmailRegex = /^([A-Z0-9._%+-]+|\*)@[A-Z0-9.-]+\.[A-Z]{2,}$/i; if (!CP_ORGANIZATION_SLUG && !CP_ORGANIZATION_FRIENDLY_NAME) { throw new Error( 'One of CP_ORGANIZATION_SLUG and CP_ORGANIZATION_FRIENDLY_NAME is required', ); } if (CP_ORGANIZATION_SLUG && !slugRegex.test(CP_ORGANIZATION_SLUG)) { throw new Error( `CP_ORGANIZATION_SLUG may only include lowecase letters, digits and dashes - received ${CP_ORGANIZATION_SLUG}`, ); } if (!CP_ORGANIZATION_ALLOWED_EMAILS) { throw new Error('CP_ORGANIZATION_ALLOWED_EMAILS is required'); } if ( !CP_ORGANIZATION_ALLOWED_EMAILS.split(',').every(email => allowedEmailRegex.test(email), ) ) { throw new Error( `CP_ORGANIZATION_ALLOWED_EMAILS must be comma-separated list of email addresses (e.g. 'john.doe@example.com') or domain wildcards (e.g. '*@example.com') - received '${CP_ORGANIZATION_ALLOWED_EMAILS}'`, ); } if (!CP_PROJECT_SLUG && !CP_PROJECT_FRIENDLY_NAME) { throw new Error( 'One of CP_PROJECT_SLUG and CP_PROJECT_FRIENDLY_NAME is required', ); } if (CP_PROJECT_SLUG && !slugRegex.test(CP_PROJECT_SLUG)) { throw new Error( `CP_PROJECT_SLUG may only include lowecase letters, digits and dashes - received ${CP_PROJECT_SLUG}`, ); } if (!CP_PROJECT_REPOSITORY_TYPE) { throw new Error('CP_PROJECT_REPOSITORY_TYPE is required'); } if (!['GitHub', 'GitLab'].includes(CP_PROJECT_REPOSITORY_TYPE)) { throw new Error( `CP_PROJECT_REPOSITORY_TYPE must be one of 'GitHub' or 'GitLab' - received ${CP_PROJECT_REPOSITORY_TYPE}`, ); } if (!CP_PROJECT_REPOSITORY_OWNER) { throw new Error('CP_PROJECT_REPOSITORY_OWNER is required'); } if (!CP_PROJECT_REPOSITORY_REPO) { throw new Error('CP_PROJECT_REPOSITORY_REPO is required'); } return { slug: CP_ORGANIZATION_SLUG || slugify(CP_ORGANIZATION_FRIENDLY_NAME), ...(CP_ORGANIZATION_FRIENDLY_NAME && { friendlyName: CP_ORGANIZATION_FRIENDLY_NAME, }), allowedEmails: CP_ORGANIZATION_ALLOWED_EMAILS.split(','), projects: [ { _id: new ObjectId(), slug: CP_PROJECT_SLUG || slugify(CP_PROJECT_FRIENDLY_NAME), ...(CP_PROJECT_FRIENDLY_NAME && { friendlyName: CP_PROJECT_FRIENDLY_NAME, }), repository: { type: CP_PROJECT_REPOSITORY_TYPE, owner: CP_PROJECT_REPOSITORY_OWNER, repo: CP_PROJECT_REPOSITORY_REPO, }, }, ], }; } /** * Converts friendly name to slug. * @param {string} name Friendly name * @returns {string} Slug */ function slugify(name) { return name .replace(/[A-Z]/g, char => char.toLowerCase()) .replace(/\s+/g, '-') .replace(/[^a-z0-9-]/, ''); }
.env
# ... # replace these values CP_ORGANIZATION_SLUG=code-pushup CP_ORGANIZATION_FRIENDLY_NAME='Code PushUp' CP_ORGANIZATION_ALLOWED_EMAILS='*@flowup.cz,*@push-based.io' CP_PROJECT_SLUG=todos-app CP_PROJECT_FRIENDLY_NAME='Todos app' CP_PROJECT_REPOSITORY_TYPE=GitHub CP_PROJECT_REPOSITORY_OWNER=code-pushup CP_PROJECT_REPOSITORY_REPO=todos-app
Start up containers with docker compose up
. Visit http://localhost:8000 in your browser to interact with the portal UI. To configure uploads, use http://localhost:4000/graphql as the API URL.
The best way to deploy Docker images in Google Cloud is with Cloud Run.
Once the customer's Google Cloud project has been created, you'll need to find the Cloud Run service agent and copy its email address (should be in the format service-<project-id>@serverless-robot-prod.iam.gserviceaccount.com
), so that it can be added by Code PushUp side as a principal with Artifact Registry Reader role in order to authorize downloading the Docker images (for more info, refer to docs on Deploying images from other Google Cloud projects).
You can deploy to Cloud Run manually, but it is more future-proof to create a CI/CD pipeline, as it makes later updates easy to deploy. The gcloud CLI needs to be installed and for CI/CD you'll need to authorize a service account (e.g. via service account key or Workflow Identity Federation) which has at least Cloud Run Admin and Service Account User roles.
The command to deploy API to Cloud Run should then look something like this (refer to API environment configuration regarding --set-env-vars
):
gcloud run deploy code-pushup-portal-api \
--image=europe-docker.pkg.dev/code-pushup/portal/portal-api:latest \
--platform=managed \
--region=... \
--allow-unauthenticated \
--set-env-vars=PORTAL_URL=... \
--set-env-vars=MONGODB_URI=... \
--set-env-vars=MONGODB_IS_REPLICA_SET=.. \
--set-env-vars=GITLAB_HOST=... \
--set-env-vars=GITLAB_TOKEN=... \
--set-env-vars=EMAIL_SERVICE=... \
--set-env-vars=EMAIL_AUTH__USER=... \
--set-env-vars=EMAIL_AUTH__PASS=... \
--set-env-vars=HMAC_SECRET=...
And the command to deploy UI to Cloud Run should look something like this (refer to UI environment configuration regarding --set-env-vars
):
gcloud run deploy code-pushup-portal-ui \
--image=europe-docker.pkg.dev/code-pushup/portal/portal-ui:latest \
--platform=managed \
--region=europe-west1 \
--allow-unauthenticated \
--port=80 \
--set-env-vars=API_URL=...
GitHub Actions example
name: Deploy Code PushUp portal
on: push
jobs:
deploy_ui:
runs-on: ubuntu-latest
name: Deploy UI
steps:
- id: auth
name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.SERVICE_ACCOUNT_KEY }}
- name: Set up Google Cloud SDK
uses: google-github-actions/setup-gcloud@v2
- name: Deploy UI image to Cloud Run
run: |
gcloud run deploy code-pushup-portal-ui \
--image=europe-docker.pkg.dev/code-pushup/portal/portal-ui:latest \
--platform=managed \
--region=europe-west4 \
--allow-unauthenticated \
--port=80 \
--set-env-vars=API_URL=https://api.code-pushup.example.com/graphql
deploy_api:
runs-on: ubuntu-latest
name: Deploy API
steps:
- id: auth
name: Authenticate to Google Cloud
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.SERVICE_ACCOUNT_KEY }}
- name: Set up Google Cloud SDK
uses: google-github-actions/setup-gcloud@v2
- name: Deploy API image to Cloud Run
run: |
gcloud run deploy code-pushup-portal-api \
--image=europe-docker.pkg.dev/code-pushup/portal/portal-api:latest \
--platform=managed \
--region=europe-west4 \
--allow-unauthenticated \
--set-env-vars=PORTAL_URL=https://code-pushup.example.com \
--set-env-vars=MONGODB_URI=${{ secrets.MONGODB_URI }} \
--set-env-vars=MONGODB_IS_REPLICA_SET=true \
--set-env-vars=GITHUB_APP_ID=197378 \
--set-env-vars=GITHUB_APP_PRIVATE_KEY="${{ secrets.GH_APP_PRIVATE_KEY }}" \
--set-env-vars=EMAIL_HOST=smtp.gmail.com \
--set-env-vars=EMAIL_PORT=465 \
--set-env-vars=EMAIL_SECURE=true \
--set-env-vars=EMAIL_AUTH__TYPE=OAuth2 \
--set-env-vars=EMAIL_AUTH__USER=john.doe@example.com \
--set-env-vars=EMAIL_AUTH__SERVICE_CLIENT=107438341143996518602 \
--set-env-vars=EMAIL_AUTH__PRIVATE_KEY="${{ secrets.EMAIL_PRIVATE_KEY }}" \
--set-env-vars=HMAC_SECRET=${{ secrets.HMAC_SECRET }}
GitLab CI/CD example
# GCP Secrets Manager configuration: https://docs.gitlab.com/ee/ci/secrets/gcp_secret_manager.html
variables:
GCP_PROJECT_NUMBER: 625211858852
GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID: gitlab-pool
GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID: gitlab-provider
.setup: &setup
image: registry.example.com:5005/platform/runner/terraform:latest
id_tokens:
GCP_ID_TOKEN:
aud: https://iam.googleapis.com/projects/${GCP_PROJECT_NUMBER}/locations/global/workloadIdentityPools/${GCP_WORKLOAD_IDENTITY_FEDERATION_POOL_ID}/providers/${GCP_WORKLOAD_IDENTITY_FEDERATION_PROVIDER_ID}
secrets:
DEPLOY_SA_KEY_PATH:
gcp_secret_manager:
name: CP_DEPLOY_SA_KEY
token: $GCP_ID_TOKEN
MONGODB_URI_PATH:
gcp_secret_manager:
name: CP_MONGODB_URI
token: $GCP_ID_TOKEN
GITLAB_TOKEN_PATH:
gcp_secret_manager:
name: CP_GITLAB_TOKEN
token: $GCP_ID_TOKEN
GMAIL_APP_PASSWD_PATH:
gcp_secret_manager:
name: CP_GMAIL_APP_PASSWD
token: $GCP_ID_TOKEN
HMAC_SECRET_PATH:
gcp_secret_manager:
name: CP_HMAC_SECRET
token: $GCP_ID_TOKEN
before_script:
- cp $DEPLOY_SA_KEY_PATH /etc/key-file.json
- gcloud auth activate-service-account --key-file=/etc/key-file.json
- rm /etc/key-file.json
- gcloud config set project code-pushup-88f57892
deploy-api:
<<: *setup
script:
- |
gcloud run deploy code-pushup-portal-api \
--image=europe-docker.pkg.dev/code-pushup/portal/portal-api:latest \
--platform=managed \
--region=europe-west1 \
--allow-unauthenticated \
--set-env-vars=PORTAL_URL=https://code-pushup.example.com \
--set-env-vars=MONGODB_URI=$(cat $MONGODB_URI_PATH) \
--set-env-vars=MONGODB_IS_REPLICA_SET=true \
--set-env-vars=GITLAB_HOST=https://gitlab.example.com \
--set-env-vars=GITLAB_TOKEN=$(cat $GITLAB_TOKEN_PATH) \
--set-env-vars=EMAIL_SERVICE=gmail \
--set-env-vars=EMAIL_AUTH__USER=code.pushup@example.com \
--set-env-vars=EMAIL_AUTH__PASS=$(cat $GMAIL_APP_PASSWD_PATH) \
--set-env-vars=HMAC_SECRET=$(cat $HMAC_SECRET_PATH)
deploy-ui:
<<: *setup
script:
- |
gcloud run deploy code-pushup-portal-ui \
--image=europe-docker.pkg.dev/code-pushup/portal/portal-ui:latest \
--platform=managed \
--region=europe-west1 \
--allow-unauthenticated \
--port=80 \
--set-env-vars=API_URL=https://api.code-pushup.example.com/graphql
You will probably want to configure custom domains because the Cloud Run URLs aren't very memorable. For available options, refer to Cloud Run's Mapping custom domains docs.
For hosting the database, the recommended way is to use MongoDB Atlas on Google Cloud - for more information refer to MongoDB environment configuration for portal.
Once you establish a database connection, you should initialize the empty database with an organization and a project to get started, as described in Adding organization and projects.
Since adding projects and organizations isn't yet part of the portal UI, some scripts are needed to create organizations and projects in the database.
Create a new Node project with npm init -y
and then install MongoDB (database driver) and Inquirer (terminal prompts) with npm install mongodb inquirer
. Then copy the following ESM scripts:
add-organization.mjs
import inquirer from 'inquirer';
import * as mongodb from 'mongodb';
if (!process.env['MONGODB_URI']) {
throw new Error('Missing MONGODB_URI environment variable');
}
const mongoClient = new mongodb.MongoClient(process.env['MONGODB_URI']);
mongoClient.connect();
inquirer
.prompt([
{
name: 'organizationName',
type: 'input',
message: 'Friendly name for your organization:',
},
{
name: 'organizationSlug',
type: 'input',
message: 'Slug for your organization:',
default: answers =>
answers.organizationName
.replace(/[A-Z]/g, char => char.toLowerCase())
.replace(/\s+/g, '-')
.replace(/[^a-z0-9-]/, ''),
validate: input =>
/^[a-z0-9-]+$/.test(input) ||
'Invalid slug - only lowercase letters, numbers and dashes are permitted',
},
{
name: 'allowedEmails',
type: 'input',
message:
'Insert emails that should have access to this organization separated by "," (wildcards are allowed, e.g. "*@example.com"):',
filter: input => input.split(',').map(email => email.trim()),
},
])
.then(answers =>
mongoClient.db('qmdb').collection('organizations').insertOne({
slug: answers.organizationSlug,
friendlyName: answers.organizationName,
allowedEmails: answers.allowedEmails,
projects: [],
}),
)
.then(() => {
console.info('Organization added ✔️');
return mongoClient.close();
});
add-project.mjs
import inquirer from 'inquirer';
import * as mongodb from 'mongodb';
if (!process.env['MONGODB_URI']) {
throw new Error('Missing MONGODB_URI environment variable');
}
const mongoClient = new mongodb.MongoClient(process.env['MONGODB_URI']);
mongoClient.connect();
const db = mongoClient.db('qmdb');
const REPO_CHOICES = ['GitHub', 'GitLab'];
inquirer
.prompt([
{
name: 'organization',
type: 'list',
message: 'Pick an organization:',
choices: async () => {
const organizations = await db
.collection('organizations')
.find()
.toArray();
return organizations.map(({ slug }) => slug);
},
},
{
name: 'projectName',
type: 'input',
message: 'Friendly name for your project:',
},
{
name: 'projectSlug',
type: 'input',
message: 'Slug for your project:',
default: answers =>
answers.projectName
.replace(/[A-Z]/g, char => char.toLowerCase())
.replace(/\s+/g, '-')
.replace(/[/_\\=]/g, '-')
.replace(/[^a-z0-9-]/g, ''),
validate: input =>
/^[a-z0-9-]+$/.test(input) ||
'Invalid slug - only lowercase letters, numbers and dashes are permitted',
},
{
name: 'repoType',
type: 'list',
message: 'Where is your repository hosted?',
choices: REPO_CHOICES,
},
{
name: 'owner',
type: 'input',
message: 'Repository owner',
},
{
name: 'repo',
type: 'input',
message: 'Repository name',
},
])
.then(answers =>
mongoClient
.db('qmdb')
.collection('organizations')
.updateOne(
{ slug: answers.organization },
{
$push: {
projects: {
_id: new mongodb.ObjectId(),
slug: answers.projectSlug,
friendlyName: answers.projectName,
repository: {
type: answers.repoType,
owner: answers.owner,
repo: answers.repo,
},
},
},
},
),
)
.then(() => {
console.info('Project added ✔️');
return mongoClient.close();
});
To create an organization, run node add-organization.mjs
and fill in the prompts:
- give the organization a friendly name (something human readable),
- accept or adjust the auto-generated slug (identifier, referred to in
code-pushup.config.ts
'supload.organization
), - set list of emails or domains which should have access to organization (domains can be set using
*@example.com
syntax).
To create a project, run node add-project.mjs
and fill in the prompts:
- select which organization the new project should be placed under (inherits access),
- give the project a friendly name and a slug (project slug referred to in
code-pushup.config.ts
'supload.project
), - select where the project's repository is hosted (
GitHub
orGitLab
are supported at present), - set the GitHub/GitLab owner and repository
- e.g. for GitHub hosted repo at
https://github.com/example/website
, the owner isexample
and the repo iswebsite
- e.g. for self-hosted GitLab repo at
https://gitlab.example.com/example/marketing/website
, the owner isexample/marketing
and the repo iswebsite
- e.g. for GitHub hosted repo at
Warning
The script doesn't check if your repository configuration is valid. Ensure the owner and repo exist, and that the portal has access to them: