migrate from netlify to GCS

[atvaccaro]

We briefly discussed future plans for the portfolio site awhile back (e.g. #715) and we decided it's time to finally start exploring this migration. We've struggled with netlify recently (specifically deploying individual sites under redirects) and we don't have paid support, so it's probably worthwhile to just migrate to a static site served from a GCS bucket.

The general steps are:

1. Create a new bucket and configure it as a GCP-hosted static site behind a load balancer; see this document created by the services team for gtfs.calitp.org
2. Change portfolio.py to write to the proper subpaths in the bucket for individual sites; this would involve replacing netlify deploy with gsutil or gcsfs
3. (Potential) May have to configure JupyterBook to render links etc. under a subpath
4. Remove _redirects and modify index.html if needed to function properly as the bucket home page
5. (Optional) Configure CI to allow deploys with a button (i.e. workflow dispatches) for at least the index

[mjumbewu]

I'm not sure whether it is new since the Cal-ITP GTFS Hosting doc was written, but GCP has a set of steps for this at https://cloud.google.com/storage/docs/hosting-static-website#command-line_1. The general steps above still apply, but that documentation is useful for the specific gcloud commands.

• Create a new bucket and configure it as a GCP-hosted static site behind a load balancer; see the GCS docs

  Looking through the existing buckets (for naming patterns), there are three that are public:

  • calitp-map-tiles which (I hope) contains map tiles
  • calitp-publish-data-analysis which contains downloadable data assets that are useful for analysis
  • calitp-metabase-data-public which contains GeoJSON boundary files for metabase to use in aggregations

  I'll stick to the calitp- prefixing and call this one calitp-data-analyses-portfolio. Note that a pattern that some use is to use the domain name as the bucket name (or as a subset of the bucket name), and this does not follow that pattern.

  Command:

  gcloud storage buckets create calitp-data-analyses-portfolio \
    --project=cal-itp-data-infra \
    --location=us-west2
    
  gcloud storage buckets add-iam-policy-binding gs://calitp-data-analyses-portfolio \
    --member=allUsers \
    --role=roles/storage.objectViewer

• Create a GCP role and a service account that can be used to deploy from GH Actions

  This role should be set up similar to the GTFS flex hosting IAM configuration. Ideally these would be configured as IAC with terraform in the data-infra repo.

  • Create IAM role

    ID projects/cal-itp-data-infra/roles/CustomGCSPublisher

    8 assigned permissions

    • resourcemanager.projects.get
    • storage.buckets.get
    • storage.buckets.list
    • storage.objects.create
    • storage.objects.delete
    • storage.objects.get
    • storage.objects.list
    • storage.objects.update

    Command:

    gcloud iam roles create CustomGCSPublisher \
      --project=cal-itp-data-infra \
      --title="Custom GCS Publisher" \
      --description="Custom role for publishing to GCS" \
      --permissions=resourcemanager.projects.get,storage.buckets.get,storage.buckets.list,storage.objects.create,storage.objects.delete,storage.objects.get,storage.objects.list,storage.objects.update

  • Create Service account

    Email gh-actions-publisher@cal-itp-data-infra.iam.gserviceaccount.com

    Assign Role projects/cal-itp-data-infra/roles/CustomGCSPublisher

    Command:

    gcloud iam service-accounts create gh-actions-publisher \
      --project=cal-itp-data-infra \
      --description="Service account for GH Actions to publish to GCS" \
      --display-name="GH Actions Publisher"

  • Create Service account key

    gcloud iam service-accounts keys create ~/key.json \
      --iam-account gh-actions-publisher@cal-itp-data-infra.iam.gserviceaccount.com

  • Add the key to the GH repo secrets

    • GCS_BUCKET: calitp-data-analyses-portfolio
    • GCS_CREDENTIALS: cat ~/key.json | base64

• Change portfolio.py to write to the proper subpaths in the bucket for individual sites; this would involve replacing netlify deploy with gcloud storage cp

• (Potential) May have to configure JupyterBook to render links etc. under a subpath

• Remove _redirects and modify index.html if needed to function properly as the bucket home page

• (Optional) Configure CI to allow deploys with a button (i.e. workflow dispatches) for at least the index

[ohrite]

@mjumbewu once you have GCS credentials set up, let's make sure there's a runbook about GitHub GCS credential rotation as part of acceptance. Maybe this is something to pair on?