Skip to content
This repository has been archived by the owner on Mar 14, 2024. It is now read-only.

External data

Jump to bottom
Ewa edited this page Jun 1, 2022 · 3 revisions

(Googlers: this was originally designed at go/dcc-external-data).

developer.chrome.com relies on a wide variety of external data. It can be built locally via npm run build-external, which eventually ends up simply running all JS binaries inside "external/build/", with data expected to be written at "external/data/". This path is not checked-in to Git.

However, most developers will implicitly sync external data as part of npm run dev—which interally will run npm run sync-external. This will pull down the last known good version of this data from the gs://external-dcc-data Cloud Storage bucket, which is publicly readable.

Populating External Data

The chrome-apps-doc Google Cloud project has a Cloud Build action which runs every hour to build external data. This does:

  • npm run build-external to generate external data
  • runs a site build to confirm the site builds successfully
  • writes the data to the Cloud Storage bucket.

This process can fail—that's by design—and this will alert the team via the Stackdriver notice. However, the last known good version will not change, so authors will be unaffected.

Failure might occur because e.g., the format or expectations around the external data has changed. Developers on the site can run npm run build-external locally to debug the issue.

Implementation notes

There are more notes in the README of the external/ folder. This includes how to test changes and additions to the external build process.

Updates to data

⚠️ If you're making major changes to the data format, consider giving it a version number and changing its filename.

This is as any change you make might not be immediately built or synced to other clients, especially external data is only considered "out of date" after a few hours (defined in "maybe-sync-external.js").

Adding a new external data source

  1. Add a data build script to /external/build/

Write a data pipeline that fetches the data from an external source and writes to storage. Example: https://github.com/GoogleChrome/developer.chrome.com/blob/main/external/build/tweets.js

  1. Verify and upload to cloud storage

Run a Cloud Build task to confirm that the output builds correctly and to write it to cloud storage for other users.

gcloud builds submit --config .cloudbuild/external.yaml .

Note: It is important that we only sync new version of the file if it is proven to be sane & working (to avoid breaking dcc build by external dependencies). If new file is incompatible for some reason, do not update cloud storage - last sane version of the file will be used instead.

  1. Sync data between cloud storage and /external/data
npm run sync-external

This will populate /external/data/... with data files from cloud storage.

  1. Expose data from /external/data to eleventy

Expose data as an eleventy _data variable, by adding a file with logic in /site/_data folder.

Example: https://github.com/GoogleChrome/developer.chrome.com/blob/main/site/_data/tweets.js#L132

  1. Use exposed variable in an eleventy template

Use that variable in an .md or .njk template file.

Example: https://github.com/GoogleChrome/developer.chrome.com/blob/main/site/_includes/layouts/home.njk#L40

Clone this wiki locally