-
Notifications
You must be signed in to change notification settings - Fork 1.6k
External data
(Googlers: this was originally designed at go/dcc-external-data).
developer.chrome.com relies on a wide variety of external data.
It can be built locally via npm run build-external
, which eventually ends up simply running all JS binaries inside "external/build/", with data expected to be written at "external/data/".
This path is not checked-in to Git.
However, most developers will implicitly sync external data as part of npm run dev
—which interally will run npm run sync-external
.
This will pull down the last known good version of this data from the gs://external-dcc-data
Cloud Storage bucket, which is publicly readable.
The chrome-apps-doc
Google Cloud project has a Cloud Build action which runs every hour to build external data.
This does:
-
npm run build-external
to generate external data - runs a site build to confirm the site builds successfully
- writes the data to the Cloud Storage bucket.
This process can fail—that's by design—and this will alert the team via the Stackdriver notice. However, the last known good version will not change, so authors will be unaffected.
Failure might occur because e.g., the format or expectations around the external data has changed.
Developers on the site can run npm run build-external
locally to debug the issue.
There are more notes in the README of the external/ folder. This includes how to test changes and additions to the external build process.
This is as any change you make might not be immediately built or synced to other clients, especially external data is only considered "out of date" after a few hours (defined in "maybe-sync-external.js").
- Add a data build script to
/external/build/
Write a data pipeline that fetches the data from an external source and writes to storage. Example: https://github.com/GoogleChrome/developer.chrome.com/blob/main/external/build/tweets.js
- Verify and upload to cloud storage
Run a Cloud Build task to confirm that the output builds correctly and to write it to cloud storage for other users.
gcloud builds submit --config .cloudbuild/external.yaml .
Note: It is important that we only sync new version of the file if it is proven to be sane & working (to avoid breaking dcc build by external dependencies). If new file is incompatible for some reason, do not update cloud storage - last sane version of the file will be used instead.
- Sync data between cloud storage and
/external/data
npm run sync-external
This will populate /external/data/...
with data files from cloud storage.
- Expose data from
/external/data
to eleventy
Expose data as an eleventy _data variable, by adding a file with logic in /site/_data
folder.
Example: https://github.com/GoogleChrome/developer.chrome.com/blob/main/site/_data/tweets.js#L132
- Use exposed variable in an eleventy template
Use that variable in an .md
or .njk
template file.