feat(gatsby): add assetPrefix to support deploying assets separate fr…

…om html (#12128)

* chore: add validation schema, start tweaking webpack config

* chore: keep iterating

* chore: yadda

* chore: keep working

* keep doing stuff

* chore: get mostly done (let's see!)

* chore: remove unused package

* chore: ensure url is normalized correctly

* chore: try try again

* chore: fix for base path

* test: tests are important; fix them

* chore: remove a silly change

* chore: fix linter

Note: this should've been fine

* fix(gatsby-plugin-offline): hard fail if assetPrefix is used

Note: very possible this may be reverted

* refactor: add a publicPath helper

* test: add some get public path tests

* chore: use correct name

* docs: add asset prefix doc, and tweak path prefix

* chore: allow relative url for assetPrefix

* test: add a few more unit tests

* test: clean up test

* chore: fix e2e-test

Note: this is a bug, will fix the underlying bug too. pathPrefix should
have no effect unless using --prefix-paths

* fix: fall back to empty string, not slash

* Update docs/docs/asset-prefix.md

* fix: handle relative paths

* feat: add withAssetPrefix helper for gatsby-link

This should rarely be used--but should be exposed

* fix: use withAssetPrefix (if available) for gatsby-plugin-manifest

* Allow using gatsby-plugin-offline with assetPrefix

* Add docs for using offline-plugin with asset prefix

* clarify docs

* feat(*): use withAssetPrefix helper from gatsby-link

BREAKING CHANGE: this is a breaking change (as currently authored) for a
few plugins (specified in this commit). I'll work on a fallback--but I
think it might make sense to just fail here. We can specify a
peerDependency in the package.json of each of these packages, too.

* test: get tests passing

* test: add a test for assetPrefix with nesting

* Update docs/docs/path-prefix.md

Co-Authored-By: DSchau <DSchau@users.noreply.github.com>

* chore: fix up merge conflicts/get tests passing

* chore: tweak version

* fix(gatsby-plugin-sitemap): work with asset prefix

* fix(gatsby): disallow both relative assetPrefix and pathPrefix

* chore: fallback to withPathPrefix, bump peerDep

* chore: remove caveat re: trailing slash

* fix: gatsby-plugin-sitemap regression

* chore: revert peer dep

* chore: use basePath if it's defined

* chore: remove eslint global comment

* chore: ensure prefixPaths is set to enable pathPrefix

* chore: fix read-only error (can't reassign imports ya dingus)

* chore: actually fallback

* Update docs/docs/asset-prefix.md

Co-Authored-By: DSchau <DSchau@users.noreply.github.com>

* Update docs/docs/path-prefix.md

Co-Authored-By: DSchau <DSchau@users.noreply.github.com>

* Update docs/docs/asset-prefix.md

Co-Authored-By: DSchau <DSchau@users.noreply.github.com>

* chore: simply/merely remove the easy term ;)

* Update docs/docs/asset-prefix.md

Co-Authored-By: DSchau <DSchau@users.noreply.github.com>

* test: write e2e test for asset prefix

Note: this very well may fail

* chore: fix package json and make isURL test stricter

* chore: fix yarn and stuff hopefully

* chore: minor clean up

* fix(gatsby): fix initial navigation not registering in history

* chore: remove unneccessary dep

* fix: use __BASE_PATH__ in development runtime too; add a test

* chore: fix @pieh nit before he finds it

.eslintrc.json

@@ -26,7 +26,9 @@
   "globals": {
     "before": true,
     "spyOn": true,
-    "__PATH_PREFIX__": true
+    "__PATH_PREFIX__": true,
+    "__BASE_PATH__": true,
+    "__ASSET_PREFIX__": true
   },
   "rules": {
     "arrow-body-style": [

docs/blog/2019-02-08-government-open-data-site-with-gatsby/index.md

@@ -223,8 +223,7 @@ We set a `BASEURL` environment variable in `gatsby-config.js` that resolves the
 const BASEURL = process.env.BASEURL || ""
 
 module.exports = {
-  // Note: it must *not* have a trailing slash.
-  // This is currently the realtive path in our Jekyll deployment. This path points to our Gatsby pages.
+  // This is currently the relative path in our Jekyll deployment. This path points to our Gatsby pages.
   // This prefix is prepended to load all our related images, code, and pages.
   pathPrefix: `${BASEURL}/gatsby-public`,
 }

docs/docs/asset-prefix.md

@@ -0,0 +1,96 @@
+---
+title: Adding an Asset Prefix
+---
+
+Gatsby produces static content that can be hosted _anywhere_ at scale in a cost-effective manner. This static content is comprised of HTML files, JavaScript, CSS, images, and more that power your great Gatsby application.
+
+In some circumstances you may want to deploy _assets_ (non-HTML resources such as JavaScript, CSS, etc.) to a separate domain. Typically this is when you're required to use a dedicated CDN for assets or need to follow company-specific hosting policies.
+
+This `assetPrefix` functionality is available starting in gatsby@2.4.0, so that you can seamlessly use Gatsby with assets hosted from a separate domain. To use this functionality, ensure that your version of `gatsby` specified in `package.json` is at least `2.4.0`.
+
+## Usage
+
+### Adding to `gatsby-config.js`
+
+```js:title=gatsby-config.js
+module.exports = {
+  assetPrefix: `https://cdn.example.com`,
+}
+```
+
+One more step - when we build out this application, we need to add a flag so that Gatsby picks up this option.
+
+### The `--prefix-paths` flag
+
+When building with an `assetPrefix`, we require a `--prefix-paths` flag. If this flag is not specified, the build will ignore this option, and build out content as if it was hosted on the same domain. To ensure we build out successfully, use the following command:
+
+```shell
+gatsby build --prefix-paths
+```
+
+That's it! We now have an application that is ready to have its assets deployed from a CDN and its core files (e.g. HTML files) can be hosted on a separate domain.
+
+## Building / Deploying
+
+Once your application is built out, all assets will be automatically prefixed by this asset prefix. For example, if we have a JavaScript file `app-common-1234.js`, the script tag will look something like:
+
+```html
+<script src="https://cdn.example.com/app-common-1234.js"></script>
+```
+
+However - if we were to deploy our application as-is, those assets would not be available! We can do this in a few ways, but the general approach will be to deploy the contents of the `public` folder to _both_ your core domain, and the CDN/asset prefix location.
+
+### Using `onPostBuild`
+
+We expose an [`onPostBuild`](/docs/node-apis/#onPostBuild) API hook. This can be used to deploy your content to the CDN, like so:
+
+```js:title=gatsby-node.js
+const assetsDirectory = `public`
+
+exports.onPostBuild = async function onPostBuild() {
+  // do something with public
+  // e.g. upload to S3
+}
+```
+
+### Using `package.json` scripts
+
+Additionally, we can use an npm script, which will let us use some command line interfaces/executables to perform some action, in this case, deploying our assets directory!
+
+In this example, I'll use the `aws-cli` and `s3` to sync the `public` folder (containing all our assets) to the `s3` bucket.
+
+```json:title=package.json
+{
+  "scripts": {
+    "build": "gatsby build --prefix-paths",
+    "postbuild": "aws s3 sync public s3://mybucket"
+  }
+}
+```
+
+Now whenever the `build` script is invoked, e.g. `npm run build`, the `postbuild` script will be invoked _after_ the build completes, therefore making our assets available on a _separate_ domain after we have finished building out our application with prefixed assets.
+
+## Additional Considerations
+
+### Usage with `pathPrefix`
+
+The [`pathPrefix`](/docs/path-prefix/) feature can be thought of as semi-related to this feature. That feature allows _all_ your website content to be prefixed with some constant prefix, for example you may want your blog to be hosted from `/blog` rather than the project root.
+
+This feature works seamlessly with `pathPrefix`. Build out your application with the `--prefix-paths` flag and you'll be well on your way to hosting an application with its assets hosted on a CDN, and its core functionality available behind a path prefix.
+
+### Usage with `gatsby-plugin-offline`
+
+When using a custom asset prefix with `gatsby-plugin-offline`, your assets can still be cached offline. However, to ensure the plugin works correctly, there are a few things you need to do.
+
+1. Your asset server needs to have the following headers set:
+
+   ```
+   Access-Control-Allow-Origin: <site origin>
+   Access-Control-Allow-Credentials: true
+   ```
+
+   Note that the origin needs to be specific, rather than using `*` to allow all origins. This is because Gatsby makes requests to fetch resources with `withCredentials` set to `true`, which disallows using `*` to match all origins. This is also why the second header is required. For local testing, use `http://localhost:9000` as the origin.
+
+2. Certain essential resources need to be available on your content server (i.e. the one used to serve pages). This includes `sw.js`, as well as resources to precache: the Webpack bundle, the app bundle, the manifest (and any icons referenced), and the resources for the offline plugin app shell.
+
+   You can find most of these by looking for the `self.__precacheManifest` variable in your generated `sw.js`. Remember to also include `sw.js` itself, and any icons referenced in your `manifest.webmanifest` if you have one. To check your service worker is functioning as expected, look in Application → Service Workers in your browser dev tools, and check for any failed resources in the Console/Network tabs.

docs/docs/gatsby-config.md

@@ -63,7 +63,6 @@ It's common for sites to be hosted somewhere other than the root of their domain
 
 ```javascript:title=gatsby-config.js
 module.exports = {
-  // Note: it must *not* have a trailing slash.
   pathPrefix: `/blog`,
 }
 ```

docs/docs/path-prefix.md

@@ -2,46 +2,82 @@
 title: Adding a Path Prefix
 ---
 
-Many sites are hosted at something other than the root of their domain.
+Many applications are hosted at something other than the root (`/`) of their domain.
 
-E.g. a Gatsby blog could live at `example.com/blog/` or a site could be hosted
-on GitHub Pages at `example.github.io/my-gatsby-site/`
+For example, a Gatsby blog could live at `example.com/blog/` or a site could be hosted on GitHub Pages at `example.github.io/my-gatsby-site/`
 
 Each of these sites need a prefix added to all paths on the site. So a link to
 `/my-sweet-blog-post/` should be rewritten to `/blog/my-sweet-blog-post`.
 
-In addition links to various resources (JavaScript, images, CSS) need the same
-prefix added (this is accomplished by setting the `publicPath` in webpack).
+In addition links to various resources (JavaScript, CSS, images, and other static content) need the same prefix, so that the site continues to function and display correctly, even if served from this path prefix.
 
-Luckily, for most sites, this work can be offloaded to Gatsby. Using
-[Gatsby's Link component](/docs/gatsby-link/) for internal links ensures those links
-will be prefixed correctly. Gatsby ensures that paths created internally and by
-webpack are also correctly prefixed.
+Let's get this functionality implemented. We'll add an option to our `gatsby-config.js`, and add a flag to our build command.
 
-## Development
+### Add to `gatsby-config.js`
 
-During development, write paths as if there was no path prefix e.g. for a blog
-hosted at `example.com/blog`, don't add `/blog` to your links. The prefix will
-be added when you build for deployment.
-
-## Production build
-
-There are two steps for building a site with path prefixes.
-
-First define the prefix in your site's `gatsby-config.js`.
-
-```javascript:title=gatsby-config.js
+```js:title=gatsby-config.js
 module.exports = {
-  // Note: it must *not* have a trailing slash.
   pathPrefix: `/blog`,
 }
 ```
 
-Then pass `--prefix-paths` cmd option to Gatsby.
+### Build
+
+Once the `pathPrefix` is specified in `gastby-config.js`, we are well on our way to a prefixed app. The final step is to build out your application with a flag `--prefix-paths`, like so:
 
 ```shell
 gatsby build --prefix-paths
 ```
 
-NOTE: When running the command without the `--prefix-paths` flag, Gatsby ignores
-your `pathPrefix`.
+If this flag is not passed, Gatsby will ignore your `pathPrefix` and build out your site as if it were hosted from the root domain.
+
+### In-app linking
+
+As a developer using this feature, it should be seamless. We provide APIs and libraries to make using this functionality a breeze. Specifically, the [`Link`](/docs/gatsby-link/) component has built-in functionality to handle path prefixing.
+
+For example, if we want to link to our `/page-2` link (but the actual link will be prefixed, e.g. `/blog/page-2`) we don't want to hard code this path prefix in all of our links. We have your back! By using the `Link` component, we will automatically prefix your paths for you. If you later migrate off of `pathPrefix` your links will _still_ work seamlessly.
+
+Let's look at a quick example.
+
+```jsx:title=src/pages/index.js
+import React from "react"
+import { Link } from "gatsby"
+import Layout from "../components/layout"
+
+function Index() {
+  return (
+    <Layout>
+      {/* highlight-next-line */}
+      <Link to="page-2">Page 2</Link>
+    </Layout>
+  )
+}
+```
+
+Without doing _anything_ and merely using the `Link` component, this link will be prefixed with our specified `pathPrefix` in `gatsby-config.js`. Woo hoo!
+
+If we want to do programatic/dynamic navigation, totally possible too! We expose a `navigate` helper, and this too automatically handles path prefixing.
+
+```jsx:title=src/pages/index.js
+import React from "react"
+import { navigate } from "gatsby"
+import Layout from "../components/layout"
+
+export default function Index() {
+  return (
+    <Layout>
+      {/* Note: this is an intentionally contrived example, but you get the idea! */}
+      {/* highlight-next-line */}
+      <button onClick={() => navigate("/page-2")}>
+        Go to page 2, dynamically
+      </button>
+    </Layout>
+  )
+}
+```
+
+### Additional Considerations
+
+The [`assetPrefix`](/docs/asset-prefix/) feature can be thought of as semi-related to this feature. That feature allows your assets (non-HTML files, e.g. images, JavaScript, etc.) to be hosted on a separate domain, for example a CDN.
+
+This feature works seamlessly with `assetPrefix`. Build out your application with the `--prefix-paths` flag and you'll be well on your way to hosting an application with its assets hosted on a CDN, and its core functionality available behind a path prefix.

e2e-tests/development-runtime/cypress/integration/navigation/linking.js

@@ -25,6 +25,16 @@ describe(`navigation`, () => {
     cy.location(`pathname`).should(`equal`, `/`)
   })
 
+  it(`can navigate back using history`, () => {
+    cy.getTestElement(`page-two`)
+      .click()
+      .waitForRouteChange()
+
+    cy.go(`back`).waitForRouteChange()
+
+    cy.location(`pathname`).should(`equal`, `/`)
+  })
+
   describe(`non-existant route`, () => {
     beforeEach(() => {
       cy.getTestElement(`broken-link`)

e2e-tests/gatsby-image/gatsby-config.js

@@ -1,7 +1,6 @@
 const path = require(`path`)
 
 module.exports = {
-  pathPrefix: `/blog`,
   siteMetadata: {
     title: `Gatsby Image e2e`,
   },

e2e-tests/path-prefix/.gitignore

@@ -3,9 +3,10 @@
 node_modules
 yarn-error.log
 
-# Build directory
+# Build assets
 /public
 .DS_Store
+/assets
 
 # Cypress output
 cypress/videos/

e2e-tests/path-prefix/cypress/integration/asset-prefix.js

@@ -0,0 +1,52 @@
+const { assetPrefix } = require(`../../gatsby-config`)
+
+const assetPrefixExpression = new RegExp(`^${assetPrefix}`)
+
+const assetPrefixMatcher = (chain, attr = `href`) =>
+  chain.should(`have.attr`, attr).and(`matches`, assetPrefixExpression)
+
+describe(`assetPrefix`, () => {
+  beforeEach(() => {
+    cy.visit(`/`).waitForRouteChange()
+  })
+
+  describe(`runtime`, () => {
+    it(`prefixes preloads`, () => {
+      assetPrefixMatcher(cy.get(`head link[rel="preload"]`))
+    })
+
+    it(`prefixes styles`, () => {
+      assetPrefixMatcher(cy.get(`head style[data-href]`), `data-href`)
+    })
+
+    it(`prefixes scripts`, () => {
+      assetPrefixMatcher(cy.get(`body script[src]`), `src`)
+    })
+  })
+
+  describe(`gatsby-plugin-manifest`, () => {
+    it(`prefixes manifest`, () => {
+      assetPrefixMatcher(cy.get(`head link[rel="manifest"]`))
+    })
+
+    it(`prefixes shortcut icon`, () => {
+      assetPrefixMatcher(cy.get(`head link[rel="shortcut icon"]`))
+    })
+
+    it(`prefixes manifest icons`, () => {
+      assetPrefixMatcher(cy.get(`head link[rel="apple-touch-icon"]`))
+    })
+  })
+
+  describe(`gatsby-plugin-sitemap`, () => {
+    it(`prefixes sitemap`, () => {
+      assetPrefixMatcher(cy.get(`head link[rel="sitemap"]`))
+    })
+  })
+
+  describe(`gatsby-plugin-feed`, () => {
+    it(`prefixes RSS feed`, () => {
+      assetPrefixMatcher(cy.get(`head link[type="application/rss+xml"]`))
+    })
+  })
+})

e2e-tests/path-prefix/cypress/integration/path-prefix.js

@@ -39,8 +39,8 @@ describe(`Production pathPrefix`, () => {
       cy.getTestElement(`page-2-link`)
         .click()
         .waitForRouteChange()
-
-      cy.go(`back`).waitForRouteChange()
+        .go(`back`)
+        .waitForRouteChange()
 
       cy.location(`pathname`).should(`eq`, withTrailingSlash(pathPrefix))
     })

e2e-tests/path-prefix/cypress/plugins/index.js

@@ -0,0 +1,17 @@
+// ***********************************************************
+// This example plugins/index.js can be used to load plugins
+//
+// You can change the location of this file or turn off loading
+// the plugins file with the 'pluginsFile' configuration option.
+//
+// You can read more here:
+// https://on.cypress.io/plugins-guide
+// ***********************************************************
+
+// This function is called when a project is opened or re-opened (e.g. due to
+// the project's config changing)
+
+module.exports = (on, config) => {
+  // `on` is used to hook into various events Cypress emits
+  // `config` is the resolved Cypress config
+}