index.html cache issue

[ketysek]

Library Affected:
workbox-webpack-plugin@4.2.0 with workbox-build@4.3.1

Browser & Platform:
all browsers

Issue or Feature Request Description:
Hi there, we've got service worker enabled for caching purposes and we also have notification bar in app which informs user about its new version. This notification bar also includes button to update app by click :-)

registerServiceWorker.js of CRA framework with little modifications has following code:

const isLocalhost = Boolean(
  window.location.hostname === "localhost" ||
    // [::1] is the IPv6 localhost address.
    window.location.hostname === "[::1]" ||
    // 127.0.0.1/8 is considered localhost for IPv4.
    window.location.hostname.match(/^127(?:\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}$/)
)

export default function register(config) {
  if (process.env.NODE_ENV === "production" && "serviceWorker" in navigator) {
    // The URL constructor is available in all browsers that support SW.
    const publicUrl = new URL(process.env.PUBLIC_URL, window.location)
    if (publicUrl.origin !== window.location.origin) {
      // Our service worker won't work if PUBLIC_URL is on a different origin
      // from what our page is served on. This might happen if a CDN is used to
      // serve assets; see https://github.com/facebookincubator/create-react-app/issues/2374
      return
    }

    window.addEventListener("load", () => {
      let isInitialServiceWorker = false
      if (!navigator.serviceWorker.controller) {
        isInitialServiceWorker = true
      }
      const swUrl = `${process.env.PUBLIC_URL}/service-worker.js`

      if (isLocalhost) {
        // This is running on localhost. Lets check if a service worker still exists or not.
        checkValidServiceWorker(swUrl, config)
      } else {
        // Is not local host. Just register service worker
        registerValidSW(swUrl, config)
      }

      navigator.serviceWorker.addEventListener("controllerchange", () => {
        if (navigator.serviceWorker.controller) {
          if (!isInitialServiceWorker) {
            window.location.reload()
          }
        }
      })
    })
  }
}

function registerValidSW(swUrl, config) {
  navigator.serviceWorker
    .register(swUrl)
    .then(registration => {
      registration.onupdatefound = () => {
        const installingWorker = registration.installing
        if (installingWorker == null) {
          return
        }
        installingWorker.onstatechange = () => {
          if (installingWorker.state === "installed") {
            if (navigator.serviceWorker.controller) {
              // At this point, the old content will have been purged and
              // the fresh content will have been added to the cache.
              // It's the perfect time to display a "New content is
              // available; please refresh." message in your web app.
              console.log("New content is available; please refresh.")
              if (config && config.onUpdate) {
                config.onUpdate(registration)
              }
            } else {
              // At this point, everything has been precached.
              // It's the perfect time to display a
              // "Content is cached for offline use." message.
              console.log("Content is cached for offline use.")
            }
          }
        }
      }
      if (registration.waiting === null) {
        setInterval(() => registration.update(), 3600000)
      } else {
        // this needs to be done because of firefox, is it doesn't have
        // serviceWorker in installing state before app load
        const waitingWorker = registration.waiting
        if (waitingWorker.state === "installed") {
          if (navigator.serviceWorker.controller) {
            console.log("New content is available; please refresh.")
            if (config && config.onUpdate) {
              config.onUpdate(registration)
            }
          }
        }
      }
    })
    .catch(error => {
      console.error("Error during service worker registration:", error)
    })
}

function checkValidServiceWorker(swUrl, config) {
  // Check if the service worker can be found. If it can't reload the page.
  fetch(swUrl)
    .then(response => {
      // Ensure service worker exists, and that we really are getting a JS file.
      if (
        response.status === 404 ||
        response.headers.get("content-type").indexOf("javascript") === -1
      ) {
        // No service worker found. Probably a different app. Reload the page.
        navigator.serviceWorker.ready.then(registration => {
          registration.unregister().then(() => {
            window.location.reload()
          })
        })
      } else {
        // Service worker found. Proceed as normal.
        registerValidSW(swUrl, config)
      }
    })
    .catch(() => {
      console.log("No internet connection found. App is running in offline mode.")
    })
}

and I append this to resulted serviceWorker.js file for "update" app functionality

self.addEventListener('message', function(event) {
  // Force SW upgrade (activation of new installed SW version)
  if ( event.data === 'skipWaiting' ) {
    /* eslint-disable-next-line no-restricted-globals */
    self.skipWaiting();
  }
});

From time to time, users are reporting app unavailability with following error in console:
SyntaxError: expected expression, got '<'

The whole problem is that sometimes index.html file is not replaced with the newest one, so it points to not-existing css/js files. How can it be possible?

[alexeigs]

I face the very same issue. My vague guess is that the window.location.reload() happens too soon before the new SW has fully taken over. Then there are still mismatching precached url parameters (precached by the old SW) which should be cleaned once you reload with the new SW in place - but potentially it has not fully taken over which prevents a full cleanup. Honestly, I'm not sure if that explanation even makes sense and I've only come to that guess because I've spent hours trying to debug myself; but I don't understand everything around SWs yet so still looking for some help here too.

1. What is really the impact of the reload? Is it true that it purges the old SW's traces in case there's a new SW in place?
2. What's the appropriate timing/chaining to switch and reload without issues?
3. May the option ignoreURLParametersMatching: [/.*/] help with anything of that?
4. What is it that I don't understand at all given my comments so far? 😅

[claymation296]

#1692 Solved this issue for me.

I had a similar issue. The root cause was that workbox-webpack-plugin was not updating its revision hash for index.html.

I followed the advice offered by @blephy and added a meta tag to my index.html file that changes its hash on each new build.

[ketysek]

@claymation296 hm, I'm not sure if this helps, I'm afraid that revision number of index.html in precache-manifest file changes ... because if not, every user will face the issue, but it's not happening now. It happens sometimes (randomly?)...

[blephy]

Hi,

@ketysek I saw your console error and it was exactly the same issue i worked around. The behavior is the same too and the context too.
It's an hash problem, maybe caused by html webpack plugin. It was more than one year ago, i don't remember.

I'm pretty sure that when this happen, your revision hash in your manifest.json is wrong (the previous one ?).

@claymation296 you are welcom 😉

[claymation296]

@ketysek aww sorry it didn't help. Originally the error was caused after updating my Webpack and Workbox packages.

I thought this would help you because I too was getting the - Uncaught SyntaxError: Unexpected token '<' - but from a .js file.

After I commented out a script tag type="application/ld+json" in my index.html, that error stopped being raised. That's when I realized that app updates weren't taking because index.html wasn't updating, which lead me to this issue.

When you say randomly, are you sure that its not from the cache expiring on its own?

I ask because that's what it seemed like to me when it first started happening. At first I looking around the internet to make sure I was using the right events from window-workbox. Thought it was a timing issue with sw lifecycle events. But nope, I was wrong about that.

I had to really dig into the cache and files with the 'Application', 'Sources' and 'Network' tabs in Chrome Dev Tools to know for sure that the WB_REVISION hash tagged onto the end of the index.html was off.

I hope this helps someone out there in the future!

[blephy]

How your server is setup for the pwa fallback ? Because it can be caused by the mime-type incorrectly setup ... ex your server is returning .html file with .js mime-type ?

I was using express in production to serve assets behind a proxy nginx. My express serveur was using express-history-api-fallback.

Only for HTML requests: Never serve mistakenly for JS or CSS or image or other static file requests. >Less debugging headaches.

[ketysek]

I'm now waiting if some customer will have an issue so I can inspect some stuff... :-)

[ketysek]

OK, it happened also to me now. So I got these records in precache manifest:

  {
    "revision": "790223494a31cd1ae60dfff0a57d7509",
    "url": "/index.html"
  },
  {
    "revision": "b493b1191316b5a5bf01",
    "url": "/static/css/2.af4906e8.chunk.css"
  },
  {
    "revision": "c899958f29abcc948c80",
    "url": "/static/css/main.5004af8b.chunk.css"
  },
  {
    "revision": "b493b1191316b5a5bf01",
    "url": "/static/js/2.cb2d1aa6.chunk.js"
  },
  {
    "revision": "8b0e5c207f6ad7048a94a7511fc6289e",
    "url": "/static/js/2.cb2d1aa6.chunk.js.LICENSE.txt"
  },
  {
    "revision": "c899958f29abcc948c80",
    "url": "/static/js/main.90d12299.chunk.js"
  },
  {
    "revision": "e83f495e06ab00f75e67",
    "url": "/static/js/runtime-main.32c89a9f.js"
  }

following things are in my browser's cache:

0 | /index.html?__WB_REVISION__=790223494a31cd1ae60dfff0a57d7509 | default | text/html; charset=utf-8 | 1,213 | 27/02/2020, 10:15:00
1 | /static/css/2.af4906e8.chunk.css?__WB_REVISION__=b493b1191316b5a5bf01 | basic | text/css; charset=utf-8 | 0 | 27/02/2020, 10:15:00
2 | /static/css/main.5004af8b.chunk.css?__WB_REVISION__=c899958f29abcc948c80 | basic | text/css; charset=utf-8 | 0 | 27/02/2020, 10:15:01
3 | /static/js/2.cb2d1aa6.chunk.js.LICENSE.txt?__WB_REVISION__=8b0e5c207f6ad7048a94a7511fc6289e | basic | text/plain; charset=utf-8 | 1,568 | 27/02/2020, 10:15:00
4 | /static/js/2.cb2d1aa6.chunk.js?__WB_REVISION__=b493b1191316b5a5bf01 | basic | application/javascript; charset=utf-8 | 0 | 27/02/2020, 10:15:00
5 | /static/js/main.90d12299.chunk.js?__WB_REVISION__=c899958f29abcc948c80 | basic | application/javascript; charset=utf-8 | 0 | 27/02/2020, 10:15:00
6 | /static/js/runtime-main.32c89a9f.js?__WB_REVISION__=e83f495e06ab00f75e67 | basic | application/javascript; charset=utf-8 | 778 | 11/02/2020, 21:34:04

So as you can see, I got same index revision in cache as it's in precache manifest, but index.html tries to load following scripts from previous version:

<script src="/static/js/2.facef6cc.chunk.js"></script><script src="/static/js/main.f4e6e53e.chunk.js"></script>

Cmd+shift+R for loading index.html without service worker load correct index.html. Weird is that my colleague did the same updating procedure as me but he has got correct index.html in cache. Why is this happening? He has exact same revision number of index.html in cache also in precache manifest of course.

[blephy]

The solution is to force webpack to change and update your index.html with your build revision hash in the head meta. read comments

[ketysek]

@blephy what it will solve? The revision number was different than the previous one. Also I'm using create-react-app, can't inject webpack.hash in index.html now.

[jeffposnick]

Whenever this has some up before, it's been due to one of two things:

• Accelerating the service worker lifecycle by calling skipWaiting() before lazily-loaded subresources have been requested. There's more information at https://pawll.glitch.me/ about why this is an issue, and the best practice is to follow something like this recipe to ensure that if/when you do trigger skipWaiting(), you also force a reload of the fresh HTML.

• Subresources being left out of the precache manifest because they are larger than 2 megabytes, which is the default maximumFileSizeToCacheInBytes value. When this happens, you'll see a warning in your build letting you know exactly what was excluded from being precached. You can either reduce the size of your subresources (which is normally a good idea in general), or manually set a larger maximumFileSizeToCacheInBytes value in your Workbox build configuration.

[ketys-from-meiro]

Hi @jeffposnick, thanks for your reply!

Ad your second point first - this is definitely not my problem because max. size of the whole bundle app file is according to console around 800kB, it also doesn't throw any warning in console during build, so I assume everything is OK.

now to your first point - I'm not using lazy loading in app, so I can show you my updating procedure while I'm developing it using create-react-app

As you probably know, there's registerValidSW(swUrl, config) function prepared in CRA's serviceWorker.js file. I slightly edited it and it looks following way in my case:

function registerValidSW(swUrl, config) {
  navigator.serviceWorker
    .register(swUrl)
    .then((registration) => {
      registration.onupdatefound = () => {
        const installingWorker = registration.installing;
        if (installingWorker == null) {
          return;
        }
        installingWorker.onstatechange = () => {
          if (installingWorker.state === "installed") {
            if (navigator.serviceWorker.controller) {
              // At this point, the updated precached content has been fetched,
              // but the previous service worker will still serve the older
              // content until all client tabs are closed.
              console.log(
                "New content is available and will be used when all " +
                  "tabs for this page are closed. See https://bit.ly/CRA-PWA."
              );

              // Execute callback, which looks like: 
              // registration => {
              //   alert("New version available! Ready to update?");
              //   if (registration && registration.waiting) { registration.waiting.postMessage({ type: "SKIP_WAITING" }); }
              // }
              if (config && config.onUpdate) {
                config.onUpdate(registration);
              }
            } else {
              // At this point, everything has been precached.
              // It's the perfect time to display a
              // "Content is cached for offline use." message.
              console.log("Content is cached for offline use.");
            }
          } else if (installingWorker.state === "activated") {
            console.log("realod when installing worker is activated!");
            window.location.reload();
          }
        };
      };
    })
    .catch((error) => {
      console.error("Error during service worker registration:", error);
    });
}

As you can see, I'm reloading window right after installingWorker.state is activated so I don't see the issue even here.

[jackdbd]

I had a similar problem with the index.html revision. I have a simple 11ty website hosted on Netlify, and I'm using workbox-build's generateSW to build the service worker.

If I am not mistaken, the Workbox revision string of an asset is a MD5 content hash of the asset itself.

The code that generates this hash should be in get-string-hash.js and looks like this:

// This is how Workbox generate the `revision` for an asset.
const workboxRevisionFromString = (string) => {
  const md5 = crypto.createHash('md5');
  md5.update(string);
  return md5.digest('hex');
};

The content of my index.html doesn't change, but its dependencies do. The fact that my site is hosted on Netlify is relevant to my solution, because:

1. Netlify advises against asset fingerprinting (i.e. cache busting), so my CSS/JS files keep the same name across Workbox revisions. This means that in my index.html I have something like my-site.css and my-site.js, not my-site-5eb63bbbe01eeed093cb22bb8f5acdc3.css and my-site-5eb63bbbe01eeed093cb22bb8f5acdc3.js.
2. I have configured several HTTP response headers in my netlify.toml, including a Content-Security-Policy with the list of allowed stylesheets and scripts.

Because of 2, even if index.html keeps the same content over time, it might have to be treated differently by the browser. For instance, if I include my-site.js in index.html and the current CSP allows the browser to fetch it, I might change the CSP and disallow my-site.js tomorrow. And because of 1, I can't include a hash in the name of my CSS/JS assets. So the very same index.html would behave differently when fetched via the network vs when fetched via the Workbox precache.

I think that the solution discussed in #1692 would also work in my case. I am not using webpack, but of course I can still generate a hash and inject it in a <meta> tag in my index.html. However, I had already came up with a solution that seems to work.

This is what I thought: since the CSP and all other headers I am setting in my netlify.toml are basically dependencies for my index.html, changing even a single thing in a header should result in a different revision for my index.html. So I decided to implement this solution:

1. read netlify.toml and index.html [*]
2. concatenate them
3. generate a hash from this concatenated string and use it as the revision string for index.html
4. add the index.html and its revision as a ManifestEntry in the additionalManifestEntries array of generateSW

[*] At first I was concatenating just my CSP and the index.html, but since the CSP is written in my netlify.toml, I decided to read the entire file instead.

[jeffposnick]

@jackdbd, this has always been one of the more difficult things to explain in our docs, but workbox-build does support that use case out of the box, via the templatedURLs option.

You provide an object whose keys are the "real" URLs that should be used as a cache key, and whose values are an array of one or more glob patterns. Anything that matches the glob patterns will be used to generate a composite hash for the corresponding URL. This allows you to specify multiple files, updates to any of which are sufficient to cause the precached entry for the "real" URL to be updated.

So in your case, you might have:

const generateSWConfig = {
  // ...other options...
  templatedURLs: {
    '/': ['index.html', 'netlify.toml'],
  },
};

You can also do this "by hand" via additionalManifestEntries or via a manifestTransforms callback, but that's the "built-in" way of doing it.

[aleclarson]

New link to templatedURLs option: https://developer.chrome.com/docs/workbox/modules/workbox-build#property-GlobPartial-templatedURLs