Opt-in method for purging (some) caches upon a QuotaExceededError #1505

jeffposnick · 2018-05-30T21:28:26Z

R: @philipwalton
CC: @raejin @Enalmada @josephliccini

There are two changes in this PR:

There's a check in workbox-core's cacheWrapper.put() (which is used "under the hood" throughout Workbox) method to detect when the underlying cache.put() fails due to a QuotaExceededError. When that happens, one or more previously registered callback functions will automatically be called, and given a chance to free up storage. There is (currently) no attempt made to retry the failed cache.put() automatically after the cleanup completes, and the QuotaExceededError will be re-thrown and bubble-up to the method that called cacheWrapper.put() originally.
There's a new purgeOnQuotaError option added to the workbox.expiration.Plugin constructor. It defaults to false. When true, it will automatically register a quota error callback that will delete the entire cache (and IndexedDB) data that the plugin is responsible for. (This is based on New workbox.expiration.Plugin.deleteCacheAndMetadata() method #1500.)

Nothing should change by default, but developers who want to opt-in to trying this automatic deletion of caches should make sure that they're using workbox.expiration.Plugin for a given strategy, and set purgeOnQuotaError: true alongside their maxEntries and/or maxAgeSeconds configuration when constructing the plugin.

At some point in the future, we might:

Automatically retry the failed cache.put() operation after all the callbacks are complete.
Default to purgeOnQuotaError: true inside of workbox.expiration.Plugin.
Add in some special logic inside workbox.precaching to handle quota errors in a different way.

In the meantime, I'd love to start getting some feedback from developers who frequently encounter quota issues with their real-world applications, and figure out how effective opting-in to this new behavior is.

It will have the most significant effect on developers whose runtime caches contribute the largest amounts towards their quota usage (perhaps because they're caching opaque responses). If you're precaching a massive amount of data, but not runtime caching much, this PR is not likely to help.

coveralls · 2018-05-30T21:41:32Z

Coverage increased (+0.1%) to 86.833% when pulling 27ae156 on quota-errors into 1185983 on master.

raejin · 2018-05-30T22:11:08Z

Thank you for working on this! This is very exciting, as the cache quota reporting chromium bug was recently fixed.

With this implementation, one is still possible to manually add logic to retry precaching assets when quota error callbacks are called.

philipwalton · 2018-06-11T22:51:26Z

packages/workbox-cache-expiration/Plugin.mjs

@@ -235,7 +242,7 @@ class Plugin {
   * There is no Workbox-specific method needed for cleanup in that case.
   */
  async deleteCacheAndMetadata() {
-    // Do this one at at a time instance of all at once via `Promise.all()` to
+    // Do this one at at a time instead of all at once via `Promise.all()` to


While we're fixing grammar, remove the duplicate "at" earlier in this line.

philipwalton

Overall I think the strategy is good, I just have a couple of issues with some of the naming/abstractions.

philipwalton · 2018-06-11T23:23:36Z

test/workbox-core/node/_private/test-quota.mjs

+    sandbox.restore();
+  });
+
+  devOnly.it(`registerCallback() should throw when passed a non-function`, async function() {


Can you wrap all the tests related to registerCallback() in a describe block and then do a separate describe block for all the tests in executeCallback().

I don't know about that—there's only one test that's just registerCallback() (which only runs in devOnly mode), and all of the rest of them rely on a combination of the two methods. That seems like overkill?

Sorry if I was unclear. I care less about how many levels of nesting we have than I do about the semantics of describe() and it(). Your test is effectively describing "registerCallback()" and then declaring that it "should throw when passed...", but those semantics are all being passed to the it() function.

While I'm sure it doesn't affect how the test is run, it reads weirdly to me (I read it literally as "it registerCallback should..."). If we're using BDD style blocks, I'd prefer if we stick to the BDD conventions.

philipwalton · 2018-06-11T23:27:54Z

packages/workbox-core/_private.mjs

  getFriendlyURL,
+  logger,
+  registerQuotaErrorCallback,


What's the reasoning behind exposing this?

Just flexibility.

While the automatic cleanup of caches that use expiration is one way of dealing with quota errors, you could imagine a web app that, for example, stores a large amount of data in IndexedDB. Developers who understand how their web app stores data could use this to register a callback that cleaned up that hypothetical IndexedDB database.

philipwalton · 2018-06-11T23:29:32Z

packages/workbox-cache-expiration/Plugin.mjs

@@ -76,6 +79,10 @@ class Plugin {
    this._config = config;
    this._maxAgeSeconds = config.maxAgeSeconds;
    this._cacheExpirations = new Map();
+
+    if (config.purgeOnQuotaError) {
+      registerCallback(() => this.deleteCacheAndMetadata());


Is the plan that registerCallback() might/will get invoked in more places in the future? From what I can tell in this PR, this is the only place it's used, which makes the quota.mjs module seem like over-kill for just this.

Yes, along the lines of #1505 (comment)

In addition to giving developers a general-purpose way of registering a "cleanup" callback, we may want to make use of it elsewhere in the Workbox codebase—potentially as part of the precaching module, along the lines of #1312.

Yeah, I get the intention of the change, I'm just wondering if it could be somehow better integrated into the rest of workbox.

Did you consider creating a new lifecycle callback: maybe something like cacheWriteDidFail? And in the same way that we can register a generic fetch failure handler, we could register generic cache failure handler.

Anyway, just a thought, I don't want to block this change if a better solution isn't obvious.

I felt like using the lifecycle callback naming conventions would be misleading, since here you're registering a callback that has a "global" (or workbox.core) scope, which will be invoked regardless of which strategy triggers the quota exception. That doesn't match the behavior of the lifecycle callbacks, which are all scoped to a given strategy.

workbox.routing.setCatchHandler() is closer in semantics, but that only supports setting a single callback, while we need support for multiple quota exception callbacks. To me, register... sounds more in line with being able to create multiple callbacks than set..., but I could change the naming if you feel strongly.

I felt like using the lifecycle callback naming conventions would be misleading, since here you're registering a callback that has a "global"...

Right, I wasn't suggesting just using the name, I was suggesting we actually add a cacheWriteDidFail lifecycle callback. While I understand that it's not global and would have to be added per strategy, that seems like a benefit to me, not a shortcoming. And users could write their strategy code in such a way that sharing the same callbacks across all their strategies isn't so bad.

My concern is that global handlers can lead to unpredictable behavior that's harder to test, especially for folks who are using Workbox in a more modular way, or anyone building a third-party plugin.

Anyway, I don't want to block this feature being added, especially if we know folks are running into quota issues today. And you've obviously thought about the trade-offs, so I'm happy with whatever you think is best.

But I think we should revisit it later. I also think it's probably worth adding a catchWriteDidFail lifecycle method at some point regardless, so strategy plugin authors can handle these cases individually without potentially affecting plugins they don't own.

I filed #1530 to capture that. To address the immediate quota issues, I'd like to go with a global approach.

philipwalton · 2018-06-11T23:37:46Z

packages/workbox-core/_private/quota.mjs

+  callbacks.add(callback);
+
+  if (process.env.NODE_ENV !== 'production') {
+    logger.log('Registered a callback to respond to quota errors.', callback);


How you checked to see how these function are logged (IIRC normal functions output their toString() result)? Maybe it makes more sense to output .name value?

Yeah, it logs the code for the actual callback function. This will normally be

() => this.deleteCacheAndMetadata()

or it might be the code for a custom callback, if you write and register your own.

I experimented with this a little bit and could go with just leaving out the logging of the function entirely, but I figured that more context for the development environment logging is better.

Since a developer could theoretically pass in whatever function they want here (in addition to using our automatically registered functions), I don't think we could count on .name being meaningful.

You probably already know this, but .name has gotten a lot better in browsers lately (e.g. it reports anonymous functions assigned to variable or property names). That being said, if the common use case is actually passing an unreferenced anonymous function then yeah, it's probably not useful.

I'd just like to leave it as-is to make sure we're providing the full context to folks who are in debug mode and reading the Console log messages.

philipwalton · 2018-06-11T23:38:58Z

test/workbox-cache-expiration/node/test-Plugin.mjs

+      });
+      plugin.deleteCacheAndMetadata = sandbox.stub();
+
+      await executeCallbacks();


Same comment here as above w.r.t the name.

Not sure what you mean—the name of what?

This is definitely a nit, but I mention it because it confused me a bit when reading the code for the first time.

Basically my point is that the name executeCallbacks() here has no context, so as a reader you have to go searching for where it's defined/imported to figure out what it's doing.

If it were called something like executeQuotaErrorCallbacks() or something like that, then it'd be more clear. Alternatively, if you imported the entire namespace as quotaErrors, then you'd have more context because it'd be quotaErrors.executeCallbacks().

Anyway, like I said this is a nit. If you think it's fine, I'm OK with it.

Note, this comment primarily applies to line 90 in cacheWrapper, not necessarily here.

Renamed to executeQuotaErrorCallbacks().

philipwalton · 2018-06-11T23:39:42Z

packages/workbox-precaching/controllers/PrecacheController.mjs

-    await Promise.all(requests.map(async (request) => {
+    // Process each request/response one at a time, deleting the temporary entry
+    // when done, to help avoid triggering quota errors.
+    for (const request of requests) {


+1 to this change. I think it'll make it a lot easier to respond to errors in a predictable way.

philipwalton · 2018-06-11T23:40:37Z

packages/workbox-core/_private/quota.mjs

+ * a quota error.
+ *
+ * @param {Function} callback
+ *


Can you make the spacing here consistent with below. I'm not sure if we prefer a space or not.

jeffposnick · 2018-06-12T20:50:09Z

I've addressed some of the comments, and provided more context.

Another thing that I'd like to do is write up a "Guide to Handling Quota Errors" or something along those lines for https://developers.google.com/web/tools/workbox/guides/ following the next release, which will explain a bit more about opting-in to the automatic cleanup and registering your own callback.

philipwalton · 2018-06-12T22:40:19Z

test/workbox-core/node/_private/test-quota.mjs

+    sandbox.restore();
+  });
+
+  devOnly.it(`registerCallback() should throw when passed a non-function`, async function() {


Sorry if I was unclear. I care less about how many levels of nesting we have than I do about the semantics of describe() and it(). Your test is effectively describing "registerCallback()" and then declaring that it "should throw when passed...", but those semantics are all being passed to the it() function.

While I'm sure it doesn't affect how the test is run, it reads weirdly to me (I read it literally as "it registerCallback should..."). If we're using BDD style blocks, I'd prefer if we stick to the BDD conventions.

philipwalton · 2018-06-12T22:55:02Z

packages/workbox-core/_private/quota.mjs

+  callbacks.add(callback);
+
+  if (process.env.NODE_ENV !== 'production') {
+    logger.log('Registered a callback to respond to quota errors.', callback);


You probably already know this, but .name has gotten a lot better in browsers lately (e.g. it reports anonymous functions assigned to variable or property names). That being said, if the common use case is actually passing an unreferenced anonymous function then yeah, it's probably not useful.

philipwalton · 2018-06-12T23:04:09Z

test/workbox-cache-expiration/node/test-Plugin.mjs

+      });
+      plugin.deleteCacheAndMetadata = sandbox.stub();
+
+      await executeCallbacks();


This is definitely a nit, but I mention it because it confused me a bit when reading the code for the first time.

Basically my point is that the name executeCallbacks() here has no context, so as a reader you have to go searching for where it's defined/imported to figure out what it's doing.

If it were called something like executeQuotaErrorCallbacks() or something like that, then it'd be more clear. Alternatively, if you imported the entire namespace as quotaErrors, then you'd have more context because it'd be quotaErrors.executeCallbacks().

Anyway, like I said this is a nit. If you think it's fine, I'm OK with it.

philipwalton · 2018-06-12T23:13:45Z

test/workbox-cache-expiration/node/test-Plugin.mjs

+      });
+      plugin.deleteCacheAndMetadata = sandbox.stub();
+
+      await executeCallbacks();


Note, this comment primarily applies to line 90 in cacheWrapper, not necessarily here.

philipwalton · 2018-06-12T23:14:14Z

packages/workbox-cache-expiration/Plugin.mjs

@@ -76,6 +79,10 @@ class Plugin {
    this._config = config;
    this._maxAgeSeconds = config.maxAgeSeconds;
    this._cacheExpirations = new Map();
+
+    if (config.purgeOnQuotaError) {
+      registerCallback(() => this.deleteCacheAndMetadata());


Yeah, I get the intention of the change, I'm just wondering if it could be somehow better integrated into the rest of workbox.

Did you consider creating a new lifecycle callback: maybe something like cacheWriteDidFail? And in the same way that we can register a generic fetch failure handler, we could register generic cache failure handler.

Anyway, just a thought, I don't want to block this change if a better solution isn't obvious.

philipwalton · 2018-06-13T16:45:15Z

packages/workbox-core/_private/cacheWrapper.mjs

 import pluginEvents from '../models/pluginEvents.mjs';
 import pluginUtils from '../utils/pluginUtils.mjs';
+import {WorkboxError} from './WorkboxError.mjs';
+import {assert} from './assert.mjs';
+import {executeCallbacks} from './quota.mjs';


Can we also change this name to executeQuotaErrorCallbacks?

philipwalton · 2018-06-13T16:46:35Z

Thanks Jeff! I left a few random thoughts for the future and one more comment, but overall everything LGTM.

workbox-pr-bot · 2018-06-13T19:37:50Z

PR-Bot Size Plugin

Changed File Sizes

File	Before	After	Change	GZipped
packages/workbox-cache-expiration/build/workbox-cache-expiration.prod.js	3.78 KB	3.88 KB	+3%	1.37 KB
packages/workbox-core/build/workbox-core.prod.js	6.86 KB	7.12 KB	+4%	2.81 KB
packages/workbox-precaching/build/workbox-precaching.prod.js	5.68 KB	5.57 KB	-2%	2.15 KB

New Files

No new files have been added.

All File Sizes

View Table

File	Before	After	Change	GZipped
packages/workbox-background-sync/build/workbox-background-sync.prod.js	3.51 KB	3.51 KB	0%	1.45 KB
packages/workbox-broadcast-cache-update/build/workbox-broadcast-cache-update.prod.js	1.12 KB	1.12 KB	0%	584 B
packages/workbox-build/build/_types.js	41 B	41 B	0%	61 B
packages/workbox-build/build/index.js	4.02 KB	4.02 KB	0%	1.57 KB
packages/workbox-cache-expiration/build/workbox-cache-expiration.prod.js	3.78 KB	3.88 KB	+3%	1.37 KB
packages/workbox-cacheable-response/build/workbox-cacheable-response.prod.js	587 B	587 B	0%	345 B
packages/workbox-cli/build/app.js	6.50 KB	6.50 KB	0%	2.16 KB
packages/workbox-cli/build/bin.js	2.32 KB	2.32 KB	0%	1.03 KB
packages/workbox-core/build/workbox-core.prod.js	6.86 KB	7.12 KB	+4%	2.81 KB
packages/workbox-google-analytics/build/workbox-google-analytics.prod.js	2.12 KB	2.12 KB	0%	1.03 KB
packages/workbox-precaching/build/workbox-precaching.prod.js	5.68 KB	5.57 KB	-2%	2.15 KB
packages/workbox-range-requests/build/workbox-range-requests.prod.js	1.63 KB	1.63 KB	0%	797 B
packages/workbox-routing/build/workbox-routing.prod.js	2.87 KB	2.87 KB	0%	1.31 KB
packages/workbox-strategies/build/workbox-strategies.prod.js	4.30 KB	4.30 KB	0%	1.14 KB
packages/workbox-streams/build/workbox-streams.prod.js	1.66 KB	1.66 KB	0%	765 B
packages/workbox-sw/build/workbox-sw.js	1.47 KB	1.47 KB	0%	792 B
packages/workbox-webpack-plugin/build/generate-sw.js	8.04 KB	8.04 KB	0%	2.57 KB
packages/workbox-webpack-plugin/build/index.js	742 B	742 B	0%	470 B
packages/workbox-webpack-plugin/build/inject-manifest.js	10.30 KB	10.30 KB	0%	3.23 KB

Workbox Aggregate Size Plugin

9.68KB gzip'ed (65% of limit)
25.18KB uncompressed

tgangso · 2018-06-14T19:56:30Z

When can we expect this to be released?
Thanks

jeffposnick added 2 commits May 30, 2018 17:07

Add purgeOnQuotaError to expiration.Plugin, and quota callbacks to core.

d7caa63

Rename the public symbol to workbox.core.registerQuotaErrorCallback

b09566d

jeffposnick requested a review from philipwalton May 30, 2018 21:28

raejin mentioned this pull request Jun 11, 2018

Handling of quota exceeded errors in workbox-precaching #1312

Closed

philipwalton reviewed Jun 11, 2018

View reviewed changes

philipwalton requested changes Jun 11, 2018

View reviewed changes

Review feedback.

0addd86

philipwalton requested changes Jun 12, 2018

View reviewed changes

More review feedback.

58c75f9

philipwalton reviewed Jun 13, 2018

View reviewed changes

philipwalton approved these changes Jun 13, 2018

View reviewed changes

jeffposnick mentioned this pull request Jun 13, 2018

New cacheWriteDidFail lifecycle callback #1530

Closed

Review feedback.

27ae156

jeffposnick merged commit b3735f3 into master Jun 13, 2018

jeffposnick deleted the quota-errors branch June 13, 2018 19:42

jeffposnick mentioned this pull request Jun 16, 2018

Cache expiration could add an option for max size limits. #1401

Closed

gugiserman mentioned this pull request Sep 18, 2018

Uncaught TypeError: quota_mjs.registerQuotaErrorCallback is not a function #1642

Closed

jflayhart mentioned this pull request Apr 21, 2020

Add support to customize the Service Worker with InjectManifest and Workbox v5 facebook/create-react-app#8485

Closed

Opt-in method for purging (some) caches upon a QuotaExceededError #1505

Opt-in method for purging (some) caches upon a QuotaExceededError #1505

Conversation

jeffposnick commented May 30, 2018

coveralls commented May 30, 2018 • edited Loading

raejin commented May 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philipwalton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffposnick commented Jun 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

philipwalton commented Jun 13, 2018

workbox-pr-bot commented Jun 13, 2018

PR-Bot Size Plugin

Changed File Sizes

New Files

All File Sizes

Workbox Aggregate Size Plugin

tgangso commented Jun 14, 2018

coveralls commented May 30, 2018 •

edited

Loading