Built-in Suspense Cache

[acdlite]

We recently landed an initial implementation of an experimental, built-in Suspense cache.

PR: facebook/react#20456

If you have experience building a Suspense cache in user space, you will likely find this feature interesting.

The built-in Cache isn't part of the alphas (yet, at least) because we haven't tested it anywhere except for the Server Components demos. We have had some initial discussions with the Relay team about how they might use it in their implementation, but that's it. There are also some known missing gaps in functionality that we need to fill.

However, I think what's already implemented could be useful enough on its own to start delivering value. Or at least to start considering about how it might into your existing cache implementations.

If we can get some folks to try out the new APIs and give us feedback, that would help us fill in those gaps, especially if you already maintain a Suspense-based data framework.

I've likely omitted or glossed over some important detail in this summary, so please ask for clarifications when necessary. There's a lot to share and some parts are more refined than others.

In addition to reading this overview, the best way to familiarize yourself with basic API is to read through the Server Components demo, which uses most of these features: https://github.com/reactjs/server-components-demo

You can also check a few sandboxes using the same APIs on the client:

• https://codesandbox.io/s/sad-banach-tcnim
• https://codesandbox.io/s/laughing-almeida-v3gk5

API Overview

unstable_getCacheForType(createInitialCache)

Returns the cache for a given type, or creates a new one if it doesn't exist. The "create" function acts as the type/key/identity for the cache. The type corresponds either to a type of data or the framework responsible for fetching it.

React stores the cache in an internal queue. You have full control over populating the entries for a given type, but React owns and handles the lifetime of the cache object itself.

Crucially, the cache object persists across multiple suspended render attempts. React will reuse the same cache (and therefore won't issue duplicate requests) on each attempt — even before the UI has mounted yet. This is a key feature that isn't currently possible in a user space implementation.

Previously, to circumvent this problem, a user space implementation would need to store the cache outside of React, as opposed to a built-in hook like useState. But this made evicting existing entries extremely complicated — there was basically no correct way to do this in a fully concurrent-friendly way, without de-opting to synchronous rendering.

With the built-in API, all that complexity is now handled by React.

If you've built a Suspense cache before, you may know there are some "cache laws" for how to properly populate the cache. We'll document these at some point, but just know for now that they haven't changed: you can add new entries to the cache, but you can't modify or delete existing ones. To evict stale data, you need to use the next feature...

unstable_useCacheRefresh

This will invalidate the current cache and create a fresh one in the background. The data in the old cache will continue to be displayed on screen until the new data is ready to display. Crucially, because both the old cache and new cache can exist simultaneously, the UI remains interactive even while we're waiting for the requests to finish in the background.

The canonical use case is to refresh after performing a server mutation (POST, UPDATE, etc). You want to rerender the UI with fresh data from the server. Often, the server mutation request will include fresh data in its response. Like when you edit a field in a form, the browser will return the updated version of that field. So we include a mechanism to seed the new cache with this initial data. (This feature needs a lot more research, though.)

We don't intend to provide support for refreshing specific entries. The idea is that you refresh everything, and rely on an additional, backing cache layer — the browser request cache, a mutable data store, etc — to deduplicate requests. Again, still lots of research required to figure out the details, but that's the basic model.

<Cache />

A Cache component defines a boundary within which all components must read consistent data. If the same data is requested in multiple parts of the tree — say, the current user's profile picture — then they must always display the same thing:

<Cache>
  <CurrentUserProfilePic />
  <CurrentUserProfilePic />
<Cache>

Because these components share a cache boundary, they will always show the same image — even if the user has just changed their profile picture. In that scenario, both component will either display the old (stale) picture or the new (fresh) picture. Imagine you're building a chat UI, where each message is accompanied by the author's photo. You'd rather show the old photo next to each message than show the new photo next to some and the old photo next to others.

However, since this actually the default behavior of the built-in cache, I prefer to think of a Cache boundary as defining which parts of the UI are allowed to be inconsistent.

<>
  <Cache>
    <Toolbar>
        <CurrentUserProfilePic />
    </Toolbar>
  </Cache>
  <Cache>
    <MessageThread>
      <CurrentUserProfilePic />
      <CurrentUserProfilePic />
    </MessageThread>
  </Cache>
</>

In this example, the message thread photos will always be consistent — either both the old photo or both the new one. But the photo in the toolbar is allowed to be different.

Data in separate boundaries can be fetched independently. That doesn't mean they will be fetched independently, though. That would be wasteful. So in this example, on initial render, React will use the same cache for both boundaries. But after initial render, you can use useCacheRefresh to load the new photo in the message thread without bothering to also refresh the toolbar.

I expect developers won't often interact with Cache boundaries directly. They'll likely be baked into some other part of the app architecture. A router is a natural place to put these — each route is wrapped with its own boundary.

There's a lot more to discuss here that I won't get into in this overview. @sebmarkbage wrote a highly influential post internally called "Here and There" that does an excellent job explaining the UX principles behind our caching model; I'll encourage him to share it in this group.

Integrating the Cache APIs into your existing Suspense cache

The React-provided cache isn't designed to act as persistent storage. There's always some layer beneath it that you will read from in order to populate the cache. Often that layer is itself a kind of cache — like the browser network cache. Or, the backing layer is a mutable data store, owned and managed by a data framework that lives outside React.

The React cache doesn't replace those layers, but it could make it easier for those layers to work with Suspense.

I like to think of the React cache as a lazily-populated, immutable snapshot of an external data source. Because React will always read from the snapshot, data in the external store can be mutated, changed, manipulated, deleted without affecting the current UI, or breaking the requirements of Concurrent React. React manages the lifetime of these snapshots, deduplicates requests, and provides high-level APIs for controlling when a new snapshot should be taken. In theory, this should shift implementation complexity away from the data framework and into React itself.

[drcmda]

i really like this! i would love to test this (on the client). currently some libraries that i work on use use-asset which either closures a cache or uses a naive global cache. to let users create cache boundaries would be more than welcome.

[markerikson]

Or, the backing layer is a mutable data store, owned and managed by a data framework that lives outside React.

The React cache doesn't replace those layers, but it could make it easier for those layers to work with Suspense.

I like to think of the React cache as a lazily-populated, immutable snapshot of an external data source. Because React will always read from the snapshot, data in the external store can be mutated, changed, manipulated, deleted without affecting the current UI, or breaking the requirements of Concurrent React. React manages the lifetime of these snapshots, deduplicates requests, and provides high-level APIs for controlling when a new snapshot should be taken. In theory, this should shift implementation complexity away from the data framework and into React itself.

Obligatory obvious question: without getting into any implementation details, would a Redux store or similar be a potentially valid "backing mutable data store" in this scenario?

[acdlite]

Because Redux is a generic data store that frequently updates, probably not. You could in theory, but it wouldn't be good for performance. The cache is optimized for data that is fetched over an IO boundary. You wouldn't use it to store, say, controlled text input state or scroll position or other UI-driven state.

Remember, the cache is append-only. The only way to update an existing entry in a Suspense cache is to refresh and populate a new cache. So Redux would have to refresh the cache on everydispatch, which is not what we designed it for. Refreshes are meant to be pretty rare. Mainly during a navigation, or after a server mutation.

A good rule of thumb is that if it's a type of data that might be preserved when you hit the browser's refresh button, then it can be put into the cache.

If the state is lost when you refresh, then it's probably UI state that belongs in a state hook.

Putting all your UI state into useState or useReducer is by far the easiest way to ensure that your library will work with all our concurrent features. (We know there are reasons why people choose not to do this, but we're exploring solutions to address these. Like making the context API faster to remove the need for subscription-based replacements. Or a new combined context + state API that is optimized specifically for cross-cutting updates that affect many different components.

If moving ownership of your data store into React isn't an option, you can use useMutableSource — which works pretty well for basic cases, but does occasionally de-opt back to synchronous mode. But useMutableSource isn't a cure all. We can prevent inconsistencies (tearing) with useMutableSource, but we can't guarantee that that all concurrent features will work perfectly, because to maintain consistency, we may have to temporarily de-opt back to synchronous rendering.

I'll leave more details about useMutableSource and its trade offs for another thread.

[drcmda]

Remember, the only way to update data in a Suspense cache is to refresh and populate a new cache.

Could you explain this a bit more in detail? Does it mean that when anything new goes into the cache everything else is invalidated?

probably UI state that belongs in a state hook.

We have a specific need for a client cache that's integrated into suspense (for fallbacks and loading state), you can see it run here: https://codesandbox.io/s/re-using-gltfs-dix1y?file=/src/App.js

A model is loaded like so:

function Model() {
  const data = useGLTF(url)
  // data is available at this point bc the hook suspends

It goes through an async fetch call, then an async wasm parser, then an additional js parser, it can take seconds and is a very heavy operation that must be optimised.

The cached item is supposed to be reused, first time it loads, second time it draws from cache:

<Model ... />
<Model ... />

The app may load different assets, like textures and so on:

const data = useTexture(url)

It is critical that this never invalidates other cached items, like that gltf model above.

Will the built-in cache help us in this regard, or is it for something entirely else?

[acdlite]

@drcmda

Does it mean that when anything new goes into the cache everything else is invalidated?

I should have been more precise. By “update” I meant “change an existing cache entry.”

Adding new entries to the cache will not affect other entries. You can append as much as you want.

It’s only when you mutate something that you have to do a refresh.

Also don’t forget that you can layer another cache underneath the React one with your own cache invalidation protocol.

Will the built-in cache help us in this regard, or is it for something entirely else?

Yeah, overall your use case does sound like an appropriate fit. Pretty neat!

[drcmda]

that is so awesome. the <Cache> bound will solve many of the pesky issues we faced.

[devknoll]

Previously, to circumvent this problem, a user space implementation would need to store the cache outside of React

The React-provided cache isn't designed to act as persistent storage. There's always some layer beneath it that you will read from in order to populate the cache

I think it might be useful to include a more thorough upfront explanation of the problem(s) that the built-in cache is solving.

At first glance, the two statements seem sort of contradictory and make me wonder: if I’ve already got a layer outside of React, why do I need one inside? Of course the answer is for eviction + consistency etc but maybe going into a little more detail into why those are trickier with concurrency might be useful.

[josephsavona]

Great point, see #35 (comment) for a writeup of the problem that Cache is solving.

[gaearon]

I've edited the post to include a few sandboxes:

https://codesandbox.io/s/sad-banach-tcnim
https://codesandbox.io/s/laughing-almeida-v3gk5

Hope these help give some intuition behind how this API works. (At least for now.)

[alvarlagerlof]

https://codesandbox.io/s/sad-banach-tcnim

I'm not sure that I understand what refresh does here. Is it talking to react-fetch? Also why does this demo not have a <Cache> boundary?

[gaearon]

Also why does this demo not have a boundary?

If you don't specify one, it's like there's an implicit default top-level one. As a convenience for small examples.

I'm not sure that I understand what refresh does here.

It clears the closest Cache (which is the default one in this example). So this causes a rerender, and a refetch.

Is it talking to react-fetch?

React Fetch keeps data in the Cache (via unstable_getCacheForType) so when the cache is cleared, that data is gone. This is not specific to React Fetch — any data fetching library integrating with the Cache would work this way.

[alvarlagerlof]

If you don't specify one, it's like there's an implicit default top-level one. As a convenience for small examples.

Maybe I just missed it, but makes sure this is really clear in the new docs. Other than that, that answers my questions. Thanks.

[drcmda]

thanks dan! good to see this work on the client. will it allow for clearing individual cached items or is it always the whole cache?

and react-fetch, so useful! had to rub my eyes until i saw the import.

[gaearon]

but makes sure this is really clear in the new docs

Just to be clear, the Cache part is still in research and it's quite a bit away from the docs. And the docs rewrite is in relatively early stage so we're working on covering much more basic features, like state and context. But yes, we'll need to be clear about this when this API is a part of a stable release.

will it allow for clearing individual cached items or is it always the whole cache?

It clears the whole cache, but there's a way to seed the cache with some content after a mutation. E.g. "posting" to a todo list can return the updated list, which will be cached to prevent another request. This part of the API is still unclear and needs more work.

[josephsavona]

Does the Cache API support cancellation? One challenge right now is that libraries don't know when to cleanup resources that are allocated during render, requiring things like setting a long timeout to clear an entry if it doesn't end up being used. This can happen when a tree starts rendering, suspends, and then never finishes (due to some other state change). Our hope was that the Cache API would offer some cleanup mechanism but I don't see that described here or in code - thoughts on this?

[acdlite]

Chatted about out-of-band, quick summary:

• For canceling pending requests, we have plans to provide an AbortController for tracking the lifetime of the cache instance. The lifetime ends either when the transition gets aborted (because a newer transition replaced it), or when a mounted Cache boundary is unmounted.
• I think you could also use this AbortController to reference count entries inside the cache. During a refresh, the old cache is cleaned up right after the new cache mounts. If AbortController isn't the right API then we could provide some similar clean-up API that serves the same purpose.

[dai-shi]

This feature is interesting to me!

Crucially, the cache object persists across multiple suspended render attempts. React will reuse the same cache (and therefore won't issue duplicate requests) on each attempt — even before the UI has mounted yet. This is a key feature that isn't currently possible in a user space implementation.

I would like to understand this. So, does this mean, even if <Cache> is inside <Suspense>, it will store data there? A user-space implementation will not keep the data inside Suspense while fallback.

A codesandbox repro: https://codesandbox.io/s/reverent-easley-x2uxo
It seems to be working so.

[acdlite]

That's right

[dai-shi]

Thanks. Are there any other differences from what a user-space impl can do?

[acdlite]

That's the main one, though we may add more special cases. Like perhaps in the server rendering layer.

[bvaughn]

Can't refresh individual Suspense caches

The current behavior of unstable_useCacheRefresh is to reset all Suspense caches rather than a the specific cache that corresponds to the createMap identity function provided (see example code below). This seems like a pretty severe limitation for the built-in caching mechanism.

When does this matter?

For example, the React DevTools uses the new Suspense APIs to "inspect" a component (load its props/state for inspection) as well as to load source and parse hook names. (These features use separate Suspense caches.) When an element's props/state change, DevTools "refreshes" that cache in order to load new data. This has the unfortunate side effect of also blowing away the cached hook names (which are much more expensive to recompute).

In this case, both caches are managed by DevTools, so it could better coordinate the refreshing of one cache to e.g. pre-seed the other, but what if I were using an external library (like Relay?) that also used the new caching APIs? It seems like it would be easy for application and library code (or different libraries) to interfere with each other in unexpected ways.

Current workarounds

One work around, which I believe is similar to what Relay itself uses under the hood, would be to manage a 2nd (module level?) cache for named hooks data. Then the Suspense code related to it would be:

1. Read first from React's cache.
   1. If so, return the value.
   2. If not, check the backup (module-level) cache to see if there's one there.
      1. If so, update React's cache and then return the value
      2. If not, continue loading and then update both caches

This seems pretty complicated for a user-space solution, although maybe we don't anticipate the cache API being used directly by applications very often?

Preferred built-in solution

The behavior I anticipated from the refresh API was that React would only refresh the cache identified by the createMap function provided to it. It seems like this would prevent caches/libraries from accidentally interfearing with each other. (Although maybe I'm over-simplifying something here. I've thought about this more as a consumer of the API rather than the implementation.)

Example of using the "refresh" function returned by unstable_useCacheRefresh

// createCacheMap is a stable function that identifies the Suspense cache
const map = createCacheMap();

// The new cache map can be seeded with key/value pairs, e.g.
map.set(key, value);

// The info returned to React is both the identify function and the new (seeded) cache
refresh([createCacheMap, map]);

[drcmda]

@acdlite @gaearon i have tried to implement the new caching model in use-asset(alpha). and it works! 🙏

the only regression i can make out is preloading and peeking. is there a plan for that?

previously i could run asset.preload(id) in global space, which would be similar to asset.read(id) in the component. this allowed us to pre-fetch stuff before even react is ready. of course react now can't determine the cache so it greets me with:

// Global space
asset.preload(id)

Error: Context can only be read while React is rendering. In classes, you can read it in the render method or getDerivedStateFromProps. In function components, you can read it directly in the function body, but not inside Hooks like useReducer() or useMemo().

▼ 3 stack frames were expanded.
_readContext
node_modules/react-dom/cjs/react-dom.development.js:12385
getCacheForType
node_modules/react-dom/cjs/react-dom.development.js:17218
getCacheForType
node_modules/react/cjs/react.development.js:1543
▲ 3 stack frames were expanded.
preload
src/use-asset.js:67
  64 |   console.log("read", cache)
  65 |   return handleAsset(fn, cache, args, lifespan)
  66 | },
> 67 | preload: (...args) => {
     | ^  68 |   const cache = getCacheForType(createCache)
  69 |   return handleAsset(fn, cache, args, lifespan, true)
  70 | },

or in the render phase (fetching the next 3 items for instance, which completely removed waiting interruptions), i used to do this inside a useEffect, now i get this:

function Post({ id }) {
  useEffect(() => {
    // Pre-load next 3 posts
    for (let i = 0; i < 3; i++) asset.preload(id + 1 + i);
  }, [id]);

Error: Context can only be read while React is rendering. In classes, you can read it in the render method or getDerivedStateFromProps. In function components, you can read it directly in the function body, but not inside Hooks like useReducer() or useMemo().
▶ 3 stack frames were collapsed.
preload
src/use-asset.js:68
  65 |   return handleAsset(fn, cache, args, lifespan)
  66 | },
  67 | preload: (...args) => {
> 68 |   const cache = getCacheForType(createCache)
     | ^  69 |   return handleAsset(fn, cache, args, lifespan, true)
  70 | },

i take it i am allowed to preload in the render function, is that correct? preload won't throw, so it should be safe?

function Post({ id }) {
  // Pre-load next 3 posts
  // Not 100% sure if running this in the render function is considered ok
  for (let i = 0; i < PRELOAD; i++) asset.preload(id + 1 + i)

it works, though. so it's probably fine ...

[gaearon]

Yes, preload should be done during rendering (rather than outside or in an effect), from my understanding.

[drcmda]

thanks for confirming. the error messages are still a bit weird, it complains about context. but otherwise, this is a super useful tool, so glad we have that soon!

[gaearon]

Yeah it technically reads cache from context, which is why you’re seeing this. The error messages are very far from being polished because the whole feature is in a very experimental stage (and likely won’t make it into 18.0, but will come after).