Deletes everything when hitting the max database size. (React Native - Android)

[ksegla]

Hi, thanks for the great module.
I'm using this (probably improperly) as a quick (and temporary) way to store data (list of things) relevant for my app users.
In any case, I ran into a problem recently. When it hits the maximum size (6MB by default for React Native onAndroid), it just deletes (or at least fails to render) any stored data. Which would have had disastrous effects if the app was already launched.
Ideally, it would just fail updating the database, not destroy it.
Thanks for considering fixing this

[rt2zz]

AsyncStorage stores everything in one JSON file, so while I have never hit the limit, I can imagine bad things happen when you do. Two possible options off the top of my head:

• implement a SQLLite or similar storage backend
• we can detect if the storage save errors and if so resave the previous state (this seems brittle however.

Open to others ideas

[ksegla]

Obviously Option 1 is better and would be great but even Option 2 would be vastly better than the state's wipe out. As I understood it, Option 1 would do more than just fix the problem I had. In my specific case, may be there could be some kind of quick check on the new information size and a verification on whether it could hit the size limit. Note that the limit for AsyncStorage can be changed in the MainApplication.java (it can be more than 6MB)

[nhayfield]

i started building a new store but haven't been able to integrate it yet: sriraman/react-native-shared-preferences#5
and
https://github.com/nhayfield/react-native-shared-preferences
i may continue but i'm going to attempt to up the storage limit for asyncstorage first based on this:
https://github.com/facebook/react-native/pull/11656/files

[nitkuk]

Even after changing maximum database size limit to 50 MB, it is wiping the state.

Here is how I did it:
In Main Application, inside getPackages(), I wrote this code..
long size = 50L * 1024L * 1024L; // 50 MB com.facebook.react.modules.storage.ReactDatabaseSupplier.getInstance(getApplicationContext()).setMaximumSize(size);

Only difference I find is app is not showing disk full warning in debug.
Can someone give some pointer?

[ptomasroos]

Thanks. I will look into this in a while. Currently shipping a release. I will let you know later this week.

[frostney]

@nitkuk If you increase the database size inside of getPackages(), it might not have the desired effect. At least that was the case for me.

Putting the call inside of onCreate of MainApplication, as outlined in facebook/react-native#11656 worked for me.

Edit: After testing this again, it turns out it didn't have any effect on where the piece of code was placed and I have the same exact issue.

[nhayfield]

@nitkuk I set it to 8gb and now it works pretty good lol

[robwalkerco]

I've just worked around this issue by writing my own storage implementation for Android that uses the file system rather than AsyncStorage.

Details at - #284 (comment)

[robwalkerco]

I've just published an npm module specifically to fix this issue.
Check out https://www.npmjs.com/package/redux-persist-filesystem-storage

[MichaelDanielTom]

Quick question @robwalkerco, is performance in your library better, worse, or the same as vanilla redux-persist that uses AsyncStorage on Android?

[robwalkerco]

I've not benchmarked it @MichaelDanielTom, but I'm using it in production and haven't noticed any issues or slowness, though I have also been using the debounce option in redux-persist to reduce the number of writes. Think I set it to 50ms.

Please do share any benchmark results though, if you run any.

[ksegla]

So unfortunately, the problem is still there, even with redux-persist-filesystem-storage :-(
Any possible lead on where to start in the source code if I wanted to fix the issue myself?
Thanks

[robwalkerco]

@ksegla, redux-persist used AsyncStorage by default, which on Android uses sqlite, which imposes a storage limit. redux-persist-filesystem-storage gets around this by using the filesystem instead of AsyncStorage, and so does not have a limit on the storage size (as far as I understand).

It therefor leads to me wonder if you are hitting the actual physical storage limit of your device? i.e. The device is refusing to store any more data in any location, as its run out of space. Could this be your issue?

[ksegla]

Hi @robwalkerco
Thanks for the answer. I still have at least 10 GB left so I don't think it's the problem. I now think the bug is not solely due to AsyncStorage limitations. The bug is probably somewhere in the core code of redux-persist (or possibly react-native itself). The thing is the behavior ("when in doubt, wipe everything" :-D) is really the worst possible for what seem to be just hiccups. I now have a cloud save of the data so when the problem happens, I download all the data back and can actually continue adding data (lots of data) for days before another incident. So it does not seem to be just a matter of hitting a hard ceiling.
(And I have the problem on different devices : S6 and S8, different Android versions: 5 and 7, as well as different React Navive versions)

It is clear that the bug appears only above a certain amount of data (> 10 MB, I'd estimate) saved to the store. I'm saying that because I have an alpha tester using my yet-to-be-released app for months now. He has about a fifth of my data and never had the problem.

@rt2zz Is it possible that even if the data is persisted in and restored from files, there is some communication with AsyncStorage in the source code? And again, any idea where to start if I wanted to fix the bug myself?

[robwalkerco]

@ksegla just trying to think about potential causes of this issue.

As you say it seems like you are not hitting a hard limit, perhaps the error is occurring when the app is closed during a write or something?

When I had this issue originally, I was getting the same error consistently when hitting the hard limit, so the fact that that is not happening for you is suggesting to me that it's a different problem.

If you can find a way to consistently replicate the error it will be much easier for you to fix. Perhaps get the app to continually increase the amount of data being stored while logging out the progress so that you can keep running until the error occurrs to see if you get as far each time etc.

Something else that may perhaps help. Each reducer is persisted to a separate storage location, so whenever a reducer changes, redux-persist only needs to write the data for that reducer rather then for the whole app. That makes for smaller, faster writes, so perhaps could help you in some way?

There is also a new version https://github.com/rt2zz/redux-persist/blob/v5/README.md on the way. Perhaps worth a look?

[ksegla]

@robwalkerco
Thanks for the very insighful and helpful comments.

Yeah, I was planning to try v5 short term. Pretty sure too the bug is indeed linked to some bad event during a write. may be app being closed at the "wrong" time. The bug is really the worst possible for my app (heavy on user generated/bookmarked data) so before release, I'll make sure to hunt it down systematically like you proposed.

Finally, your information about how each reducer goes to a separate storage is really significant for me. I knew I had a dis-proportionally large reducer and I even strongly suspected, without giving it much thought and investigation, that the only data getting erased was coming from there. This is probably what I'll take a hard look at first. I can certainly spread around a bit more the data being handled by that reducer. Thanks very much for that hint.

[ksegla]

Follow-up: for what it's worth, v5 seems way way worse (with default AsyncStorage) for the problem and even deletes very small stored data. Either that or I'm doing a terrible job updating from v4 (I have way more crashes and after each of those, data is gone). I did check that the data is persisted through restarts (app and device) and all but I don't know: it may very well be me. I'm reporting anyway in case it could help.
Will try with files, then downgrade to v4. will spread data through a bunch of reducers and see.

[rt2zz]

@ksegla have you tried one of the fs storage adapters for react-native? There are a couple of options here: https://github.com/rt2zz/redux-persist/blob/v5/README.md#storage-engines

The underlying problem here is that redux-persist does not have any elegant way to deal with storage limit max. This is not an easy problem because even if we know you are hitting the storage limit, there is no remediation that we can take automatically which is safe.

At present the best options are

1. use a storage engine with a larger limit
2. have your app check storage size and clear out lower priority data as needed
3. store more data on the server

That is it as far as I know, as the problem is not specific to redux-persist, but moreso an inherent limitation of the platform. Open to suggestions for how the lib can help mitigate the problem of course.

[ksegla]

@rt2zz
Ok. Thanks. Will try the alternatives. Will update with news.

[ksegla]

@rt2zz
I may be doing something wrong but while v4 works with redux-persist-filesystem-storage, v5 doesn't seem to.
As reported, v5 does work with AsyncStorage (with more severe wipe-out problem). However, there is no persistence at all when I just replace
import storage from 'redux-persist/lib/storage'
by
import storage from 'redux-persist-filesystem-storage'
@robwalkerco Have you tested v5 with your module? No reason it shouldn't work (same required api as v4) but somehow it doesn't. Feeling pretty lost right now.

[rt2zz]

@ksegla sorry for the churn, it looks like redux-persist-filesystem-storage does not yet support the promise api which v5 requires. I created a pull request to add support here robwalkerco/redux-persist-filesystem-storage#4 but note I have never run this code and it may not work.

[pedrosimao]

@robwalkerco Did anyone benchmark redux-filesystem-storage?
I wonder how it compares to Db solutions like Realm.
I have a project that might need to store large quantity of historical data. I am trying to use Realm, but it is way too buggy. I am facing several issues with it.
I wonder if redux-persist with filesystem-storage, instead of asyncStorage would be too slow for me.

[robwalkerco]

@pedrosimao there has not been any benchmarks done as far as I know.
We are actually using Realm together with a fork or ‘redux-persist-realm’
https://github.com/papertrailio/redux-persist-realm/tree/always-update which is working in for us at the moment.

[ItsNoHax]

@robwalkerco What are you using nowadays then? We tried to use redux-persist with react-native-fs storage and it gave Android a severe start up time since we wait for the store to persist.

[robwalkerco]

@ItsNoHax I've been using https://github.com/papertrailio/redux-persist-realm/tree/always-update on a project that's persisting a lot of data, and just AsyncStorage on a very small project. I've not worked on trying to improve Android startup time. Let me know if you find a solution that helps with that!

[ItsNoHax]

I looked into that @robwalkerco but it seems that it is written for Redux-Persist V4. I changed it to work for V5 but i'm running into some errors.

[raphaelrk]

Our android users' apps have been crashing + logging out and after a week of debugging it seems this is the cause.

Fixed it by setting a whitelist as we only really needed to save a couple kilobytes of text (vs ~6MB). Would be nice as a dev to have some more visibility when running into this issue, like a console.log that redux persist crashed, or that the store is getting too large. Had a tough time figuring out what was going on.

[hariDasu]

@robwalkerco can you provide a link to or snippet of your usage of debounce in redux-persist? I have a concern on the data being persisted too often on redux-form onChange's, but don’t mean to hijack this thread...this android file size limitation is a bug on our android app I am attempting to debug as well

[TheDSCPL]

Hey, guys! I know this is an old issue but now there's an oficial way to solve this!

[ckalika]

Is this still happening?

[nhayfield]

Is this still happening?

Not sure, I have moved on from that position and not using the library currently.

[shehzadosama]

For me, this solution worked.
Add this piece of code at the end of onCreate() in MainApplication.java

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //100MB
  } catch (Exception e) {
 if (BuildConfig.DEBUG) {
  e.printStackTrace();
  }
  }

Also import these on top of the MainApplication.java

import android.database.CursorWindow;
import java.lang.reflect.Field;

[ThushalIntervest]

@shehzadosama Error storing data [Error: database or disk is full (code 13 SQLITE_FULL[13])]. still get this error using redux-persist

[kangfenmao]

Data lost solution

1. Change persist config

const persistConfig: {
   ...
   timeout: 0, // The code base checks for falsy, so 0 disables
};

2. Change Android CursorWindow size to 1000MB

// node_modules/@react-native-async-storage/async-storage/android/src/main/java/com/reactnativecommunity/asyncstorage/AsyncStorageModule.java

try {
  Field field = CursorWindow.class.getDeclaredField("sCursorWindowSize");
  field.setAccessible(true);
  field.set(null, 100 * 1024 * 1024); //the 100MB is the new size
} catch (Exception e) {
  e.printStackTrace();
}

Put the patch file blow into patches folder then exec npx patch-package

If your persist data is very large, increase CursorWindow size will slow your app launch speed.

@react-native-async-storage+async-storage+1.17.11.patch