Try persistent storage on Firefox

[estellecomment]

Firefox has an API for persistent storage for IndexedDB :
https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria#Firefox_specifics

[estellecomment]

Added storage: 'persistent' to the args to Pouch. Removed the worker-pouch for now, it was eating up the storage param.

Got it to work on desktop, though only after some time, not sure why :

On mobile, I never got that popup, and the data doesn't show up in the permanent folder, so I don't think I'm actually using permanent storage.

Also, Bug : the sync finishes successfully and then the app seems to lose access to the indexedDB (500, indexeddb_went_bad). Reproduced every time.
(user has <300 docs, phone has 4GB free)

From then on it's over. Reloading doesn't help. Clearing browsing data either. Clearing Firefox app data in App Manager resets the whole thing.

The db itself doesn't look corrupted though: you can access it in console, get info, query views :

[garethbowen]

@estellecomment There might be some weird restriction like it only works on localhost or https (it seems like all cool features are restricted like this these days). No idea why the indexed db went bad but it's probably related to the empty permanent folder? You're not running in private mode or something? I might have a look too and see if I can reproduce it.

Also using the web worker is a big win for performance so I'll see if I can get the storage option passed through.

[garethbowen]

It looks like the web worker is dropping the options we pass in, which explains why the auto compaction isn't working...

[garethbowen]

It looks like the web worker is dropping the options we pass in

It's dropping all except for the name, auto_compaction, and revs_limit. I've created an issue to pass through storage also.

[garethbowen]

@estellecomment I've got persistent storage working for me. In my profile persistent storage is > 0 and temporary storage is empty. Once you've removed the web worker (or patched it to pass through storage) it should work.

When I first tried I forgot to add it in both places that PouchDB is instantiated - did you remember to add the storage option to the bootstrap code?

I've created an issue to fix this inconsistency: #2625

[garethbowen]

I'm getting fatal errors trying to get IDB working on Firefox mobile - I'm not even getting the prompt. My best guess is it's disabled for anything other than localhost or https. If I could find a way to do port forwarding like in Chrome I'd be good to go...

If necessary we can update alpha.dev to try this stuff out in a real world situation.

[estellecomment]

did you remember to add the storage option to the bootstrap code?

Aaaaaaaarrrrrrrrrghghghghhrrrrraaaarrrrrhhhh

[estellecomment]

That explains why the replication would finish, and then the indexeddb would get corrupted. It's because the replication didn't use the persistent storage! Mystery solved.

[estellecomment]

Disabled pouch web worker to test on alpha.dev.

[estellecomment]

Ussing https on alpha.dev doesn't fix it, still getting indexed_db_went_bad :

[garethbowen]

Maybe we don't need persistent storage. Maybe on Firefox even default storage will allow unlimited usage. It looks like it may just prompt the user when 5MB is hit: http://www.html5rocks.com/en/tutorials/offline/quota-research/ . However this conflicts with Firefox's own documentation: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria

Some testing required...

[estellecomment]

You can go over 5MB on default storage on firefox, just tested that. You don't get a popup.
That guy's research is from 2.5 yrs ago, no doubt things have evolved since then. I would trust the documentation more.

[garethbowen]

Never trust documentation.

[garethbowen]

I presume that was meant to read "You can't go over 5MB"? Because otherwise just use Firefox and we're done?

[estellecomment]

What I mean is, there is no hard limit at 5 or 50MB. You don't get an error, you can go over, I tried that.

That doesn't mean there's no limit at all :) But it's not hard-set at 5 or 50MB.

[garethbowen]

Was this on mobile? If so it sounds like a Firefox PWA without persistent storage is the way to go. Persistent storage would be good but not necessary for increasing storage and we've had trouble getting it to work on mobile.

[estellecomment]

There's still the quota depending on the free space. I haven't tested it on firefox.
https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria#Storage_limits

[estellecomment]

With 150MB of free space, and an app of 57MB, I'm not sure if I have a quota issue or not. I'm not sure what the quota issue looks like on Firefox mobile.

The app is not throwing the "db connection is closing" errors, which is what chrome does over quota.
But it's so slow that it's really not functional. Tasks and forms load, contacts don't. I'm not able to query pouch from console.
That could be a quota overload, or just a cheap phone with a lot of data and not much free disk space.

To be sure I'd have to reproduce the quota issue with more free space, i.e. with a user that has more data than that.

[estellecomment]

@browndav is testing browsers I think. What's up @browndav ?

[ghost]

I think I have a pretty good handle on this now, after writing a quick script to interact with IndexedDB directly. Short answer: not great news.

There are two documented ways I found to turn persistent storage on. MDN documents a variant of IDBFactory.open for Firefox, where the version argument can be replaced with an options argument of the form { version: 1, storage: 'persistent' }. This works perfectly on the desktop version of Firefox 48 after a single pop-up prompt, and appeares to have no practical size limit save for the size of the underlying storage device. This method doesn't work at all on the current release or beta versions of Firefox for Android – an InvalidStateError is thrown directly from the call to IDBFactory.open. I took a look through the Firefox test suite to see if there was some other calling convention/structure that I'd missed, but came up empty.

The second documented method of obtaining persistent storage is Chrome-specific – navigator.storage.persist(). This is in an "origin trial" in Chrome 52 (release), which means you need to apply to Google for a feature-enabling token on a per-domain basis. I've already submitted two requests to Google, one for lg.app and one for lg.dev. If we get a good response, we'd at least be able to test this second method.

I also did a bunch of quota tests using default (non-persistent) storage. For Firefox 48 and the latest Firefox betas, on both mobile and desktop, I was able to consistently occupy up to min(0.10 * free_space, 2 gigabytes) of device space per top-level-domain before hitting the first QuotaExceededError. That size includes Firefox's underlying SQLite databases, the write-ahead logs, everything.

One unknown piece of information remains IMO: assuming we were able to turn persistent storage on using a Google origin trial key, is the storage allowance unlimited? If not, what's the limit, and is it controllable via something like the quota API?

My instinct is for us to hedge and start figuring out how to ship a bundled Chromium 52 with our container. The patch to give us special quota treatment would be relatively straightforward IMO, but the build process/tooling setup seems high-risk.

[ghost]

Update: we have origin trial keys. I'll perform another round of quota tests using Chrome 52's navigator.storage.persist() this week and report back.

[ngaruko]

No AT required. Looked at this along with @estellecomment