Offline mode #351
Comments
|
Notes from brief research on IndexedDb: |
We have very limited needs for SQL in smuggler today: only access-by-key and iteration over for a search. Is this supported by IndexDB at least? |
Yes it does! Setup of "tables", indexes and query APIs are all completely alien to what we are used to, but the capabilities are there. More notes:
|
|
This is concerning. Are the limitations on a size and persistence of indexedDB equality strict on chrome and edge? Do we know how much disk space does Mazed node takes on average? As far as I remember we do not use relations between tables in smuggler, so we should be safe in that regard. Do I miss anything? |
No, we do not. At the moment performance and size are secondary concerns from my perspective - but see the note about them at the bottom, search for "OPFS"!
I believe you are correct. More notes
Next stepsIssues with data persistence look manageable on Chrome and Firefox. On Safari they appear unacceptable long-term, but short-term we have no Safari support plans anyway. So persistence doesn't block our immediate goals. "Same-origin policy" complications on the other hand are less clear to me. It seems bad, but docs across the web are not very clear on what can and can't be done, so as a next step🦵I'll need to run some experiments. |
|
On sharing a data store between truthsayer and archaeologist:
|
|
<estimates moved to the top> |
|
Thanks a lot for the detailed write-up! A few questions:
|
We can push implementation for Firefox to the end of this list. The goal is to make it work for engineers in tech companies with a special policy on installed software, I know none who use Firefox. We can get to the point without Firefox at first |
As far as I remember, you wanted to use WASM to do it, didn't you? Can we add some small piece of code with WASM first and try to release an extension to the Chrome web store? Just to make sure they won't bloc/ban us for this. |
The biggest in my opinion was communication between truthsayer and archaeologist, but I made it work. SQLite comes second and will be my next step!
Yes, archaeologist will become a mandatory requirement to use it.
The most obvious one is to "push implementation for Firefox to the end of this list", I agree that we should use it. Just thinking at the level of big chunks of work estimated above, I don't see any obvious ones that can be removed. But once we start working on them we'll probably see sub-parts that aren't needed.
Yes, good one to cut! (see the previous answer)
Yes, sqlite itself is WASM-only in JS as far as I understand. It's a good idea to trial a WASM release, let's use the outcome of "Plug in SQLite with a trivial proof that it works" step for that. |
|
Minor update on how to pull sqlite into our code: prior to the current "official" (as in "by maintainers of sqlite itself") effort to provide WASM builds there have been a number of projects that experimented in a similar territory (see the "Attribution" section). All of them has been published to npm and can be consumed easily:
So based on the above, it's appealing to use the "official" offerings. I was surprised to find that although WASM is officially supported, there are currently no official NPM packages - we are expected to download the binaries manually from their website. This is obviously inconvenient, and luckily some folks have created a paper-thin NPM-compatible wrapper over the official build -- sqlite-wasm-esm; they also have an ongoing conversation with sqlite maintainers to help them eventually offer the same convenience out of the box. That's the package I'll experiment with first ⭐ |
Let's experiment with something smaller that sqlite and less "experimental", something that works 💯 . Any stable package with wasm inside would do. My concern is that Chrome Web Store might block Archaeologist entirely for using WASM - for some reason they afraid of it |
|
I managed to make our build process to pick up WASM dependencies properly and, as we expected, after that their usage is the same as regular JS - just do an We are, however, not good on the persistence front and as a result will not proceed with SQLite at this time. See the second part of the comment for more context if curious. What's next?In summary,
Next - implementation of something with direct usage of IndexedDb or SQLite storage optionsAside from the in-memory DB, SQLite comes with two more types of storage: Storage option 1 - OPFSThis is the one I was hoping to use, but it won't work right now. It does work "within workers", but turns out there are multiple kinds of workers. There are
SQLite's OPFS implementation requires this API and their docs say
As you might have guessed by this point, Is it possible to spawn a "dedicated worker" from background?In Manifest V2 - yes. In Manifest V3 - no, but it has been identified as an undesired regression. Chrome expressed intent to fix this, that's being tracked here. So at some point we'll be able to use OPFS in our usecase 👍, but that day is not today. Storage option 2 -
|
|
Wow, you digging really deep! Read it like an adventure story😲 Just as an idea, is there some politfill package for or dependency injection trick to replace 'window.localStorage' with 'browser.storage' without changing code of SQLite at all? |
|
So what's your plan now then? |
|
It's as described in "What's next?" section: instead of
|
Probably not -- these APIs have 1 fundamental difference which I missed originally, one is synchronous and another is asynchronous. Maintainers of sqlite were kind enough to look into this very quickly with the intension to patch their code in the next release, but identified this as the severe complication. This means we can do our hand-written implementation without sqlite with a peace of mind -- we have ruled out all the possibilities where it wouldn't cost us an arm and a leg to get sqlite to work. |
|
First pass of a background implementation of
The API is inherently time-based and yet neither of the usecases actually cares about time-related parameters of Since time-based lookups are very unattractive in a KV-storage I think I may need to split these into two distinct APIs, otherwise I'll either make |
|
Core functionality is in place for Chromium-based browsers. Bugs to fix (mostly extracted from #404 comments):
|
Only images uploading doesn't work, text files uploading works fine |
Update: Looked closer, it looks like i found the issue. @SergNikitin , if you are not looking at it already, I'll take it |
|
Done! |
Implementation plan: - ~50 hours total
Add db to archaeologist - around 8 hours totalPlug in SQLite with a trivial proof that it works - around 4 hours add POC sqlite integration to background #387Mirror smuggler's db table layout - around 4 hoursfor background, using SQL directly - around 16 hoursbrowser.storage.local- around 16 hours - implement StorageApi for browser.storage.StorageArea #396The text was updated successfully, but these errors were encountered: