[IO] Rewrite strategy

[ferndot]

This is an issue to track and explain the implementation of the new IO script structure.

Explanation

IO.js will now offer a public API that allows the addition of storage methods (we call them "systems"). These systems will be defined in other, API-specific scripts. When IO.js receives a read/write/enumerate/etc... request, it will channel the request to the script corresponding to the storage method. The benefit of this implementation is a flexible and modular IO script that can adapt to various storage methods, but still provide an easy API for the app to use.

Get involved

We need help planning and documenting the API. If you have insights to offer, please suggest them in this thread. Our WIP API specification is here.

Coding will be done in the filesystem-2 branch.

[twiss]

Are you currently working on this (the filesystem-2 branch)? If not I'll try and get it in a working state.

[ferndot]

@twiss: I was...until my computer started having major issues that prevent me from using git. I need to set up an online IDE to continue development.

It would be wonderful if you could work on this! Have you taken a look at the WIP API specification? If it needs changes, feel free to suggest them here. Also, it might be a good idea to fix #194 before proceeding. I will try to do that soon.

[twiss]

Thanks for picking up #194, is your pc behaving better? I don't much mind helping with this, but you'd probably do a quicker and better job. Do you want me to take it anyway?

[ferndot]

My computer is not doing any better, but I created a Codio environment so that I could develop again (https://codio.com/joshua-s/firetext).

I could use all the help I can get with this issue. It is actually quite extensive (this is a meta issue), and will be somewhat complicated to implement.

[twiss]

Questions after reading the spec:

(Much of this isn't super important and if you've already started or don't feel like arguing about function signatures feel free to just move forward.)

io.initialized

Event that is called when the IO script has been completely loaded. It is recommended for storage systems to wait for this before trying to add themselves.

Why is this needed?

io.isInitialized

Nit: it's unclear what the value of this is while io.initialized is called.

io.systems.add()

systemid is important because it enables systems to be "remembered" over multiple sessions.

What does this function do, exactly? This sentence talks about sessions, implying that systems are stored in sessionStorage or so, which seems impossible. Same for storages.

It would be nice to have a little intro which talks about systems and storages and when to add and remove them and what that does.

Nit: what about (id,api,callback)?

Also, for systems and storages, do we need callbacks or are they for consistency?

(...) error code

Are those strings? In any case it would be good to have a table of them here at some point.

io.storages.add()

Nit: what about (name,systemId,callback)? Or is there a logical use case for manual ids?

io.storages.get()

Nit: what about getAll() or ids()?

Also, how do you get the storages' names? Maybe storages.getAll() -> [{id, name}...]?

io.files.enumerate()

(...) io.files.enumerate(storageid,directory,callback,deep)

Are we really going to use (and implement) deep? (That's partly a genuine question, we should look at how we (are going to) use enumerate and decide based on that. And performance, meaning defer judgment.) Also nit: I'd prefer directory[,deep],callback. (#174)

io.files.write()

Nit: I understand the sorting, but maybe put this below add()?

---

Listening for changes is missing. (Do all storage methods support that at all? If not maybe we should implement polling, or drag to refresh, but we should still have an api for those that do.)

This depends on how we're going to implement renaming, and on whether any storage methods support it, but maybe it would be nice to have a specialistic rename()/move() method. (#119)

Both Device Storage and Dropbox have a mechanism where you get a file revision id on read and pass that on write (although the behavior between the two is different). Maybe we should support that at some point? (#35 (comment) 3 + 4)

Nit: for readability, it would be nice to subdivide Events, Systems, etc. with headers.

---

Looking forward to this! (And again, not necessarily all of this has to happen immediately.)

[ferndot]

io.initialized

Why is this needed?

I am using it to provide something for storage scripts to wait for before they add themselves. See for example deviceStorage.js.

io.isInitialized

Nit: it's unclear what the value of this is while io.initialized is called.

I added this for the case where a storage script is added after the module is loaded. It may not be necessary...

io.systems.add()
systemid is important because it enables systems to be "remembered" over multiple sessions.

What does this function do, exactly? This sentence talks about sessions, implying that systems are stored in sessionStorage or so, which seems impossible. Same for storages.

It would be nice to have a little intro which talks about systems and storages and when to add and remove them and what that does.

Nit: what about (id,api,callback)?

Also, for systems and storages, do we need callbacks or are they for consistency?

This function "registers" an API script with the main IO handler. I added this function so that the IO script does not depend upon any of the APIs.

I was intending for the app to "remember" systems and storages so that we could remember recently-opened documents. Unfortunately, storages like the sdcard do not have a system-set unique identifier, so I don't know how we could do this. The other purpose for the systemid is to uniquely identify the system/storage during the current session. The app will use these IDs instead of passing around storage objects.

I will put an intro together.

I used the (system,callback,systemid) order because systemid is optional. The script passes it in if it received a systemid in a previous session (See here).

(...) error code

Are those strings? In any case it would be good to have a table of them here at some point.

I think that they will need to be strings. A table would definitely be useful!

io.storages.add()

Nit: what about (name,systemId,callback)? Or is there a logical use case for manual ids?

If the storage has been given an ID previously, it could pass in that ID so that a new one would not be generated. This would enable us to remember recent documents for that storage. I don't know if this is feasible or not.

io.storages.get()

Nit: what about getAll() or ids()?

Also, how do you get the storages' names? Maybe storages.getAll() -> [{id, name}...]?

That is definitely less ambiguous.

io.files.enumerate()
(...) io.files.enumerate(storageid,directory,callback,deep)

Are we really going to use (and implement) deep? (That's partly a genuine question, we should look at how we (are going to) use enumerate and decide based on that. And performance, meaning defer judgment.) Also nit: I'd prefer directory[,deep],callback. (#174)

I originally thought that we could enumerate all files and store them in an array. That way, we could change directories without needing to enumerate again. Unfortunately, that may not be performant with large storages, so maybe we should remove that option.

io.files.write()

Nit: I understand the sorting, but maybe put this below add()?

Makes sense!

[ferndot]

Listening for changes is missing. (Do all storage methods support that at all? If not maybe we should implement polling, or drag to refresh, but we should still have an api for those that do.)

I know that deviceStorage supports it. Maybe, we should create a io.storages.notifyUpdate() function that scripts could call when they detect a change. That way, each individual API could detect changes themselves?

This depends on how we're going to implement renaming, and on whether any storage methods support it, but maybe it would be nice to have a specialistic rename()/move() method. (#119)

That would indeed be nice. Should we delegate this to individual API scripts, or use the IO script to perform the following?:

• io.files.read() where the filepath is the old location/name.
• io.files.add() where the filepath is the target location/name and the blob is the old file.
• io.files.remove() where the filepath is the old location/name.

Note to self: add() should have a filepath parameter instead of a name parameter.

Both Device Storage and Dropbox have a mechanism where you get a file revision id on read and pass that on write (although the behavior between the two is different). Maybe we should support that at some point? (#35 (comment) 3 + 4)

That is a good idea. How would we accomplish this?

Nit: for readability, it would be nice to subdivide Events, Systems, etc. with headers.

👍!

[twiss]

io.initialized

Why is this needed?

I am using it to provide something for storage scripts to wait for before they add themselves. See for example deviceStorage.js.

Yes, but why should they wait? Is it for performance, or are they waiting for something? All scripts in index.html already have defer.

(...)

io.systems.add()
systemid is important because it enables systems to be "remembered" over multiple sessions.

What does this function do, exactly? This sentence talks about sessions, implying that systems are stored in sessionStorage or so, which seems impossible. Same for storages.

It would be nice to have a little intro which talks about systems and storages and when to add and remove them and what that does.

Nit: what about (id,api,callback)?

Also, for systems and storages, do we need callbacks or are they for consistency?

This function "registers" an API script with the main IO handler. I added this function so that the IO script does not depend upon any of the APIs.

I was intending for the app to "remember" systems and storages so that we could remember recently-opened documents.

I see, I didn't think about that.

Unfortunately, storages like the sdcard do not have a system-set unique identifier, so I don't know how we could do this. The other purpose for the systemid is to uniquely identify the system/storage during the current session. The app will use these IDs instead of passing around storage objects.

I will put an intro together.

I used the (system,callback,systemid) order because systemid is optional. The script passes it in if it received a systemid in a previous session (See here).

Thanks. My main confusion was that this function somehow does something with systems and sessions, but it doesn't, that's left to the caller.

(...)

io.files.enumerate()
(...) io.files.enumerate(storageid,directory,callback,deep)

Are we really going to use (and implement) deep? (That's partly a genuine question, we should look at how we (are going to) use enumerate and decide based on that. And performance, meaning defer judgment.) Also nit: I'd prefer directory[,deep],callback. (#174)

I originally thought that we could enumerate all files and store them in an array. That way, we could change directories without needing to enumerate again. Unfortunately, that may not be performant with large storages, so maybe we should remove that option.

Maybe we could prefetch one extra level of directories as to make directory opening fast.

(...)

As to the last points, mostly I don't know. Particularly,

Device Storage [has] a mechanism where you get a file revision id on read and pass that on write

I thought I read this in the docs somewhere, but I can't find it now. Maybe they removed it or I confused it with something else. Mostly we'll have to figure it out when implementing.