Architecture.md

Architecture

The diagram above roughly describes the functionality of the library, but the general flow is as follows:

1. User creates FileData object (code link)
2. We store the Stream to a file on disk (code link)
3. We try to upload the file (with incremental backoffs on failure) (code link)
4. We update the FileData with the url of the newly uploaded file (code link)

The major components are briefly documented below.

FileData

This is an embedded object with the following structure:

{
    Id: "uuid",
    Status: "Local" | "Remote",
    Url: "string?",
    Name: "string?",
}

This is the EmbeddedObject that holds the metadata associated with the file. It is constructed with a binary source, which it then persists in a temporary location. When the object is added to Realm, we schedule the upload.

LFSManager

This is the class that contains most of the orchestration logic and for the most part deals with local files. It exposes internal methods for filesystem manipulation (reading, deleting, etc.), as well as enqueuing files for upload. There are two main file locations:

1. Temporary: this is where we store all files for FileData-s that were created, but have not yet been added to Realm. The default location is Documents/realm-lfs but can be customized by specifying LFSOptions.PersistenceLocation.
2. *path-to-realm*/.lfs: this is where we store all files that are pending upload. When FileData.GetStream() is called on a FileData with Status == Local, this is where we try to find the file from. When Status == Remote, this is where we download the file to (so the client that created the image will almost never have to download it again).

RemoteStorageManager

This is the class that handles uploads and downloads from a remote file server. Every Realm gets its own file manager and each file manager uploads files to *hashed-realm-url*/*FileData.Id*.

The interesting pieces of that class deal with parallelizing uploads. All uploads are enqueued to _uploadQueue and picked up by Executor instances. If the size of the queue exceeds twice the number of executors, a new executor is added up until MaxExecutors. Each executor dequeues an item from the queue and executes UploadItem until the queue is empty, after which it removes itself.

Before each upload, we're checking whether the item should be uploaded. There are two main cases when this check would return false:

1. We're trying to upload an item that was not created on this device (so the file doesn't exist). This would happen if we've received a FileData from another device with Status == Local and then we enqueued all pending uploads, which happens after an app restart. This could obviously be done by filtering the items at time of enqueuing them, but since everything happens on a background thread, it doesn't make much difference.
2. We're trying to upload a local item, that has already been deleted. This could be the case when the user deleted the associated FileData before the executors could pick up the item from the queue.

Note on multithreadedness: Since Realm doesn't allow accessing objects from multiple threads and we're dealing with a lot of background work here, it's important to take care not to access Realm objects/instances across different threads. This is achieved by using the BackgroundRunner class, which dispatches all Realm access on a single thread.