Sync Methods for FileSystemSyncAccessHandle in File System Access API

[dslee414]

Wotcher TAG!

I'm requesting a TAG review of Sync Methods for FileSystemSyncAccessHandle in File System Access API.

FileSystemSyncAccessHandle is a file primitive that provides performant access to local files. One of its main use cases is applications porting C/C++ code to Wasm; however, asynchronous calls are not fully supported on Wasm yet, and using Asyncify library as an alternative has substantially degraded performance. Also, a split async/sync interface is not ergonomic, as Wasm-based applications expect a synchronous, POSIX-like file API.

Therefore, the proposal is to update asynchronous getSize(), truncate(), flush() and close() methods in FileSystemSyncAccessHandle in File System Access API to synchronous method, in line with synchronous read() and write() methods.

This change can potentially cause breakage if async methods were used the Promise .then()/.catch()/.finally() methods directly. (If used with await, there is no breakage). However, the current usage of FileSystemSyncAccessHandle is very low (e.g. zero usage queried by HttpArchive, recent shipping on Chrome-only); therefore, the scope and the impact of breakage is expected to be very minimal. Also, FileSystemSyncAccessHandle was shipped recently on Chrome M102 and is in the process of being implemented in Firefox, so it is preferred to make this change as soon as possible before the API gets used more widely.

• Explainer: https://github.com/WICG/file-system-access/blob/main/EXPLAINER.md
• User research: N/A
• Security and Privacy self-review²: https://github.com/WICG/file-system-access/blob/main/security-privacy-questionnaire.md
• GitHub repo: whatwg/fs
• Primary contacts (and their relationship to the specification):
  • Austin Sullivan (@a-sully), Google Chrome
  • Daseul Lee (@dslee414), Google Chrome
• Organization/project driving the design: Google/Chromium
• External status/issue trackers for this feature: https://chromestatus.com/feature/5149644305203200

Further details:

• I have reviewed the TAG's Web Platform Design Principles
• The group where the incubation/design work on this is being done (or is intended to be done in the future): whatwg (initial work was started WICG, but the relevant parts to this proposal have been moved to whatwg)
• The group where standardization of this work is intended to be done ("unknown" if not known): whatwg
• Existing major pieces of multi-stakeholder review or discussion of this design:
  FileSystemSyncAccessHandle contains async methods whatwg/fs#7
  Multiple asynchronous operations whatwg/fs#28
• Major unresolved issues with or opposition to this design:
• This work is being funded by: Google

You should also know that...

• FileSystemSyncAccessHandle is available in Worker-only, thereby all synchronous calls are invoked from Workers, not from window.
• There is ongoing discussion and developer demand around supporting an Async-version of access handle (name TBD for this performant file primitive). Note that this proposal is about making all methods sync within FileSystemSyncAccessHandle, and separately, we plan on supporting an async access handle, which may be more appropriate for common use cases, following asynchronous design patterns.

We'd prefer the TAG provide feedback as (please delete all but the desired option):
💬 leave review feedback as a comment in this issue and @-notify [github usernames]
whatwg/fs#7
@dslee414, @a-sully

[littledan]

I'm wondering, what's the motivation for the sync API? I think async APIs should be friendly to WebAssembly in the near future, given that WebAssembly JavaScript Promise Integration reached Phase 3 in the Wasm CG.

[a-sully]

There's a lot of great discussion by WASM developers on whatwg/fs#7 which illustrate the need for this change. This comment (sampled below) provides a pretty good summary of my thoughts:

From my perspective as a Chromium engineer, we are currently focused on making SyncAccessHandles as performant as possible to support C(++) applications being ported to the web via WASM, but we are very interested to hear developer feedback about whether an asynchronous alternative to SyncAccessHandles is worth pursuing. One could imagine an async alternative could be more webby: available from Window contexts etc, using a streams-based interface with built-in queueing, etc. To this point we haven't seen much developer demand for this async alternative. If you have compelling use cases for an async alternative, we'd love to hear your feedback :)

Okay, so here's the reality of the current landscape:

• asynchronous WASM is not performant enough to support the most powerful web applications (for now!)
• applications porting C(++) code to WASM expect a synchronous, Posix-like file API
• a split sync/async interface is a no-win compromise

I'm excitedly following the developments to improve async support in WASM. My hope is that this will allow an async alternative to SyncAccessHandles to have comparable performance characteristics, at which point any web developer looking for fast storage will have no reason to choose the more restrictive SyncAccessHandles.

That being said, it's also unclear to me whether C(++) applications being ported to WASM would be able to use or benefit from this async alternative as easily as SyncAccessHandles. There may always be a need for a synchronous file API to support porting these applications to the web.

The current split interface is the worst of both worlds. We'll never be able to expose SyncAccessHandles to anything other than DedicatedWorkers due to the existence of sync methods. However, the split sync/async interface forces the application to write effectively sync code anyways, since you can't queue operations due to the sync methods. The result is a significantly slower and more complicated interface than if all the methods were sync. I'd like to see SyncAccessHandles be the MOST useful for WASM-compiled applications, while keeping an ear out for WASM ecosystem async support + developer demand for an async alternative.

[littledan]

I am having trouble following this thread. It sounds like the user-facing requirement is that some C++ code can make a blocking call. My understanding is that this is what WebAssembly JS Promise integration provides. These are two ways to implement the same high-level user requirement; it's not the kind of thing user demand will inform one way or the other.

Is the concern that JSPI is lower performance? Do you have any benchmarks that show this penalty? I wonder if this benchmark can be used to fix/further optimize JSPI (whether in its core design or various implementations of it), regardless of in what form this feature ships.

The reason I am interested in this area is, I worked on the design of JavaScript WeakRefs, where we also had a sync method created for Wasm, and removed it for the reasons @fgmccabe mentions in the thread. I am wondering how we should handle similar cases going forward (as this question can reasonably come up for many async APIs, hence it being a relevant concern for the TAG).

[a-sully]

If the high-level concern here is that we don't want to end up in a world with sync and async versions of every API, then, well, I agree :) This API is a particularly performance-oriented exception, not the rule.

The purpose of this API is performance. SyncAccessHandles are intended to provide the most low-level persistent file primitive to the web. This primitive allows sites to bring their own database to the web via Wasm in a way that's orders of magnitude faster than IndexedDB, for example.

Also I realize this isn't clear based on the issue description, but SyncAccessHandles are only available from DedicatedWorkers specifically because we don't want to impact the responsiveness of sites.

[hcldan]

This broke our shipped code. We were using an API that landed in a release of chrome.
There needs to be a better way to handle things like this.

We have a POSIX filesystem built on top of these APIs (we do not have the problems of asyncify, not using it) that we've shipped to customers, we were relying on the APIs that landed in a release of chrome (after a long time of being roped off, mind you) and this change broke our customers. We now have to release an emergency fix. Please, PLEASE, find other ways to do this in the future.

[tomayac]

Hi @hcldan! Really sorry to hear that. We did have a blog post aimed at developers that described the breaking change: https://developer.chrome.com/blog/sync-methods-for-accesshandles/. I recommend you subscribe to the blog’s RSS feed if you haven’t already. Again, sorry this broke your product. If there’s better ways we could have reached developers like you or if there are places you looked at but that didn’t hint at the upcoming breakage, please let me know. Ideally in private, as to not spam this thread with unrelated stuff. Thanks!

[hober]

What's the current state here? Given that the main motivation for this is WASM and the WASM people are saying they are working on it from their end (WASM Promises Integration is at Phase 3), is it still necessary to have this feature?

[a-sully]

The fully-synchronous interface* is shipped and in active use on all major browsers. Given the usage, I don't think deprecating this feature if feasible, if that's what you're asking.

You bring up a good point, though. Given the progress we're seeing with JSPI would we create the FileSystemSyncAccessHandle interface at all if we had to do it over again?... I'm not sure. The interface is commonly implemented as a very thin wrapper around a file descriptor. The sync-ness of the JavaScript API continues down the stack - once you hit C++ it's sync calls all the way down**.

I'm not sure what the performance delta is between JSPI and Asyncify, but presumably the basically-no-overhead approach of FileSystemSyncAccessHandle is faster than what JSPI could possibly be. Which suggests that a sync interface has merits even in a world with JSPI :)

That being said, JSPI makes an async alternative to FileSystemSyncAccessHandle (which could be used from the main thread) much more attractive. WASM folks have already expressed interest in this: whatwg/fs#41 (comment)

*note that this specific issue only tracks making the entire FileSystemSyncAccessHandle interface sync. It was initially shipped with a split sync/async interface, which, as I described, in whatwg/fs#7 (comment), was a "worst of both worlds" situation. By the time this issue was filed it was less a question of "should this sync API exist" but "how can we fix what we've already shipped"

**until you hit the device driver, at least. Though arguably the decision for browsers to implement the FileSystemSyncAccessHandle interface using synchronous file IO is just an implementation detail, since browsers could choose to use async IO under the hood

[plinss]

Given that this shipped we're closing the review. While we see the use cases, we do have concerns about adding more synchronous APIs where async may have been more appropriate. We hope that once WASM Promises Integration is available that there is some thought given to deprecating these methods.