Cancel queue + async usercall interface #404

mzohreva · 2022-07-28T23:45:57Z

Combined and rebased two long open PRs (#286 and #291) in a separate branch (some commits are quashed).

raoulstrackx · 2022-08-09T14:23:25Z

intel-sgx/fortanix-sgx-abi/src/lib.rs

+        pub fn async_queues(
+            usercall_queue: *mut FifoDescriptor<Usercall>,
+            return_queue: *mut FifoDescriptor<Return>,
+            cancel_queue: *mut FifoDescriptor<Cancel>


This could lead to issues like:

thread 'main' panicked at 'assertion failed: `(left == right)` left: `94410652213776`, right: `0`: Usercall async_queues: expected 3rd argument to be 0', intel-sgx/enclave-runner/src/usercalls/abi.rs:311:1 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This happens when on older enclave runner is used to run a new async enclave that provides a cancel_queue to the async_queues usercall. I'm wondering if we can provide a nicer error message as users has ran into this issue and probably will again when this PR is merged an published.

Probably, but we'll need to update the crates in a separate PR and publish enclave-runner before this PR lands and hope that users use that version before switching to this one (older ones will still panic with the same error). I'm not sure it's worth the effort. I don't think we can reliably detect the enclave-runner version in the enclave to produce a better error in the enclave.

Shouldn't we have some explicit versioning in the runner/enclave interface so the runner and the enclave can negotiate whether they can successful run? Something like an API version that the runner can pass to the enclave, and the enclave can return an error if that API version isn't supported?

Which won't help us right now, since we don't have the versioning mechanism in place yet. But I think we should assume that we need to make more breaking changes in the future at some point. Having a mechanism in place that lets us get a clear error message when there's incompatibility between the runner and the enclave will make it easier in the future when such transitions happen again.

I created a separate issue tracking this: #443

intel-sgx/enclave-runner/src/usercalls/mod.rs

raoulstrackx · 2022-08-09T14:42:09Z

intel-sgx/enclave-runner/src/usercalls/mod.rs

@@ -515,7 +516,7 @@ struct PendingEvents {

 impl PendingEvents {
    // Will error if it doesn't fit in a `u64`
-    const EV_MAX_U64: u64 = (EV_USERCALLQ_NOT_FULL | EV_RETURNQ_NOT_EMPTY | EV_UNPARK) + 1;
+    const EV_MAX_U64: u64 = (EV_USERCALLQ_NOT_FULL | EV_RETURNQ_NOT_EMPTY | EV_UNPARK | EV_CANCELQ_NOT_FULL) + 1;


Shouldn't EV_ABORT be added here as well?

No, this type handles abort separately, see line 512 and associated functions of PendingEvents

async-usercalls/src/queues.rs

async-usercalls/src/duplicated.rs

intel-sgx/async-usercalls/src/duplicated.rs

intel-sgx/async-usercalls/src/provider_api.rs

intel-sgx/async-usercalls/src/lib.rs

405: Add cancel queue to SGX ABI r=raoulstrackx a=mzohreva We need to publish `fortanix-sgx-abi` with this change and then update libstd's dependency on `fortanix-sgx-abi` so we can remove the hacks in #404. Co-authored-by: Mohsen Zohrevandi <mohsen.zohrevandi@fortanix.com>

Update fortanix-sgx-abi to 0.5.0 to add support for cancel queue (see fortanix/rust-sgx#405 and fortanix/rust-sgx#404). Export some useful traits for processing SGX usercall. This is needed for fortanix/rust-sgx#404 to avoid duplication.

…queue, r=Mark-Simulacrum Update fortanix-sgx-abi and export some useful SGX usercall traits Update `fortanix-sgx-abi` to 0.5.0 to add support for cancel queue (see fortanix/rust-sgx#405 and fortanix/rust-sgx#404). Export some useful traits for processing SGX usercall. This is needed for fortanix/rust-sgx#404 to avoid duplication. cc `@raoulstrackx` and `@jethrogb`

Update fortanix-sgx-abi to 0.5.0 to add support for cancel queue (see fortanix/rust-sgx#405 and fortanix/rust-sgx#404). Export some useful traits for processing SGX usercall. This is needed for fortanix/rust-sgx#404 to avoid duplication.

…Mark-Simulacrum Update fortanix-sgx-abi and export some useful SGX usercall traits Update `fortanix-sgx-abi` to 0.5.0 to add support for cancel queue (see fortanix/rust-sgx#405 and fortanix/rust-sgx#404). Export some useful traits for processing SGX usercall. This is needed for fortanix/rust-sgx#404 to avoid duplication. cc `@raoulstrackx` and `@jethrogb`

Taowyoo · 2023-04-26T20:43:33Z

I cherry pick the 6b2a2cb to fix the CI, need to remove it once the #433 is addressed

intel-sgx/async-usercalls/src/lib.rs

raoulstrackx · 2023-04-28T09:21:29Z

intel-sgx/async-usercalls/test.sh

+#!/bin/bash
+
+# Run this in parallel with:
+# $ cargo test --target x86_64-fortanix-unknown-sgx --release -- --nocapture --ignored echo


Can we avoid this script?

raoulstrackx · 2023-04-28T12:44:39Z

ipc-queue/src/lib.rs

+/// arranged as a ring buffer, we can assign a position to each value written/
+/// read to/from the queue. This is useful in case we want to know whether or
+/// not a particular value written to the queue has been read.
+pub struct PositionMonitor<T: 'static> {


The implementation of PositionMonitor is in position.rs. That's confusing. Can you move it to there as well, and export it here?

raoulstrackx · 2023-04-28T12:47:05Z

ipc-queue/src/lib.rs

+/// read to/from the queue. This is useful in case we want to know whether or
+/// not a particular value written to the queue has been read.
+pub struct PositionMonitor<T: 'static> {
+    read_epoch: Arc<AtomicU64>,


What's the purpose of read_epoch?

raoulstrackx · 2023-04-28T12:48:51Z

ipc-queue/src/position.rs

+    pub fn read_position(&self) -> ReadPosition {
+        let current = self.fifo.current_offsets(Ordering::Relaxed);
+        let read_epoch = self.read_epoch.load(Ordering::Relaxed);
+        ReadPosition(((read_epoch as u64) << 32) | (current.read_offset() as u64))


read_epoch itself is a u64. This assumes it will never grow larger than 2^32 before things start wrapping around. How reasonable is this? Can we make read_epoch an AtomicU32 instead?

TODO: check if that is actually reachable

Alternatively, panic instead of silently do the wrong stuff.

raoulstrackx · 2023-04-28T13:57:03Z

intel-sgx/async-usercalls/src/queues.rs

+    let return_queue = unsafe { User::<FifoDescriptor<Return>>::from_raw(return_q) }.to_enclave();
+
+    // FIXME: once `WithId` is exported from `std::os::fortanix_sgx::usercalls::raw`, we can remove
+    // `transmute` calls here and use FifoDescriptor/WithId from std everywhere including in ipc-queue.


I think the fortanix-sgx-abi crate has been update in the compiler and we can get rid of these transmutes

raoulstrackx · 2023-04-28T14:55:38Z

intel-sgx/async-usercalls/src/io_bufs.rs

+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.read == self.write


Shouldn't the modulo of these offsets be taken here as well?

Using len() would keep most of these details in the len() and offsets() functions.

raoulstrackx · 2023-04-28T14:58:56Z

intel-sgx/async-usercalls/src/io_bufs.rs

+
+    fn is_full(&self) -> bool {
+        let (read_offset, write_offset) = self.offsets();
+        read_offset == write_offset && self.read != self.write


Using len() would keep most of these details in the len() and offsets() functions.

`UsercallEvent::Start` was being sent in `fn handle_usercall`, which is too late. It needs to be sent before we receive the next usercall from the enclave so we can maintain the invariant that "we only need to keep track of cancels received before the actual usercall if the read position has not moved past the write position when cancel was received."

- Move async-usercalls to intel-sgx directory - Remove hacks/unsafe_typecasts.rs - Fix some typos in docs - Use marker trait for MakeSend to avoid warnings about issue #93367 - Update crossbeam and crossbeam-channel dependencies - Use nightly Rust in CI - Use SGX target for generating docs when crate has `feature(sgx_platform)`

Credits for this commit go to: Mohsen: fortanix#404 YxC: fortanix#441 This commit is an attempt to have the async-usercalls finally merged into the main codebase (master branch).

Credits for this commit go to: Mohsen: #404 YxC: #441 This commit is an attempt to have the async-usercalls finally merged into the main codebase (master branch).

Credits for this commit go to: Mohsen: fortanix#404 YxC: fortanix#441 This commit is an attempt to have the async-usercalls finally merged into the main codebase (master branch).

Taowyoo · 2024-01-27T00:33:56Z

I am going to close this since merged #515 covered code change in this PR.
Please be free to reopen this if needed!

mzohreva requested review from aandyl, jethrogb and raoulstrackx July 28, 2022 23:45

mzohreva force-pushed the mz/new-async-usercalls branch 2 times, most recently from 78e5b39 to f0f6beb Compare August 1, 2022 18:14

raoulstrackx reviewed Aug 12, 2022

View reviewed changes

mzohreva mentioned this pull request Aug 12, 2022

Add cancel queue to SGX ABI #405

Merged

mzohreva mentioned this pull request Aug 16, 2022

Update fortanix-sgx-abi and export some useful SGX usercall traits rust-lang/rust#100642

Merged

mzohreva force-pushed the mz/new-async-usercalls branch from f0f6beb to 58056e7 Compare August 16, 2022 20:13

Taowyoo force-pushed the mz/new-async-usercalls branch from 39895b5 to e93163f Compare April 26, 2023 20:10

Taowyoo mentioned this pull request Apr 26, 2023

Bump dependency of mbedtls to 0.9.0 #439

Closed

Taowyoo requested review from raoulstrackx and Pagten and removed request for aandyl April 26, 2023 20:41

Taowyoo mentioned this pull request Apr 27, 2023

Bump dependency of mbedtls to 0.10.0 #441

Closed

raoulstrackx reviewed Apr 28, 2023

View reviewed changes

mzohreva mentioned this pull request Apr 28, 2023

Design a versioning mechanism for enclave/runner interface #443

Open

mzohreva and others added 6 commits June 8, 2023 13:16

Cancel queue

4be56c4

Async usercall interface for SGX enclaves

20a97a1

Verify allocation FifoDescriptor

6de101b

Making read_epoch AtomicU64

a962ba5

mzohreva and others added 5 commits June 8, 2023 13:16

Update fortanix-sgx-abi dependency and restore path dependencies

7263daa

Add documentation for async-usercalls and address other minor issues

b34df5e

Remove almost all hacks

08ff823

resolve rebase conflict

0bf1a93

Pin rust toolchain version in travis.yml

8249d26

Taowyoo force-pushed the mz/new-async-usercalls branch from 6b2a2cb to 8249d26 Compare June 8, 2023 20:16

BelalH mentioned this pull request Jul 21, 2023

Upstream SGX Async I/O #447

Open

Taowyoo mentioned this pull request Jul 26, 2023

Support asynchronous wait usercalls #290

Open

vn971 mentioned this pull request Aug 28, 2023

Async usercall interface for SGX enclaves #515

Merged

raoulstrackx pushed a commit that referenced this pull request Jan 4, 2024

Async usercall interface for SGX enclaves

0dc531f

Credits for this commit go to: Mohsen: #404 YxC: #441 This commit is an attempt to have the async-usercalls finally merged into the main codebase (master branch).

vn971 pushed a commit that referenced this pull request Jan 4, 2024

Async usercall interface for SGX enclaves

1e6a9d4

Credits for this commit go to: Mohsen: #404 YxC: #441 This commit is an attempt to have the async-usercalls finally merged into the main codebase (master branch).

Taowyoo closed this Jan 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cancel queue + async usercall interface #404

Cancel queue + async usercall interface #404

mzohreva commented Jul 28, 2022

raoulstrackx Aug 9, 2022

mzohreva Aug 12, 2022

arai-fortanix Apr 28, 2023

arai-fortanix Apr 28, 2023

mzohreva Apr 28, 2023

raoulstrackx Aug 9, 2022

mzohreva Aug 12, 2022

Taowyoo commented Apr 26, 2023

raoulstrackx Apr 28, 2023

raoulstrackx Apr 28, 2023

raoulstrackx Apr 28, 2023

raoulstrackx Apr 28, 2023

vn971 Sep 4, 2023

vn971 Sep 4, 2023

raoulstrackx Apr 28, 2023

raoulstrackx Apr 28, 2023

raoulstrackx Apr 28, 2023

raoulstrackx Apr 28, 2023

Taowyoo commented Jan 27, 2024

Cancel queue + async usercall interface #404

Cancel queue + async usercall interface #404

Conversation

mzohreva commented Jul 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Taowyoo commented Apr 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Taowyoo commented Jan 27, 2024