add `{.async: (raises).}` to `libp2p/stream` modules #1050

etan-status · 2024-03-02T22:48:20Z

Annotate stream modules with {.async: (raises).} to simplify exception handling.

close: {.async: (raises: []).}
read/write: {.async: (raises: [CancelledError, LPStreamError]).}

All modules need to be updated atomically, otherwise the method mechanism does not find overrides anymore. Also, because close functions have a tendency to start writing RST packets and reading EOF packets, close and read/write need to be done together.

Higher modules have not been touched unless necessary, e.g., to remain compatible when plugging in callbacks.

Annotate `stream` modules with `{.async: (raises).}` to simplify exception handling. - `close`: `{.async: (raises: []).}` - `read`/`write`: `{.async: (raises: [CancelledError, LPStreamError]).}` All modules need to be updated atomically, otherwise the `method` mechanism does not find overrides anymore. Also, because `close` functions have a tendency to start writing RST packets and reading EOF packets, `close` and `read`/`write` need to be done together. Higher modules have not been touched unless necessary, e.g., to remain compatible when plugging in callbacks.

codecov · 2024-03-02T23:50:15Z

Codecov Report

Attention: Patch coverage is 83.92371% with 59 lines in your changes are missing coverage. Please review.

Project coverage is 82.76%. Comparing base (8294d5b) to head (393ae49).

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1050      +/-   ##
============================================
+ Coverage     82.73%   82.76%   +0.03%     
============================================
  Files            91       91              
  Lines         15586    15666      +80     
============================================
+ Hits          12895    12966      +71     
- Misses         2691     2700       +9

Files	Coverage Δ
libp2p/muxers/mplex/coder.nim	`92.68% <100.00%> (+2.93%)`	⬆️
libp2p/protocols/connectivity/relay/rconn.nim	`97.36% <100.00%> (-2.64%)`	⬇️
libp2p/transports/wstransport.nim	`84.97% <93.33%> (+1.06%)`	⬆️
libp2p/upgrademngrs/muxedupgrade.nim	`85.24% <88.88%> (+1.37%)`	⬆️
libp2p/muxers/mplex/lpchannel.nim	`82.60% <93.33%> (+0.74%)`	⬆️
libp2p/protocols/secure/noise.nim	`90.09% <83.33%> (+0.43%)`	⬆️
libp2p/stream/bufferstream.nim	`93.84% <88.88%> (+4.69%)`	⬆️
libp2p/stream/connection.nim	`91.13% <90.90%> (+1.95%)`	⬆️
libp2p/stream/chronosstream.nim	`84.46% <86.95%> (+1.13%)`	⬆️
libp2p/protocols/secure/secure.nim	`72.13% <80.00%> (-0.60%)`	⬇️
... and 5 more

... and 14 files with indirect coverage changes

etan-status · 2024-03-03T23:31:03Z

define proper parent error type for YamuxError #1040

tests/helpers.nim

arnetheduck

this broadly LGTM - I think it's useful to merge this PR as a starting point since most of it is semi-mechanical - then we can deal with polish separately

Co-authored-by: Jacek Sieka <jacek@status.im>

dryajov

Agree, this looks good enough right now.

dryajov · 2024-03-05T01:40:28Z

Bumping to latest master with the GitHub update seems to break it.

diegomrsantos · 2024-03-13T19:40:25Z

libp2p/muxers/mplex/lpchannel.nim

    elif s.name != $s.oid and s.name.len > 0:
      &"{shortLog(s.conn.peerId)}:{s.oid}:{s.name}"
    else: &"{shortLog(s.conn.peerId)}:{s.oid}"
  except ValueError as exc:
-    raise newException(Defect, exc.msg)
+    raiseAssert(exc.msg)


@etan-status should we be raising a Defect here?

raiseAssert raises a Defect :)

libp2p/muxers/mplex/lpchannel.nim

diegomrsantos · 2024-03-13T19:57:43Z

libp2p/muxers/mplex/mplex.nim

@@ -68,19 +68,19 @@ proc cleanupChann(m: Mplex, chann: LPChannel) {.async, inline.} =
      libp2p_mplex_channels.set(
        m.channels[chann.initiator].len.int64,
        labelValues = [$chann.initiator, $m.connection.peerId])
-  except CatchableError as exc:
+  except CancelledError as exc:


can't m.channels[chann.initiator] raise KeyError?

no, m.channels is an array[bool, ...], not a Table, so both false and true always exist.

diegomrsantos · 2024-03-13T20:06:51Z

libp2p/muxers/muxer.nim

-  T: typedesc[MuxerProvider],
-  creator: MuxerConstructor,
-  codec: string): T {.gcsafe.} =
+method handle*(m: Muxer): Future[void] {.base, async: (raises: []).} = discard


should we use raiseAssert("Not implemented!") here too?

It wasn't done before, so rather didn't want to change behaviour. Maybe the base is sometimes called for default behaviour

diegomrsantos · 2024-03-13T20:16:39Z

libp2p/muxers/yamux/yamux.nim

-method handle*(m: Yamux) {.async.} =
+  await m.streamHandler(channel)
+  trace "finished handling stream"
+  doAssert(channel.isClosed, "connection not closed by handler!")


do we want to raise a Defect here?

it's not a new Defect, it's the same code from before, including the assert, except with CatchableError handling removed because m.streamHandler cannot raise errors.

diegomrsantos · 2024-03-13T20:23:18Z

libp2p/muxers/yamux/yamux.nim

+    try:
+      await m.connection.write(YamuxHeader.goAway(ProtocolError))
+    except CancelledError, LPStreamError:
+      discard


we can extract it to a proc and avoid the repetition

sure, but it's only 2 occurrences. the proc has about the same overhead.

diegomrsantos · 2024-03-13T20:43:00Z

libp2p/protocols/secure/noise.nim

    else: &"{shortLog(conn.peerId)}:{conn.oid}"
  except ValueError as exc:
-    raise newException(Defect, exc.msg)
+    raiseAssert(exc.msg)


should we raise a Defect here?

raiseAssert is very similar to raise newException(Defect)

diegomrsantos · 2024-03-13T21:02:02Z

libp2p/protocols/secure/secio.nim

    else: &"{shortLog(conn.peerId)}:{conn.oid}"
  except ValueError as exc:
-    raise newException(Defect, exc.msg)
+    raiseAssert(exc.msg)


Same as before.

diegomrsantos · 2024-03-13T21:03:12Z

libp2p/stream/bufferstream.nim

    else: &"{shortLog(s.peerId)}:{s.oid}"
  except ValueError as exc:
-    raise newException(Defect, exc.msg)
+    raiseAssert(exc.msg)


Same as before.

diegomrsantos · 2024-03-13T21:04:41Z

libp2p/stream/bufferstream.nim

+    s: BufferStream,
+    pbytes: pointer,
+    nbytes: int
+): Future[int] {.async: (raises: [CancelledError, LPStreamError]).} =
  doAssert(nbytes > 0, "nbytes must be positive integer")
  doAssert(not s.reading,


should we be raising Defects here?

Defects can always be raised, regardless of {.async: (raises).} annotation. they are not newly introduced

diegomrsantos · 2024-03-13T21:06:42Z

libp2p/stream/bufferstream.nim

@@ -209,8 +208,8 @@ method closeImpl*(s: BufferStream): Future[void] =
        if not s.readQueue.empty():
          discard s.readQueue.popFirstNoWait()
  except AsyncQueueFullError, AsyncQueueEmptyError:
-    raise newException(Defect, getCurrentExceptionMsg())
+    raiseAssert(getCurrentExceptionMsg())


Same as before.

diegomrsantos · 2024-03-13T21:07:28Z

libp2p/stream/chronosstream.nim

    else: &"{shortLog(conn.peerId)}:{conn.oid}"
  except ValueError as exc:
-    raise newException(Defect, exc.msg)
+    raiseAssert(exc.msg)


Same as before.

diegomrsantos · 2024-03-13T21:19:26Z

libp2p/stream/connection.nim

@@ -58,23 +58,28 @@ method initStream*(s: Connection) =

  procCall LPStream(s).initStream()

-  doAssert(isNil(s.timerTaskFut))
+  doAssert(s.timerTaskFut == nil)


Same as previous comments about Defects.

diegomrsantos · 2024-03-13T21:20:30Z

libp2p/stream/connection.nim

-      s.timeoutHandler = proc(): Future[void] =
-        trace "Idle timeout expired, closing connection", s
-        s.close()
+    if s.timeoutHandler == nil:


why == nil instead of isNil?

status-im/nimbus-eth2#4590 (comment)

diegomrsantos · 2024-03-13T21:23:16Z

libp2p/stream/connection.nim

-      debug "Timeout handler cancelled", s
-    except CatchableError as exc: # Shouldn't happen
-      warn "exception in timeout handler", s, exc = exc.msg
+    await s.timeoutHandler()


Why has the try been removed?

timeoutHandler has {.async: (raises: []).} so is statically checked at compiletime to never raise. so, the exception handlers are unreachable.

diegomrsantos · 2024-03-13T21:27:26Z

libp2p/stream/lpstream.nim

+  let fut = newFuture[void]()
+  fut.complete()
+  fut


is this necessary?

with raw: true, yes. but one can also use {.async.} without raw: true to have this be implicitly introduced as part of the async transformation. either should be fine (skipping the transformation is a bit more efficient -- we are not doing anything async in this function)

diegomrsantos · 2024-03-13T21:29:52Z

libp2p/stream/lpstream.nim

@@ -304,9 +340,9 @@ proc closeWithEOF*(s: LPStream): Future[void] {.async, public.} =
    var buf: array[8, byte]
    if (await readOnce(s, addr buf[0], buf.len)) != 0:
      debug "Unexpected bytes while waiting for EOF", s
+  except CancelledError:


why has this been changed?

for CancelledError: the callers just end up discarding it. discarding it earlier allows {.async: (raises: []).}

for CatchableError, it cannot be raised by readOnce, catching LPStreamError is enough, and the compiler guarantees exhaustiveness because the proc itself is tagged as {.async: (raises: [].}, so it would alert us if an exception is forget to be caught.

diegomrsantos · 2024-03-13T21:33:31Z

libp2p/transports/wstransport.nim

+method closeImpl*(s: WsStream): Future[void] {.async: (raises: []).} =
+  try:
+    await s.session.close()
+  except CatchableError:


is it better to swallow the error here?

yeah, if the session close fails for whatever reason, we should still proceed to close the underlying connection.

diegomrsantos · 2024-03-13T21:46:47Z

libp2p/upgrademngrs/muxedupgrade.nim

    trace "Starting stream handler", conn
    try:
      await upgrader.ms.handle(conn) # handle incoming connection
    except CancelledError as exc:
-      raise exc
+      return


why has this been changed?

streamHandler is launched using asyncSpawn, and raising an error from asyncSpawn crashes the entire app (defect).

diegomrsantos · 2024-03-13T21:52:52Z

tests/errorhelpers.nim

+proc allFuturesThrowing*[T](futs: varargs[Future[T]]): Future[void] =
+  allFuturesThrowing(futs.mapIt(FutureBase(it)))
+
+proc allFuturesThrowing*[T, E](
+    futs: varargs[InternalRaisesFuture[T, E]]): Future[void] =
+  allFuturesThrowing(futs.mapIt(FutureBase(it)))


Why is it better than the previous code? Mentioning InternalRaisesFuture doesn't seem like a good idea.

It's needed to support calling allFuturesThrowing with Future[T].Raising(...) futures. otherwise, only those without Raising restrictions can be passed.

And yes, it's ugly, I agree. It's the same logic as allFutures and race use, though, so seems like the best way right now.

It's solely used in test code, so should be good enough.

etan-status · 2024-03-18T15:01:29Z

Async exception tracking #962

etan-status added 3 commits March 3, 2024 19:37

Merge branch 'unstable' into dev/etan/ex-stream

a592dc9

Merge branch 'unstable' into dev/etan/ex-stream

8854722

split other PRs

7eab192

etan-status marked this pull request as ready for review March 3, 2024 23:30

Merge branch 'unstable' into dev/etan/ex-stream

b19c20b

arnetheduck reviewed Mar 4, 2024

View reviewed changes

tests/helpers.nim Outdated Show resolved Hide resolved

arnetheduck previously approved these changes Mar 4, 2024

View reviewed changes

Merge branch 'unstable' into dev/etan/ex-stream

ce42579

etan-status enabled auto-merge (squash) March 4, 2024 18:26

etan-status disabled auto-merge March 4, 2024 18:27

avoid Raising unless necessary

d6cd5c7

Co-authored-by: Jacek Sieka <jacek@status.im>

etan-status dismissed arnetheduck’s stale review via d6cd5c7 March 4, 2024 18:33

syntax

a416478

etan-status enabled auto-merge (squash) March 4, 2024 23:32

Merge branch 'unstable' into dev/etan/ex-stream

393ae49

dryajov approved these changes Mar 5, 2024

View reviewed changes

etan-status merged commit 2860959 into unstable Mar 5, 2024
11 checks passed

etan-status deleted the dev/etan/ex-stream branch March 5, 2024 07:06

arnetheduck mentioned this pull request Mar 11, 2024

Raises #547

Closed

diegomrsantos reviewed Mar 13, 2024

View reviewed changes

libp2p/muxers/mplex/lpchannel.nim Show resolved Hide resolved

diegomrsantos reviewed Mar 13, 2024

View reviewed changes

arnetheduck mentioned this pull request Mar 18, 2024

add support for setting protocol handlers with {.raises.} annotation #1064

Merged

add {.async: (raises).} to libp2p/stream modules #1050

add {.async: (raises).} to libp2p/stream modules #1050

Conversation

etan-status commented Mar 2, 2024

codecov bot commented Mar 2, 2024 • edited Loading

Codecov Report

etan-status commented Mar 3, 2024

arnetheduck left a comment

Choose a reason for hiding this comment

dryajov left a comment

Choose a reason for hiding this comment

dryajov commented Mar 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

diegomrsantos Mar 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etan-status commented Mar 18, 2024

add `{.async: (raises).}` to `libp2p/stream` modules #1050

add `{.async: (raises).}` to `libp2p/stream` modules #1050

codecov bot commented Mar 2, 2024 •

edited

Loading

diegomrsantos Mar 13, 2024 •

edited

Loading

etan-status Mar 15, 2024 •

edited

Loading