High rate of tvu ingress traffic causes delinquency

[leoluk]

A high rate of inbound shreds to the tvu port (~50 kpps) causes validators to fall out of sync with the network and become delinquent.

This how one of the recent TdS networks died (aided the the repair issues at the time that prevented nodes from getting back in sync, so they could be attacked one by one vs. having to target 1/3).

Still works, tested on our TdS node by replaying the same valid shred a lot of times:

(you can see where it switches to its own fork and goes delinquent until traffic stops)

[mvines]

Do you know if the same is true if the payload is an invalid shred? That would lower the bar for this attack from "anybody who gets enough stake to be in the leader schedule" to "anybody"

[leoluk]

Works with invalid shreds, too. Tried by replaying a SLP shred on TDS, still killed the node.

[leoluk]

I did not investigate how much of that is due to the TVU stalling vs. overloading the host OS.

[sakridge]

@leoluk can you share the hardware setup you used to produce this?

[leoluk]

16 vCores @ 2.3 GHz (Skylake Xeon), 64G RAM, NVMe disk

[leoluk]

Reproduction instructions: https://gist.github.com/leoluk/bbdcb6173ec3dd3da8b2f6ca9512b2e9