gossipsub: lazy prefix detection

[arnetheduck]

Consider the following case:

• we receive a message from peer A and start validating it
• we start receiving a message from B and store the first N bytes in an "in-flight" map
• we finish validating and are ready to re-broadcast to peer B which is in our mesh

At this point, if the message we're about to send matches any "in-flight" entry, delay sending the message to that peer so as to potentially avoid a duplicate send.

This is expected to work well for larger messages whose prefixes typically are unique.

[Menduist]

An alternative to this might be to send (if the message is big enough to warrant this):

• IHAVE <msgid>
• <msg id content>
  (in two separate messages)

Then the other side can know who has it, before even sending receiving the full message once.
Requires small spec backward-compatible change though (would have to write "send IHAVE before sending a message more than N bytes, and don't send to a mesh peer if he sent you IHAVE before")

We could even imagine sending IHAVE to our peers before receiving the full content (and fallback to IWANT if the original peer doesn't end up sending us the thing)

[arnetheduck]

Requires small spec backward-compatible change though

hm, the question here is what happens when a peer receives IHAVE for a message they don't have, but is inflight from other peers - it doesn't quite make sense to query for it with IWANT for example - there's a semantic difference: "I have the message and will now send it to you" vs "I have the message, come and ask for it" that drives a difference in response as well, potentially

[Menduist]

Sure, maybe that should instead be a new field in the protobuf, which would make backward compatibility even better (if you don't support this extension you'll just ignore it)

[Menduist]

This ended up being IDontWant, which has been implemented #934