Fix task handling bugs, so peers are more likely to be available

[teor2345]

Motivation

Some of our crawler and connection rates can lead to weird interactions between crawlers, such as flapping or task synchronisation.

This is unexpected work in sprint 24, but it's related to the upcoming fanout and notfound work.

Solution

Timer changes:

• Use prime numbers for crawler and heartbeat intervals, so they are almost always happening at different times
• Make sure each peer gets at least one connection attempt in each reconnection interval, even if the address book is full
• Add tests for relative timings
• Move some tests into their own modules

Task order changes:

• Let other tasks run between fanouts and retries, so we're more likely to choose different peers
• Let other tasks run after peer crawler DemandDrop, so we're more likely to get ready peers
• Yields are really difficult to test for, we'll replace this code soon in Limit fanouts to the number of active peers, Credit: Equilibrium #2214 and Move syncer block download retries inside the peer set #3235

Review

@jvff or @dconnolly can review this PR.

This is urgent because syncs are very slow, and #3191 or #3244 should help fix that.

This PR is based on PR #3167, because they change the same import lines.

Reviewer Checklist

• Code implements Specs and Designs
• Tests for Expected Behaviour
• Tests for Errors

Follow Up Work

• Decide if we need shorter timeouts on PeerSet fanout requests #3136
• Limit fanouts to the number of active peers, Credit: Equilibrium #2214
• Make Zebra send notfound, and use received notfound to finish requests #2726
• Security: Send notfound inv items to the inv collector, Credit: Equilibrium #2156
• maybe Security: rate-limit outbound messages, Credit: Equilibrium #2153

[teor2345]

I've marked this as draft until PR #3167 merges, because they change the same import lines.

[teor2345]

The Google Cloud build failed, I opened:

• Add retries for the sapling and sprout parameter downloads #3239

[teor2345]

I've marked this PR and all its dependencies as a high priority, because this is a known bug that has actually happened before, and seems to still be happening.

I'm not sure if this PR fixes the bug, but it's an easy first step.

[mpguerra]

I've marked this PR and all its dependencies as a high priority, because this is a known bug that has actually happened before, and seems to still be happening.

I'm not sure if this PR fixes the bug, but it's an easy first step.

@teor2345 should this still be in draft and are you looking for an additional review at this time? I see @jvff already approved a previous version of this PR a while ago.

[conradoplg]

I believe this is just waiting for #3167 to be merged and is a draft just to prevent it from merging before that one.

It looks good - I don't have much experience with this part of the code but was approved before, so I'm approving it now and leave it unmerged for a bit if anyone else wants to take a look.

[conradoplg]

I rebased after merging #3167, @jvff could you please check and reapprove?

[teor2345]

I believe this is just waiting for #3167 to be merged and is a draft just to prevent it from merging before that one.

Yep, that's the only way to stop a GitHub PR that is based on another PR from merging into the wrong branch.

@mpguerra here's the process we agreed on a few months ago:

• if you finish a task or you're looking for work, check for PRs that need reviews (but you can skip draft PRs)
• if you want a review on a draft PR, ask for one, which I did:
  • https://discord.com/channels/676527656170160146/716086297210650634/920797363885441054
  • https://discord.com/channels/676527656170160146/716086297210650634/921193869570367548
  • and by tagging people in the PR description

If there's something else I need to do, let's talk about it at the Engineering sync. And then let's document it in the onboarding doc.

[teor2345]

This got approved then rebased, and it's urgent, so I just admin-merged it.