Retrieval deals fail if running many in parallel

[mishmosh]

From @stefangeorgerobert:

I’ve tried running parallel retrievals for approx 10 hours on our dev environment and most of the retrievals failed with timeout.

2020/07/31:43:23:984  [ INFO ]   RetrieveDeals: TOTAL : 61
2020/07/31:43:23:984  [ INFO ]   RetrieveDeals: SUCCESSFUL : 3

• In order to run a retrieval bot that doesn't crash its lotus node, we are heavily constraining the retrieval volume. MAX_RETRIEVE_DEALS_RUNS is currently set to 2, that means the bot can request only 2 and wait for the result.
• Serial retrievals seems more successful as on calibration.spacerace dashboard the Average Retrieval Deal Success Rate is 26% at the moment

Based on @jimpick's manual testing:

Based on manual testing with my custom testground instance, if I do a single retrieval, it completes in 1 minute. If I do 2 in parallel, each takes 2 minutes to complete. If I do 6 in parallel, each takes 7 minutes to complete.

@dirkmc has a PR that may address this, but not confirmed.
#2640

[whyrusleeping]

Thanks for the report! I think this should be resolved by #2640 , but am not 100% sure.

@dirkmc what do you think?

[mishmosh]

Short-term, this is causing major delays to the retrieval deal-making rate from the Space Race bot. Proceeding with the workaround, which is to run more parallel retrieval dealbot nodes.

Long-term, the ability to do multiple retrievals in parallel is essential for production usage.

[jimpick]

#2640 has landed and when I test parallel deals using my testground setup, they run as fast as single deals.

[TippyFlitsUK]

Hi @mishmosh

This looks like a reminder. Should we close it or keep it open?

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

[github-actions]

This issue was closed because it is missing author input.