Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

concurrent deals from a single client might fail due to no funds available on chain #4395

Closed
nonsense opened this issue Oct 14, 2020 · 1 comment

Comments

@nonsense
Copy link
Member

nonsense commented Oct 14, 2020

Describe the bug
One of the lotus-soup tests is exercising FIL deals with Lotus, by creating a network with a few clients and miners, and clients concurrently submitting deals to miners. Since deals are created concurrently within the span of a single block, sometimes the funds are not available on chain and the client sends immediately a proposal to the miner, which results in a rejected deal.

Ultimately the client fails with the following error:

ERROR storagemarket_impl	clientstates/client_states.go:289	deal bafyreicnknok2ndpsaqv2f2il545gm5jarckc65evthxlpeb44kwclwde4 failed: unexpected deal status while waiting for data request: 11 (StorageDealFailing). Provider message: deal rejected: clientMarketBalance.Available too small

To Reproduce
Steps to reproduce the behavior:

  1. Trigger a CI job on TaaS with testground run composition -f _compositions/deals-stress-concurrent-natural-k8s.toml from the filecoin-project/oni repo, lotus-soup directory.
  2. See error on https://ci.testground.ipfs.team/
    (This test runs on a regular basis, so it is possible to see a past failure as well).
    (TaaS is not yet available to everyone, so talk to me if you need access).

Expected behavior
Clients should be able to send a few deals concurrently to a given miner successfully.

Version (run lotus version):
v0.10.0

Additional context

  • Markets code calls node.EnsureFunds() 6 times, each time increasing by 640297000 (the cost of one of the deals). This means it calls EnsureFunds(640297000), EnsureFunds(640297000 * 2), etc.
  • The node processes one of these calls out of order. This is ok, because the amount requested will still cover all 6 deals. What happens in the logs is it calls in this order: EnsureFunds(640297000), EnsureFunds(640297000 * 3), EnsureFunds(640297000 * 2), ...
  • So the 3rd call sees that there is enough funds for its requirement, and exits without sending an AddBalance message. This should generally be ok, because the we are still ensuring that there is enough to cover outstanding deals.
  • however, in the markets code, we assume that if no AddBalance message was sent, funds are available, and we continue deal negotiation, immediately sending the proposal to the miner (https://github.com/filecoin-project/go-fil-markets/blob/master/storagemarket/impl/clientstates/client_states.go#L70-L73)
  • The miner rejects the deal because the AddBalance calls have not appeared on chain yet.

cc @ingar @hannahhoward

@rjan90
Copy link
Contributor

rjan90 commented Jun 3, 2024

Closing this ticket as the Legacy Lotus/Lotus-Miner Markets sub-system reached EOL at the end of the 31st January 2023, and is being completely removed from the code here:

@rjan90 rjan90 closed this as completed Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants