Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storagefsm: Fix expired ticket retry loop #4876

Merged
merged 3 commits into from
Nov 24, 2020
Merged

storagefsm: Fix expired ticket retry loop #4876

merged 3 commits into from
Nov 24, 2020

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented Nov 16, 2020

Fixes #4865

@jennijuju jennijuju mentioned this pull request Nov 16, 2020
Closed
@magik6k
Copy link
Contributor Author

magik6k commented Nov 18, 2020

(broke something in windowpost test)

@s0nik42
Copy link

s0nik42 commented Nov 18, 2020

Is this error related to the bug :

2020-11-18T23:52:09.020+0100 WARN sectors storage-sealing/fsm.go:475 sector 1405 got error event sealing.SectorSealPreCommit1Failed: getting ticket failed: getting precommit info: sectorNumber is allocated, but PreCommit info wasn't found on chain

@dodohack
Copy link

dodohack commented Nov 21, 2020 •
edited
Loading

I encountered similar bug as well, here is the log.

2020-11-21T10:01:36.379Z	WARN	sectors	storage-sealing/fsm.go:507	sector 19 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 249141, head: 254883
2020-11-21T10:01:36.380Z	INFO	sectors	storage-sealing/states_failed.go:26	SealPreCommit1Failed(19), waiting 59.619750759s before retrying
2020-11-21T10:02:36.014Z	DEBUG	advmgr	sector-storage/sched.go:354	SCHED 1 queued; 10 open windows

Basically, the sector 19 had finished PC1 and was running PC2 for more than 20 mins, but when this error "ticket expired" message happened, it caused sector 19 from running PC2, back to re-run PC1 instead.

My lotus/miner/worker version is: version 1.2.1+git.df66f48f6

I saw so many PC1 -> PC2 -> PC1... loop when I switched from ntwk-calibration to mainnet recently.

@s0nik42
Copy link

s0nik42 commented Nov 21, 2020 •
edited
Loading

@magik6k is it going to be corrected on the next release or should we remove the sectors manually ? my miners is saturated of them.
Thanks

@Shekelme
Copy link

2020-11-21T16:54:15.076+0200 WARN sectors storage-sealing/fsm.go:507 sector 4 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 250201, head: 255468

What should I do with this sector in endless sealing loop?

@jennijuju jennijuju mentioned this pull request Nov 23, 2020
Closed
Kubuxu
Kubuxu reviewed Nov 24, 2020
View reviewed changes
extern/storage-sealing/states_sealing.go
case exitcode.SysErrOutOfGas:
// gas estimator guessed a wrong number
// gas estimator guessed a wrong number / out of funds:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to see this, as at the upgrade epoch we might hit it.

@magik6k magik6k merged commit 214f46b into master Nov 24, 2020
@magik6k magik6k deleted the fix/miner-ticket-loop branch November 24, 2020 18:54
@jennijuju jennijuju mentioned this pull request Nov 25, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arajasek arajasek arajasek approved these changes

@Kubuxu Kubuxu Kubuxu approved these changes

@whyrusleeping whyrusleeping Awaiting requested review from whyrusleeping

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

PC tickets expire, seems new ticket is already expired at retrieval time
6 participants
@magik6k @s0nik42 @dodohack @Shekelme @Kubuxu @arajasek