Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Loop between P1 and P2 if ticket expired happen #4843

Closed
askender opened this issue Nov 13, 2020 · 8 comments
Closed

Loop between P1 and P2 if ticket expired happen #4843

askender opened this issue Nov 13, 2020 · 8 comments
Assignees
@jennijuju
Labels
support

Comments

@askender
Copy link

Describe the problem

SectorSealPreCommit1Failed got ticket expired error, then SectorRetrySealPreCommit1, then SectorSealPreCommit1Failed got ticket expired error, then SectorRetrySealPreCommit1, then ...

Sectors status

Event Log:
0.	2020-11-09 01:35:57 +0800 CST:	[event;sealing.SectorStartCC]	{"User":{"ID":478,"SectorType":3,"Pieces":[{"Piece":{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}},"DealInfo":null}]}}
1.	2020-11-09 01:35:57 +0800 CST:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":null}}
2.	2020-11-09 06:38:53 +0800 CST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWd..........","TicketEpoch":217451}}
3.	2020-11-09 08:19:05 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
4.	2020-11-09 11:15:48 +0800 CST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbgaws4atofyxsgaqi4dr2fsw5d2skrd3kk5rsbm25ox23k4awyke"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
5.	2020-11-11 16:14:54 +0800 CST:	[event;sealing.SectorRestart]	{"User":{}}
6.	2020-11-11 16:14:54 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	ticket expired: ticket expired: seal height: 218351, head: 225869
7.	2020-11-11 16:15:54 +0800 CST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}

There are other logs like this... , I will comment below

I think there should be a check of ticket-time before SectorRetrySealPreCommit1
If spend a lot time before SectorRetrySealPreCommit1 (various situation), there is no need to retry.

logs from Charles Cao - NBFS:
we have it as well…
24. 2020-11-13 03:07:55 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {“User”:{}}
ticket expired: ticket expired: seal height: 225431, head: 231015
25. 2020-11-13 03:08:55 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {“User”:{}}
26. 2020-11-13 03:08:55 +0000 UTC: [event;sealing.SectorOldTicket] {“User”:{}}
27. 2020-11-13 03:08:55 +0000 UTC: [event;sealing.SectorTicket] {“User”:{“TicketValue”:“N7MQJOUatucpog3C6mlNvxAZTwYfsTrR+td9MSNy/OY=“,”TicketEpoch”:230117}}

Version

The output of lotus --version.
Daemon: 1.1.2+git.d4cdc6d3+api0.17.0
Local: lotus version 1.1.2+git.d4cdc6d3

Realted slack thread
https://filecoinproject.slack.com/archives/CEGN061C5/p1605148865426100?thread_ts=1605088411.414500&cid=CEGN061C5
@jennijuju
Copy link
Member

miner logs is required for investigation.

@askender
Copy link
Author

askender commented Nov 13, 2020 •
edited
Loading

Another log (I make it short for easily review):

8.	2020-11-10 14:14:03 +0800 CST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdp....","TicketValue":"AsY2bob8CMHEdFdaFlkhqfHaFqq4bxWhrOAnQMb3zQM=","TicketEpoch":221106}}
9.	2020-11-10 15:06:07 +0800 CST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcatcanyork5xwtherrcyw7ia77qziuufessgvt2jqsh7fqan4ogdl"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
10.	2020-11-10 15:06:15 +0800 CST:	[event;sealing.SectorPreCommitted]	{"User":{"Message":{"/":"bafy2bzacec7luofybr35l7bo5tuesrkt6pxnuqeldnan6qafmdaki2f7elvpu"},"PreCommitDeposit":"128855159304685409","PreCommitInfo":{"SealProof":3,"SectorNumber":525,"SealedCID":{"/":"bagboea4b5abcatcanyork5xwtherrcyw7ia77qziuufessgvt2jqsh7fqan4ogdl"},"SealRandEpoch":221106,"DealIDs":[],"Expiration":1776190,"ReplaceCapacity":false,"ReplaceSectorDeadline":0,"ReplaceSectorPartition":0,"ReplaceSectorNumber":0}}}
11.	2020-11-10 15:09:30 +0800 CST:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIg4Xh6rcbLTeL2yL17c9/kaI3RA5i64XY1SzBh3ICEAgIBcaDkAiDoIAGZPU5Mvxi0oWzjv02ynF+4JznmTiuAHzSXly1C/AFxoOQCIHZW6x3HAW1R61TjOrNdudLIS15T5eDNs1sL2dk1pX+8AXGg5AIgybGP14hnQMhstdl0YPeQG4ATdm4i3ACoPef5ljG4FNsBcaDkAiAhqnXVPAvCQC6Z7EH/Awus649sJ9lXesz+jdcuwtw8fgFxoOQCIKO62XMi36aH7eI0gGVCp3jDWTtoHZ6kegGDUo9pt5cdAXGg5AIg4DmtYn96ePtvOczX9p060PqaT2CddqvWSGckxvj2IjM="}}
12.	2020-11-10 16:24:30 +0800 CST:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"43EbzxmJTv7EEzcBPcNSdAYrRIE0sMf9BwALGsX2ci8=","SeedEpoch":223003}}

13.	2020-11-10 16:24:30 +0800 CST:	[event;sealing.SectorComputeProofFailed]	{"User":{}}
	computing seal proof failed(1): StandaloneSealCommit: could not read file p_aux="/mypath/cache/s-t019362-525/p_aux"
Caused by:
    No such file or directory (os error 2)
14.	2020-11-10 16:25:30 +0800 CST:	[event;sealing.SectorRetryComputeProof]	{"User":{}}
15.	  same as 13
16.	  same as 14  
17.	same as 13
18.	2020-11-10 16:27:34 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive compute fails

19.	2020-11-10 16:28:34 +0800 CST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
20.	2020-11-11 02:25:19 +0800 CST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdp....","TicketValue":"AsY2bob8CMHEdFdaFlkhqfHaFqq4bxWhrOAnQMb3zQM=","TicketEpoch":221106}}
21.	2020-11-11 03:01:50 +0800 CST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbkliazfgv5gtjo4mbeb62jkkjzkyk7qdl75hdukjii77f2kbyjqk"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
22.	2020-11-11 03:01:50 +0800 CST:	[event;sealing.SectorPreCommitLanded]	{"User":{"TipSet":"AXGg5AIgJXn88so+Maw8oVnrzQUYBLmgSStcfx83WMKTXstLaQ4BcaDkAiBCyvxaNgpi3ItrIqToBzAcQUo0jrXfa7nM5PQwUzS5/A=="}}
23.	2020-11-11 03:01:50 +0800 CST:	[event;sealing.SectorSeedReady]	{"User":{"SeedValue":"43EbzxmJTv7EEzcBPcNSdAYrRIE0sMf9BwALGsX2ci8=","SeedEpoch":223003}}
24.	2020-11-11 04:41:14 +0800 CST:	[event;sealing.SectorCommitted]	{"User":{"Proof":"p1dhB+sg..."}}
25.	2020-11-11 04:41:14 +0800 CST:	[event;sealing.SectorCommitFailed]	{"User":{}}
	commit check error: invalid proof (compute error?)
26.	2020-11-11 04:42:14 +0800 CST:	[event;sealing.SectorRetryInvalidProof]	{"User":{}}
27.	2020-11-11 06:55:52 +0800 CST:	[event;sealing.SectorCommitted]	{"User":{"Proof":"rZwK9RO/...."}}
28.	2020-11-11 06:55:52 +0800 CST:	[event;sealing.SectorCommitFailed]	{"User":{}}
	commit check error: invalid proof (compute error?)
29.	2020-11-11 06:56:52 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	consecutive invalid proofs

30.	2020-11-11 06:57:52 +0800 CST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
31.	2020-11-11 12:16:01 +0800 CST:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdp....","TicketValue":"AsY2bob8CMHEdFdaFlkhqfHaFqq4bxWhrOAnQMb3zQM=","TicketEpoch":221106}}
32.	2020-11-11 13:23:53 +0800 CST:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbkliazfgv5gtjo4mbeb62jkkjzkyk7qdl75hdukjii77f2kbyjqk"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
33.	2020-11-11 13:23:53 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	ticket expired: ticket expired: seal height: 222006, head: 225527

@jennijuju jennijuju added the need/author-input Hint: Needs Author Input label Nov 13, 2020
@jennijuju jennijuju self-assigned this Nov 13, 2020
@askender
Copy link
Author

askender commented Nov 13, 2020 •
edited
Loading

miner logs is required for investigation.

I checked the miner log: there is only one warn log:
2020-11-11T16:14:54.671+0800 WARN sectors storage-sealing/fsm.go:475 sector 478 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 218351, head: 225869

There is a little things in logs to be improved: repeat ticket expired: in this log.

I think ticket expired is related ticket-created-time, If spend a lot of time before SectorRetrySealPreCommit1, this will happen every time.

By the way, SectorRetrySealPreCommit1 means redo: P1-P2, means spend filecoin again. Will you offer an option that not redo any thing, just stuck there before things is checked. Auto-retry is not always the best choice.

@askender
Copy link
Author

related issue (about ticket expired):
#3141 Got expired ticket, sector stucked

@askender
Copy link
Author

a sector out of the loop

51.	2020-11-14 18:28:24 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	ticket expired: ticket expired: seal height: 230835, head: 234776
52.	2020-11-14 18:29:24 +0800 CST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
53.	2020-11-14 18:29:24 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	getting ticket failed: getting precommit info: sectorNumber is allocated, but PreCommit info wasn't found on chain
54.	2020-11-14 18:30:24 +0800 CST:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
55.	2020-11-14 18:30:24 +0800 CST:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}

But it would be better to estimate if ticket will expire before do sealing.SectorRetrySealPreCommit1.

@askender askender changed the title Endless loop between P1 and P2 if ticket expired happen Loop between P1 and P2 if ticket expired happen Nov 16, 2020
@askender
Copy link
Author

related issue: PC tickets expire, seems new ticket is already expired at retrieval time #4865

@jennijuju
Copy link
Member

jennijuju commented Nov 16, 2020 •
edited
Loading

thanks for reporting, duplicated #4865

@jjxx2021
Copy link

@

thanks for reporting, duplicated #4865

Please check whether it is a new bug or dup to #4896
I have the same issue on 1.2.1.

jjxx@fc02:/bin$ ./lotus-miner --version
lotus-miner version 1.2.1+git.df66f48f6
jjxx@fc02:/bin$

@TippyFlitsUK TippyFlitsUK removed the need/author-input Hint: Needs Author Input label Mar 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jennijuju jennijuju

Labels
support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@askender @TippyFlitsUK @jennijuju @jjxx2021