PC tickets expire, seems new ticket is already expired at retrieval time #4865

karalabe · 2020-11-16T05:56:40Z

All my sectors during the weekend got stuck in PC2 due to some new GPU scheduling bug in 1.1.3. I've reconfigured things to not have concurrent GPU handling, after which PC2 seems to have worked, but all the sectors started looping between PC2->PC1->PC2, etc. First two time it said the sector ticket expired. Two bugs though:

If the ticket expired, why on earth try it again with the same expired ticket. Surely we don't expect it to go back in time and finish it the second time before the deadline.
After 2 failures, the logs seem to show that a new ticket was requested, which was already expired upon request:

Event Log:
0.	2020-11-14 12:28:35 +0200 EET:	[event;sealing.SectorStartCC]	{"User":{"ID":569,"SectorType":3,"Pieces":[{"Piece":{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}},"DealInfo":null}]}}
1.	2020-11-14 12:28:35 +0200 EET:	[event;sealing.SectorPacked]	{"User":{"FillerPieces":null}}
2.	2020-11-14 12:28:37 +0200 EET:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"M1t/hKgo2wxZEn6AjiYgtgotBAsf0yzmxuY5J4q4MnE=","TicketEpoch":233877}}
3.	2020-11-14 14:10:30 +0200 EET:	[event;sealing.SectorRestart]	{"User":{}}
4.	2020-11-14 14:10:31 +0200 EET:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: worker restarted
5.	2020-11-14 14:11:31 +0200 EET:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
6.	2020-11-14 14:12:50 +0200 EET:	[event;sealing.SectorRestart]	{"User":{}}
7.	2020-11-14 14:23:56 +0200 EET:	[event;sealing.SectorRestart]	{"User":{}}
8.	2020-11-14 14:23:57 +0200 EET:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: worker restarted
9.	2020-11-14 14:24:57 +0200 EET:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
10.	2020-11-14 14:25:07 +0200 EET:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: acquiring sector paths: local acquire error: finding best storage for allocating : no good path found
11.	2020-11-14 14:26:07 +0200 EET:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
12.	2020-11-14 21:59:30 +0200 EET:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	seal pre commit(1) failed: acquiring sector paths: reserving storage space: can't reserve 484472310988 bytes in '/mnt/worker-hddb' (id:4d8739f3-b6f0-48a3-aca6-965fc7fdced5), only 43306869556 available
13.	2020-11-14 22:00:30 +0200 EET:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
14.	2020-11-15 03:00:49 +0200 EET:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjEiLCJsYWJlbHMiOnsiU3RhY2tlZERyZzMyR2lCVjEiOnsibGFiZWxzIjpbeyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci0xIiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbW50L3dvcmtlci1oZGRhL2NhY2hlL3MtdDAxOTEwMC01NjkiLCJpZCI6ImxheWVyLTIiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoibGF5ZXItMyIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci00Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbW50L3dvcmtlci1oZGRhL2NhY2hlL3MtdDAxOTEwMC01NjkiLCJpZCI6ImxheWVyLTUiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoibGF5ZXItNiIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci03Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbW50L3dvcmtlci1oZGRhL2NhY2hlL3MtdDAxOTEwMC01NjkiLCJpZCI6ImxheWVyLTgiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoibGF5ZXItOSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci0xMCIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci0xMSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N31dLCJfaCI6bnVsbH19LCJjb25maWciOnsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoidHJlZS1kIiwic2l6ZSI6MjE0NzQ4MzY0Nywicm93c190b19kaXNjYXJkIjo3fSwiY29tbV9kIjpbNywxMjYsOTUsMjIyLDUzLDE5NywxMCwxNDcsMywxNjUsODAsOSwyMjcsNzMsMTM4LDc4LDE5MCwyMjMsMjQzLDE1Niw2NiwxODMsMTYsMTgzLDQ4LDIxNiwyMzYsMTIyLDE5OSwxNzUsMTY2LDYyXX0="}}
15.	2020-11-15 20:04:00 +0200 EET:	[event;sealing.SectorRestart]	{"User":{}}
16.	2020-11-15 20:59:47 +0200 EET:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbv6obb4fqbx2gammxd4g2agzjyjkoo3bcgdlmtf3ywlo2vl4ucsu"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
17.	2020-11-15 20:59:47 +0200 EET:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	ticket expired: ticket expired: seal height: 234777, head: 238679
18.	2020-11-15 21:00:47 +0200 EET:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
19.	2020-11-16 03:02:54 +0200 EET:	[event;sealing.SectorPreCommit1]	{"User":{"PreCommit1Out":"eyJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjEiLCJsYWJlbHMiOnsiU3RhY2tlZERyZzMyR2lCVjEiOnsibGFiZWxzIjpbeyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci0xIiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbW50L3dvcmtlci1oZGRhL2NhY2hlL3MtdDAxOTEwMC01NjkiLCJpZCI6ImxheWVyLTIiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoibGF5ZXItMyIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci00Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbW50L3dvcmtlci1oZGRhL2NhY2hlL3MtdDAxOTEwMC01NjkiLCJpZCI6ImxheWVyLTUiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoibGF5ZXItNiIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci03Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvbW50L3dvcmtlci1oZGRhL2NhY2hlL3MtdDAxOTEwMC01NjkiLCJpZCI6ImxheWVyLTgiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoibGF5ZXItOSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci0xMCIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL21udC93b3JrZXItaGRkYS9jYWNoZS9zLXQwMTkxMDAtNTY5IiwiaWQiOiJsYXllci0xMSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N31dLCJfaCI6bnVsbH19LCJjb25maWciOnsicGF0aCI6Ii9tbnQvd29ya2VyLWhkZGEvY2FjaGUvcy10MDE5MTAwLTU2OSIsImlkIjoidHJlZS1kIiwic2l6ZSI6MjE0NzQ4MzY0Nywicm93c190b19kaXNjYXJkIjo3fSwiY29tbV9kIjpbNywxMjYsOTUsMjIyLDUzLDE5NywxMCwxNDcsMywxNjUsODAsOSwyMjcsNzMsMTM4LDc4LDE5MCwyMjMsMjQzLDE1Niw2NiwxODMsMTYsMTgzLDQ4LDIxNiwyMzYsMTIyLDE5OSwxNzUsMTY2LDYyXX0="}}
20.	2020-11-16 04:38:42 +0200 EET:	[event;sealing.SectorPreCommit2]	{"User":{"Sealed":{"/":"bagboea4b5abcbv6obb4fqbx2gammxd4g2agzjyjkoo3bcgdlmtf3ywlo2vl4ucsu"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
21.	2020-11-16 04:38:42 +0200 EET:	[event;sealing.SectorSealPreCommit1Failed]	{"User":{}}
	ticket expired: ticket expired: seal height: 234777, head: 239597
22.	2020-11-16 04:39:42 +0200 EET:	[event;sealing.SectorRetrySealPreCommit1]	{"User":{}}
23.	2020-11-16 07:34:49 +0200 EET:	[event;sealing.SectorRestart]	{"User":{}}
24.	2020-11-16 07:34:49 +0200 EET:	[event;sealing.SectorOldTicket]	{"User":{}}
25.	2020-11-16 07:34:50 +0200 EET:	[event;sealing.SectorTicket]	{"User":{"TicketValue":"zM0j4kiTXVjVETOpa1VqI6w2q057Akra0QkX9lVzIfU=","TicketEpoch":239049}}

The text was updated successfully, but these errors were encountered:

rjan90 · 2020-11-16T08:40:07Z

Just wanted to chime in here with more ticket expired bugs.

Got a lot of sectors with ticket expired after they have been stuck in mpool due to base fee's being higher over the weekend. If you don't replace gas, and it takes too long for the messages to get on chain from "PreCommitWait" & "CommitWait", the tickets will expire and the sectors start looping. And there is no other way than removing the sectors to clean it up.

2020-11-15T16:16:08.423+0100 WARN sectors storage-sealing/fsm.go:494 sector 768 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 233977, head: 238232 2020-11-15T16:53:52.618+0100 WARN sectors storage-sealing/fsm.go:494 sector 774 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234329, head: 238307 2020-11-15T17:32:02.793+0100 WARN sectors storage-sealing/fsm.go:494 sector 779 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234442, head: 238384 2020-11-15T20:12:29.369+0100 WARN sectors storage-sealing/fsm.go:494 sector 777 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234442, head: 238704 2020-11-15T20:20:53.761+0100 WARN sectors storage-sealing/fsm.go:494 sector 773 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234719, head: 238721 2020-11-15T20:33:53.023+0100 WARN sectors storage-sealing/fsm.go:494 sector 775 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234442, head: 238747 2020-11-15T20:55:12.373+0100 WARN sectors storage-sealing/fsm.go:494 sector 776 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234442, head: 238790 2020-11-15T20:57:52.450+0100 WARN sectors storage-sealing/fsm.go:494 sector 778 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234442, head: 238795 2020-11-15T23:07:05.432+0100 WARN sectors storage-sealing/fsm.go:494 sector 780 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234443, head: 239054 2020-11-15T23:28:43.163+0100 WARN sectors storage-sealing/fsm.go:494 sector 774 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 234329, head: 239097

jennijuju · 2020-11-16T21:01:54Z

@askender @rjan90 @karalabe, could you please try #4876 and see if the fix works for you?

s0nik42 · 2020-11-18T20:14:55Z

I've got my tickets expired too
What is the expiration time ??
was that bug already there in 1.1.2 (I didn't upgraded yet to 1.1.3)

https://filecoinproject.slack.com/archives/CEGN061C5/p1605723685085300

dodohack · 2020-11-21T10:23:03Z

I had the same issue with v1.2.1:

#4876 (comment)

jennijuju · 2020-12-14T22:55:48Z

This happened again with v1.2.2 , the sector in question can not be removed or update-state to Removedor Removing

sector log
https://gist.github.com/karalabe/d8ca4a0f6d2e081b30d1a7e167a5113b

shaodan · 2020-12-15T06:19:30Z

@karalabe have you tried abort jobs of sectors after run remove command #5081 (comment)

William8Work · 2020-12-25T21:43:17Z

My miner ran into this multiple times, and still happening. I am troubleshooting what the root causes are. For now, here are some information to share: (in the following case, it's related to sector 656), initially it looked like caused by worker overloaded. But later got moved to other worker for PC1, and PC2, no success. Kept doing loops and back to PC1.

2020-12-24T07:51:35.591-0800 WARN advmgr sector-storage/manager_calltracker.go:149 canceling started (not running) work seal/v0/precommit/1([[{"ID":{"Miner":8399,"Number":656},"ProofType":8},"2gxfSK2BRU6saSR35fxxCKp3WnimYu9IWvdNTCIO1x8=",[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]]])
2020-12-24T07:51:35.592-0800 WARN sectors storage-sealing/fsm.go:511 sector 656 got error event sealing.SectorSealPreCommit1Failed: seal pre commit(1) failed: storage call error 0: failed to acquire sector {8399 656} from remote (tried [{c90229a6-e411-4c34-8417-95494c70db9f [http://10.1.18.161:3456/remote/unsealed/s-t08399-656] 10 true false true}]): 1 error occurred:
* fetch error http://10.1.18.161:3456/remote/unsealed/s-t08399-656 (storage c90229a6-e411-4c34-8417-95494c70db9f) -> /media/lotus/worker/unsealed/fetching/s-t08399-656: do request: Get "http://10.1.18.161:3456/remote/unsealed/s-t08399-656": dial tcp 10.1.18.161:3456: connect: connection refused

2020-12-24T07:51:47.207-0800 WARN stores stores/index.go:204 sector {8399 656} redeclared in c90229a6-e411-4c34-8417-95494c70db9f

2020-12-25T13:00:20.208-0800 WARN sectors storage-sealing/fsm.go:511 sector 656 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 348153, head: 354120

flyworker · 2021-01-02T04:58:04Z

Same issue here
Daemon: 1.4.0+git.e9989d0e4+api1.0.0
Local: lotus version 1.4.0+git.e9989d0e4
SectorID: 4358
Status: PreCommit1
CIDcommD:
CIDcommR:
Ticket: e6a41afcb8114496b5ca3d9d82340375b3c2111c7a83572f202a647e064e38cd
TicketH: 371545
Seed:
SeedH: 0
Precommit:
Commit:
Proof:
Deals: [1467573 1467620]
Retries: 0

Event Log:
0. 2021-01-01 04:41:20 +0000 UTC: [event;sealing.SectorStart] {"User":{"ID":4358,"SectorType":8}}

2021-01-01 04:51:41 +0000 UTC: [event;sealing.SectorAddPiece] {"User":{"NewPiece":{"Piece":{"Size":17179869184,"PieceCID":{"/":"baga6ea4seaqhydz7brcpkhs7uafbkb5qztx7p4gv34fstogfmvsrkog4jxqwumq"}},"DealInfo":{"PublishCid":{"/":"bafy2bzacedsyhz3rfhu7ppwdtubjx3ac3ryysrga3ldngawiwmnctd7mgbm64"},"DealID":1467573,"DealSchedule":{"StartEpoch":385181,"EndEpoch":1439398},"KeepUnsealed":true}}}}
2021-01-01 05:22:25 +0000 UTC: [event;sealing.SectorAddPiece] {"User":{"NewPiece":{"Piece":{"Size":17179869184,"PieceCID":{"/":"baga6ea4seaqjejb5svfk7fw3jjbu2nf6rmnn25yosig5jwxbjjgtp4pfk6poqey"}},"DealInfo":{"PublishCid":{"/":"bafy2bzacec6yjvoeyayyfanaqnnpo5rk4uzzomzi7pvo3hqpn6xblhflqbjda"},"DealID":1467620,"DealSchedule":{"StartEpoch":385280,"EndEpoch":1439398},"KeepUnsealed":true}}}}
2021-01-01 05:42:36 +0000 UTC: [event;sealing.SectorStartPacking] {"User":{}}
2021-01-01 05:42:36 +0000 UTC: [event;sealing.SectorPacked] {"User":{"FillerPieces":null}}
2021-01-01 05:42:39 +0000 UTC: [event;sealing.SectorTicket] {"User":{"TicketValue":"5qQa/LgRRJa1yj2dgjQDdbPCERx6g1cvICpkfgZOOM0=","TicketEpoch":371545}}
2021-01-01 19:13:43 +0000 UTC: [event;sealing.SectorRestart] {"User":{}}
2021-01-01 19:19:56 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}}
seal pre commit(1) failed: storage call error 0: task aborted
2021-01-01 19:20:56 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}
2021-01-02 02:23:13 +0000 UTC: [event;sealing.SectorForceState] {"User":{"State":"PreCommit1"}}
2021-01-02 02:23:13 +0000 UTC: [event;sealing.SectorPreCommit1] {"User":{"PreCommit1Out":"eyJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSIsImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJsYWJlbHMiOlt7InBhdGgiOiIvaG9tZS9uYmZzL2xvdHVzd29ya2VyL2NhY2hlL3MtdDAyNDAxLTQzNTgiLCJpZCI6ImxheWVyLTEiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9ob21lL25iZnMvbG90dXN3b3JrZXIvY2FjaGUvcy10MDI0MDEtNDM1OCIsImlkIjoibGF5ZXItMiIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2hvbWUvbmJmcy9sb3R1c3dvcmtlci9jYWNoZS9zLXQwMjQwMS00MzU4IiwiaWQiOiJsYXllci0zIiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvaG9tZS9uYmZzL2xvdHVzd29ya2VyL2NhY2hlL3MtdDAyNDAxLTQzNTgiLCJpZCI6ImxheWVyLTQiLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9ob21lL25iZnMvbG90dXN3b3JrZXIvY2FjaGUvcy10MDI0MDEtNDM1OCIsImlkIjoibGF5ZXItNSIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2hvbWUvbmJmcy9sb3R1c3dvcmtlci9jYWNoZS9zLXQwMjQwMS00MzU4IiwiaWQiOiJsYXllci02Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvaG9tZS9uYmZzL2xvdHVzd29ya2VyL2NhY2hlL3MtdDAyNDAxLTQzNTgiLCJpZCI6ImxheWVyLTciLCJzaXplIjoxMDczNzQxODI0LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LHsicGF0aCI6Ii9ob21lL25iZnMvbG90dXN3b3JrZXIvY2FjaGUvcy10MDI0MDEtNDM1OCIsImlkIjoibGF5ZXItOCIsInNpemUiOjEwNzM3NDE4MjQsInJvd3NfdG9fZGlzY2FyZCI6N30seyJwYXRoIjoiL2hvbWUvbmJmcy9sb3R1c3dvcmtlci9jYWNoZS9zLXQwMjQwMS00MzU4IiwiaWQiOiJsYXllci05Iiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvaG9tZS9uYmZzL2xvdHVzd29ya2VyL2NhY2hlL3MtdDAyNDAxLTQzNTgiLCJpZCI6ImxheWVyLTEwIiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fSx7InBhdGgiOiIvaG9tZS9uYmZzL2xvdHVzd29ya2VyL2NhY2hlL3MtdDAyNDAxLTQzNTgiLCJpZCI6ImxheWVyLTExIiwic2l6ZSI6MTA3Mzc0MTgyNCwicm93c190b19kaXNjYXJkIjo3fV0sIl9oIjpudWxsfX0sImNvbmZpZyI6eyJwYXRoIjoiL2hvbWUvbmJmcy9sb3R1c3dvcmtlci9jYWNoZS9zLXQwMjQwMS00MzU4IiwiaWQiOiJ0cmVlLWQiLCJzaXplIjoyMTQ3NDgzNjQ3LCJyb3dzX3RvX2Rpc2NhcmQiOjd9LCJjb21tX2QiOlszLDUwLDExMCwxOCwyMjksMTYwLDI0NCwxNDUsNzQsNDksNzIsMzcsNzEsMTE5LDcwLDIzNiwxMjIsMjEyLDE4OCwxNTEsMTAsMjE3LDE1NywxODYsMjcsNzAsMSwxMTYsMTM3LDE4NSwxNjYsNTVdfQ=="}}

CharlesH76 · 2021-03-02T05:17:38Z

I am also experiencing a similar issue with a sector stuck on repeating PreCommit1. I am trying to see if I can complete the sealing process on other sectors. Is there anyone to prevent it or manually correct this problem?

lotus-miner sealing jobs

ID Sector Worker Hostname Task State Time
f28ddd40 3 1fe869a7 lotus000 PC1 running 18h34m8.1s
e917d7f5 10 1fe869a7 lotus000 PC1 running 9h18m13.2s
33e83432 11 1fe869a7 lotus000 PC1 running 9h12m38.2s

lotus version

Daemon: 1.4.2+mainnet+git.d1abc3cb5.dirty+api1.1.0
Local: lotus version 1.4.2+mainnet+git.d1abc3cb5.dirty

lotus-miner sectors status 3

SectorID: 3
Status: PreCommit1
CIDcommD: baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq
CIDcommR: bagboea4b5abcb6y6t3bsjacnd3vsypptzcctkujr5zmnzgh7ywi4e5tvnrxx7mkj
Ticket: 5ed2fc15fa54e45e3caf19887d5f8c9a2c6c7d65ebd0257350c01bb62601cc4d
TicketH: 542020
Seed:
SeedH: 0
Precommit:
Commit:
Proof:
Deals: [0]
Retries: 0

Miner.log
2021-03-01T17:37:59.476+0800 ^[[35mDEBUG^[[0m advmgr sector-storage/sched.go:354 SCHED 0 queued; 2 open windows
2021-03-01T17:37:59.476+0800 ^[[35mDEBUG^[[0m advmgr sector-storage/worker_local.go:129 acquired sector {{229378 3} 8} (e:6; a:0): {{0 0} /sealing_space/sealed/s-t0229378-3 /seali
ng_space/cache/s-t0229378-3}
2021-03-01T18:19:15.379+0800 ^[[33mWARN^[[0m sectors storage-sealing/fsm.go:535 sector 3 got error event sealing.SectorSealPreCommit1Failed: ticket expired: ticket expired: seal height: 53
5185, head: 542918
2021-03-01T18:19:15.384+0800 ^[[34mINFO^[[0m sectors storage-sealing/states_failed.go:25 SealPreCommit1Failed(3), waiting 59.61595246s before retrying
2021-03-01T18:20:15.029+0800 ^[[35mDEBUG^[[0m advmgr sector-storage/sched.go:354 SCHED 1 queued; 2 open windows

Thank You

jennijuju · 2021-08-16T10:15:48Z

this should be resolve in lotus v1.9.0 and up - Please open a new ticket if you are running into this issue on the latest version of lotus!

askender mentioned this issue Nov 16, 2020

Loop between P1 and P2 if ticket expired happen #4843

Closed

jennijuju added area/mining Area: Mining need/analysis Hint: Needs Analysis labels Nov 16, 2020

magik6k mentioned this issue Nov 16, 2020

storagefsm: Fix expired ticket retry loop #4876

Merged

dodohack mentioned this issue Nov 21, 2020

Lots of PreCommit1 sectors unscheduled to idle workers #4955

Closed

magik6k closed this as completed in #4876 Nov 24, 2020

jennijuju reopened this Dec 14, 2020

jennijuju assigned magik6k Dec 14, 2020

jennijuju added area/sealing and removed need/analysis Hint: Needs Analysis area/mining Area: Mining labels Jan 20, 2021

jennijuju closed this as completed Aug 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PC tickets expire, seems new ticket is already expired at retrieval time #4865

PC tickets expire, seems new ticket is already expired at retrieval time #4865

karalabe commented Nov 16, 2020

rjan90 commented Nov 16, 2020 •

edited

Loading

jennijuju commented Nov 16, 2020 •

edited

Loading

s0nik42 commented Nov 18, 2020 •

edited

Loading

dodohack commented Nov 21, 2020

jennijuju commented Dec 14, 2020

shaodan commented Dec 15, 2020

William8Work commented Dec 25, 2020

flyworker commented Jan 2, 2021 •

edited

Loading

CharlesH76 commented Mar 2, 2021

jennijuju commented Aug 16, 2021

PC tickets expire, seems new ticket is already expired at retrieval time #4865

PC tickets expire, seems new ticket is already expired at retrieval time #4865

Comments

karalabe commented Nov 16, 2020

rjan90 commented Nov 16, 2020 • edited Loading

jennijuju commented Nov 16, 2020 • edited Loading

s0nik42 commented Nov 18, 2020 • edited Loading

dodohack commented Nov 21, 2020

jennijuju commented Dec 14, 2020

shaodan commented Dec 15, 2020

William8Work commented Dec 25, 2020

flyworker commented Jan 2, 2021 • edited Loading

CharlesH76 commented Mar 2, 2021

lotus-miner sealing jobs

lotus version

lotus-miner sectors status 3

jennijuju commented Aug 16, 2021

rjan90 commented Nov 16, 2020 •

edited

Loading

jennijuju commented Nov 16, 2020 •

edited

Loading

s0nik42 commented Nov 18, 2020 •

edited

Loading

flyworker commented Jan 2, 2021 •

edited

Loading