Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(SnapDeals) Sector get stuck because it want to assign the miner for PR2 //SectorProveReplicaUpdateFailed #8226

Closed
8 of 18 tasks
Reiers opened this issue Mar 2, 2022 · 3 comments · Fixed by #8420
Closed
8 of 18 tasks

(SnapDeals) Sector get stuck because it want to assign the miner for PR2 //SectorProveReplicaUpdateFailed #8226

Reiers opened this issue Mar 2, 2022 · 3 comments · Fixed by #8420
Assignees
@magik6k
Labels
dif/easy An existing lotus user should be able to pick this up kind/bug Kind: Bug need/analysis Hint: Needs Analysis P2 P2: Should be resolved SnapDeals
Milestone
v1.15.1

Comments

@Reiers
Copy link

Reiers commented Mar 2, 2022

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not an enhancement request. If it is, please file a improvement suggestion instead.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to lotus.

Lotus component

  • lotus daemon - chain sync
  • lotus miner - mining and block production
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt)
  • lotus miner/market - storage deal
  • lotus miner/market - retrieval deal
  • lotus miner/market - data transfer
  • lotus client
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other

Lotus Version

Daemon:  1.14.2+mainnet+git.6347daf84+api1.5.0
Local: lotus version 1.14.2+mainnet+git.6347daf84

Describe the Bug

I have set everything to false in the miner config.toml - but the scheduler want to assign the miner anyway and sector get stuck in a loop. The only option is to abort-upgrade. and it goes without saying - you will lose the deal.

Here is worker ID for the miner, so we can see that in corresponds with the logs below.

Worker 5432b068-93c1-46e5-863a-a40858a5de04, host miner
        CPU:  [                                                                ] 0/128 core(s) in use
        RAM:  [|||                                                             ] 5% 25.77 GiB/503.5 GiB
        VMEM: [|||                                                             ] 5% 26.27 GiB/505.5 GiB
        GPU:  [                                                                ] 0% 0.00/2 gpu(s) in use
        GPU: NVIDIA RTX A5000, not used
        GPU: NVIDIA RTX A5000, not used

Logging Information

2022-03-01T22:12:31.989+0100	WARN	sectors	storage-sealing/fsm.go:733	sector 20782 got error event sealing.SectorProveReplicaUpdateFailed: prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
2022-03-01T22:12:31.997+0100	ERROR	sectors	storage-sealing/states_failed.go:227	bad PR1: +nil PR2 proof
2022-03-01T22:12:32.983+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 1 queued; 24 open windows
2022-03-01T22:12:33.032+0100	DEBUG	advmgr	sector-storage/sched.go:452	SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]}]
2022-03-01T22:12:33.032+0100	DEBUG	advmgr	sector-storage/sched.go:453	SCHED Acceptable win: [[22 23 7 8]]
2022-03-01T22:12:33.032+0100	DEBUG	advmgr	sector-storage/sched.go:467	SCHED try assign sqi:0 sector 20782 to window 22
2022-03-01T22:12:33.032+0100	DEBUG	advmgr	sector-storage/sched.go:476	SCHED ASSIGNED sqi:0 sector 20782 task seal/v0/provereplicaupdate/1 to window 22
2022-03-01T22:12:33.032+0100	DEBUG	advmgr	sector-storage/sched_worker.go:375	assign worker sector 20782
2022-03-01T22:12:33.032+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 0 queued; 24 open windows
2022-03-01T22:12:33.034+0100	WARN	advmgr	sector-storage/manager_calltracker.go:149	canceling started (not running) work seal/v0/provereplicaupdate/1(bca15f223d1a98f94a2cbcccd77165d08f1d0371cccfe68d0a02791bef200dea)
2022-03-01T22:12:33.034+0100	DEBUG	advmgr	sector-storage/sched_worker.go:280	task done	{"workerid": "5432b068-93c1-46e5-863a-a40858a5de04"}
2022-03-01T22:12:33.034+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 0 queued; 24 open windows
2022-03-01T22:12:33.038+0100	WARN	sectors	storage-sealing/fsm.go:733	sector 20782 got error event sealing.SectorProveReplicaUpdateFailed: prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
2022-03-01T22:12:33.046+0100	ERROR	sectors	storage-sealing/states_failed.go:227	bad PR1: +nil PR2 proof
2022-03-01T22:12:34.087+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 1 queued; 24 open windows
2022-03-01T22:12:34.134+0100	DEBUG	advmgr	sector-storage/sched.go:452	SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]}]
2022-03-01T22:12:34.134+0100	DEBUG	advmgr	sector-storage/sched.go:453	SCHED Acceptable win: [[22 23 7 8]]
2022-03-01T22:12:34.134+0100	DEBUG	advmgr	sector-storage/sched.go:467	SCHED try assign sqi:0 sector 20782 to window 22
2022-03-01T22:12:34.134+0100	DEBUG	advmgr	sector-storage/sched.go:476	SCHED ASSIGNED sqi:0 sector 20782 task seal/v0/provereplicaupdate/1 to window 22
2022-03-01T22:12:34.134+0100	DEBUG	advmgr	sector-storage/sched_worker.go:375	assign worker sector 20782
2022-03-01T22:12:34.134+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 0 queued; 24 open windows
2022-03-01T22:12:34.136+0100	DEBUG	advmgr	sector-storage/sched_worker.go:280	task done	{"workerid": "5432b068-93c1-46e5-863a-a40858a5de04"}
2022-03-01T22:12:34.136+0100	WARN	advmgr	sector-storage/manager_calltracker.go:149	canceling started (not running) work seal/v0/provereplicaupdate/1(bca15f223d1a98f94a2cbcccd77165d08f1d0371cccfe68d0a02791bef200dea)
2022-03-01T22:12:34.136+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 0 queued; 24 open windows
2022-03-01T22:12:34.139+0100	WARN	sectors	storage-sealing/fsm.go:733	sector 20782 got error event sealing.SectorProveReplicaUpdateFailed: prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
2022-03-01T22:12:34.148+0100	ERROR	sectors	storage-sealing/states_failed.go:227	bad PR1: +nil PR2 proof
2022-03-01T22:12:35.088+0100	DEBUG	advmgr	sector-storage/sched.go:356	SCHED 1 queued; 24 open windows

Repo Steps

snap-up sectors - have everything set to false on the miner (sealing), and wait suddenly one sector will get assign to miner and it will get stuck.
@Reiers Reiers added P2 P2: Should be resolved kind/bug Kind: Bug SnapDeals labels Mar 2, 2022
@jennijuju jennijuju added this to the v1.15.1 milestone Mar 7, 2022
@jennijuju jennijuju added dif/easy An existing lotus user should be able to pick this up need/analysis Hint: Needs Analysis labels Mar 7, 2022
@llifezou
Copy link
Contributor

llifezou commented Mar 11, 2022 •
edited
Loading

WARN sectors storage-sealing/fsm.go:733 sector 20782 got error event sealing.SectorProveReplicaUpdateFailed: prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector This is the reason for failure.
Can you provide the lotus-miner sectors status --log

@alyshare-ipfs
Copy link

I built Calibnet environment and tried the snap-deal but I've got he same error.
Any suggestions of the settings what didn't changed or make?

Already enable the AllowReplicaUpdate = true AllowProveReplicaUpdate2 = true AllowRegenSectorKey = true

Got the snap-deal sector with ProveReplicaUpdate but changed continuously into ReplicaUpdateFailed

Worker

lotus-worker --worker-repo=replica-update2 --listen 0.0.0.0:27011 --addpiece=false --precommit1=false --unseal=false --precommit2=false --commit=false --replica-update=true --prove-replica-update2=true

Miner Log

2022-03-15T16:06:01.082+0900    DEBUG   advmgr  sector-storage/sched.go:356     SCHED 1 queued; 20 open windows
2022-03-15T16:06:01.119+0900    DEBUG   advmgr  sector-storage/sched.go:452     SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:0 cpuUse:0 cond:<nil> waiting:0} todo:[]} 
2022-03-15T16:06:01.119+0900    DEBUG   advmgr  sector-storage/sched.go:453     SCHED Acceptable win: [[18 19 6 15]]
2022-03-15T16:06:01.119+0900    DEBUG   advmgr  sector-storage/sched.go:467     SCHED try assign sqi:0 sector 1 to window 18
2022-03-15T16:06:01.119+0900    DEBUG   advmgr  sector-storage/sched.go:476     SCHED ASSIGNED sqi:0 sector 1 task seal/v0/provereplicaupdate/1 to window 18
2022-03-15T16:06:01.119+0900    DEBUG   advmgr  sector-storage/sched_worker.go:375      assign worker sector 1
2022-03-15T16:06:01.119+0900    DEBUG   advmgr  sector-storage/sched.go:356     SCHED 0 queued; 20 open windows
2022-03-15T16:06:01.132+0900    DEBUG   advmgr  sector-storage/sched_worker.go:280      task done       {"workerid": "2dd3af9f-7aec-4996-bc3f-0108ddde6d38"}
2022-03-15T16:06:01.132+0900    DEBUG   advmgr  sector-storage/sched.go:356     SCHED 0 queued; 20 open windows
2022-03-15T16:06:01.132+0900    WARN    advmgr  sector-storage/manager_calltracker.go:149       canceling started (not running) work seal/v0/provereplicaupdate/1(6a59e7d0887714d018d43ae96c5185086b4846f2d8c04351a6029215e81915f9)
2022-03-15T16:06:01.159+0900    WARN    sectors storage-sealing/fsm.go:749      sector 1 got error event sealing.SectorProveReplicaUpdateFailed: prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
2022-03-15T16:06:01.278+0900    ERROR   sectors storage-sealing/states_failed.go:227    bad PR1: +nil PR2 proof

Sector Log

15.     2022-03-15 15:17:37 +0900 JST:  [event;sealing.SectorStartPacking]      {"User":{}}
16.     2022-03-15 15:17:37 +0900 JST:  [event;sealing.SectorPacked]    {"User":{"FillerPieces":null}}
17.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorReplicaUpdate]     {"User":{"Out":{"NewSealed":{"/":"bagboea4b5abca24zve43revygej6f2tp5sgea2aemhqn6xcklevxx2kiikltfeqm"},"NewUnsealed":{"/":"baga6ea4seaqjhcd6weipdczmx5ilnr4b5d
cnqzmibogzybeyrbzvaauzg36zgey"}}}}
18.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
19.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
20.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
21.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
22.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
23.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
24.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
25.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
26.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
27.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
28.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
29.     2022-03-15 15:42:33 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
30.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
31.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
32.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
33.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
34.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}
        prove replica update (1) failed: storage call error 0: allocate local sector for fetching: couldn't find a suitable path for a sector
35.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorRetryProveReplicaUpdate]   {"User":{}}
36.     2022-03-15 15:42:34 +0900 JST:  [event;sealing.SectorProveReplicaUpdateFailed]  {"User":{}}

Then to stop the logging error continuously, need to abort the snap-deal upgrade using command
lotus-miner sectors abort-upgrade --really-do-it 1

What is missing?

@Reiers
Copy link
Author

Reiers commented Mar 30, 2022

So after some more reproducing on latest tag - the scheduler still assigns workers that have these flag set to false:
--replica-update=false --prove-replica-update2=false --regen-sector-key=false

and it want the miner to perform even though everything is set to false in config.toml

Example

Worker 65108336-1663-4631-981c-46184f3a43c3, host amd-worker
        CPU:  [                                                                ] 0/32 core(s) in use
        RAM:  [||                                                              ] 2% 3.424 GiB/125.8 GiB
        VMEM: [|                                                               ] 1% 7.492 GiB/381.8 GiB
        GPU:  [                                                                ] 0% 0.00/1 gpu(s) in use
        GPU: NVIDIA GeForce RTX 3090, not used
Worker 6f5d0c12-ce5d-4c80-8f86-ab41478b50f1, host amd-miner
        CPU:  [                                                                ] 0/128 core(s) in use
        RAM:  [||||                                                            ] 5% 28.28 GiB/503.5 GiB
        VMEM: [||||                                                            ] 5% 29.04 GiB/505.5 GiB
        GPU:  [                                                                ] 0% 0.00/2 gpu(s) in use
        GPU: NVIDIA RTX A5000, not used
        GPU: NVIDIA RTX A5000, not used

Miner log:

2022-03-30T19:52:47.073+0200    WARN    advmgr  sector-storage/manager_calltracker.go:149   canceling started (not running) work seal/v0/replicaupdate([[{"ID":{"Miner":22352,"Number":8210},"ProofType":8},[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqpb2oayf5sh5deovtew3efncuteyk4ulpzovgr2guhmmikj5uf2lq"}}]]])
2022-03-30T19:52:47.073+0200    DEBUG   advmgr  sector-storage/sched.go:356 SCHED 0 queued; 26 open windows
2022-03-30T19:52:47.074+0200    WARN    sectors storage-sealing/fsm.go:757  sector 8210 got error event sealing.SectorUpdateReplicaFailed: replica update failed: Schedule: storage call error 0: %!w(allocate local sector for fetching: couldn't find a suitable path for a sector [Hostname: amd-worker])
2022-03-30T19:52:47.075+0200    INFO    sectors storage-sealing/states_failed.go:28 ReplicaUpdateFailed(8210), waiting 59.924147663s before retrying
2022-03-30T19:53:04.094+0200    WARN    advmgr  sector-storage/manager_calltracker.go:149   canceling started (not running) work seal/v0/provereplicaupdate/1(c49a84795d8d9a82e9c87428091caa32f3f3e132f2e6fd21cf45149f3d6ddcd1)
2022-03-30T19:53:04.094+0200    DEBUG   advmgr  sector-storage/sched_worker.go:280  task done   {"workerid": "6f5d0c12-ce5d-4c80-8f86-ab41478b50f1"}
2022-03-30T19:53:04.094+0200    DEBUG   advmgr  sector-storage/sched.go:356 SCHED 0 queued; 26 open windows
2022-03-30T19:53:04.095+0200    WARN    sectors storage-sealing/fsm.go:757  sector 8211 got error event sealing.SectorProveReplicaUpdateFailed: prove replica update (1) failed: storage call error 0: %!w(allocate local sector for fetching: couldn't find a suitable path for a sector [Hostname: amd-miner])
2022-03-30T19:53:04.096+0200    INFO    sectors storage-sealing/states_failed.go:28 ReplicaUpdateFailed(8211), waiting 59.903554932s before retrying

Both amd-worker and amd-miner does not have the flags set to do the job, but still it tries. Could be minutes, or hours.

This will loop over and over again until it finds the correct workers.

Expected behavior

Only assagin the workers that have these flags set to true:
--replica-update=true --prove-replica-update2=true --regen-sector-key=true

Lotus version

Daemon:  1.15.1-rc4+mainnet+git.6a88a94a8+api1.5.0
Local: lotus version 1.15.1-rc4+mainnet+git.6a88a94a8

cc: @magik6k

@magik6k magik6k mentioned this issue Mar 31, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@magik6k magik6k

Labels
dif/easy An existing lotus user should be able to pick this up kind/bug Kind: Bug need/analysis Hint: Needs Analysis P2 P2: Should be resolved SnapDeals
Projects
None yet
Milestone
v1.15.1
Development

Successfully merging a pull request may close this issue.

fix: sealing: Fix PR1 worker selection filecoin-project/lotus
5 participants
@magik6k @jennijuju @llifezou @Reiers @alyshare-ipfs