Sealing scheduler improvement suggestion

[jennijuju]

A place for our community members to share ideas and suggestion to help us improve the sealing scheduler 🧠 .

A good suggestion should contain:

• The idea.
• The benefit.
• Possible solution 🌟
• Code that implements it 🔥

Discussion is welcomed! And don't forget to give a 👍 to the ideas you like as well!

[jennijuju]

Danny from Filecoin slack suggests:

Right now, the scheduler tries to prioritise sectors that are the latest in the stage. i.e. it will prioritise a commit over a PC2.
While the idea sounds reasonable, in practice this is harmful to performance for non-pipelined setups. It causes sectors to be "bunched up". Instead of evenly spreading work, it will result in sectors being PC1'd at the same time, PC2'd at the same time, WaitSeed at the same time (resulting in idle miners), Commit2 at the same time, etc.
I recommend updating the priority to prioritise whichever stage has the most sectors.

Example:
2 PC1
1 PC2
3 Commit
-> Scheduler will prioritise Commit because there it is more than anything else; then prioritise PC1

The advantage of this is that you will have smooth moving of all of your sectors, resulting in best utilisation of hardware resources, and constant sector pledging, instead of bundling up.

[liuxing111]

Miner调度程序应该可根据用户进行个性化参数设置，可设置是否绑定AP、P1、P2、C2，可根据用户需求自行设置(使用参数)。在Worker中可设置是否使用双CPU、双GPU，如果选择后可动态调整任务分配。

[mxz9102]

Is there a way to let a worker to complete P1, P2, C2 for a sector on the same machine to avoid transferring about 500GB on the network?

[flyworker]

We found that the P1 will be redo after several time trying available P2 worker. if we don't have enough P2 worker, it will accumulate more and more P1 tasks and repeat again and again, which make the small system low efficiency.

Can we set longer period for P1 re-do?

[joshyoonchoi]

• sometimes the scheduler assigns multiple jobs to one worker, even though it should only assign one job

• sometimes the scheduler does not assign jobs to available workers

• sometimes the scheduler doesn't assign a certain type of job (Committing) to my workers

• I wish the scheduler would take into consideration the available disk space and current ongoing jobs before beginning the jobs on newly pledged sectors.

• I wish the lotus-miner sealing sched-diag output was more human-readable and easily understandable.

• I wish I could manually assign jobs to some workers because when I have the problem listed above, I have no idea what to do.

[Meatball13]

• Would like a way to cleanly pause all work (and have it be able to pick back up where it left off) without causing any problems with sectors when reboots, changes to configs, etc., warrant restarts or shutdowns of the miner/workers.

• Would like to have to capability to force a job to move from a worker to another.

• Would like a scheduler flag to force workers to keep all work for a sector on the same machine.

• Would like to have a way to 'restart' a sector from scratch. If there was some sort of problem with the sector through the process, it would be nice to have the capability of zeroing out the sector and having it repack/restart (assuming the deals won't expire before the job will complete) instead of basically having to mark the sector as failed.

• Seems as though the scheduler gets stuck on not allowing some jobs to go if any cores are in use. For example, AP/Packing or PC2's will just pause if PC1 is running. So maybe instead of requiring all cores be free to do some of these other jobs, maybe do (Total Cores - 2) must be available for those types of tasks to run.

• This may not be scheduler specific, but I've noticed recently that my jobs that are working on the same box as the daemon/lotus-miner/scheduler are slowly taking longer to complete. PC1's that used to take 4 hours on the machine now after just one week of mining are taking upwards of 7 hours. (Have a machine with the exact same hardware that is still completing PC1's in 4 hours). Might be the extra overhead for the daemon/scheduler, but seems excessive. Some additional info here

[hyunmoon]

1. A flag to stop receiving additional jobs. A graceful shutdown of lotus-worker will work as well. It will allow the worker to finish up its assigned jobs before shutting down.

2. Friendly worker groups. A worker in the same group is to be favored over the other in another group for the next step.
   If all workers have GPUs (PC1 & PC2 on the same machine), setting a group for each worker will prevent transferring 450+GB of files to a different worker. If only some workers have GPUs (PC1 & PC2 on different machines), it will try to keep the files circulating within the same network switch and reduce the need for a large up-link bandwidth among the network switches.

3. Weight on workers. If everything else is equal, a worker with a higher weight is to be selected similar to storage weight. Being able to set this value on runtime will be helpful for optimizing performance and scheduling downtime for maintenance.

[William8Work]

1. Allow safe shutdown option so that the worker can finish jobs at hand before crashing down through cache, or pause. Even better if it predicts how long it would take to reach the safe shutdown state so that user has a choice if to interrupt and force shutdown or wait for the safe shutdown.

2. Provide a safe upgrade scheduling to inform the worker finish current job then automatically shutdown and restart. Even better if allows to specify the upgrade path.

3. Schedule jobs through predicting the max use cluster resources and best matching job type.

[Shekelme]

I would like to see the weights of storage paths really worked. In my example:

rabinovitch@burstmine:~$ lotus-miner storage list
a9487030-7c6a-4e68-b37e-a97e53906354:
[ ] 982.6 GiB/80.54 TiB 1%
Unsealed: 9; Sealed: 18; Caches: 18; Reserved: 0 B
Weight: 10; Use: Seal Store
Local: /filecoin
URL: http://127.0.0.1:2345/remote

e4a134ac-96cc-48f5-acc6-173ee96764a9:
[##### ] 89.86 GiB/782 GiB 11%
Unsealed: 0; Sealed: 0; Caches: 0; Reserved: 0 B
Weight: 15; Use: Seal
Local: /home/rabinovitch/.lotusminer
URL: http://127.0.0.1:2345/remote

I see that e4a134ac-96cc-48f5-acc6-173ee96764a9 is never being occupied with sealing jobs. All processing is being performed on /filecoin (it's a Raidz2, but sure not as fast as NVME).

Also, there is an issue described in Slack, I'd like to see this solution implemented out of the box, so it would be not a user's trick, but an improvement. ))

[f8-ptrk]

why you run seal+store storage in the first place?

[rjan90]

I would like the scheduler to not override the arguments set by the workers. So a worker set with:
lotus-worker run --addpiece=true --precommit1=true --precommit2=false should only do Add Piece and PreCommit 1, even though it has available resources or are a bit idle waiting for a new sector. Currently, it seems that the scheduler overrides this.

When building out a pipeline of different servers set to different tasks, it would be easier to see where your bottlenecks are, and where you are able to add a new server to fill out that bottleneck.

[s0nik42]

Danny from Filecoin slack suggests:

Right now, the scheduler tries to prioritise sectors that are the latest in the stage. i.e. it will prioritise a commit over a PC2.
While the idea sounds reasonable, in practice this is harmful to performance for non-pipelined setups. It causes sectors to be "bunched up". Instead of evenly spreading work, it will result in sectors being PC1'd at the same time, PC2'd at the same time, WaitSeed at the same time (resulting in idle miners), Commit2 at the same time, etc.
I recommend updating the priority to prioritise whichever stage has the most sectors.

Example:
2 PC1
1 PC2
3 Commit
-> Scheduler will prioritise Commit because there it is more than anything else; then prioritise PC1

The advantage of this is that you will have smooth moving of all of your sectors, resulting in best utilisation of hardware resources, and constant sector pledging, instead of bundling up.

the problem of this is if NEw sectors are produced for any reason quicker

I would like to see the weights of storage paths really worked. In my example:

rabinovitch@burstmine:~$ lotus-miner storage list
a9487030-7c6a-4e68-b37e-a97e53906354:
[ ] 982.6 GiB/80.54 TiB 1%
Unsealed: 9; Sealed: 18; Caches: 18; Reserved: 0 B
Weight: 10; Use: Seal Store
Local: /filecoin
URL: http://127.0.0.1:2345/remote
e4a134ac-96cc-48f5-acc6-173ee96764a9:
[##### ] 89.86 GiB/782 GiB 11%
Unsealed: 0; Sealed: 0; Caches: 0; Reserved: 0 B
Weight: 15; Use: Seal
Local: /home/rabinovitch/.lotusminer
URL: http://127.0.0.1:2345/remote

I see that e4a134ac-96cc-48f5-acc6-173ee96764a9 is never being occupied with sealing jobs. All processing is being performed on /filecoin (it's a Raidz2, but sure not as fast as NVME).

Also, there is an issue described in Slack, I'd like to see this solution implemented out of the box, so it would be not a user's trick, but an improvement. ))

@Shekelme . I read from Magick that the prioritisation is done by comparing StorageWeight*StorageAvailableSpace of each Storage.

So in your case you have 80Tb versus 800Gb, your weight difference has to be much much bigger than 5.

[s0nik42]

@Shekelme, futhermore, if I remember well your setup (just one box) better not failling-over the HDD. Otherwise you take the risk of having to many sectors sealing at the same time and entering an infernal loop. In my case (single machine). I just seal on the NVMe, it limits the number of sealing sectors in // , when there is not enough space, the next block just wait and keep taking deals.

[s0nik42]

I would like :

• PC1/PC2 or PC1/P2 being able to run in parallel on the same machine. (if not possible, can we do that setting up a local worker and isolating a CPU ?!?)
• being able to see in lotus-miner show sectors the remaining time before a WaitingDeals will go for packing
• being able to see in lotus-miner the sector queue order of all sectors not proving.
• prioritization of sectors with deals over pledge sectors

[Shekelme]

While my last sector being sealed was in PC2 phase, I was trying to pledge another one. Miner does not allows this with:

2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:430 SCHED try assign sqi:0 sector 25 to window 0
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:437 SCHED ASSIGNED sqi:0 sector 25 task seal/v0/addpiece to window 0
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:596 assign worker sector 25
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:324 SCHED 0 queued; 2 open windows
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched_resources.go:78 sched: not scheduling on worker 0 for withResources; not enough threads, need 1, 16 in use, target 16
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:415 SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:} todo:[]}]
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:416 SCHED Acceptable win: []

But why not to start sealing of new sector? CPU can start PC1 with 1 thread occupied and PC1 does not require GPU...

[s0nik42]

While my last sector being sealed was in PC2 phase, I was trying to pledge another one. Miner does not allows this with:

2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:430 SCHED try assign sqi:0 sector 25 to window 0
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:437 SCHED ASSIGNED sqi:0 sector 25 task seal/v0/addpiece to window 0
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:596 assign worker sector 25
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:324 SCHED 0 queued; 2 open windows
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched_resources.go:78 sched: not scheduling on worker 0 for withResources; not enough threads, need 1, 16 in use, target 16
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:415 SCHED windows: [{allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:} todo:[]} {allocated:{memUsedMin:0 memUsedMax:0 gpuUsed:false cpuUse:0 cond:} todo:[]}]
2020-09-04T18:04:17.589+0700 DEBUG advmgr sector-storage/sched.go:416 SCHED Acceptable win: []

But why not to start sealing of new sector? CPU can start PC1 with 1 thread occupied and PC1 does not require GPU...

Because lotus doesn't allow to run multiple threaded process a the same time of something else (another multithreaded process or single threaded process)

[Shekelme]

Yes, that is what I am talking about. Lotus doesn't allow. But isn't it a subject to change?

[Shekelme]

@s0nik42 OK, I managed to get around this limitation. But as for the storage weights, I changed them, but it still doesn't work.

NVME is still not used.

[bobjiang]

Improvement suggestion for scheduler:

From my view, I suggest there are 2 policies for scheduler:

1. default policy for scheduler
2. customized policy for scheduler

For advanced users/miners, they could customize the priority of one worker. Let's say following cases:

• Miner (daemon)
• worker 1 (PC1)
• worker 2 (PC2)
• worker 3 (C1&C2)

so as miners, they could set priority for each worker. Example commands could be:

# set priority for current worker
lotus-worker priority 10
# list priority of worker
lotus-worker priority list

[iversonpuppets]

Is there a way to let a worker to complete P1, P2, C2 for a sector on the same machine to avoid transferring about 1TB on the network?

[f8-ptrk]

i do understand the issue.

just saying that with 2 machines for sealing it almost always is better to let at least one of them run PC1 only and not PC1+PC2

[iversonpuppets]

Semi manual and fully manual are actually good idea.
Sounds better than current scheduler(dumb).

[f8-ptrk]

i do not want to touch what the current scheduler is doing at all. i just want to add the possibility to manipulate it from the outide

[iversonpuppets]

here is the explanation how the current scheduler works
filecoin-project/community-china#69
I had a hard time to understand the code.

[f8-ptrk]

plugging into that with an API will not be that hard i believe - not sure if the scheduler has a state or if it is stateless - but if there are major problems occurring i will look at it in detail - for now it is just fine i think

[hxg8202]

Scheduling very silly

[f8-ptrk]

we need a fully externally controlled scheduler. via API interface.

• auto (as is now)
• manual (fully manually scheduled)
• semiauto (manual override possible)

every decision the scheduler makes needs to be able to be done by hand and we need to be able to overwrite decisions the auto scheduler made.

we need to be able to pull the state of the scheduler, listen to new events, etc.

but this here is a year old - i will just goon to define the API and then make the proposal public as soon as it's ready...

[iversonpuppets]

We can manipulate the ResourceTable to control the scheduler. (not a smart/perfect way but possible to get configuration from config.toml).
https://github.com/filecoin-project/lotus/blob/fadc79a4875bf965faa9c749856fb5038527d8b2/extern/sector-storage/resources.go

As windows[wnd].allocated.canHandleRequest(needRes, wid, "schedAssign", wr) variable needRes from resources.go in sched.go

[dayou5168]

Can we turn this into an issue? There are many voices and suggestions from the community

[iversonpuppets]

Please go ahead.

[SgtCoin]

Archiving for future reference. Discussion will remain available for review