Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: sealing: flag to run data_cid untied from addpiece #10797

Merged
merged 5 commits into from
May 9, 2023

Conversation

snadrus
Copy link
Collaborator

@snadrus snadrus commented May 2, 2023

Related Issues

#10351

Proposed Changes

data_cid flag that workers can use to enable/disable that task irrespective of addpiece. Backwards compat is maintained (addpiece starts data_cid)

Additional Info

CLI description of usage added.

@snadrus snadrus linked an issue May 3, 2023 that may be closed by this pull request
18 tasks
@snadrus snadrus marked this pull request as ready for review May 4, 2023 14:21
@snadrus snadrus requested a review from a team as a code owner May 4, 2023 14:21
@rjan90
Copy link
Contributor

rjan90 commented May 5, 2023

I ran some tests with this on a local network with all sealing tasks disabled on the internal Lotus-Miner worker:

  1. Ran a worker with lotus-worker run --no-default --data-cid=true

  2. Check the worker info, to see which tasks it can run:

lotus-worker info
Worker version:  1.7.0
CLI version: lotus-worker version 1.23.2-dev+2k+git.702ef3336

Session: 4aadb451-860d-4295-ab6d-26f851717233
Enabled: true
Hostname: Ubuntu-2004-focal-amd64-base
CPUs: 12; GPUs: []
RAM: 2.915 GiB/62.78 GiB; Swap: 0 B/0 B
Task types: FUS FIN GET FRU C1 PR1 DC 

5414c867-6251-45b9-9d40-922776c0d020:
        Weight: 10; Use: Seal 
        Local: /root/.lotusworker
  1. Ran a compute dataCID task and check that its successful:
lotus-miner sealing data-cid /root/text.txt
baga6ea4seaqar7r7nt75pkhf7n64t733md2e3pv277h3q5xhugwrepdhvzkgwei   256

I also did some other runs, checking that you can disable the dataCID-task when addpiece is enabled:

  1. lotus-worker run --no-default --addpiece=true --data-cid=false
  2. The worker info correctly shows that the dataCID task is disabled, while addpiece is enabled:
lotus-worker info
Worker version:  1.7.0
CLI version: lotus-worker version 1.23.2-dev+2k+git.702ef3336

Session: 5e39d7bb-ae85-4aef-a9a7-89d9c45fd021
Enabled: true
Hostname: Ubuntu-2004-focal-amd64-base
CPUs: 12; GPUs: []
RAM: 4.763 GiB/62.78 GiB; Swap: 0 B/0 B
Task types: FUS FIN GET FRU C1 PR1 AP 

5414c867-6251-45b9-9d40-922776c0d020:
        Weight: 10; Use: Seal 
        Local: /root/.lotusworker
  1. Sanity checked that I would not be able to compute a dataCID in this case. The lotus-miner sealing data-cid /root/text.txt cmd would now just hang, until it was canceled.

Potential enhancements

One thing that I noted, was that when computing the dataCID there where no outputs in the lotus-workers log:

lotus-worker run --no-default --data-cid=true 
2023-05-05T05:00:17.728-0400    INFO    main    lotus-worker/main.go:295        Starting lotus worker
2023-05-05T05:00:17.732-0400    INFO    main    lotus-worker/main.go:358        Remote version 1.23.2-dev+2k+git.702ef3336+api1.5.0
...........
2023-05-05T05:00:17.809-0400    INFO    main    lotus-worker/main.go:679        Making sure no local tasks are running
2023-05-05T05:00:17.847-0400    INFO    main    lotus-worker/main.go:702        Worker registered successfully, waiting for tasks

It would be nice if we could log every time a lotus-worker is running/computing a dataCID-task, so its easier to track where the dataCID task was run.

Copy link
Contributor

@magik6k magik6k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct, tho two small comments.

The logging thing can be a separate PR, just adding a log to storage/sealer/worker_local.go (really we could add structured logs for every task there, those will be more useful when we move away from the centralized scheduler)

cmd/lotus-worker/main.go Outdated Show resolved Hide resolved
cmd/lotus-worker/main.go Outdated Show resolved Hide resolved
@magik6k magik6k merged commit 3b4fa77 into master May 9, 2023
@magik6k magik6k deleted the sealerFlagForDataCid3 branch May 9, 2023 14:26
@rjan90 rjan90 mentioned this pull request Jul 11, 2023
40 tasks
@Shekelme
Copy link

I am running lotus-worker with this string:
nohup lotus-worker --worker-repo /home/admfc/worker1 run --listen=192.168.11.17:3456 --name lenovo7 --precommit1=true --addpiece=true --unseal=true --data-cid=true --no-default > ~/worker1.log 2>&1 &

But there is stil no DC in supported tasks list:

Worker version:  1.7.0
CLI version: lotus-worker version 1.25.1+mainnet+git.924af4294

Session: c89bc57f-a271-44a2-9537-9dab7e36a3c9
Enabled: true
Hostname: lenovo7
CPUs: 32; GPUs: [NVIDIA GeForce RTX 3090]
RAM: 8.684 GiB/503.6 GiB; Swap: 0 B/0 B
Task types: FUS FIN GET FRU UNS C1 PC1 PR1 AP

9a6b44a8-52c6-4c1f-bade-a285e7143076:
        Weight: 10; Use: Seal
        Local: /home/admfc/worker1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TTDataCid (DC) is missing from lotus-worker CLI (run --option)
4 participants