decoupling post worker #7600

siriusyim · 2021-11-05T06:45:21Z

based on：
#6943

siriusyim · 2021-11-05T06:58:34Z

Hi，@jennijuju
These ci errors are due to the fact that the PR has not been merged: filecoin-project/filecoin-ffi#197

After this part is merged, I will modify it again

Kubuxu · 2021-11-17T15:43:29Z

extern/sector-storage/ffiwrapper/types.go

+
+var GenerateSingleVanillaProof = ffi.GenerateSingleVanillaProof
+
+var MergeWindowPoStPartitionProofs = ffi.MergeWindowPoStPartitionProofs


ffi types should not be used as part of the interface

Ok, resubmitted

magik6k

Sorry for taking so long on this, lets work on landing this soon.

(reviewing the ffiwrapper parts for now)

I'm not too sure if we need DeDuplicatePrivateSectorInfo, it seems like it should be easy to avoid it

magik6k · 2021-11-05T10:42:08Z

extern/sector-storage/stores/http_handler.go

+	}
+	body, err := ioutil.ReadAll(r.Body)
+	if err != nil {
+		failedRet(500, err)


Could use http.Error

magik6k · 2021-11-05T11:00:18Z

extern/sector-storage/sealtasks/task.go

+	TTGenerateWindowPoSt:  "GWDP",
+	TTGenerateWinningPoSt: "GWNP",


Looks tiny bit cleaner:

Suggested change

TTGenerateWindowPoSt: "GWDP",

TTGenerateWinningPoSt: "GWNP",

TTGenerateWindowPoSt: "WDP",

TTGenerateWinningPoSt: "WNP",

magik6k · 2021-11-05T11:02:58Z

api/api_worker.go

+	GenerateWinningPoSt(ctx context.Context, mid abi.ActorID, privsectors ffiwrapper.SortedPrivateSectorInfo, randomness abi.PoStRandomness, sectorChallenges *ffiwrapper.FallbackChallenges) ([]proof.PoStProof, error)                                      //perm:admin
+	GenerateWindowPoSt(ctx context.Context, mid abi.ActorID, privsectors ffiwrapper.SortedPrivateSectorInfo, partitionIdx int, offset int, randomness abi.PoStRandomness, postChallenges *ffiwrapper.FallbackChallenges) (ffiwrapper.WindowPoStResult, error) //perm:admin


Nit: move above TaskDisable, so that calls which do work are grouped together.

magik6k · 2021-12-07T12:28:36Z

extern/sector-storage/ffiwrapper/verifier_cgo.go

+
+	privsectors := ffi.NewSortedPrivateSectorInfo(out...)
+
+	deDuplicationPrivsectors, err := ffi.DeDuplicatePrivateSectorInfo(ctx, &privsectors)


Could we, instead of adding this function, do the following:

Add a seen := map[abi.SectorNumber]struct{} map above

Keep track of what sectors we've added to out in that map

Skip appending to out if we've seen the sector.

This way we should be able to avoid adding this function in filecoin-ffi

Thank you, this is a good idea for deduplication, but after deduplication according to the underlying logic of ffi, the sector set must be sorted according to sectorNumber

jennijuju · 2021-12-15T04:32:27Z

Hi @siriusmz how is this going?

siriusyim · 2021-12-15T05:47:16Z

Hi @siriusmz how is this going?
Based on the logic of the ffi library, we believe that the post cannot be modified in accordance with the review comments after the post is split. The reasons are as follows

siriusyim · 2021-12-15T05:51:52Z

Hi @siriusmz how is this going?

@jennijuju Others have been modified in accordance with the review comments

hail100 · 2022-02-21T04:47:52Z

hi @siriusmz @jennijuju @magik6k，can this feature merge before march 4th？

magik6k · 2022-03-26T00:32:54Z

Thanks for the PR!

Merged with #7971

siriusyim requested a review from a team as a code owner November 5, 2021 06:45

Kubuxu reviewed Nov 17, 2021

View reviewed changes

siriusyim force-pushed the feat/decoupling-post-worker branch from 24ba063 to a1bcd4a Compare November 18, 2021 04:39

magik6k reviewed Dec 7, 2021

View reviewed changes

siriusyim force-pushed the feat/decoupling-post-worker branch from a1bcd4a to b387349 Compare December 8, 2021 06:07

siriusyim force-pushed the feat/decoupling-post-worker branch from b387349 to 9e4447b Compare December 22, 2021 02:22

jennijuju added the P1 P1: Must be resolved label Dec 29, 2021

jennijuju modified the milestones: v1.13.3, v1.15.0 Dec 29, 2021

mz-sirius added 2 commits January 5, 2022 01:50

decoupling winningpost and windowpost from lotus-miner

3fd55fa

update ffi

6867f01

siriusyim force-pushed the feat/decoupling-post-worker branch 3 times, most recently from 95a787c to 057efae Compare January 5, 2022 11:57

fix ci err

793b5c7

siriusyim force-pushed the feat/decoupling-post-worker branch from 057efae to 793b5c7 Compare January 5, 2022 13:41

magik6k mentioned this pull request Jan 10, 2022

feat: Manually triggered window PoSt #7915

Closed

5 tasks

magik6k mentioned this pull request Jan 19, 2022

feat: PoSt workers #7971

Merged

5 tasks

magik6k self-assigned this Feb 10, 2022

magik6k merged commit cf7dd36 into filecoin-project:master Mar 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

decoupling post worker #7600

decoupling post worker #7600

siriusyim commented Nov 5, 2021

siriusyim commented Nov 5, 2021 •

edited

Loading

Kubuxu Nov 17, 2021

siriusyim Nov 18, 2021

magik6k left a comment

magik6k Nov 5, 2021

siriusyim Dec 8, 2021

magik6k Nov 5, 2021

magik6k Nov 5, 2021

magik6k Dec 7, 2021

siriusyim Dec 8, 2021

siriusyim Dec 8, 2021

jennijuju commented Dec 15, 2021

siriusyim commented Dec 15, 2021

siriusyim commented Dec 15, 2021 •

edited

Loading

hail100 commented Feb 21, 2022 •

edited

Loading

magik6k commented Mar 26, 2022


		var GenerateSingleVanillaProof = ffi.GenerateSingleVanillaProof

		var MergeWindowPoStPartitionProofs = ffi.MergeWindowPoStPartitionProofs

		GenerateWinningPoSt(ctx context.Context, mid abi.ActorID, privsectors ffiwrapper.SortedPrivateSectorInfo, randomness abi.PoStRandomness, sectorChallenges *ffiwrapper.FallbackChallenges) ([]proof.PoStProof, error) //perm:admin
		GenerateWindowPoSt(ctx context.Context, mid abi.ActorID, privsectors ffiwrapper.SortedPrivateSectorInfo, partitionIdx int, offset int, randomness abi.PoStRandomness, postChallenges *ffiwrapper.FallbackChallenges) (ffiwrapper.WindowPoStResult, error) //perm:admin


		privsectors := ffi.NewSortedPrivateSectorInfo(out...)

		deDuplicationPrivsectors, err := ffi.DeDuplicatePrivateSectorInfo(ctx, &privsectors)

decoupling post worker #7600

decoupling post worker #7600

Conversation

siriusyim commented Nov 5, 2021

siriusyim commented Nov 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jennijuju commented Dec 15, 2021

siriusyim commented Dec 15, 2021

siriusyim commented Dec 15, 2021 • edited Loading

hail100 commented Feb 21, 2022 • edited Loading

magik6k commented Mar 26, 2022

siriusyim commented Nov 5, 2021 •

edited

Loading

siriusyim commented Dec 15, 2021 •

edited

Loading

hail100 commented Feb 21, 2022 •

edited

Loading