Decoupling WinningPoSt and WindowPoSt Process from lotus-miner

[jennijuju]

Author: @siriusmz

Design summary

• Miner gets sector storage info and sector path for PoSt worker.
• PoSt worker gets VanillaPoSt by HTTP from remote workers.
• Miner schedules window PoSt workers to calculate WindowPost in parallel for batch.
• Miner schedules PoSt workers based on GPU .
• Multiple window workers can be deployed according to the size of the cluster.

Detailed design

Add interface

• Add filecoin-ffi interface

// Sorting and de-duplication sector info
func DeDuplicatePrivateSectorInfo(ctx context.Context, privsectors *SortedPrivateSectorInfo) (SortedPrivateSectorInfo, error)

// Get sector info from start to end
func SplitSortedPrivateSectorInfo(ctx context.Context, privsectors SortedPrivateSectorInfo, start int, end int) (SortedPrivateSectorInfo, error)

// Get the number of sectors that need to be certified for each partition of the window
func GetPartitionSectorCount(ctx context.Context, proofType int, repslen int) (int, error)

// Get vanilla PoSt params of partition
func GetPartitionVanillaParams(ctx context.Context,proofType abi.RegisteredPoStProof,sectorsLen uint64,) (partitionVanillaParams string, err error)

// Get vanilla PoSt params of sector
func GetSectorVanillaParams(ctx context.Context, offset int, partitionVanillaParams string) （sectorVanillaParams string, err error)

// Generate window vanilla PoSt for the given sectors which compose a partition
func GenerateWindowPoStVanilla(ctx context.Context,minerID abi.ActorID,privsector PrivateSectorInfo,randomness abi.PoStRandomness,sectorVanillaParams string) (string, error)

// Generate window PoSt with generated window vanilla PoSt result
func GenerateWindowPoStWithVanillas(ctx context.Context,minerID abi.ActorID,privsectors SortedPrivateSectorInfo,randomness abi.PoStRandomness,vanillas []string) ([]proof5.PoStProof, error)

// Generate winning vanilla PoSt for the given sector
func GenerateWinningPoStVanilla(ctx context.Context,minerID abi.ActorID,privsector PrivateSectorInfo,randomness abi.PoStRandomness) (vanilla string, err error)

// Generate winning PoSt with generated winning vanilla PoSt result
func GenerateWinningPoStWithVanillas(ctx context.Context,minerID abi.ActorID,privsectors SortedPrivateSectorInfo,randomness abi.PoStRandomness,vanillas []string) ([]proof5.PoStProof, error)

• Add index interface

Used to obtain sector's URL

StorageGetUrl(ctx context.Context, s abi.SectorID, ft storiface.SectorFileType) (string, error)

• Extend ffiwraper.storage interface

Used to support storage to call ffi-wrapper interface to obtain vanilla PoSt result

type WorkerProver interface {
	// worker_local generate window vanilla PoSt
    GenerateWindowPoStWithVanilla(ctx context.Context, mid abi.ActorID, privsectors SortedPrivateSectorInfo, randomness abi.PoStRandomness, vanillas []string) ([]proof.PoStProof, error)
    
    // worker_local generate winning vanilla PoSt
    GenerateWinningPoStWithVanilla(ctx context.Context, mid abi.ActorID, privsectors SortedPrivateSectorInfo, randomness abi.PoStRandomness, vanillas []string) ([]proof.PoStProof, error)
    
    // worker_local get vanilla PoSt params
    GetSectorVanillaParams(ctx context.Context, offset int, partitionVanillaParams string) string
}

• Extend SectorManager interface

Support the manager to call the ffiwrapper interface to obtain the basic information that needed when generating PoSt, including sector information, vanilla PoSt params and the number of sectors in each partition.

type MinerProver interface {
	// SectorManager gets sorted and de-duplication sectors info.
    PubSectorToPriv(ctx context.Context, mid abi.ActorID, sectorInfo []proof5.SectorInfo, faults []abi.SectorNumber, rpt func(abi.RegisteredSealProof) (abi.RegisteredPoStProof, error)) (SortedPrivateSectorInfo, []abi.SectorID, func(), error)
    
    // SectorManager splits the sector sets according to the number of sectors in each partition.
    SplitSortedPrivateSectorInfo(ctx context.Context, privsectors SortedPrivateSectorInfo, off int, end int) (SortedPrivateSectorInfo, error)
    
    GetPartitionSectorsCount(ctx context.Context, prooftype abi.RegisteredPoStProof, sectorsLen int) (int, error)
    
    GetPartitionVanillaParams(ctx context.Context, proofType abi.RegisteredPoStProof, sectorsLen uint64) (string, error)
}

• Extend workerCalls interface

Proof interface supported by PoSt worker

type WorkerCalls interface {
    GenerateWindowPoSt(ctx context.Context, mid abi.ActorID, privsectors SortedPrivateSectorInfo, partitionVanillaParams string, offset int, randomness abi.PoStRandomness) (WindowPoStResult, error)
    
    GenerateWinningPoSt(ctx context.Context, mid abi.ActorID, privsectors SortedPrivateSectorInfo, randomness abi.PoStRandomness) ([]proof.PoStProof, error)
}

• Add stores.Store interface

Obtain the sector's vanilla PoSt from storage by HTTP

//Obtain the vanilla PoSt by HTTP
GenerateWindowPoStVanilla(ctx context.Context, minerID abi.ActorID, privsector *ffiwrapper.PrivateSectorInfo, vanillaParams string, randomness abi.PoStRandomness) (string, error)
GenerateWinningPoStVanilla(ctx context.Context, minerID abi.ActorID, privsector *ffiwrapper.PrivateSectorInfo, randomness abi.PoStRandomness) (string, error)

PoSt worker scheduler

type poStScheduler struct {
    lk       sync.RWMutex
    // PoSt worker map
    workers  map[WorkerID]*workerHandle
    cond     *sync.Cond
    // PoSt type TTGenerateWindowPoSt or TTGenerateWinningPoSt
    postType sealtasks.TaskType
}
// add post worker
func (ps *poStScheduler) AddWorker(wid WorkerID, w *workerHandle) bool

// del post worker
func (ps *poStScheduler) delWorker(wid WorkerID) *workerHandle 

// determine whether there are workers that can be scheduled
func (ps *poStScheduler) CanSched(ctx context.Context) bool 

// schedule PoSt worker
func (ps *poStScheduler) SchedWorker(ctx context.Context, primary bool) (Worker, PoStDone, error) 

// determine whether the worker can handle the task
func (ps *poStScheduler) canHandleWorkers() (bool, []WorkerID)

// get resource
func (ps *poStScheduler) addResource(wid WorkerID)

// free resource
func (ps *poStScheduler) freeResource(wid WorkerID) 

// determine whether the worker is valid 
func (ps *poStScheduler) watch(wid WorkerID, worker *workerHandle) 

// clean up worker when lost connection
func (ps *poStScheduler) workerCleanup(wid WorkerID, w *workerHandle) 

// close PoSt worker scheduler
func (ps *poStScheduler) schedClose()

// get worker's status
func (ps *poStScheduler) WorkerStats(cb func(wid WorkerID, worker *workerHandle))

• The scheduling of window PoSt and winning PoSt is realized by instantiating PoSt scheduler.

type scheduler struct {
...
    // window scheduler
    windowPoStSched *poStScheduler
    
    // winning scheduler
    winningPoStSched *poStScheduler

...
}

• Through the above method, the window worker machine can be expanded to multiple.

Compatible with original PoSt

func (m *Manager) GenerateWinningPoSt(ctx context.Context, minerID abi.ActorID, sectorInfo []proof.SectorInfo, randomness abi.PoStRandomness) ([]proof.PoStProof, error) {
    if !m.sched.winningPoStSched.CanSched(ctx) {
        return m.Prover.GenerateWinningPoSt(ctx, minerID, sectorInfo, randomness)
    }
    
    // Perform scheduling to calculate winning PoSt
    return m.generateWinningPoSt(ctx, minerID, sectorInfo, randomness)
}

func (m *Manager) GenerateWindowPoSt(ctx context.Context, minerID abi.ActorID, sectorInfo []proof.SectorInfo, randomness abi.PoStRandomness) (proof []proof.PoStProof, skipped []abi.SectorID, err error) {
    if !m.sched.windowPoStSched.CanSched(ctx) {
        return m.Prover.GenerateWindowPoSt(ctx, minerID, sectorInfo, randomness)
    }
    
	// Perform scheduling to calculate window PoSt
    return m.generateWindowPoSt(ctx, minerID, sectorInfo, randomness)
}

• The generateWindowPoSt function splits the batch to partitions and calculates them in parallel.

Compute each batch in parallel

• Modify the logic of the runPoStCycle function to calculate in parallel between multiple batches

Get vanilla PoSt

• http_handler implements generating vanilla PoSt interface

func (handler *FetchHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
...
    mux.HandleFunc("/remote/vanilla/window", handler.generateWindowPoStVanilla).Methods("POST")
    mux.HandleFunc("/remote/vanilla/winning", handler.generateWinningPoStVanilla).Methods("POST")
...
}

• The PoSt workers call the GenerateWindowPoStVanilla or GenerateWinningPoStVanilla interface which implemented by stores.Store to obtain vanilla PoSt result from the sector's storage.

[jennijuju]

@magik6k @raulk I feel like a new design requirement for this would be - it need to work with the Subsystem architecture introduced in v1.11.1. What do you think?

[f8-ptrk]

i am not raul or magic, but i see the same "problem".

[f8-ptrk]

one question i see when looking at the config vars for the subsystems: will they implement the same thing but via subsystems with, in a later stage, a N:M setup?

[siriusyim]

hey @jennijuju i think it can work with the Subsystem architecture introduced in v1.11.1

[siriusyim]

Now this task is coding and debugging on devnet

[siriusyim]

hey @jennijuju ，I am testing this post decoupling function. I will submit the code here temporarily. If there is no problem in the test, I will submit a PR: https://github.com/wusirdcenter/lotus/tree/feat/decoupling-post-worker
https://github.com/wusirdcenter/filecoin-ffi/tree/feat/decoupling-post-worker
https://github.com/wusirdcenter/rust-fil-proofs/tree/feat/decoupling-post-worker
https://github.com/wusirdcenter/rust-filecoin-proofs-api/tree/feat/decoupling-post-worker

[KIMQI001]

it works perfectly 👍

[PerrywLiu]

@KIMQI001 Now we are discussing with PL about the rectification of the rust interface, please do not go to the production environment

How is it going? Is it ready for production environment now？

[siriusyim]

@KIMQI001 Now we are discussing with PL about the rectification of the rust interface, please do not go to the production environment

How is it going? Is it ready for production environment now？

PR has been submitted, waiting for code approval and merge

[PerrywLiu]

Great ! I'm going to study and test it locally, Looking forward to an early merger~！

[f8-ptrk]

If the lotus-worker process is started on the miner machine, do not let him support wn｜wd, such as lotus-worker run --windowpost=false --winningpost=false

will this be the case in the finished "product"? this needs to be possible

[jennijuju]

@siriusmz could you please confirm if the wdpost worker can:

• read the sector info for proving from remote storage
• miners can move the data from the current miner storage to the wdpost worker?

[siriusyim]

I understand that these two questions mean the same thing, no, can’t do this

[jennijuju]

could you please elaborate more on why it cant be done?

[magik6k]

read the sector info for proving from remote storage

Sector info (the metadata) lives on the main mining node, not in sector storage, and it's sent to the PoSt worker when the mining node request PoSt to be computed. The PoSt worker will then just read challenges over http from other workers in the system.

(basically PoSt workers don't have any local storage, they just compute things, at least that's how I understand this design (and I like it being this way))

[siriusyim]

To get sector challenges, need to get the sector path information (pubSectorToPriv) first. If the sector is stored in the work, it will trigger the sector fetch (worker-->miner).

[siriusyim]

I will implement my ideas to solve this problem

[siriusyim]

Hi,@jennijuju We have started to submit PRs,
here is the PRs related to the algorithm：
filecoin-project/rust-fil-proofs#1497
filecoin-project/rust-filecoin-proofs-api#59
filecoin-project/filecoin-ffi#197

[jennijuju]

@siriusmz ffi v10.1.0 is tagged and has the partition API you needed. Could you please start working on the lotus side of the changes and try to have a PR ready for review in mid Nov?

[jennijuju]

Ideally id like wdpost worker to be supported in lotus v1.13.2

[siriusyim]

I am revising and testing and will submit PR as soon as possible

[RobQuistNL]

@siriusmz are you still working on this? We could really use it :+)

[siriusyim]

The PR has been submitted, is being reviewed and communicated
filecoin-project/filecoin-ffi#197

/pull/7600

[RobQuistNL]

awesome, thanks!