Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bypass task scheduler for reading unsealed pieces #6280

Merged
merged 25 commits into from
Jun 7, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
670835f
bypass task scheduler for reading unsealed pieces
aarshkshah1992 May 18, 2021
65eb610
docs, logs and green ci
aarshkshah1992 May 18, 2021
c853350
Apply suggestions from code review
aarshkshah1992 May 19, 2021
db5c881
address review comments
aarshkshah1992 May 19, 2021
759d8f0
test http handler
aarshkshah1992 May 19, 2021
74372d3
fix linting problems
aarshkshah1992 May 19, 2021
6879ae9
feat: TestPieceProviderReadPiece
dirkmc May 19, 2021
77b5e8d
use an actual worker in the integration tests
aarshkshah1992 May 20, 2021
0d88800
use mockgen
aarshkshah1992 May 20, 2021
4efaa8d
unit tests for the remote store Reader
aarshkshah1992 May 20, 2021
35a0dbf
fix go mod
aarshkshah1992 May 20, 2021
ad4b182
remove read task type and run gen and docsgen
aarshkshah1992 May 20, 2021
acfa3d7
finish integration tests
aarshkshah1992 May 21, 2021
22f3648
more logging
aarshkshah1992 May 21, 2021
3b792a3
better logging
aarshkshah1992 May 21, 2021
73f7825
clean up logging
aarshkshah1992 May 21, 2021
207f0d9
integration test should remove unsealed files
aarshkshah1992 May 21, 2021
ec6a496
logs to debug read and unseal
aarshkshah1992 May 21, 2021
2a13488
logs to debug read & unseal
aarshkshah1992 May 21, 2021
8d9cef1
changes as per review
aarshkshah1992 May 21, 2021
21e6b50
finished rebasing PR
aarshkshah1992 Jun 7, 2021
1a90d3b
update ffi
aarshkshah1992 Jun 7, 2021
ddd9bf6
fix CI
aarshkshah1992 Jun 7, 2021
7124cd5
fix lotus soup build
aarshkshah1992 Jun 7, 2021
8625da3
Bump miner/worker api versions
magik6k Jun 7, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -331,7 +331,7 @@ jobs:
- run: cd extern/filecoin-ffi && make
- run:
name: "go get lotus@master"
command: cd testplans/lotus-soup && go mod edit -replace=github.com/filecoin-project/lotus=../..
command: cd testplans/lotus-soup && go mod edit -replace=github.com/filecoin-project/lotus=../.. && go mod tidy
- run:
name: "build lotus-soup testplan"
command: pushd testplans/lotus-soup && go build -tags=testground .
Expand Down
2 changes: 0 additions & 2 deletions api/api_worker.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ package api

import (
"context"
"io"

"github.com/google/uuid"
"github.com/ipfs/go-cid"
Expand Down Expand Up @@ -43,7 +42,6 @@ type Worker interface {
ReleaseUnsealed(ctx context.Context, sector storage.SectorRef, safeToFree []storage.Range) (storiface.CallID, error) //perm:admin
MoveStorage(ctx context.Context, sector storage.SectorRef, types storiface.SectorFileType) (storiface.CallID, error) //perm:admin
UnsealPiece(context.Context, storage.SectorRef, storiface.UnpaddedByteIndex, abi.UnpaddedPieceSize, abi.SealRandomness, cid.Cid) (storiface.CallID, error) //perm:admin
ReadPiece(context.Context, io.Writer, storage.SectorRef, storiface.UnpaddedByteIndex, abi.UnpaddedPieceSize) (storiface.CallID, error) //perm:admin
Fetch(context.Context, storage.SectorRef, storiface.SectorFileType, storiface.PathType, storiface.AcquireMode) (storiface.CallID, error) //perm:admin

TaskDisable(ctx context.Context, tt sealtasks.TaskType) error //perm:admin
Expand Down
820 changes: 410 additions & 410 deletions api/mocks/mock_full.go

Large diffs are not rendered by default.

11 changes: 0 additions & 11 deletions api/proxy_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion api/test/deals.go
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ func TestOfflineDealFlow(t *testing.T, b APIBuilder, blocktime time.Duration, st
require.Eventually(t, func() bool {
cd, _ := s.client.ClientGetDealInfo(s.ctx, *proposalCid)
return cd.State == storagemarket.StorageDealCheckForAcceptance
}, 1*time.Second, 100*time.Millisecond, "actual deal status is %s", storagemarket.DealStates[cd.State])
}, 30*time.Second, 1*time.Second, "actual deal status is %s", storagemarket.DealStates[cd.State])

// Create a CAR file from the raw file
carFileDir, err := ioutil.TempDir(os.TempDir(), "test-make-deal-car")
Expand Down
816 changes: 408 additions & 408 deletions api/v0api/v0mocks/mock_full.go

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions api/version.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ var (
FullAPIVersion0 = newVer(1, 3, 0)
FullAPIVersion1 = newVer(2, 1, 0)

MinerAPIVersion0 = newVer(1, 0, 1)
WorkerAPIVersion0 = newVer(1, 0, 0)
MinerAPIVersion0 = newVer(1, 1, 0)
WorkerAPIVersion0 = newVer(1, 1, 0)
)

//nolint:varcheck,deadcode
Expand Down
Binary file modified build/openrpc/full.json.gz
Binary file not shown.
Binary file modified build/openrpc/miner.json.gz
Binary file not shown.
Binary file modified build/openrpc/worker.json.gz
Binary file not shown.
51 changes: 26 additions & 25 deletions cli/servicesmock_test.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions cmd/lotus-seal-worker/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -362,9 +362,10 @@ var runCmd = &cli.Command{
return xerrors.Errorf("could not get api info: %w", err)
}

remote := stores.NewRemote(localStore, nodeApi, sminfo.AuthHeader(), cctx.Int("parallel-fetch-limit"))
remote := stores.NewRemote(localStore, nodeApi, sminfo.AuthHeader(), cctx.Int("parallel-fetch-limit"),
&stores.DefaultPartialFileHandler{})

fh := &stores.FetchHandler{Local: localStore}
fh := &stores.FetchHandler{Local: localStore, PfHandler: &stores.DefaultPartialFileHandler{}}
remoteHandler := func(w http.ResponseWriter, r *http.Request) {
if !auth.HasPerm(r.Context(), nil, api.PermAdmin) {
w.WriteHeader(401)
Expand Down
14 changes: 12 additions & 2 deletions cmd/lotus-storage-miner/init.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"os"
"path/filepath"
"strconv"
Expand Down Expand Up @@ -453,14 +454,23 @@ func storageMinerInit(ctx context.Context, cctx *cli.Context, api v1api.FullNode
wsts := statestore.New(namespace.Wrap(mds, modules.WorkerCallsPrefix))
smsts := statestore.New(namespace.Wrap(mds, modules.ManagerWorkPrefix))

smgr, err := sectorstorage.New(ctx, lr, stores.NewIndex(), sectorstorage.SealerConfig{
si := stores.NewIndex()

lstor, err := stores.NewLocal(ctx, lr, si, nil)
if err != nil {
return err
}
stor := stores.NewRemote(lstor, si, http.Header(sa), 10, &stores.DefaultPartialFileHandler{})

smgr, err := sectorstorage.New(ctx, lstor, stor, lr, si, sectorstorage.SealerConfig{
ParallelFetchLimit: 10,
AllowAddPiece: true,
AllowPreCommit1: true,
AllowPreCommit2: true,
AllowCommit: true,
AllowUnseal: true,
}, nil, sa, wsts, smsts)
}, wsts, smsts)

if err != nil {
return err
}
Expand Down
37 changes: 0 additions & 37 deletions documentation/en/api-v0-methods-worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@
* [MoveStorage](#MoveStorage)
* [Process](#Process)
* [ProcessSession](#ProcessSession)
* [Read](#Read)
* [ReadPiece](#ReadPiece)
* [Release](#Release)
* [ReleaseUnsealed](#ReleaseUnsealed)
* [Seal](#Seal)
Expand Down Expand Up @@ -263,41 +261,6 @@ Inputs: `null`

Response: `"07070707-0707-0707-0707-070707070707"`

## Read


### ReadPiece


Perms: admin

Inputs:
```json
[
{},
{
"ID": {
"Miner": 1000,
"Number": 9
},
"ProofType": 8
},
1040384,
1024
]
```

Response:
```json
{
"Sector": {
"Miner": 1000,
"Number": 9
},
"ID": "07070707-0707-0707-0707-070707070707"
}
```

## Release


Expand Down
Loading