Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

migrate to DAG store + CARv2 blockstores for storage and retrieval #576

Merged
merged 1 commit into from
Aug 16, 2021
Merged

migrate to DAG store + CARv2 blockstores for storage and retrieval #576

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ orbs:
executors:
golang:
docker:
- image: circleci/golang:1.14-node
- image: circleci/golang:1.16.4
resource_class: large

commands:
Expand Down
2 changes: 2 additions & 0 deletions discovery/migrations/migrations_cbor_gen.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions discovery/types_cbor_gen.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 27 additions & 11 deletions docs/retrievalclient.mmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ stateDiagram-v2
state "DealStatusWaitForAcceptanceLegacy" as DealStatusWaitForAcceptanceLegacy
state "DealStatusWaitingForLastBlocks" as DealStatusWaitingForLastBlocks
state "DealStatusPaymentChannelAddingInitialFunds" as DealStatusPaymentChannelAddingInitialFunds
state "DealStatusErroring" as DealStatusErroring
state "DealStatusRejecting" as DealStatusRejecting
state "DealStatusDealNotFoundCleanup" as DealStatusDealNotFoundCleanup
state "DealStatusFinalizingBlockstore" as DealStatusFinalizingBlockstore
DealStatusNew : On entry runs ProposeDeal
DealStatusPaymentChannelCreating : On entry runs WaitPaymentChannelReady
DealStatusPaymentChannelAddingFunds : On entry runs WaitPaymentChannelReady
Expand All @@ -42,24 +46,28 @@ stateDiagram-v2
DealStatusCancelling : On entry runs CancelDeal
DealStatusRetryLegacy : On entry runs ProposeDeal
DealStatusPaymentChannelAddingInitialFunds : On entry runs WaitPaymentChannelReady
DealStatusErroring : On entry runs FailsafeFinalizeBlockstore
DealStatusRejecting : On entry runs FailsafeFinalizeBlockstore
DealStatusDealNotFoundCleanup : On entry runs FailsafeFinalizeBlockstore
DealStatusFinalizingBlockstore : On entry runs FinalizeBlockstore
[*] --> DealStatusNew
note right of DealStatusNew
The following events are not shown cause they can trigger from any state.

ClientEventWriteDealProposalErrored - transitions state to DealStatusErrored
ClientEventWriteDealProposalErrored - transitions state to DealStatusErroring
ClientEventUnknownResponseReceived - transitions state to DealStatusFailing
ClientEventDataTransferError - transitions state to DealStatusErrored
ClientEventWriteDealPaymentErrored - transitions state to DealStatusErrored
ClientEventDataTransferError - transitions state to DealStatusErroring
ClientEventWriteDealPaymentErrored - transitions state to DealStatusErroring
ClientEventProviderCancelled - transitions state to DealStatusCancelling
ClientEventCancel - transitions state to DealStatusCancelling
end note
DealStatusNew --> DealStatusNew : ClientEventOpen
DealStatusNew --> DealStatusWaitForAcceptance : ClientEventDealProposed
DealStatusRetryLegacy --> DealStatusWaitForAcceptanceLegacy : ClientEventDealProposed
DealStatusWaitForAcceptance --> DealStatusRetryLegacy : ClientEventDealRejected
DealStatusWaitForAcceptanceLegacy --> DealStatusRejected : ClientEventDealRejected
DealStatusWaitForAcceptance --> DealStatusDealNotFound : ClientEventDealNotFound
DealStatusWaitForAcceptanceLegacy --> DealStatusDealNotFound : ClientEventDealNotFound
DealStatusWaitForAcceptanceLegacy --> DealStatusRejecting : ClientEventDealRejected
DealStatusWaitForAcceptance --> DealStatusDealNotFoundCleanup : ClientEventDealNotFound
DealStatusWaitForAcceptanceLegacy --> DealStatusDealNotFoundCleanup : ClientEventDealNotFound
DealStatusWaitForAcceptance --> DealStatusAccepted : ClientEventDealAccepted
DealStatusWaitForAcceptanceLegacy --> DealStatusAccepted : ClientEventDealAccepted
DealStatusPaymentChannelCreating --> DealStatusFailing : ClientEventPaymentChannelErrored
Expand Down Expand Up @@ -95,8 +103,8 @@ stateDiagram-v2
DealStatusOngoing --> DealStatusBlocksComplete : ClientEventAllBlocksReceived
DealStatusFundsNeededLastPayment --> DealStatusSendFundsLastPayment : ClientEventAllBlocksReceived
DealStatusBlocksComplete --> DealStatusBlocksComplete : ClientEventAllBlocksReceived
DealStatusCheckComplete --> DealStatusCompleted : ClientEventAllBlocksReceived
DealStatusWaitingForLastBlocks --> DealStatusCompleted : ClientEventAllBlocksReceived
DealStatusCheckComplete --> DealStatusFinalizingBlockstore : ClientEventAllBlocksReceived
DealStatusWaitingForLastBlocks --> DealStatusFinalizingBlockstore : ClientEventAllBlocksReceived
DealStatusFundsNeeded --> DealStatusFundsNeeded : ClientEventBlocksReceived
DealStatusSendFunds --> DealStatusOngoing : ClientEventBlocksReceived
DealStatusSendFundsLastPayment --> DealStatusOngoing : ClientEventBlocksReceived
Expand Down Expand Up @@ -129,10 +137,15 @@ stateDiagram-v2
DealStatusOngoing --> DealStatusCheckComplete : ClientEventComplete
DealStatusFundsNeededLastPayment --> DealStatusCheckComplete : ClientEventComplete
DealStatusBlocksComplete --> DealStatusCheckComplete : ClientEventComplete
DealStatusFinalizing --> DealStatusCompleted : ClientEventComplete
DealStatusCheckComplete --> DealStatusCompleted : ClientEventCompleteVerified
DealStatusCheckComplete --> DealStatusErrored : ClientEventEarlyTermination
DealStatusFinalizing --> DealStatusFinalizingBlockstore : ClientEventComplete
DealStatusCheckComplete --> DealStatusFinalizingBlockstore : ClientEventCompleteVerified
DealStatusCheckComplete --> DealStatusErroring : ClientEventEarlyTermination
DealStatusCheckComplete --> DealStatusWaitingForLastBlocks : ClientEventWaitForLastBlocks
DealStatusErroring --> DealStatusErrored : ClientEventBlockstoreFinalized
DealStatusRejecting --> DealStatusRejected : ClientEventBlockstoreFinalized
DealStatusDealNotFoundCleanup --> DealStatusDealNotFound : ClientEventBlockstoreFinalized
DealStatusFinalizingBlockstore --> DealStatusCompleted : ClientEventBlockstoreFinalized
DealStatusFinalizingBlockstore --> DealStatusErrored : ClientEventFinalizeBlockstoreErrored
DealStatusFailing --> DealStatusErrored : ClientEventCancelComplete
DealStatusCancelling --> DealStatusCancelled : ClientEventCancelComplete
DealStatusInsufficientFunds --> DealStatusCheckFunds : ClientEventRecheckFunds
Expand Down Expand Up @@ -166,3 +179,6 @@ stateDiagram-v2

note left of DealStatusPaymentChannelAddingInitialFunds : The following events only record in this state.<br><br>ClientEventLastPaymentRequested<br>ClientEventPaymentRequested<br>ClientEventAllBlocksReceived<br>ClientEventBlocksReceived


note left of DealStatusFinalizingBlockstore : The following events only record in this state.<br><br>ClientEventWaitForLastBlocks

Binary file modified docs/retrievalclient.mmd.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions docs/retrievalclient.mmd.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/storageprovider.mmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ stateDiagram-v2
4 --> 11 : ProviderEventMultistoreErrored
4 --> 11 : ProviderEventDealHandoffFailed
4 --> 29 : ProviderEventDealHandedOff
29 --> 11 : <invalid Value>
29 --> 5 : <invalid Value>
29 --> 11 : ProviderEventDealPrecommitFailed
29 --> 5 : ProviderEventDealPrecommitted
5 --> 11 : ProviderEventDealActivationFailed
5 --> 6 : ProviderEventDealActivated
29 --> 6 : ProviderEventDealActivated
Expand Down
Binary file modified docs/storageprovider.mmd.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions docs/storageprovider.mmd.svg
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
24 changes: 16 additions & 8 deletions filestore/filestore.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,12 @@ type fileStore struct {
}

// NewLocalFileStore creates a filestore mounted on a given local directory path
func NewLocalFileStore(basedirectory OsPath) (FileStore, error) {
base := filepath.Clean(string(basedirectory))
info, err := os.Stat(string(base))
func NewLocalFileStore(baseDir OsPath) (FileStore, error) {
base, err := checkIsDir(string(baseDir))
if err != nil {
return nil, fmt.Errorf("error getting %s info: %s", base, err.Error())
}
if !info.IsDir() {
return nil, fmt.Errorf("%s is not a directory", base)
return nil, err
}
return &fileStore{string(base)}, nil
return &fileStore{base}, nil
}

func (fs fileStore) filename(p Path) string {
Expand Down Expand Up @@ -73,3 +69,15 @@ func (fs fileStore) CreateTemp() (File, error) {
filename := filepath.Base(f.Name())
return &fd{File: f, basepath: fs.base, filename: filename}, nil
}

func checkIsDir(baseDir string) (string, error) {
base := filepath.Clean(string(baseDir))
info, err := os.Stat(base)
if err != nil {
return "", fmt.Errorf("error getting %s info: %s", base, err.Error())
}
if !info.IsDir() {
return "", fmt.Errorf("%s is not a directory", base)
}
return base, nil
}
4 changes: 3 additions & 1 deletion filestore/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package filestore

import "io"
import (
"io"
)

// Path represents an abstract path to a file
type Path string
Expand Down
37 changes: 21 additions & 16 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ module github.com/filecoin-project/go-fil-markets
go 1.13

require (
github.com/filecoin-project/go-address v0.0.3
github.com/filecoin-project/dagstore v0.4.2
github.com/filecoin-project/go-address v0.0.5
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7
github.com/filecoin-project/go-data-transfer v1.7.0
Expand All @@ -12,40 +13,44 @@ require (
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
github.com/filecoin-project/go-multistore v0.0.3
github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1
github.com/filecoin-project/go-state-types v0.0.0-20201102161440-c8033295a1fc
github.com/filecoin-project/go-state-types v0.1.1-0.20210506134452-99b279731c48
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe
github.com/filecoin-project/go-statestore v0.1.0
github.com/filecoin-project/go-statestore v0.1.1
github.com/filecoin-project/specs-actors v0.9.13
github.com/filecoin-project/specs-actors/v2 v2.3.2
github.com/filecoin-project/specs-actors/v2 v2.3.5-0.20210114162132-5b58b773f4fb
github.com/hannahhoward/cbor-gen-for v0.0.0-20200817222906-ea96cece81f1
github.com/hannahhoward/go-pubsub v0.0.0-20200423002714-8d62886cc36e
github.com/ipfs/go-block-format v0.0.2
github.com/ipfs/go-blockservice v0.1.4-0.20200624145336-a978cec6e834
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-block-format v0.0.3
github.com/ipfs/go-blockservice v0.1.5
github.com/ipfs/go-cid v0.0.8-0.20210716091050-de6c03deae1c
github.com/ipfs/go-cidutil v0.0.2
github.com/ipfs/go-datastore v0.4.5
github.com/ipfs/go-filestore v1.0.0
github.com/ipfs/go-graphsync v0.6.4
github.com/ipfs/go-ipfs-blockstore v1.0.3
github.com/ipfs/go-ipfs-blockstore v1.0.4
github.com/ipfs/go-ipfs-blocksutil v0.0.1
github.com/ipfs/go-ipfs-chunker v0.0.5
github.com/ipfs/go-ipfs-ds-help v1.0.0
github.com/ipfs/go-ipfs-exchange-offline v0.0.1
github.com/ipfs/go-ipfs-files v0.0.8
github.com/ipfs/go-ipld-format v0.2.0
github.com/ipfs/go-log/v2 v2.1.2-0.20200626104915-0016c0b4b3e4
github.com/ipfs/go-log/v2 v2.3.0
github.com/ipfs/go-merkledag v0.3.2
github.com/ipfs/go-unixfs v0.2.4
github.com/ipfs/go-unixfs v0.2.6
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d
github.com/ipld/go-car/v2 v2.0.2
github.com/ipld/go-ipld-prime v0.5.1-0.20201021195245-109253e8a018
github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c
github.com/jpillora/backoff v1.0.0
github.com/libp2p/go-libp2p v0.12.0
github.com/libp2p/go-libp2p-core v0.7.0
github.com/libp2p/go-libp2p v0.14.0
github.com/libp2p/go-libp2p-core v0.8.5
github.com/multiformats/go-multiaddr v0.3.1
github.com/multiformats/go-multibase v0.0.3
github.com/stretchr/testify v1.6.1
github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd
golang.org/x/net v0.0.0-20201021035429-f5854403a974
github.com/multiformats/go-multihash v0.0.15
github.com/stretchr/testify v1.7.0
github.com/whyrusleeping/cbor-gen v0.0.0-20210713220151-be142a5ae1a8
golang.org/x/exp v0.0.0-20210715201039-d37aa40e8013
golang.org/x/net v0.0.0-20210423184538-5f58ad60dda6
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
)

Expand Down
Loading