Skip to content

Commit

Permalink
migrate to DAG store + CARv2 blockstores for storage and retrieval (#576
Browse files Browse the repository at this point in the history 
)

Co-authored-by: Dirk McCormick <dirkmdev@gmail.com>
Co-authored-by: Raúl Kripalani <raul@protocol.ai>
Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com>
  • Loading branch information
@raulk @dirkmc @aarshkshah1992
3 people authored Aug 16, 2021
1 parent b7a9091 commit 9859795
Show file tree
Hide file tree
Showing 96 changed files with 4,151 additions and 1,655 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ orbs:
executors:
golang:
docker:
- image: circleci/golang:1.14-node
- image: circleci/golang:1.16.4
resource_class: large

commands:
Expand Down
2 changes: 2 additions & 0 deletions discovery/migrations/migrations_cbor_gen.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions discovery/types_cbor_gen.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 27 additions & 11 deletions docs/retrievalclient.mmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ stateDiagram-v2
state "DealStatusWaitForAcceptanceLegacy" as DealStatusWaitForAcceptanceLegacy
state "DealStatusWaitingForLastBlocks" as DealStatusWaitingForLastBlocks
state "DealStatusPaymentChannelAddingInitialFunds" as DealStatusPaymentChannelAddingInitialFunds
state "DealStatusErroring" as DealStatusErroring
state "DealStatusRejecting" as DealStatusRejecting
state "DealStatusDealNotFoundCleanup" as DealStatusDealNotFoundCleanup
state "DealStatusFinalizingBlockstore" as DealStatusFinalizingBlockstore
DealStatusNew : On entry runs ProposeDeal
DealStatusPaymentChannelCreating : On entry runs WaitPaymentChannelReady
DealStatusPaymentChannelAddingFunds : On entry runs WaitPaymentChannelReady
Expand All @@ -42,24 +46,28 @@ stateDiagram-v2
DealStatusCancelling : On entry runs CancelDeal
DealStatusRetryLegacy : On entry runs ProposeDeal
DealStatusPaymentChannelAddingInitialFunds : On entry runs WaitPaymentChannelReady
DealStatusErroring : On entry runs FailsafeFinalizeBlockstore
DealStatusRejecting : On entry runs FailsafeFinalizeBlockstore
DealStatusDealNotFoundCleanup : On entry runs FailsafeFinalizeBlockstore
DealStatusFinalizingBlockstore : On entry runs FinalizeBlockstore
[*] --> DealStatusNew
note right of DealStatusNew
The following events are not shown cause they can trigger from any state.
ClientEventWriteDealProposalErrored - transitions state to DealStatusErrored
ClientEventWriteDealProposalErrored - transitions state to DealStatusErroring
ClientEventUnknownResponseReceived - transitions state to DealStatusFailing
ClientEventDataTransferError - transitions state to DealStatusErrored
ClientEventWriteDealPaymentErrored - transitions state to DealStatusErrored
ClientEventDataTransferError - transitions state to DealStatusErroring
ClientEventWriteDealPaymentErrored - transitions state to DealStatusErroring
ClientEventProviderCancelled - transitions state to DealStatusCancelling
ClientEventCancel - transitions state to DealStatusCancelling
end note
DealStatusNew --> DealStatusNew : ClientEventOpen
DealStatusNew --> DealStatusWaitForAcceptance : ClientEventDealProposed
DealStatusRetryLegacy --> DealStatusWaitForAcceptanceLegacy : ClientEventDealProposed
DealStatusWaitForAcceptance --> DealStatusRetryLegacy : ClientEventDealRejected
DealStatusWaitForAcceptanceLegacy --> DealStatusRejected : ClientEventDealRejected
DealStatusWaitForAcceptance --> DealStatusDealNotFound : ClientEventDealNotFound
DealStatusWaitForAcceptanceLegacy --> DealStatusDealNotFound : ClientEventDealNotFound
DealStatusWaitForAcceptanceLegacy --> DealStatusRejecting : ClientEventDealRejected
DealStatusWaitForAcceptance --> DealStatusDealNotFoundCleanup : ClientEventDealNotFound
DealStatusWaitForAcceptanceLegacy --> DealStatusDealNotFoundCleanup : ClientEventDealNotFound
DealStatusWaitForAcceptance --> DealStatusAccepted : ClientEventDealAccepted
DealStatusWaitForAcceptanceLegacy --> DealStatusAccepted : ClientEventDealAccepted
DealStatusPaymentChannelCreating --> DealStatusFailing : ClientEventPaymentChannelErrored
Expand Down Expand Up @@ -95,8 +103,8 @@ stateDiagram-v2
DealStatusOngoing --> DealStatusBlocksComplete : ClientEventAllBlocksReceived
DealStatusFundsNeededLastPayment --> DealStatusSendFundsLastPayment : ClientEventAllBlocksReceived
DealStatusBlocksComplete --> DealStatusBlocksComplete : ClientEventAllBlocksReceived
DealStatusCheckComplete --> DealStatusCompleted : ClientEventAllBlocksReceived
DealStatusWaitingForLastBlocks --> DealStatusCompleted : ClientEventAllBlocksReceived
DealStatusCheckComplete --> DealStatusFinalizingBlockstore : ClientEventAllBlocksReceived
DealStatusWaitingForLastBlocks --> DealStatusFinalizingBlockstore : ClientEventAllBlocksReceived
DealStatusFundsNeeded --> DealStatusFundsNeeded : ClientEventBlocksReceived
DealStatusSendFunds --> DealStatusOngoing : ClientEventBlocksReceived
DealStatusSendFundsLastPayment --> DealStatusOngoing : ClientEventBlocksReceived
Expand Down Expand Up @@ -129,10 +137,15 @@ stateDiagram-v2
DealStatusOngoing --> DealStatusCheckComplete : ClientEventComplete
DealStatusFundsNeededLastPayment --> DealStatusCheckComplete : ClientEventComplete
DealStatusBlocksComplete --> DealStatusCheckComplete : ClientEventComplete
DealStatusFinalizing --> DealStatusCompleted : ClientEventComplete
DealStatusCheckComplete --> DealStatusCompleted : ClientEventCompleteVerified
DealStatusCheckComplete --> DealStatusErrored : ClientEventEarlyTermination
DealStatusFinalizing --> DealStatusFinalizingBlockstore : ClientEventComplete
DealStatusCheckComplete --> DealStatusFinalizingBlockstore : ClientEventCompleteVerified
DealStatusCheckComplete --> DealStatusErroring : ClientEventEarlyTermination
DealStatusCheckComplete --> DealStatusWaitingForLastBlocks : ClientEventWaitForLastBlocks
DealStatusErroring --> DealStatusErrored : ClientEventBlockstoreFinalized
DealStatusRejecting --> DealStatusRejected : ClientEventBlockstoreFinalized
DealStatusDealNotFoundCleanup --> DealStatusDealNotFound : ClientEventBlockstoreFinalized
DealStatusFinalizingBlockstore --> DealStatusCompleted : ClientEventBlockstoreFinalized
DealStatusFinalizingBlockstore --> DealStatusErrored : ClientEventFinalizeBlockstoreErrored
DealStatusFailing --> DealStatusErrored : ClientEventCancelComplete
DealStatusCancelling --> DealStatusCancelled : ClientEventCancelComplete
DealStatusInsufficientFunds --> DealStatusCheckFunds : ClientEventRecheckFunds
Expand Down Expand Up @@ -166,3 +179,6 @@ stateDiagram-v2

note left of DealStatusPaymentChannelAddingInitialFunds : The following events only record in this state.<br><br>ClientEventLastPaymentRequested<br>ClientEventPaymentRequested<br>ClientEventAllBlocksReceived<br>ClientEventBlocksReceived


note left of DealStatusFinalizingBlockstore : The following events only record in this state.<br><br>ClientEventWaitForLastBlocks

Binary file modified docs/retrievalclient.mmd.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions docs/retrievalclient.mmd.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions docs/storageprovider.mmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -77,8 +77,8 @@ stateDiagram-v2
4 --> 11 : ProviderEventMultistoreErrored
4 --> 11 : ProviderEventDealHandoffFailed
4 --> 29 : ProviderEventDealHandedOff
29 --> 11 : <invalid Value>
29 --> 5 : <invalid Value>
29 --> 11 : ProviderEventDealPrecommitFailed
29 --> 5 : ProviderEventDealPrecommitted
5 --> 11 : ProviderEventDealActivationFailed
5 --> 6 : ProviderEventDealActivated
29 --> 6 : ProviderEventDealActivated
Expand Down
Binary file modified docs/storageprovider.mmd.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions docs/storageprovider.mmd.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
24 changes: 16 additions & 8 deletions filestore/filestore.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,12 @@ type fileStore struct {
}

// NewLocalFileStore creates a filestore mounted on a given local directory path
func NewLocalFileStore(basedirectory OsPath) (FileStore, error) {
base := filepath.Clean(string(basedirectory))
info, err := os.Stat(string(base))
func NewLocalFileStore(baseDir OsPath) (FileStore, error) {
base, err := checkIsDir(string(baseDir))
if err != nil {
return nil, fmt.Errorf("error getting %s info: %s", base, err.Error())
}
if !info.IsDir() {
return nil, fmt.Errorf("%s is not a directory", base)
return nil, err
}
return &fileStore{string(base)}, nil
return &fileStore{base}, nil
}

func (fs fileStore) filename(p Path) string {
Expand Down Expand Up @@ -73,3 +69,15 @@ func (fs fileStore) CreateTemp() (File, error) {
filename := filepath.Base(f.Name())
return &fd{File: f, basepath: fs.base, filename: filename}, nil
}

func checkIsDir(baseDir string) (string, error) {
base := filepath.Clean(string(baseDir))
info, err := os.Stat(base)
if err != nil {
return "", fmt.Errorf("error getting %s info: %s", base, err.Error())
}
if !info.IsDir() {
return "", fmt.Errorf("%s is not a directory", base)
}
return base, nil
}
4 changes: 3 additions & 1 deletion filestore/types.go
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
package filestore

import "io"
import (
"io"
)

// Path represents an abstract path to a file
type Path string
Expand Down
37 changes: 21 additions & 16 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ module github.com/filecoin-project/go-fil-markets
go 1.13

require (
github.com/filecoin-project/go-address v0.0.3
github.com/filecoin-project/dagstore v0.4.2
github.com/filecoin-project/go-address v0.0.5
github.com/filecoin-project/go-cbor-util v0.0.0-20191219014500-08c40a1e63a2
github.com/filecoin-project/go-commp-utils v0.1.1-0.20210427191551-70bf140d31c7
github.com/filecoin-project/go-data-transfer v1.7.0
Expand All @@ -12,40 +13,44 @@ require (
github.com/filecoin-project/go-fil-commp-hashhash v0.1.0
github.com/filecoin-project/go-multistore v0.0.3
github.com/filecoin-project/go-padreader v0.0.0-20210723183308-812a16dc01b1
github.com/filecoin-project/go-state-types v0.0.0-20201102161440-c8033295a1fc
github.com/filecoin-project/go-state-types v0.1.1-0.20210506134452-99b279731c48
github.com/filecoin-project/go-statemachine v0.0.0-20200925024713-05bd7c71fbfe
github.com/filecoin-project/go-statestore v0.1.0
github.com/filecoin-project/go-statestore v0.1.1
github.com/filecoin-project/specs-actors v0.9.13
github.com/filecoin-project/specs-actors/v2 v2.3.2
github.com/filecoin-project/specs-actors/v2 v2.3.5-0.20210114162132-5b58b773f4fb
github.com/hannahhoward/cbor-gen-for v0.0.0-20200817222906-ea96cece81f1
github.com/hannahhoward/go-pubsub v0.0.0-20200423002714-8d62886cc36e
github.com/ipfs/go-block-format v0.0.2
github.com/ipfs/go-blockservice v0.1.4-0.20200624145336-a978cec6e834
github.com/ipfs/go-cid v0.0.7
github.com/ipfs/go-block-format v0.0.3
github.com/ipfs/go-blockservice v0.1.5
github.com/ipfs/go-cid v0.0.8-0.20210716091050-de6c03deae1c
github.com/ipfs/go-cidutil v0.0.2
github.com/ipfs/go-datastore v0.4.5
github.com/ipfs/go-filestore v1.0.0
github.com/ipfs/go-graphsync v0.6.4
github.com/ipfs/go-ipfs-blockstore v1.0.3
github.com/ipfs/go-ipfs-blockstore v1.0.4
github.com/ipfs/go-ipfs-blocksutil v0.0.1
github.com/ipfs/go-ipfs-chunker v0.0.5
github.com/ipfs/go-ipfs-ds-help v1.0.0
github.com/ipfs/go-ipfs-exchange-offline v0.0.1
github.com/ipfs/go-ipfs-files v0.0.8
github.com/ipfs/go-ipld-format v0.2.0
github.com/ipfs/go-log/v2 v2.1.2-0.20200626104915-0016c0b4b3e4
github.com/ipfs/go-log/v2 v2.3.0
github.com/ipfs/go-merkledag v0.3.2
github.com/ipfs/go-unixfs v0.2.4
github.com/ipfs/go-unixfs v0.2.6
github.com/ipld/go-car v0.1.1-0.20201119040415-11b6074b6d4d
github.com/ipld/go-car/v2 v2.0.2
github.com/ipld/go-ipld-prime v0.5.1-0.20201021195245-109253e8a018
github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c
github.com/jpillora/backoff v1.0.0
github.com/libp2p/go-libp2p v0.12.0
github.com/libp2p/go-libp2p-core v0.7.0
github.com/libp2p/go-libp2p v0.14.0
github.com/libp2p/go-libp2p-core v0.8.5
github.com/multiformats/go-multiaddr v0.3.1
github.com/multiformats/go-multibase v0.0.3
github.com/stretchr/testify v1.6.1
github.com/whyrusleeping/cbor-gen v0.0.0-20210219115102-f37d292932f2
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd
golang.org/x/net v0.0.0-20201021035429-f5854403a974
github.com/multiformats/go-multihash v0.0.15
github.com/stretchr/testify v1.7.0
github.com/whyrusleeping/cbor-gen v0.0.0-20210713220151-be142a5ae1a8
golang.org/x/exp v0.0.0-20210715201039-d37aa40e8013
golang.org/x/net v0.0.0-20210423184538-5f58ad60dda6
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
)

Expand Down
Loading

0 comments on commit 9859795

Please sign in to comment.