feat: implement p2p protocol and rpc service #221

joeylichang · 2023-03-18T13:53:17Z

Description

Implement p2p service.

Rationale

P2P node implements 2 protocols:

Ping protocol

Customized ping service, it implemented dynamic update of permanent nodes. As usual, permanent nodes should cover as many SP as possible, which is more decentralized, and also meets SP requirements, eg: getting an approval request needs at least 6 or more replies from different storage providers but p2p nodes are offline, and replacement, which is an inevitable problem, If permanent nodes belonging to the same SP all fail and are replaced, then the SP will be unable to communicate, this requires dynamic updates permanent nodes.
PeerProvider implements the pruning function of permanent nodes. Ping service only adds a permanent node, for zombie nodes, PeerProvider's pruning strategy takes into account the information of the storage provider dimension, and uses a very conservative pruning strategy. Nodes are only pruned if there are enough backups and multiple failed interactions, can try to keep each storage provider with enough nodes to try to connect so that each SP has an equal opportunity to receive requests

Approval protocol

The primary SP needs to send a GetApproval request to ask whether the other SP is willing to serve the request. the request is sent by p2p protocol.
The GetApproval will be signed by the remote SP operator account, and when syncing the payload data to remote SP will be with the Approval and the remote SP will verify the Approval.

Example

N/A

Changes

Notable changes:

pkg/p2p
service/p2p

README.md

sysvm · 2023-03-21T05:41:31Z

cmd/storage_provider/init.go

@@ -3,6 +3,7 @@ package main
 import (
 	"fmt"

+	"github.com/bnb-chain/greenfield-storage-provider/service/p2p"


fix the location of imported pkg

sysvm · 2023-03-21T05:42:12Z

config/config.go

@@ -64,6 +67,7 @@ var DefaultStorageProviderConfig = &StorageProviderConfig{
 		model.TaskNodeService:   model.TaskNodeGRPCAddress,
 		model.SignerService:     model.SignerGRPCAddress,
 		model.MetadataService:   model.MetadataGRPCAddress,
+		model.P2pService:        model.P2PGRPCAddress,


P2pService -> P2PService

sysvm · 2023-03-21T05:42:19Z

config/config.go

@@ -74,13 +78,15 @@ var DefaultStorageProviderConfig = &StorageProviderConfig{
 		model.TaskNodeService:   model.TaskNodeGRPCAddress,
 		model.SignerService:     model.SignerGRPCAddress,
 		model.MetadataService:   model.MetadataGRPCAddress,
+		model.P2pService:        model.P2PGRPCAddress,


sysvm · 2023-03-21T05:43:01Z

config/subconfig.go

@@ -173,6 +174,11 @@ func (cfg *StorageProviderConfig) MakeTaskNodeConfig() (*tasknode.TaskNodeConfig
 	} else {
 		return nil, fmt.Errorf("missing signer gRPC address configuration for task node service")
 	}
+	if _, ok := cfg.Endpoint[model.P2pService]; ok {
+		snCfg.P2PGrpcAddress = cfg.Endpoint[model.P2pService]


fix snCfg.P2PGrpcAddress to snCfg.P2PGRPCAddress

sysvm · 2023-03-21T05:44:38Z

model/const.go

+	// P2PGRPCAddress default gRPC address of p2p service
+	P2PGRPCAddress = "localhost:9833"
+	// P2PGListenAddress default p2p protocol listen address of p2p node
+	P2PGListenAddress = "127.0.0.1:9933"


error spelling of P2PGListenAddress which should be P2PListenAddress

sysvm · 2023-03-21T05:45:21Z

model/errors/rpc_error.go

+	// ErrSPMismatch defines the SP's operate address mismatch error
+	ErrSPMismatch = errors.New("SP's operate address mismatch")
+	// ErrApprovalExpire defines the SP's operate address mismatch error
+	ErrApprovalExpire = errors.New("approval expire")


fix ErrApprovalExpire to ErrApprovalExpired
fix errors.New("approval expire") to errors.New("approval expired")

sysvm · 2023-03-21T05:47:22Z

model/errors/rpc_error.go

@@ -62,6 +62,12 @@ var (
 	ErrCheckPaymentAccountActive = errors.New("payment account is not active")
 	// ErrCheckQuotaEnough defines check quota is enough
 	ErrCheckQuotaEnough = errors.New("quota is not enough")
+	// ErrSPMismatch defines the SP's operate address mismatch error
+	ErrSPMismatch = errors.New("SP's operate address mismatch")


fix ErrSPMismatch to ErrMismatchedSP
fix errors.New("SP's operate address mismatch") to errors.New("the operator address of SP is a mismatch");
comment also should be fixed

sysvm · 2023-03-21T05:49:35Z

model/errors/rpc_error.go

+// task node service error
+var (
+	// ErrSPApprovalNumber defines failed to insufficient SPs' approvals from p2p server
+	ErrSPApprovalNumber = errors.New("failed to get sufficient SPs' approvals from p2p server")


fix errors.New("failed to get sufficient SPs' approvals from p2p server") to errors.New("failed to get sufficient approvals of SPs from p2p server")

sysvm · 2023-03-21T05:50:16Z

pkg/p2p/approval.go

+
+// pattern: /protocol-name/request-or-response-message/version
+const GetApprovalRequest = "/approval/request/0.0.1"
+const GetApprovalResponse = "/approval/response/0.0.1"


move these constants to const.go?

The p2p protocol independent, and has no interaction with other parts. It is recommended to separate it first.

sysvm · 2023-03-21T06:08:05Z

pkg/p2p/ping.go

+	defer func() {
+		if err != nil {
+			n.peers.DeletePeer(peerID)
+			log.Warnw("failed to receive pong", "peer_id", peerID, "error", err)


use log.Errorw

sysvm · 2023-03-21T06:08:16Z

pkg/p2p/ping.go

+
+	err = types.VerifySignature(pong.GetSpOperatorAddress(), pong.GetSignBytes(), pong.GetSignature())
+	if err != nil {
+		log.Warnw("failed to verify pong msg signature", "local", s.Conn().LocalPeer(), "remote", s.Conn().RemotePeer(), "error", err)


the same ， the ping msg fail is normal

sysvm · 2023-03-21T06:08:58Z

proto/pkg/p2p/types/p2p.proto

+  // sp_operator_address define sp operator public key
+  string sp_operator_address = 2;
+  // time_out defines the approval valid deadline
+  int64 time_out = 3;


fix time_out to timeout

sysvm · 2023-03-21T06:09:11Z

proto/pkg/p2p/types/p2p.proto

+  // time_out defines the approval valid deadline
+  int64 time_out = 3;
+  // refuse_reason defines the reason of refusing
+  string refuse_reason = 4;


fix refuse_reason to refused_reason

sysvm · 2023-03-21T06:09:25Z

proto/service/p2p/types/p2p.proto

+  // expect_accept defines the number of expecting accept approval
+  int64 expect_accept = 3;
+  // time_out defines approval request time out
+  int64 time_out = 4;


fix time_out to timeout

sysvm · 2023-03-21T06:09:36Z

proto/service/p2p/types/p2p.proto

+  // approval defines the object info for approval
+  pkg.p2p.types.GetApprovalRequest approval = 1;
+  // expect_accept defines the number of expecting accept approval
+  int64 expect_accept = 3;


fix expect_accept to expected_accept

sysvm · 2023-03-21T06:10:29Z

service/gateway/request_util.go

+	if strings.Compare(g.config.SpOperatorAddress, approval.GetSpOperatorAddress()) != 0 {
+		log.Errorw("failed to verify replicate approval's SP operate address",
+			"approval_operate_address", approval.GetSpOperatorAddress(),
+			"self_operate_address", g.config.SpOperatorAddress)


fix self_operate_address to own_operate_address

will-2012 · 2023-03-21T06:03:42Z

README.md

+[P2PConfig]
+ListenAddress = "127.0.0.1:9933"
+# p2p node msg Secp256k1 encryption key, it is different from other SP's addresses
+PrivKey = ""


Aligned with the above, no abbreviation. use PrivateKey?

will-2012 · 2023-03-21T06:07:24Z

deployment/localup/localup.sh

@@ -189,6 +191,7 @@ main() {
    reset_sp $2
    ;;
  --start)
+    stop_sp


Add a restart case to ensure to be clearer.

I think this is okay for the time being, because minimizing the commands of local setup script also reduces the complexity of use, and in the scenario of local setup script, it is not possible to run more than 7 SPs at the same time.

will-2012 · 2023-03-21T13:54:47Z

deployment/localup/env.info

+Node4="0b5a6480fe664671fd01979fc4278fa393f0abc08c95980f4a900056f035ca7e"
+Node5="896caab68f3f3fdbf053ba64a29ce0292c8a5536a2790713b67dd1851f4d77fc"
+Node6="ec09def8ee42ac9c23394080ce5bfcdcb1dc57255048a8afc74e9deaae4399cd"


Uniformly use uppercase for shell environment variables.

will-2012 · 2023-03-21T14:01:39Z

model/const.go

@@ -24,6 +24,8 @@ var (
 	BlockSyncerService = strings.ToLower("BlockSyncer")
 	// ManagerService defines the name of manager service
 	ManagerService = strings.ToLower("Manager")
+	// P2PService defines the name of p2p service
+	P2PService = strings.ToLower("p2p")


p2p as a module name is a bit general, what aboutP2PNode?

Yes, you are right.
But it is more user-friendly. In the './gnfd-sp -server p2p' scenario, and other services also does not have 'service' or 'node' suffixes, p2p may be more suitable.

will-2012 · 2023-03-21T14:02:12Z

model/const.go

@@ -37,6 +39,7 @@ var SpServiceDesc = map[string]string{
 	SignerService:      "Sign the transaction and broadcast to chain",
 	MetadataService:    "Provides the ability to query meta data",
 	BlockSyncerService: "Syncs block data to db",
+	P2PService:         "Service for p2p protocol interaction between SPs",


Consistent with above, the verb begins with.

will-2012 · 2023-03-21T14:27:38Z

proto/service/p2p/types/p2p.proto

+  // approval defines the object info for approval
+  pkg.p2p.types.GetApprovalRequest approval = 1;
+  // expected_accept defines the number of expecting accept approval
+  int64 expected_accept = 3;


will-2012 · 2023-03-21T14:28:38Z

proto/service/signer/types/signer.proto

+
+// SignGetApprovalReqResponse is response type for the SignReplicateApprovalReqMsg RPC method
+message SignReplicateApprovalReqMsgResponse {
+  pkg.p2p.types.GetApprovalRequest approval = 1;


GetApprovalResponse?

singer RPC have Object/Bucket approval interface. so the approval should have prefix. ReplicateApproval is distinguish from ObjectApproval.

Is the request here a typo? Replace GetApprovalRequest with GetApprovalResponse?

will-2012 · 2023-03-21T14:29:58Z

service/gateway/request_util.go

+			"own_operate_address", g.config.SpOperatorAddress)
+		return errors.ErrSPMismatch
+	}
+	if time.Now().Unix() > approval.Timeout {


Timeout means expired_time.

will-2012 · 2023-03-21T14:33:19Z

service/p2p/p2p_config.go

+
+type P2PConfig struct {
+	SpOperatorAddress string
+	GrpcAddress       string


Use GRPCAddress, align with other services.

will-2012 · 2023-03-21T14:34:32Z

service/p2p/p2p_config.go

+type P2PConfig struct {
+	SpOperatorAddress string
+	GrpcAddress       string
+	SignerGRPCAddress string


SignerGrpcAddress, align with other services config.

will-2012 · 2023-03-22T02:06:57Z

cmd/p2p/p2p.go

+	},
+	Category: "P2P COMMANDS",
+	Description: `
+The p2p.create.key creates 'n'' sets of Secp256k1 key pairs, each key pair contains a private key 


'n'' It seems that there is an extra single quotation mark？

will-2012 · 2023-03-22T02:09:18Z

cmd/utils/flags.go

-		Name: "configremote",
-		Usage: "Flag load config from remote db,if 'configremote' be set, the db.user, " +
+		Name: "config.remote",
+		Usage: "Flag load config from remote db,if 'config.remote' be set, the db.user, " +
 			"db.password and db.address flags are needed, otherwise use default value",


otherwise use the default value

will-2012 · 2023-03-22T02:11:55Z

config/config_template.toml

+Dsn = "localhost:3308"
+
+[P2PConfig]
+ListenAddress = "127.0.0.1:9933"


what is the difference between this ListenAddress and ListAddress.p2p?

P2PConfig.ListenAddress p2p protocol , another is rpc address

will-2012 · 2023-03-22T02:18:15Z

model/const.go

@@ -37,6 +39,7 @@ var SpServiceDesc = map[string]string{
 	SignerService:      "Sign the transaction and broadcast to chain",
 	MetadataService:    "Provides the ability to query meta data",
 	BlockSyncerService: "Syncs block data to db",
+	P2PService:         "Communicate with SPs on p2p protocol",


polish Communicate to Communicates, and polish the above SignerService description Signs and Broadcasts if you get time.

will-2012 · 2023-03-22T02:35:31Z

proto/service/signer/types/signer.proto

+
+// SignGetApprovalReqResponse is response type for the SignReplicateApprovalReqMsg RPC method
+message SignReplicateApprovalReqMsgResponse {
+  pkg.p2p.types.GetApprovalRequest approval = 1;


Is the request here a typo? Replace GetApprovalRequest with GetApprovalResponse?

will-2012 · 2023-03-22T02:38:27Z

service/p2p/p2p_service.go

+		Accept: accept,
+		Refuse: refuse,
+	}
+	log.CtxInfow(ctx, "success to get approval", "object_id", objectInfo.Id.Uint64(), "accept", len(accept), "refuse", len(refuse))


Is the request here a typo? Replace GetApprovalRequest with GetApprovalResponse?
no , it is the GetApprovalRequest

fynnss · 2023-03-22T03:54:45Z

pkg/p2p/approval.go

+)
+
+// pattern: /protocol-name/request-or-response-message/version
+const GetApprovalRequest = "/approval/request/0.0.1"


Is the 0.0.1 a protocol version? if yes, I suggest to use v0.0.1

fynnss · 2023-03-22T04:00:16Z

pkg/p2p/approval.go

+// maintains requests for getting approvals in memory
+type ApprovalProtocol struct {
+	node     *Node
+	response map[uint64]chan *types.GetApprovalResponse


If the key in the map is created first, is there no need to use a lock?

33 lines define the mutex

fynnss · 2023-03-22T04:02:44Z

pkg/p2p/approval.go

+func (a *ApprovalProtocol) cancelApprovalRequest(id uint64) {
+	a.mux.Lock()
+	defer a.mux.Unlock()
+	if _, ok := a.response[id]; !ok {


Does !ok mean that it has been canceled?

fynnss · 2023-03-22T04:04:06Z

pkg/p2p/approval.go

+
+// hangApprovalRequest records the approval request in memory for response to router
+// notice: the caller need to call cancelApprovalRequest to delete the record
+func (a *ApprovalProtocol) hangApprovalRequest(id uint64) (chan *types.GetApprovalResponse, error) {


What's the id means? OperatorAddress of sp?

sysvm · 2023-03-22T06:48:48Z

README.md

-Dsn = "localhost:3306"
+Dsn = "localhost:3308"
+# p2p node configuration
+[P2PConfig]


fix P2PConfig to P2PCfg to keep the same style with other services

sysvm · 2023-03-22T06:49:09Z

cmd/p2p/p2p.go

+	Flags: []cli.Flag{
+		number,
+	},
+	Category: "P2P COMMANDS",


define as a constant

next PR, the metrics PR has defined the constant, it should be at the same file

sysvm

LGTM

* fix: fix verify permission bug (#225) * feat: merge release v0.0.4 fix to develop (#230) * fix: fix route bug * build: change signer wiltlist ip * fix: fix https sync failed * fix: router getUserBucketsRouterName path (#229) * fix: router getUserBucketsRouterName path and update router comments Co-authored-by: BarryTong65 <122767193+BarryTong65@users.noreply.github.com> Co-authored-by: will-2012 <will.w@nodereal.io> * ci: add gosec action (#231) * feat: implement p2p protocol and rpc service (#221) * feat: implement p2p protocol node and RPC service * feat: p2p RPC server * fix: config port conflict * feat: get secondary SP approval from p2p server * feat: verify approval in gateway * fix: golang ci lint error * fix: integration testing bugs * feat: load p2p private key from ENV var * feat: implement p2p protocol msg sign and verify * feat: p2p protocol msg sign in signer service * feat: signer client method for sign p2p protocol msg * feat: signature verify in related modules * fix: manager grace stop * fix: object Id proto clone to deep copy panic * feat: p2p create key pairs command * feat: p2p config to local up script * fix: local up script deploy p2p error * fix: pr comments * fix: change p2p config name for following the project specification * chore: refine error code * feat: sp services add metrics (#211) * feat: sp service adds metrics * fix: add metrics config * refactor: refactor metrics service * fix: update go.mod * fix: fix config toml template * chore: coding adjustments * fix: golang ci lint error --------- Co-authored-by: DylanYong <dylan.y@nodereal.io> Co-authored-by: joeylichang <joeycli0919@gmail.com> * feat: implement metadata payment apis (#235) * feat: implement metadata payment apis fix: update commnets and db schema fix: update OutFlows type json fix: update metadata * fix: add the errDesciption and fix requestID * fix: fix Id & CheckSums * feat: update bucket id (#241) * feat: metadata schema update fix: update config * fix: update SettleTimestamp and format imports * fix: fix owner and end_block_number * fix: adjust check condition to avoid panic (#243) Co-authored-by: DylanYong <dylan.y@nodereal.io> * feat: update the juno version (#244) * feat: resource manager (#246) * feat: resouce manager * feat: add resource manager flags * feat: add resource manager to dowmloader service * feat: add resource manager to challenge service * feat: add resource manager to task node service * chore: rich log info for rcmgr * feat: init global resource manager * fix: integration testing bugs * feat: return preparations error to client during put object * chore: fix cr comments * docs: release changlog for v0.0.5 (#249) * docs: release changlog for v0.0.5 * build: update the greenfield chain version (#248) * build: update the greenfield chain version * feat:update juno version * fix: update metadata for chain v0.0.10 fix: update metadata for chain v0.0.10 fix: update comments for bsdb schema * fix: update from OneRowId to OneRowID --------- Co-authored-by: constwz <changbohao30@gmail.com> Co-authored-by: BarryTong65 <barrytong.work@gmail.com> --------- Co-authored-by: joeycli <joeycli0919@gmail.com> Co-authored-by: BarryTong65 <122767193+BarryTong65@users.noreply.github.com> Co-authored-by: VM <112189277+sysvm@users.noreply.github.com> Co-authored-by: DylanYong <dylan.y@nodereal.io> Co-authored-by: krish-nr <krish.z@nodereal.io> Co-authored-by: krish-z <122767080+krish-nr@users.noreply.github.com> Co-authored-by: constwz <122766871+constwz@users.noreply.github.com> Co-authored-by: dylanhuang <j75689@gmail.com> Co-authored-by: constwz <changbohao30@gmail.com> Co-authored-by: BarryTong65 <barrytong.work@gmail.com>

joeylichang added 6 commits March 17, 2023 23:33

feat: imple p2p protocol node and rpc service

842a805

feat: p2p rpc server

4a2f09b

fix: config port conflict

94d70f0

feat: get secondary sp approval from p2p server

c086c14

feat: verify approval in gateway

d264515

fix: golang ci lint error

813a9dd

joeylichang requested review from will-2012 and sysvm and removed request for will-2012 March 18, 2023 14:02

joeylichang added the wip Working in process label Mar 18, 2023

joeylichang added 6 commits March 19, 2023 00:30

fix: integration testing bugs

a544c3d

feat: load p2p private key from ENV var

d90d15a

feat: implement p2p protocaol msg sign and verify

dcb2d24

feat: p2p protocol msg sign in signer service

eddb874

feat: signer client method for sign p2p protocol msg

52ec0d1

feat: signature verify in related modules

75a34ce

joeylichang requested review from will-2012 and RenRick March 20, 2023 04:59

joeylichang added 2 commits March 20, 2023 20:59

fix: manager grace stop

e9b8b86

fix: object Id proto clone to deep copy panic

ccb6c98

sysvm reviewed Mar 21, 2023

View reviewed changes

README.md Show resolved Hide resolved

sysvm reviewed Mar 21, 2023

View reviewed changes

joeylichang added 4 commits March 21, 2023 14:59

feat: p2p create key pairs command

14b1a58

feat: p2p config to localup script

8ec7e7e

fix: local up script deploy p2p error

765e221

fix: pr comments

edcdf4f

joeylichang added r4r Ready for review and removed wip Working in process labels Mar 21, 2023

joeylichang requested a review from fynnss March 21, 2023 11:34

will-2012 reviewed Mar 21, 2023

View reviewed changes

fix: pr comments

7dd1722

joeylichang requested review from will-2012 and sysvm March 22, 2023 00:51

will-2012 reviewed Mar 22, 2023

View reviewed changes

fynnss reviewed Mar 22, 2023

View reviewed changes

joeylichang added 2 commits March 22, 2023 14:18

Merge branch 'develop' into feat-p2p

43d9a69

fix: pr comments

5518c61

will-2012 approved these changes Mar 22, 2023

View reviewed changes

sysvm reviewed Mar 22, 2023

View reviewed changes

sysvm approved these changes Mar 22, 2023

View reviewed changes

fix: change p2p config name for folllowing the project specification

399b97e

joeylichang merged commit 28bed3e into develop Mar 22, 2023

joeylichang deleted the feat-p2p branch March 22, 2023 07:10

feat: implement p2p protocol and rpc service #221

feat: implement p2p protocol and rpc service #221

Conversation

joeylichang commented Mar 18, 2023 • edited Loading

Description

Rationale

Example

Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sysvm Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sysvm Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sysvm Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joeylichang commented Mar 18, 2023 •

edited

Loading

sysvm Mar 21, 2023 •

edited

Loading

sysvm Mar 21, 2023 •

edited

Loading

sysvm Mar 21, 2023 •

edited

Loading