feat: stone node sends piece data to gateway #128

sysvm · 2023-02-19T08:44:21Z

Description

In previous code, stone node straightly send piece data to syncer just for test. But this is not standard format.

I fix it that stone node send piece data to gateway through HTTP and gateway send data to syncer through RPC.
stone node get downstream gateway through P2P.

Rationale

Storage provider works in production environment.

Example

add an example CLI or API response...

Changes

Notable changes:

add each change in a bullet point here
...

joeylichang · 2023-02-21T01:36:09Z

config/config.toml

    StoneHubServiceAddress = "127.0.0.1:9333"
-    SyncerServiceAddress = ["127.0.0.1:9543", "127.0.0.1:9553", "127.0.0.1:9563", "127.0.0.1:9573", "127.0.0.1:9583", "127.0.0.1:9593"]
+    SyncerAddress = ["127.0.0.1:9543", "127.0.0.1:9553", "127.0.0.1:9563", "127.0.0.1:9573", "127.0.0.1:9583", "127.0.0.1:9593"]


syncer address should be replaced by gateway address?

joeylichang · 2023-02-21T01:41:02Z

service/stonenode/alloc_stone_job.go

-		node.reportErrToStoneHub(ctx, allocResp, merrors.ErrSyncerNumber)
-		return merrors.ErrSyncerNumber
+	secondaryGatewayList := node.cfg.GatewayAddress
+	if len(secondaryPieceData) > len(secondarySPs) || len(secondaryGatewayList) != len(secondarySPs) {


why len(secondaryGatewayList) != len(secondarySPs) ?
it ie ok, len(secondaryGatewayList) >= len(secondarySPs).

Not required for checking mock logic

joeylichang · 2023-02-21T01:42:17Z

service/stonenode/http_client.go

+		"spID", syncerInfo.GetStorageProviderId(), "index", syncerInfo.GetPieceIndex(), "count", syncerInfo.GetPieceCount(),
+		"redundancyType", syncerInfo.GetRedundancyType().String())
+	req = addReqHeader(req, syncerInfo, traceID)
+	log.Infow("request header", "header", req.Header)


joeylichang · 2023-02-21T01:43:22Z

service/stonenode/http_client.go

+		log.Errorw("copy request body failed", "error", err)
+		return err
+	}
+	log.Error("error response body")


delete the log?

joeylichang · 2023-02-21T01:47:16Z

proto/service/types/v1/stone_hub.proto

-  PieceJob piece_job = 3;
-  ErrMessage err_message = 4;
+  string bucket_name = 3;
+  string object_name = 4;


add the field of pb between stone hub and stone node.
shoud add fill the field in stone hub?

will-2012 · 2023-02-21T08:52:25Z

service/gateway/sync_piece_handler.go

+	w.Header().Set(model.GnfdSealSignatureHeader, sig)
+}
+
+func handlePieceChecksum(pieceChecksum [][]byte) string {


Functions with similar functions in util/util.go maybe be reused

will-2012 · 2023-02-21T08:52:33Z

service/gateway/sync_piece_handler.go

+	return strings.Join(list, ",")
+}
+
+func transferRedundancyType(redundancyType string) (ptypes.RedundancyType, error) {


Functions with similar functions in util/util.go maybe be reused

we use different transfer type, so this similar function in util/util.go doesn't satisfy mine.

will-2012 · 2023-02-21T08:54:04Z

service/gateway/syncer_processor.go

+)
+
+// syncPiece send rpc request to secondary storage provider to sync the piece data
+func (g *Gateway) syncPiece(ctx context.Context, syncerInfo *stypes.SyncerInfo, pieceData [][]byte, traceID string) (


At present, there is no abstract layer of processor. Consider unifying it in the future

fix it later

will-2012 · 2023-02-21T09:06:54Z

model/const.go

-	RangeHeader         = "Range"
-	ContentRangeHeader  = "Content-Range"
+	// StoneNode to gateway request header
+	GnfdTraceIDHeader           = "X-Gnfd-Trace-ID"


reuse GnfdRequestIDHeader？

will-2012

LGTM

joeylichang · 2023-02-21T10:04:34Z

service/gateway/sync_piece_handler.go

@@ -62,6 +62,7 @@ func (g *Gateway) syncPieceHandler(w http.ResponseWriter, r *http.Request) {
 		errDescription = InternalError
 	}
 	addRespHeader(resp, w)
+	log.Info("sync piece handler reply response to stone node")


suggestion: add detail log at input and output site, it will comfortable to debug

* fix: docker push bug (#115) * feat: add stone load and gc workflow (#117) * feat: add iterator and batch interface to db * chore: delete memory job db v1 code * feat: impl the iterator and batch of job memory db * feat: add stone load and gc db * chore: delete job sql db v1 impl * feat: job sql add iterator and batcher interface * refactor: rename job_db v2 interface to job_db interface --------- Co-authored-by: joeylichang <joeycli0919@gmail.com> * feat: implement signer service (#119) * feat: implement signer service * feat: add api auth and ip whitelist for signer * refactor: redesign the interface of singer service * refactor: use greenfield-go-sdk * fix: msg to be signed should not be pre-hashed * feat: add VerifyBucketApproval, VerifyObjectApproval interface * feat: add signer client --------- Co-authored-by: joeylichang <joeycli0919@gmail.com> * feat: client opt (#129) * feat: implement gateway challenge workflow (#127) * feat: implement gateway challenge workflow * feat: support get range object * read:block_syncer * read:block_syncer * read:block-syncer * read:block-syncer * fix: use env var to get bucket url (#130) * fix: use env var to get bucket url * fix: improve loading config from env function * fix: rename variable names * fix: change piece store readme.md --------- Co-authored-by: DylanYong <dylan.y@nodereal.io> * read:toml * feat: stone node sends piece data to gateway (#128) * feat: stone node sends piece data to gateway * feat: alloc resp add bucket name and object name * fix: fix errors of stone node uint test * fix: fix gateway config * fix: add gateway syncer rpc ut * fix: add one log in gateway sync piece handler func * fix: add detailed log in gateway sync piece handler --------- Co-authored-by: DylanYong <dylan.y@nodereal.io> * read:cmd_complete * read:del * read:test-job * read:block-syncer-frame * read:fix * fix: upgrade greenfield version to fix the signing bug (#133) * ci: fix release job * dep: upgrade greenfield version * fix: unit test for gnfd sign client * fix: signer adapt to on-chain logic (#136) * feat: add metadata service to sp (#132) * feat: init commit * feat: register metadata service and run on 9733 port * feat: impl ListObjectsByBucketName api & model update * feat: cleanup files and add metadata service sdk * feat: add mock and unit tests * feat: uniform error return values * feat: metadata service case_driver update --------- Co-authored-by: BarryTong65 <barrytong.work@gmail.com> Co-authored-by: joeycli <joeycli0919@gmail.com> * read:del * read:block-syncer-update * read:block-syncer-db * fix: use greenfield-go-sdk and upgrade lib version (#135) * fix: use greenfield-go-sdk and upgrade lib version * ci: rename main to master * fix: update greenfield-common version --------- Co-authored-by: DylanYong <dylan.y@nodereal.io> * feat: add chain client to sp (#131) * feat: impl the interaction between sp and chain interface * fix: return bucket id from auth object * feat: add check authorization * feat: add chain client to stonehub * feat: add chain client to uploader/downloader * feat: implement get approval workflow * feat: implement object seal and integrity hash sign workflow * chore: update dependencies * feat:implement secondary sp integrity hash sign workflow * test: add signer and chain to onebox * fix: fix itergrated test bug --------- Co-authored-by: joeylichang <joeycli0919@gmail.com> --------- Co-authored-by: joeycli <joeycli0919@gmail.com> Co-authored-by: dylanhuang <j75689@gmail.com> Co-authored-by: constwz <changbohao30@gmail.com> Co-authored-by: VM <112189277+sysvm@users.noreply.github.com> Co-authored-by: DylanYong <dylan.y@nodereal.io> Co-authored-by: constwz <122766871+constwz@users.noreply.github.com> Co-authored-by: Annie <108039750+annielz@users.noreply.github.com> Co-authored-by: BarryTong65 <barrytong.work@gmail.com>

sysvm changed the title ~~fix: stone node sends piece data to gateway through HTTP~~ feat: stone node sends piece data to gateway through HTTP Feb 19, 2023

will-2012 changed the title ~~feat: stone node sends piece data to gateway through HTTP~~ feat: stone node sends piece data to gateway Feb 19, 2023

joeylichang reviewed Feb 21, 2023

View reviewed changes

sysvm force-pushed the syncer_to_gateway branch from ee6b2a3 to 9da80d5 Compare February 21, 2023 03:54

yzhaoyu added 2 commits February 21, 2023 11:56

feat: stone node sends piece data to gateway

685eeaf

feat: alloc resp add bucket name and object name

bc31299

sysvm force-pushed the syncer_to_gateway branch from 9da80d5 to bc31299 Compare February 21, 2023 03:56

yzhaoyu added 4 commits February 21, 2023 12:07

Merge branch 'develop' into syncer_to_gateway

cfbc50d

fix: fix errors of stone node uint test

20582f6

fix: fix gateway config

a579c7b

fix: add gateway syncer rpc ut

4bf940c

will-2012 reviewed Feb 21, 2023

View reviewed changes

will-2012 approved these changes Feb 21, 2023

View reviewed changes

fix: add one log in gateway sync piece handler func

83d4edc

joeylichang reviewed Feb 21, 2023

View reviewed changes

yzhaoyu added 2 commits February 21, 2023 18:58

Merge branch 'develop' into syncer_to_gateway

c610ba5

fix: add detailed log in gateway sync piece handler

845cc91

sysvm force-pushed the syncer_to_gateway branch from 6a1c2e3 to 845cc91 Compare February 21, 2023 12:27

joeylichang merged commit d26b18b into develop Feb 21, 2023

sysvm deleted the syncer_to_gateway branch February 21, 2023 12:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: stone node sends piece data to gateway #128

feat: stone node sends piece data to gateway #128

sysvm commented Feb 19, 2023

joeylichang Feb 21, 2023

sysvm Feb 21, 2023

joeylichang Feb 21, 2023

sysvm Feb 21, 2023 •

edited

Loading

joeylichang Feb 21, 2023

sysvm Feb 21, 2023

joeylichang Feb 21, 2023

sysvm Feb 21, 2023

joeylichang Feb 21, 2023

sysvm Feb 21, 2023

will-2012 Feb 21, 2023

sysvm Feb 21, 2023

will-2012 Feb 21, 2023

sysvm Feb 21, 2023

will-2012 Feb 21, 2023

sysvm Feb 21, 2023

will-2012 Feb 21, 2023

sysvm Feb 21, 2023

will-2012 left a comment

joeylichang Feb 21, 2023

sysvm Feb 21, 2023

feat: stone node sends piece data to gateway #128

feat: stone node sends piece data to gateway #128

Conversation

sysvm commented Feb 19, 2023

Description

Rationale

Example

Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sysvm Feb 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

will-2012 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sysvm Feb 21, 2023 •

edited

Loading