refactor: update listobjects & blocksyncer modules #471

BarryTong65 · 2023-05-24T11:48:11Z

Description

Block Syncer:

Create a prefix tree specifically for the "/" delimiter, and then optimize the query.
Refactor blocksyncer modules
Metadata:
Update to accommodate SlashPrefixTreeNode in the listobjects interface.
Restore unit tests and add readme
Update logic for getObjectMeta
Download:
Fix getObject and Universal Endpoint headers

Rationale

To optimize sql and speed up query efficiency

Example

N/A

Changes

Notable changes:

create/delete SlashPrefixTreeNode according to different events
update listobjects query
refactor blocksyncer modules
update GetObjectByName query
fix downloader's getObject and Universal Endpoint header

krish-nr

LGTM for code related to blocksyncer

will-2012 · 2023-05-25T05:32:56Z

go.mod

@@ -1,6 +1,6 @@
 module github.com/bnb-chain/greenfield-storage-provider

-go 1.20
+go 1.19


Need to update the golang version?

it's now fixed.

joeylichang · 2023-05-25T06:18:41Z

core/README.md

@@ -65,6 +65,8 @@ Infrastructure interface includes:
  must be a singleton.
 * [Uploader](./module/README.md): Uploader is the modular to handle put object request from user 
  account, and store it in primary SP's piece store.
+* [Metadata](./module/README.md): Metadata is the modular to handle the get request of metadata of objects,


Metadata module have not define the interface in core level, it is can not offer the customization ability.
This is not one of the standardized processes in the gfsp framework, so it is not recommended to add the introduction in the core layer. This will cause misleading, you can add it in modular/metadata/README.md

joeylichang · 2023-05-25T06:18:50Z

core/module/README.md

+
+## Metadata
+Metadata is the modular to handle the get request of metadata of objects,


joeylichang · 2023-05-25T06:20:25Z

modular/gater/object_handler.go

 	w.Header().Set(model.ContentTypeHeader, objectInfo.GetContentType())
 	if isRange {
 		w.Header().Set(model.ContentRangeHeader, "bytes "+util.Uint64ToString(uint64(low))+
 			"-"+util.Uint64ToString(uint64(high)))
 	} else {
 		w.Header().Set(model.ContentLengthHeader, util.Uint64ToString(objectInfo.GetPayloadSize()))
 	}
+	w.Write(data)


why change the location for write body?

@will-2012 told me that frontend peers found if the write of data is before the header, it will cause the headers to not be written correctly. This was fixed in develop branch by Will, but not in this refactor code, so adding it here.

joeylichang · 2023-05-25T06:25:17Z

modular/blocksyncer/modules/registrar.go

@@ -0,0 +1,51 @@
+package registrar


blocksyncer/modular/modules seems not good name.
modules is special for ‘metadata’, 'uploader' etc.
here modules have some confusion with the above.

We have expanded the interface of Juno in SP and implemented it. In Juno, every data module is conceptualized as 'modules', so we have aligned it here.
cc @krish-nr

fix: typo

* fix: data query issues caused by character set replacement (#431) * fix:blocksyncer statement policy id * refactor: add gfsp interface, frame, moduler refactor: add modular entry refactor: delete redundant code refactor: add modular options refactor: tidy entry of gfsp refactor: delete config dir feat: add intergrity interface impl docs: add task readme and annotation docs: add core modular readme docs: add piece store docs docs: add core readme docs: add error code readme docs: add bass app readme refactor: add register modular in main chore: tidy error code feat: add manager status feat: add executor mertic and state feat: add task info interface for log and debug feat: add compute func for secondary approval expired height feat: code tidy for ask approval workflow feat: code tidy for challenge workflow feat: code tidy for replicate piece workflow feat: code tidy for bucket quata workflow feat: code tidy for upload and download object workflow feat: code tidy for gfsp config and main entry feat: code tidy for gfsp client feat: code tidy for gfsp app dir feat: code tidy for metrics feat: code tidy for executor feat: code tidy for manager * Release refactor v0.2.1 query cli (#444) * feat: add query task service and cli command * fix: add register query service to grpc server * feat: add p2p ant address config (#446) * fix: fix check actions (#440) * fix: fix e2e action * fix: fix lint action * fix: fix ut action --------- Co-authored-by: will-2012 <xibaow2020@qq.com> * chore: add code format (#443) * fix: update go.mod (#452) Co-authored-by: VM <arimas@foxmail.com> * fix: fix new piece store and spdb (#460) Co-authored-by: VM <arimas@foxmail.com> * fix: base app rcmgr span panic (#462) * chore: refine the refactored branch codes (#451) * fix: task queue gc delay when call has method (#464) * refactor:blocksyncer (#450) * refactor: add block syncer modular --------- Co-authored-by: constwz <changbohao@gmail.com> * feat: metadata refactor v0.2.1 (#449) feat: migrate all metadata methods clients and services add getObjects handler fix: runnable retriever feat: migrate get bucket meta and get object meta feat: migrate universal endpoint and related methods feat: migrate switch db feat: unify naming from retriever to metadata fix: tests and switch db logic fix: remove outdated metadata proto fix: protobuf lint fix: golangci-lint fix: db switch and e2e test fix:typo fix: e2e script fix: tests fix: update switch db logic fix: naming of methods fix: restore auth * chore: add error for cal nil model (#468) * refactor: update listobjects & blocksyncer modules (#471) * feat: update listobjects * fix: add bsdb metadata readme, restore tests, fix get object ue * fix: adapt juno latest version * refactor: update go mod * refactor: update go version to 1.20 * fix: move metadata readme * fix: bsdb init allow error * feat: more readme contexts fix: typo * fix: update juno version --------- Co-authored-by: annielz <lmlmzhen@sina.com> Co-authored-by: krish-nr <krish.z@nodereal.io> * fix: swtich query error (#476) * fix: fix sp metrics var name (#474) Co-authored-by: VM <arimas@foxmail.com> * refactor: update stop serving module (#473) * refactor: update stop serving module * refactor: refactor codes and fix review comments * fix: fix nonce issue * chore: add default discontinue bucket keep alive days --------- Co-authored-by: joeylichang <joeycli0919@gmail.com> * build: update greenfield chain version (#475) * chore: refine some codes * fix: fix error code judgment logic * chore: refine e2e yml * build: update greenfield chain dependency --------- Co-authored-by: will-2012 <xibaow2020@qq.com> * fix: metrics items name error (#477) * build: update e2e yml (#479) Co-authored-by: will-2012 <xibaow2020@qq.com> * fix: add get public object auth (#481) * fix: check public object auth * chore: rm log file --------- Co-authored-by: Barry <122767193+BarryTong65@users.noreply.github.com> Co-authored-by: constwz <changbohao30@gmail.com> Co-authored-by: constwz <122766871+constwz@users.noreply.github.com> Co-authored-by: actioncli <joeycli0919@qq.com> Co-authored-by: will-2012 <117156346+will-2012@users.noreply.github.com> Co-authored-by: will-2012 <xibaow2020@qq.com> Co-authored-by: Fynn <zcheng1004@gmail.com> Co-authored-by: VM <112189277+sysvm@users.noreply.github.com> Co-authored-by: VM <arimas@foxmail.com> Co-authored-by: constwz <changbohao@gmail.com> Co-authored-by: Annie <108039750+annielz@users.noreply.github.com> Co-authored-by: annielz <lmlmzhen@sina.com> Co-authored-by: krish-nr <krish.z@nodereal.io> Co-authored-by: forcodedancing <just.haha.it@gmail.com>

feat: update listobjects

50f72c7

BarryTong65 added the r4r Ready for review label May 24, 2023

BarryTong65 requested review from constwz and krish-nr May 24, 2023 12:39

BarryTong65 changed the title ~~feat: update listobjects~~ refactor: update listobjects & db switch May 25, 2023

BarryTong65 changed the title ~~refactor: update listobjects & db switch~~ refactor: update listobjects & blocksyncer modules May 25, 2023

BarryTong65 requested a review from annielz May 25, 2023 02:23

fix: add bsdb metadata readme, restore tests, fix get object ue

f79e4fd

annielz force-pushed the refactor-listobjectsapi branch from 65aa9a5 to f79e4fd Compare May 25, 2023 03:55

krish-nr and others added 2 commits May 25, 2023 12:03

fix: adapt juno latest version

3b0dd58

refactor: update go mod

3edc8f1

BarryTong65 requested review from joeylichang, will-2012, sysvm and ruojunm May 25, 2023 04:14

krish-nr approved these changes May 25, 2023

View reviewed changes

will-2012 reviewed May 25, 2023

View reviewed changes

refactor: update go version to 1.20

cedcd7f

joeylichang reviewed May 25, 2023

View reviewed changes

annielz added 2 commits May 25, 2023 14:58

fix: move metadata readme

a7078f2

fix: bsdb init allow error

f1afdfb

annielz mentioned this pull request May 25, 2023

feat: metadata service and universal endpoint refactor v0.2.1 #449

Merged

annielz force-pushed the refactor-listobjectsapi branch from 7c79b79 to fc8d501 Compare May 25, 2023 08:13

feat: more readme contexts

9f1acd6

fix: typo

krish-nr force-pushed the refactor-listobjectsapi branch from fc8d501 to 9f1acd6 Compare May 25, 2023 08:31

fix: update juno version

d3cffb6

actioncli approved these changes May 25, 2023

View reviewed changes

annielz merged commit 424ea6a into release-refactor-v0.2.1 May 25, 2023

sysvm deleted the refactor-listobjectsapi branch May 26, 2023 09:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: update listobjects & blocksyncer modules #471

refactor: update listobjects & blocksyncer modules #471

BarryTong65 commented May 24, 2023 •

edited by annielz

Loading

krish-nr left a comment

will-2012 May 25, 2023

BarryTong65 May 25, 2023 •

edited

Loading

joeylichang May 25, 2023

annielz May 25, 2023

joeylichang May 25, 2023

annielz May 25, 2023

joeylichang May 25, 2023

mliu233 May 25, 2023

joeylichang May 25, 2023

BarryTong65 May 25, 2023 •

edited

Loading

BarryTong65 May 25, 2023


		## Metadata
		Metadata is the modular to handle the get request of metadata of objects,

refactor: update listobjects & blocksyncer modules #471

refactor: update listobjects & blocksyncer modules #471

Conversation

BarryTong65 commented May 24, 2023 • edited by annielz Loading

Description

Rationale

Example

Changes

krish-nr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BarryTong65 May 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BarryTong65 May 25, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BarryTong65 commented May 24, 2023 •

edited by annielz

Loading

BarryTong65 May 25, 2023 •

edited

Loading

BarryTong65 May 25, 2023 •

edited

Loading