feat: verfier and server #1

BarryTong65 · 2024-07-11T07:08:48Z

Description

Rationale

The purpose of the Greenfield BSC Archiver is to store the headers and bodies of BSC historical blocks, and to upload the block information to Greenfield via the bundle service.

The Syncer is responsible for retrieving block information through Ethereum calls, storing it in a database, and uploading it to Greenfield through the bundle service. The range of block heights uploaded can be customized; currently, it is set to 100 blocks.

The Verifier periodically verifies the correctness of the objects uploaded to Greenfield. If an error is detected, it updates and uploads the correct version of the object to the bundle service, and also updates the bundle name of the corresponding block in the local block database.

The Server provides RPC interfaces, with parameters currently adapted to Ethereum calls. Variables are customized and generated using a Swagger.yaml file.

eth_getBlockByNumber
eth_getBundledBlockByNumber
eth_blockNumber
eth_getBlockByHash
/bsc/v1/blocks/{block_id}/bundle/name

Further Improvement
Currently, the process of retrieving block heights and storing them in the database is sequential, raising concerns that the high concurrency might overwhelm the bundle service. If the current version’s block tracking speed is slow, an update will be made to employ multiple workers to request and retrieve block heights into memory, and then collectively store them in both the bundle service and the database.

Example

Method: POST
URL: https://gnfd-bsc-archiver-testnet.bnbchain.org/
eth_getBlockByNumber
request

{
    "method": "eth_getBlockByNumber",
    "params": [
        "0x2625A00"
    ],
    "id": 100,
    "jsonrpc": "2.0"
}

response

{
    "id": 100,
    "jsonrpc": "2.0",
    "result": {
        "difficulty": "0x2",
        "extraData": "0xd883010002846765746888676f312e31332e34856c696e757800000000000000924cd67a1565fdd24dd59327a298f1d702d6b7a721440c063713cecb7229f4e162ae38be78f6f71aa5badeaaef35cea25061ee2100622a4a1631a07e862b517401",
        "gasLimit": "0x25ff7a7",
        "gasUsed": "0x300b37",
        "hash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
        "logsBloom": "0x08000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000300000000000000000000000000000",
        "miner": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
        "mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
        "nonce": "0x0000000000000000",
        "number": "0x1",
        "parentHash": "0x0d21840abff46b96c84b2ac9e10e4f5cdaeb5693cb665db62a2f3b02d2d57b5b",
        "receiptsRoot": "0xfc7c0fda97e67ed8ae06e7a160218b3df995560dfcb209a3b0dddde969ec6b00",
        "sha3Uncles": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
        "size": "0x558",
        "stateRoot": "0x1db428ea79cb2e8cc233ae7f4db7c3567adfcb699af668a9f583fdae98e95588",
        "timestamp": "0x5f49ca59",
        "totalDifficulty": "0x3",
        "transactions": [
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0xbaf8ffa0b475a67cfeac3992d24422804452f0982e4e21a8816db2e0c9e5f224",
                "input": "0xe1c7392a",
                "nonce": "0x0",
                "r": "0xba64c6614028e6f7fa4cf2b218719160b9850c591279218def6894586ab73157",
                "s": "0x69cd1b3041eaa9b282a522c49b294e3c51801a1347bb8999015a0bd29bed3bbf",
                "to": "0x0000000000000000000000000000000000001000",
                "transactionIndex": "0x0",
                "type": "0x0",
                "v": "0x93",
                "value": "0x0"
            },
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0x8ea486df4eafaf713fbbe3b4b0b4196e50fbd1ea93daf66675accf3bf3f59d00",
                "input": "0xe1c7392a",
                "nonce": "0x1",
                "r": "0xd6541593c8d1bce8024f2bccfa6f73fc3a107f08e02b39a4571a027863558b3b",
                "s": "0x3448e68c416c115137e9cd1b04a9a09e10a53eadd958d9606962b4170ffffbe7",
                "to": "0x0000000000000000000000000000000000001001",
                "transactionIndex": "0x1",
                "type": "0x0",
                "v": "0x93",
                "value": "0x0"
            },
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0x9ebc5237eabb339a103a34daf280db3d9498142b49fa47f1af71f64a605acffa",
                "input": "0xe1c7392a",
                "nonce": "0x2",
                "r": "0x151f5a54ed865ed852eafe84be6557c13c6d23535ff0ec212e043f48ba5c275d",
                "s": "0x17fa5ca73322133191df498ecb7a9b7fa57a81c403312704314716d81666f848",
                "to": "0x0000000000000000000000000000000000001003",
                "transactionIndex": "0x2",
                "type": "0x0",
                "v": "0x93",
                "value": "0x0"
            },
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0xc043c5d33f8c3a6d6c0853ff8cbe88ebdf746f8092cb763b18de65db45246a6e",
                "input": "0xe1c7392a",
                "nonce": "0x3",
                "r": "0xf1d2414ecd63f567980be648645197d8a0d940921e6e8cde82f9da0ba7259be",
                "s": "0x5add78f836ad168f37cf8fe38432ca01c7827ffbccb8b1981b151f4463651b90",
                "to": "0x0000000000000000000000000000000000001006",
                "transactionIndex": "0x3",
                "type": "0x0",
                "v": "0x94",
                "value": "0x0"
            },
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0x2f64d7e926e6fb62f906e18258097af179c213f0c87a717476cce1b334049797",
                "input": "0xe1c7392a",
                "nonce": "0x4",
                "r": "0x51c8f79cbe3eabe7b8c90fff02b6186c26a31681acea887557450916eaf7b67b",
                "s": "0x63d36286391f2976842e9f541a0ab07aa7e662b79b420f78e59a1e9a487658c9",
                "to": "0x0000000000000000000000000000000000001004",
                "transactionIndex": "0x4",
                "type": "0x0",
                "v": "0x93",
                "value": "0x0"
            },
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0x463f0a179a89f47b055df14897dd7c55a2d819351568045dcb0496f2875c71ee",
                "input": "0xe1c7392a",
                "nonce": "0x5",
                "r": "0x36ffe2a7b51ea50b94dc75e02171e494e1e876902371586f24708a8c8ddb1b1b",
                "s": "0x3164d01a18795962b3a317dfec2ed4393dc0a6c34a2dc2bbb0a986d348f134df",
                "to": "0x0000000000000000000000000000000000001005",
                "transactionIndex": "0x5",
                "type": "0x0",
                "v": "0x93",
                "value": "0x0"
            },
            {
                "accessList": null,
                "blockHash": "0x04055304e432294a65ff31069c4d3092ff8b58f009cdb50eba5351e0332ad0f6",
                "blockNumber": "0x1",
                "chainId": "0x38",
                "from": "0x2a7cdd959bFe8D9487B2a43B33565295a698F7e2",
                "gas": "0x7fffffffffffffff",
                "gasPrice": "0x0",
                "hash": "0xc02fd5fc71fe8bdc4fec3f97a019a4dc9961eb95e5251c55fcb3da76f5cb5bca",
                "input": "0xe1c7392a",
                "nonce": "0x6",
                "r": "0x410b03142901b97ff5c61ed962de42f2aa052799eb1d196a7c1a2aca839e012a",
                "s": "0x599d8118ea6c2bbfe6a2338d21b46667b6ebd7d698221c8c3c580b59f3d49a06",
                "to": "0x0000000000000000000000000000000000002000",
                "transactionIndex": "0x6",
                "type": "0x0",
                "v": "0x94",
                "value": "0x0"
            }
        ],
        "transactionsRoot": "0x53a8743b873570daa630948b1858eaf5dc9bb0bca2093a197e507b2466c110a0",
        "uncles": [],
        "withdrawals": null
    }
}

clydemeng · 2024-07-17T01:23:53Z

.idea/greenfield-bsc-archiver.iml

shall we remove this Intellij IDE related file, as intelij is not a must for this project?

clydemeng · 2024-07-17T01:48:36Z

external/cmn/bundle_client.go

@@ -280,7 +280,7 @@ func (c *BundleClient) UploadObject(fileName, bucketName, bundleName, contentTyp
 	return nil


When will be the method "func (c *BundleClient) UploadObject" used? Seemingly it is not being used now.

Currently, I use the function UploadAndFinalizeBundle(bundleName, bucketName, bundleDir, bundlePath string) to upload to Greenfield via the bundle service. This function is now obsolete and has been removed.

config/example/config-syncer.json

db/dao.go

syncer/syncer.go

service/block.go

unclezoro · 2024-07-23T10:33:22Z

README.md

+        "stateRoot": "0x1db428ea79cb2e8cc233ae7f4db7c3567adfcb699af668a9f583fdae98e95588",
+        "timestamp": "0x5f49ca59",
+        "totalDifficulty": "0x3",
+        "transactions": [


The example is too long, we dont have to show all of them. It can improve the readability of the readme.

unclezoro · 2024-07-23T10:34:36Z

README.md

+2. eth_getBundledBlockByNumber
+3. eth_blockNumber
+4. eth_getBlockByHash
+5. /bsc/v1/blocks/{block_id}/bundle/name


TODO: put the public endpoint of the service after deploy

add the endpoint https://gnfd-bsc-archiver-testnet.bnbchain.org/ to the readme

unclezoro · 2024-07-23T10:36:57Z

config/example/config-server.json

@@ -1,14 +1,14 @@
 {
  "chain": "BSC",
-  "bucket_name": "yourbucketname",
+  "bucket_name": "bsc-historical-blocks-testnet-concurrency",


name concurrency is unnecessary, user dont have to aware of this

change to yourbucketname

README.md

db/dao.go

external/cmn/sp_client.go

restapi/handlers/block.go

syncer/syncer.go

alexgao001

the .idea folder should be added to gitignore

BarryTong65 · 2024-07-25T03:24:18Z

the .idea folder should be added to gitignore

removed the .idea file and add it to the gitignore

BarryTong65 added 17 commits July 11, 2024 15:08

feat: verfier and server

def8790

feat: verfier and server

a394084

feat: verfier and server

16c68ca

feat: verfier and server

0ffbf33

fix: golang ci lint

5741088

fix: response json

34617b4

feat: update struct and swagger yaml

02d76da

feat: update struct and swagger yaml

e29a257

feat: update struct and swagger yaml

8760b11

fix: verfier logic

c67606c

fix: golang ci lint

7f37447

fix: api name issue and config verfier

2efaf1a

fix: docker file

9937122

fix: docker file

f945757

fix: docker file

8bb6012

fix: format code

3271be8

fix: fix nil pointer

b042fe6

BarryTong65 added the r4r label Jul 16, 2024

BarryTong65 requested review from ruojunm, alexgao001 and 1nanosecond July 16, 2024 06:49

BarryTong65 added 3 commits July 16, 2024 16:49

fix: fix nil pointer

da70ccb

fix: fix nil pointer

8ae5632

fix: gitignore

115349c

clydemeng reviewed Jul 17, 2024

View reviewed changes

BarryTong65 added 2 commits July 17, 2024 12:44

feat: update getBlockByNumber latest feature and indexer concurrency

6b15916

fix: remove upload object function

d0d99e3

alexgao001 reviewed Jul 17, 2024

View reviewed changes

fix: retry and cofig setting

ef55e36

BarryTong65 added 12 commits July 17, 2024 20:50

fix: retry process()

c6ff9f5

fix: interval config

f91934f

fix: interval

377c4be

fix: for loop

12e30cb

fix: for loop

8ba0beb

fix: error message

bef0c27

fix: get earliest verfied block

b7e473d

fix: map memory overflow

ded2966

fix: map memory overflow

0ac2741

docs: add readme

4b4200c

docs: add readme

4c06645

fix: change block index

4b18b93

unclezoro reviewed Jul 23, 2024

View reviewed changes

alexgao001 reviewed Jul 23, 2024

View reviewed changes

BarryTong65 added 5 commits July 24, 2024 11:17

fix: update block hash index

1395fcb

fix: update block hash index

9681fe0

fix: all the comments

c0dc138

fix: add lock to check map size

2cd8169

fix: return verified block for get bundle name by block number

5218f6d

alexgao001 reviewed Jul 25, 2024

View reviewed changes

fix: Remove .idea directory from repository

d2416c7

BarryTong65 added 4 commits July 26, 2024 08:55

fix: retry err

c1540b4

fix: tiny fix

43f06b8

fix: tiny fix

c79fc32

fix: check object is sealed from chain

2de81ca

alexgao001 approved these changes Jul 29, 2024

View reviewed changes

BarryTong65 merged commit 38eabae into main Jul 29, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: verfier and server #1

feat: verfier and server #1

BarryTong65 commented Jul 11, 2024 •

edited

Loading

clydemeng Jul 17, 2024

BarryTong65 Jul 17, 2024

clydemeng Jul 17, 2024

BarryTong65 Jul 17, 2024

unclezoro Jul 23, 2024

BarryTong65 Jul 25, 2024

unclezoro Jul 23, 2024

BarryTong65 Jul 25, 2024

unclezoro Jul 23, 2024

BarryTong65 Jul 25, 2024

alexgao001 left a comment

BarryTong65 commented Jul 25, 2024 •

edited

Loading

		@@ -280,7 +280,7 @@ func (c *BundleClient) UploadObject(fileName, bucketName, bundleName, contentTyp
		return nil

feat: verfier and server #1

feat: verfier and server #1

Conversation

BarryTong65 commented Jul 11, 2024 • edited Loading

Description

Rationale

Example

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexgao001 left a comment

Choose a reason for hiding this comment

BarryTong65 commented Jul 25, 2024 • edited Loading

BarryTong65 commented Jul 11, 2024 •

edited

Loading

BarryTong65 commented Jul 25, 2024 •

edited

Loading