indexer: index more details about blocks and transactions #1329
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
altergui
force-pushed
the
feat/index-blocks
branch
4 times, most recently
from
29dccc8 to
dbdd78f
Compare
|
@mvdan wdyt in general? and in particular, TestRestoreBackupAndMigrate is failing, since i changed the db columns. what's the best fix? recreate the sqlite-backup-0009.sql.zst ? i understand this would not happen if i avoided modifying 0006_create_table_blocks.sql and instead created a 0012_add_columns_to_table_blocks.sql , which i can also do. |
p4u
reviewed
Jun 11, 2024
altergui
force-pushed
the
feat/index-blocks
branch
from
dbdd78f to
711f82e
Compare
|
Yes, please add a new migration step. As far as I can tell, and from what we agreed with @p4u, we are treating the database as stable and we should avoid any breaking changes that force us to delete it and re-index everything. In most cases, like adding new columns, it should be easy to do it as a new step anyway. |
mvdan
reviewed
Jun 14, 2024
altergui
force-pushed
the
feat/index-blocks
branch
2 times, most recently
from
f620fc0 to
514c499
Compare
|
TestRestoreBackupAndMigrate is still failing, with the new migration 0013, complaining about a missing column. digging in 🕵️ |
altergui
force-pushed
the
feat/index-blocks
branch
2 times, most recently
from
1d9efe3 to
5184f2d
Compare
Pull Request Test Coverage Report for Build 9545233328Details
💛 - Coveralls |
altergui
force-pushed
the
feat/index-blocks
branch
2 times, most recently
from
af03b7a to
5da65a6
Compare
this was failing due to my buggy migration 0013. fixed, all tests are passing now |
Pull Request Test Coverage Report for Build 9548791745Details
💛 - Coveralls |
Pull Request Test Coverage Report for Build 10701283702Details
💛 - Coveralls |
altergui
force-pushed
the
feat/index-blocks
branch
4 times, most recently
from
40c4b1c to
377af13
Compare
altergui
force-pushed
the
feat/index-blocks
branch
from
377af13 to
6bc1f7e
Compare
altergui
force-pushed
the
feat/index-blocks
branch
5 times, most recently
from
3a59d53 to
fdf41fa
Compare
altergui
force-pushed
the
feat/index-blocks
branch
5 times, most recently
from
9709c56 to
0fcf9c8
Compare
This was
linked to
issues
Sep 3, 2024
* api: remove endpoint /chain/transactions/reference/index/{index}
indexer migration:
* 0013_recreate_table_transactions.sql
renames:
* transactionHash -> hash
* blockHeight -> height
* transactionIndex -> index
* transactionType -> type
affected endpoints:
/chain/blocks/{height}/transactions/page/{page}
/chain/transactions/page/{page}
/chain/transactions
/chain/transactions/{height}/{index}
/chain/transactions/reference/{hash}
since the CreateBlock now fetches the whole block, we need app.NodeClient initialized in all nodes (including seeds), else seeds panic on first Commit
altergui
force-pushed
the
feat/index-blocks
branch
from
0fcf9c8 to
9e99ff1
Compare
|
This PR introduces the following changes in the developer-portal documentation: diff --git a/swaggers/vocdoni-api.yaml b/swaggers/vocdoni-api.yaml
index 858b707..d473e57 100644
--- a/swaggers/vocdoni-api.yaml
+++ b/swaggers/vocdoni-api.yaml
@@ -1,7 +1,7 @@
openapi: 3.0.0
info:
contact: {}
- description: "\nThe Vocdoni API is a REST API that substitutes the previous RPCs in order to make it easier for developers/integrators to build on top of the voting protocol. This API facilitates the creation of voting processes with Vocdoni, without the hassle of integrating with a complex distributed stack with blockchain components. The API allows integrators to perform all the features enabled by the Vocdoni voting protocol, such as creating accounts, organizations, voting processes, and censuses, as well as casting votes. The API is designed to abstract away the complexity of the Vocdoni protocol as much as possible, offering a simple and straightforward way to performing these actions. \n\nThe API contains the following endpoints: \n\n- [**Chain**](chain): The Vocdoni blockchain is named Vochain. It is a Byzantine fault-tolerant network based on Tendermint that executes the Vocdoni Protocol logic represented as a state machine. Its main purpose is to register votes to a decentralized data store that is able to guarantee univeral verifiability. The chain endpoints allow you to consult the state of the chain, estimate transactions costs, list organizations, and get more Vochain info.\n- [**Accounts**](accounts): Identified by an Ethereum-like address. An account can create and manage elections, transfer tokens, give power to other accounts on its behalf (delegation) and manage its metadata. This endpoint allows users to set the metadata associated with an existing account and to query for information related to existing accounts.\n- [**Elections**](elections): The elections endpoint serves information related to elections such as basic election information, election keys, and submitted votes, as well as enabling users to create a new election and modify existing ones. There is a set of [options, specifications, and lifecycle states](https://developer.vocdoni.io/protocol#elections) that determine the behavior of an election and how votes are counted. \n- [**Censuses**](censuses): The census is a key component of any voting process. It specifies the set of users (each identified by a public key or address) eligible to participate in an election. The various types of census are documented [here](https://developer.vocdoni.io/protocol/census). This endpoint provides census information like the Merkle root, type, total weight, and size of a census. It also allows you to import/export censuses and create new ones.\n- [**Votes**](votes): This endpoint serves all the information associated with any specific vote, including its validity. It is also how users can cast votes.\n- [**Wallet**](wallet): The wallet endpoint facilitates the creation of accounts on the Vochain. This endpoint fulfills requests relating to the token balance held by a given account. \n- [**SIK**](sik): The Secret Identity Key is a user-generated piece of information that proves the user's identity without revealing it. It is the hash of the user's address, the signature of a public message, and an optional secret part. It is used to ensure anonymous voting. All registered accounts or anonymous voters must register a SIK, and these keys are all stored in a Merkle tree. The `/siks` endpoints helps to generate a proof of membership, get the current valid SIK roots, or check if an account has a valid SIK.\n\n\n### Errors \n\nBackend error messages list are defined here: https://github.com/vocdoni/vocdoni-node/blob/master/api/errors.go\n\nAbout the **204 no content** error: this message will be returned only if the asset being queried cannot be found but no other errors have occurred. This response is commonly used to prevent Javascript errors that may arise when a client is waiting for a transaction to be published. During this waiting period, the client can repeatedly query the endpoint until a successful response with a status code of 200 is received, thereby avoiding any errors that may occur due to the transaction not being published yet."
+ description: "\nThe Vocdoni API is a REST API that substitutes the previous RPCs in order to make it easier for developers/integrators to build on top of the voting protocol. This API facilitates the creation of voting processes with Vocdoni, without the hassle of integrating with a complex distributed stack with blockchain components. The API allows integrators to perform all the features enabled by the Vocdoni voting protocol, such as creating accounts, organizations, voting processes, and censuses, as well as casting votes. The API is designed to abstract away the complexity of the Vocdoni protocol as much as possible, offering a simple and straightforward way to performing these actions.\n\nVocdoni API URLs: \n\n- **Production**: https://api.vocdoni.io/v2\n- **Staging**: https://api-stg.vocdoni.net/v2\n- **Development**: https://api-dev.vocdoni.net/v2\n\nThe API contains the following endpoints: \n\n- [**Chain**](chain): The Vocdoni blockchain is named Vochain. It is a Byzantine fault-tolerant network based on Tendermint that executes the Vocdoni Protocol logic represented as a state machine. Its main purpose is to register votes to a decentralized data store that is able to guarantee univeral verifiability. The chain endpoints allow you to consult the state of the chain, estimate transactions costs, list organizations, and get more Vochain info.\n- [**Accounts**](accounts): Identified by an Ethereum-like address. An account can create and manage elections, transfer tokens, give power to other accounts on its behalf (delegation) and manage its metadata. This endpoint allows users to set the metadata associated with an existing account and to query for information related to existing accounts.\n- [**Elections**](elections): The elections endpoint serves information related to elections such as basic election information, election keys, and submitted votes, as well as enabling users to create a new election and modify existing ones. There is a set of [options, specifications, and lifecycle states](https://developer.vocdoni.io/protocol#elections) that determine the behavior of an election and how votes are counted. \n- [**Censuses**](censuses): The census is a key component of any voting process. It specifies the set of users (each identified by a public key or address) eligible to participate in an election. The various types of census are documented [here](https://developer.vocdoni.io/protocol/census). This endpoint provides census information like the Merkle root, type, total weight, and size of a census. It also allows you to import/export censuses and create new ones.\n- [**Votes**](votes): This endpoint serves all the information associated with any specific vote, including its validity. It is also how users can cast votes.\n- [**Wallet**](wallet): The wallet endpoint facilitates the creation of accounts on the Vochain. This endpoint fulfills requests relating to the token balance held by a given account. \n- [**SIK**](sik): The Secret Identity Key is a user-generated piece of information that proves the user's identity without revealing it. It is the hash of the user's address, the signature of a public message, and an optional secret part. It is used to ensure anonymous voting. All registered accounts or anonymous voters must register a SIK, and these keys are all stored in a Merkle tree. The `/siks` endpoints helps to generate a proof of membership, get the current valid SIK roots, or check if an account has a valid SIK.\n\n\n### Errors \n\nBackend error messages list are defined here: https://github.com/vocdoni/vocdoni-node/blob/master/api/errors.go\n\nAbout the **204 no content** error: this message will be returned only if the asset being queried cannot be found but no other errors have occurred. This response is commonly used to prevent Javascript errors that may arise when a client is waiting for a transaction to be published. During this waiting period, the client can repeatedly query the endpoint until a successful response with a status code of 200 is received, thereby avoiding any errors that may occur due to the transaction not being published yet."
title: Vocdoni API
version: 2.0.0
servers:
@@ -1309,6 +1309,11 @@ paths:
name: limit
schema:
type: number
+ - description: Tx hash
+ in: query
+ name: hash
+ schema:
+ type: string
- description: Block height
in: query
name: height
@@ -1319,17 +1324,27 @@ paths:
name: type
schema:
type: string
+ - description: Tx subtype
+ in: query
+ name: subtype
+ schema:
+ type: string
+ - description: Tx signer
+ in: query
+ name: signer
+ schema:
+ type: string
responses:
'200':
content:
application/json:
schema:
$ref: '#/components/schemas/api.TransactionsList'
- description: List of transactions references
+ description: List of transactions (metadata only)
tags:
- Chain
description: >-
- To get full transaction information use [/chain/transaction/{blockHeight}/{txIndex}](transaction-by-block-index).\nWhere transactionIndex is the index of the transaction on the containing block.
+ To get full transaction information use [/chain/transaction/{hash}](transaction-by-hash).
summary: List transactions
post:
responses:
@@ -1407,7 +1422,7 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/api.TransactionsList'
- description: List of transactions references
+ description: List of transactions (metadata only)
tags:
- Chain
deprecated: true
@@ -1415,16 +1430,16 @@ paths:
To get full transaction information use [/chain/transaction/{blockHeight}/{txIndex}](transaction-by-block-index).\nWhere transactionIndex is the index of the transaction on the containing block.
(deprecated, in favor of /chain/transactions?page=xxx)
- summary: List transactions
- '/chain/transactions/reference/index/{index}':
+ summary: List transactions (legacy)
+ '/chain/transactions/reference/{hash}':
get:
parameters:
- - description: Index of the transaction
+ - description: Transaction hash
in: path
- name: index
+ name: hash
required: true
schema:
- type: integer
+ type: string
responses:
'200':
content:
@@ -1437,9 +1452,9 @@ paths:
tags:
- Chain
description: >-
- Get transaction by its index. This is not transaction reference (hash), and neither the block height and block index. The transaction index is an incremental counter for each transaction. You could use the transaction `block` and `index` to retrieve full info using [transaction by block and index](transaction-by-block-index).
- summary: Transaction by index
- '/chain/transactions/reference/{hash}':
+ Using a transaction's hash, returns the `block` and `index` that contains the transaction. You can use this `block` and `index` to retrieve the full transaction info using [transaction by block and index](transaction-by-block-index)
+ summary: Transaction by hash
+ '/chain/transactions/{hash}':
get:
parameters:
- description: Transaction hash
@@ -1453,14 +1468,14 @@ paths:
content:
application/json:
schema:
- $ref: '#/components/schemas/indexertypes.Transaction'
+ $ref: '#/components/schemas/api.GenericTransactionWithInfo'
description: OK
'204':
description: 'See [errors](vocdoni-api#errors) section'
tags:
- Chain
description: >-
- Using a transaction's hash, returns the `block` and `index` that contains the transaction. You can use this `block` and `index` to retrieve the full transaction info using [transaction by block and index](transaction-by-block-index)
+ Get transaction full information by hash. It returns JSON transaction protobuf encoded. Depending of transaction type will return different types of objects. Current transaction types can be found calling `/chain/transactions/cost`
summary: Transaction by hash
'/chain/transactions/{height}/{index}':
get:
@@ -1488,8 +1503,11 @@ paths:
description: 'See [errors](vocdoni-api#errors) section'
tags:
- Chain
+ deprecated: true
description: >-
Get transaction full information by block height and index. It returns JSON transaction protobuf encoded. Depending of transaction type will return different types of objects. Current transaction types can be found calling `/chain/transactions/cost`
+
+ (deprecated, in favor of /chain/transactions/{hash})
summary: Transaction by block height and index
/chain/transfers:
get:
@@ -2245,16 +2263,12 @@ components:
type: object
api.Block:
properties:
- data:
- $ref: '#/components/schemas/types.Data'
- evidence:
- $ref: '#/components/schemas/types.EvidenceData'
hash:
type: string
header:
$ref: '#/components/schemas/github_com_cometbft_cometbft_types.Header'
- last_commit:
- $ref: '#/components/schemas/types.Commit'
+ txCount:
+ type: integer
type: object
api.BlockList:
properties:
@@ -2726,7 +2740,7 @@ components:
$ref: '#/components/schemas/api.Pagination'
transactions:
items:
- $ref: '#/components/schemas/indexertypes.Transaction'
+ $ref: '#/components/schemas/indexertypes.TransactionMetadata'
type: array
type: object
api.TransfersList:
@@ -3163,22 +3177,59 @@ components:
type: object
indexertypes.Transaction:
properties:
- blockHeight:
+ hash:
+ example: 75e8f822f5dd13973ac5158d600f0a2a5fea4bfefce9712ab5195bf17884cfad
+ type: string
+ height:
example: 64924
format: int32
type: integer
- transactionHash:
+ index:
+ example: 0
+ format: int32
+ type: integer
+ signer:
+ example: 0e45513942cf95330fc5e9020851b8bdd9b9c9df
+ type: string
+ subtype:
+ example: set_process_census
+ type: string
+ type:
+ enum:
+ - vote
+ - newProcess
+ - admin
+ - setProcess
+ - registerKey
+ - mintTokens
+ - sendTokens
+ - setTransactionCosts
+ - setAccount
+ - collectFaucet
+ - setKeykeeper
+ example: Vote
+ type: string
+ type: object
+ indexertypes.TransactionMetadata:
+ properties:
+ hash:
example: 75e8f822f5dd13973ac5158d600f0a2a5fea4bfefce9712ab5195bf17884cfad
type: string
- transactionIndex:
- example: 0
+ height:
+ example: 64924
format: int32
type: integer
- transactionNumber:
- example: 944
- format: int64
+ index:
+ example: 0
+ format: int32
type: integer
- transactionType:
+ signer:
+ example: 0e45513942cf95330fc5e9020851b8bdd9b9c9df
+ type: string
+ subtype:
+ example: set_process_census
+ type: string
+ type:
enum:
- vote
- newProcess
@@ -3839,73 +3890,6 @@ components:
parts:
$ref: '#/components/schemas/types.PartSetHeader'
type: object
- types.BlockIDFlag:
- enum:
- - 1
- - 2
- - 3
- type: integer
- x-enum-varnames:
- - BlockIDFlagAbsent
- - BlockIDFlagCommit
- - BlockIDFlagNil
- types.Commit:
- properties:
- block_id:
- $ref: '#/components/schemas/types.BlockID'
- height:
- description: >-
- NOTE: The signatures are in order of address to preserve the bonded
-
- ValidatorSet order.
-
- Any peer with a block can gossip signatures by index with a peer without
-
- recalculating the active ValidatorSet.
- type: integer
- round:
- type: integer
- signatures:
- items:
- $ref: '#/components/schemas/types.CommitSig'
- type: array
- type: object
- types.CommitSig:
- properties:
- block_id_flag:
- $ref: '#/components/schemas/types.BlockIDFlag'
- signature:
- items:
- type: integer
- type: array
- timestamp:
- type: string
- validator_address:
- items:
- type: integer
- type: array
- type: object
- types.Data:
- properties:
- txs:
- description: >-
- Txs that will be applied by state @ block.Height+1.
-
- NOTE: not all txs here are valid. We're just agreeing on the order first.
-
- This means that block.AppHash does not include these txs.
- items:
- items:
- type: integer
- type: array
- type: array
- type: object
- types.EvidenceData:
- properties:
- evidence:
- items: {}
- type: array
- type: object
types.PartSetHeader:
properties:
hash:
|
blocks: * indexerdb: add block details chain_id, proposer_address and last_block_hash * indexerdb: rename method GetBlock -> GetBlockByHeight * indexerdb: make CreateBlock an UPSERT * indexer: new method BlockByHeight (and indexerdb.GetBlockByHash) * indexer: new method SearchBlocks (and indexerdb.SearchBlocks) transactions: * indexerdb: add raw_tx, subtype, signature and signer in transactions table * indexerdb: make CreateTransaction an UPSERT * indexer: new methods CountTransactionsByHeight and SearchTransactions * indexer: rename GetTransaction* methods -> GetTxMetadata* indexer migrations: * 0014_alter_columns_table_blocks.sql * 0015_alter_columns_table_transactions.sql indexer: reindex blocks and transactions after migrations indexer: new method ReindexBlocks * vochaintx: add tx.TxSubtype method
altergui
force-pushed
the
feat/index-blocks
branch
from
9e99ff1 to
6e87af5
Compare
…Store
these endpoints now fetch blocks from indexer, return just `header` and include `txCount`
* /chain/blocks
* /chain/blocks/{height}
* /chain/blocks/hash/{hash}
this new endpoint fetches the full tx from indexer, and includes `subtype` and `signer` fields
* /chain/transactions/{hash}
this legacy endpoint is now marked as deprecated
* /chain/transactions/{height}/{index}
this endpoint now accepts more filter params (subtype, signer)
* /chain/transactions
refactor:
* api: rename chainBlockHandler -> chainBlockByHeightHandler
altergui
force-pushed
the
feat/index-blocks
branch
from
6e87af5 to
1a47b20
Compare
altergui
changed the title
This was referenced Sep 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
needed since we started pruning the blockstore