Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tendermint2] re-implement tx indexing #275

Closed
jaekwon opened this issue Jul 9, 2022 · 2 comments · Fixed by #546
Closed

[tendermint2] re-implement tx indexing #275

jaekwon opened this issue Jul 9, 2022 · 2 comments · Fixed by #546
Assignees
Labels
📦 🌐 tendermint v2 Issues or PRs tm2 related 🌱 feature New update to Gno

Comments

@jaekwon
Copy link
Contributor

jaekwon commented Jul 9, 2022

The tx indexing and query system was removed to attempt a more modular and minimal implementation.

The goal I have in mind is to simply log to file all EventSwitch events (serialized using amino:binary, using protobuf for speed), and then to make readers that can idempotent-ly index or filter these events for whatever purpose, including tx indexing, but also streaming of filtered events (from the RPC using websockets, say). These streaming readers would be independent processes from the full node, so there would be possibly no resource contention depending on the configuration. Since log files could be read by any process, the node wouldn't even have to know who are the listeners are.

We could possibly use an existing persistent pubsub system to implement this, but if we're going to use such an external dependency, it needs to be implemented in Go, be minimal and simple, and mature. Alternatively we should implement one from scratch if a suitable implementation cannot be found.

The log file could use pkgs/autofile, or a fork that doesn't automatically delete older chunk files, and we could leave it up to the user to run a background service that deletes older chunk files as needed.

I'm open to alternative solutions, but this seems to be the most conceptually simple solution. Compared to Tendermint1, instead of an asynchronous pubsub, Tendermint2 has the original synchronous simple event switch. Tendermint1 bakes in several pubsub listeners, but Tendermint2 would be more minimal and simply log all events to the filesystem.

@jaekwon
Copy link
Contributor Author

jaekwon commented Aug 4, 2022

@dongwon8247

@moul
Copy link
Member

moul commented Dec 2, 2022

copied here: tendermint/tendermint2#2

@zivkovicmilos zivkovicmilos self-assigned this Feb 27, 2023
@ajnavarro ajnavarro added 🌱 feature New update to Gno 📦 🌐 tendermint v2 Issues or PRs tm2 related labels May 15, 2023
zivkovicmilos added a commit that referenced this issue Oct 19, 2023
# Description

This PR introduces file-based transaction indexing. As discussed in
#275, transaction index parsing should be done as a separate process
from the main node, meaning other services can be instantiated to index
transactions as readers.

The general architecture of the transaction indexers in this PR can be
described with the following image:
<img width="1481" alt="Architecture"
src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png">

Each concrete indexer implementation decides how to handle transaction
events, and where to store them.
Independent processes from the indexers themselves read these events (by
parsing files, logs, executing RPC queries...).

## File Indexer

The `file` transaction indexer that is included in this PR utilizes
`autofile.Group`s to write down transaction events.

Users can now specify to use the file-based indexer with the following
added flags to the `gnoland` command:
- `--tx-indexer-type` - specify the type of indexer (none is default)
- `--tx-indexer-path` - path for the file-based tx indexer

# Changes include

- [ ] Bugfix (non-breaking change that solves an issue)
- [ ] Hotfix (change that solves an urgent issue, and requires immediate
attention)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that is not backwards-compatible and/or
changes current functionality)

# Checklist (for contributors)

- [x] I have assigned this PR to myself
- [x] I have added at least 1 reviewer
- [x] I have added the relevant labels
- [ ] I have updated the official documentation
- [x] I have added sufficient documentation in code

# Testing

- [x] I have tested this code with the official test suite
- [x] I have tested this code manually

## Manual tests

- Manually executed transactions and verified they were saved to disk.
- Added unit tests that cover all added functionality.

# Additional comments

- [Relevant tendermint2
issue](tendermint/tendermint2#2)
- Resolves #275 

EDIT:
After comments from @jaekwon, this `Indexer` functionality has been
renamed to `EventStore`, and work on an independent indexer process
(process that can read from the event store) will begin soon that will
offer indexing functionality

cc @ilgooz
thehowl pushed a commit to thehowl/gno that referenced this issue Oct 21, 2023
# Description

This PR introduces file-based transaction indexing. As discussed in
gnolang#275, transaction index parsing should be done as a separate process
from the main node, meaning other services can be instantiated to index
transactions as readers.

The general architecture of the transaction indexers in this PR can be
described with the following image:
<img width="1481" alt="Architecture"
src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png">

Each concrete indexer implementation decides how to handle transaction
events, and where to store them.
Independent processes from the indexers themselves read these events (by
parsing files, logs, executing RPC queries...).

## File Indexer

The `file` transaction indexer that is included in this PR utilizes
`autofile.Group`s to write down transaction events.

Users can now specify to use the file-based indexer with the following
added flags to the `gnoland` command:
- `--tx-indexer-type` - specify the type of indexer (none is default)
- `--tx-indexer-path` - path for the file-based tx indexer

# Changes include

- [ ] Bugfix (non-breaking change that solves an issue)
- [ ] Hotfix (change that solves an urgent issue, and requires immediate
attention)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that is not backwards-compatible and/or
changes current functionality)

# Checklist (for contributors)

- [x] I have assigned this PR to myself
- [x] I have added at least 1 reviewer
- [x] I have added the relevant labels
- [ ] I have updated the official documentation
- [x] I have added sufficient documentation in code

# Testing

- [x] I have tested this code with the official test suite
- [x] I have tested this code manually

## Manual tests

- Manually executed transactions and verified they were saved to disk.
- Added unit tests that cover all added functionality.

# Additional comments

- [Relevant tendermint2
issue](tendermint/tendermint2#2)
- Resolves gnolang#275 

EDIT:
After comments from @jaekwon, this `Indexer` functionality has been
renamed to `EventStore`, and work on an independent indexer process
(process that can read from the event store) will begin soon that will
offer indexing functionality

cc @ilgooz
gfanton pushed a commit to gfanton/gno that referenced this issue Nov 9, 2023
# Description

This PR introduces file-based transaction indexing. As discussed in
gnolang#275, transaction index parsing should be done as a separate process
from the main node, meaning other services can be instantiated to index
transactions as readers.

The general architecture of the transaction indexers in this PR can be
described with the following image:
<img width="1481" alt="Architecture"
src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png">

Each concrete indexer implementation decides how to handle transaction
events, and where to store them.
Independent processes from the indexers themselves read these events (by
parsing files, logs, executing RPC queries...).

## File Indexer

The `file` transaction indexer that is included in this PR utilizes
`autofile.Group`s to write down transaction events.

Users can now specify to use the file-based indexer with the following
added flags to the `gnoland` command:
- `--tx-indexer-type` - specify the type of indexer (none is default)
- `--tx-indexer-path` - path for the file-based tx indexer

# Changes include

- [ ] Bugfix (non-breaking change that solves an issue)
- [ ] Hotfix (change that solves an urgent issue, and requires immediate
attention)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that is not backwards-compatible and/or
changes current functionality)

# Checklist (for contributors)

- [x] I have assigned this PR to myself
- [x] I have added at least 1 reviewer
- [x] I have added the relevant labels
- [ ] I have updated the official documentation
- [x] I have added sufficient documentation in code

# Testing

- [x] I have tested this code with the official test suite
- [x] I have tested this code manually

## Manual tests

- Manually executed transactions and verified they were saved to disk.
- Added unit tests that cover all added functionality.

# Additional comments

- [Relevant tendermint2
issue](tendermint/tendermint2#2)
- Resolves gnolang#275 

EDIT:
After comments from @jaekwon, this `Indexer` functionality has been
renamed to `EventStore`, and work on an independent indexer process
(process that can read from the event store) will begin soon that will
offer indexing functionality

cc @ilgooz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
📦 🌐 tendermint v2 Issues or PRs tm2 related 🌱 feature New update to Gno
Projects
Status: 🚀 Needed for Launch
Development

Successfully merging a pull request may close this issue.

4 participants