Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

re-implement tx indexing #2

Open
moul opened this issue Dec 2, 2022 · 2 comments
Open

re-implement tx indexing #2

moul opened this issue Dec 2, 2022 · 2 comments

Comments

@moul
Copy link
Member

moul commented Dec 2, 2022

The tx indexing and query system was removed to attempt a more modular and minimal implementation.

The goal I have in mind is to simply log to file all EventSwitch events (serialized using amino:binary, using protobuf for speed), and then to make readers that can idempotent-ly index or filter these events for whatever purpose, including tx indexing, but also streaming of filtered events (from the RPC using websockets, say). These streaming readers would be independent processes from the full node, so there would be possibly no resource contention depending on the configuration. Since log files could be read by any process, the node wouldn't even have to know who are the listeners are.

We could possibly use an existing persistent pubsub system to implement this, but if we're going to use such an external dependency, it needs to be implemented in Go, be minimal and simple, and mature. Alternatively we should implement one from scratch if a suitable implementation cannot be found.

The log file could use pkgs/autofile, or a fork that doesn't automatically delete older chunk files, and we could leave it up to the user to run a background service that deletes older chunk files as needed.

I'm open to alternative solutions, but this seems to be the most conceptually simple solution. Compared to Tendermint1, instead of an asynchronous pubsub, Tendermint2 has the original synchronous simple event switch. Tendermint1 bakes in several pubsub listeners, but Tendermint2 would be more minimal and simply log all events to the filesystem.

Original issue: gnolang/gno#275
@moul moul mentioned this issue Dec 2, 2022
Closed
@zivkovicmilos
Copy link

@moul

Can you please explain this part more? I'm not sure why we'd need a pubsub system in the first place

We could possibly use an existing persistent pubsub system to implement this, but if we're going to use such an external dependency, it needs to be implemented in Go, be minimal and simple, and mature. Alternatively we should implement one from scratch if a suitable implementation cannot be found.

@moul
Copy link
Member Author

moul commented Feb 10, 2023

Make sure that we can have a standalone collector feeding a pubsub system. And another standalone processor that consumes the pubsub.

PubSub could be a simple file, a database, not necessarily an advanced API.

@zivkovicmilos zivkovicmilos mentioned this issue Feb 28, 2023
Merged
11 tasks
zivkovicmilos added a commit to gnolang/gno that referenced this issue Oct 19, 2023
@zivkovicmilos
feat: add file-based transaction indexing (#546)
0600d41 
# Description

This PR introduces file-based transaction indexing. As discussed in
#275, transaction index parsing should be done as a separate process
from the main node, meaning other services can be instantiated to index
transactions as readers.

The general architecture of the transaction indexers in this PR can be
described with the following image:
<img width="1481" alt="Architecture"
src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png">

Each concrete indexer implementation decides how to handle transaction
events, and where to store them.
Independent processes from the indexers themselves read these events (by
parsing files, logs, executing RPC queries...).

## File Indexer

The `file` transaction indexer that is included in this PR utilizes
`autofile.Group`s to write down transaction events.

Users can now specify to use the file-based indexer with the following
added flags to the `gnoland` command:
- `--tx-indexer-type` - specify the type of indexer (none is default)
- `--tx-indexer-path` - path for the file-based tx indexer

# Changes include

- [ ] Bugfix (non-breaking change that solves an issue)
- [ ] Hotfix (change that solves an urgent issue, and requires immediate
attention)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that is not backwards-compatible and/or
changes current functionality)

# Checklist (for contributors)

- [x] I have assigned this PR to myself
- [x] I have added at least 1 reviewer
- [x] I have added the relevant labels
- [ ] I have updated the official documentation
- [x] I have added sufficient documentation in code

# Testing

- [x] I have tested this code with the official test suite
- [x] I have tested this code manually

## Manual tests

- Manually executed transactions and verified they were saved to disk.
- Added unit tests that cover all added functionality.

# Additional comments

- [Relevant tendermint2
issue](tendermint/tendermint2#2)
- Resolves #275 

EDIT:
After comments from @jaekwon, this `Indexer` functionality has been
renamed to `EventStore`, and work on an independent indexer process
(process that can read from the event store) will begin soon that will
offer indexing functionality

cc @ilgooz
thehowl pushed a commit to thehowl/gno that referenced this issue Oct 21, 2023
@zivkovicmilos @thehowl
feat: add file-based transaction indexing (gnolang#546)
8042f54 
# Description

This PR introduces file-based transaction indexing. As discussed in
gnolang#275, transaction index parsing should be done as a separate process
from the main node, meaning other services can be instantiated to index
transactions as readers.

The general architecture of the transaction indexers in this PR can be
described with the following image:
<img width="1481" alt="Architecture"
src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png">

Each concrete indexer implementation decides how to handle transaction
events, and where to store them.
Independent processes from the indexers themselves read these events (by
parsing files, logs, executing RPC queries...).

## File Indexer

The `file` transaction indexer that is included in this PR utilizes
`autofile.Group`s to write down transaction events.

Users can now specify to use the file-based indexer with the following
added flags to the `gnoland` command:
- `--tx-indexer-type` - specify the type of indexer (none is default)
- `--tx-indexer-path` - path for the file-based tx indexer

# Changes include

- [ ] Bugfix (non-breaking change that solves an issue)
- [ ] Hotfix (change that solves an urgent issue, and requires immediate
attention)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that is not backwards-compatible and/or
changes current functionality)

# Checklist (for contributors)

- [x] I have assigned this PR to myself
- [x] I have added at least 1 reviewer
- [x] I have added the relevant labels
- [ ] I have updated the official documentation
- [x] I have added sufficient documentation in code

# Testing

- [x] I have tested this code with the official test suite
- [x] I have tested this code manually

## Manual tests

- Manually executed transactions and verified they were saved to disk.
- Added unit tests that cover all added functionality.

# Additional comments

- [Relevant tendermint2
issue](tendermint/tendermint2#2)
- Resolves gnolang#275 

EDIT:
After comments from @jaekwon, this `Indexer` functionality has been
renamed to `EventStore`, and work on an independent indexer process
(process that can read from the event store) will begin soon that will
offer indexing functionality

cc @ilgooz
gfanton pushed a commit to gfanton/gno that referenced this issue Nov 9, 2023
@zivkovicmilos @gfanton
feat: add file-based transaction indexing (gnolang#546)
6f4056b 
# Description

This PR introduces file-based transaction indexing. As discussed in
gnolang#275, transaction index parsing should be done as a separate process
from the main node, meaning other services can be instantiated to index
transactions as readers.

The general architecture of the transaction indexers in this PR can be
described with the following image:
<img width="1481" alt="Architecture"
src="https://user-images.githubusercontent.com/16712663/221845299-ff552470-8efc-4134-8c3d-e71e74929acc.png">

Each concrete indexer implementation decides how to handle transaction
events, and where to store them.
Independent processes from the indexers themselves read these events (by
parsing files, logs, executing RPC queries...).

## File Indexer

The `file` transaction indexer that is included in this PR utilizes
`autofile.Group`s to write down transaction events.

Users can now specify to use the file-based indexer with the following
added flags to the `gnoland` command:
- `--tx-indexer-type` - specify the type of indexer (none is default)
- `--tx-indexer-path` - path for the file-based tx indexer

# Changes include

- [ ] Bugfix (non-breaking change that solves an issue)
- [ ] Hotfix (change that solves an urgent issue, and requires immediate
attention)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (change that is not backwards-compatible and/or
changes current functionality)

# Checklist (for contributors)

- [x] I have assigned this PR to myself
- [x] I have added at least 1 reviewer
- [x] I have added the relevant labels
- [ ] I have updated the official documentation
- [x] I have added sufficient documentation in code

# Testing

- [x] I have tested this code with the official test suite
- [x] I have tested this code manually

## Manual tests

- Manually executed transactions and verified they were saved to disk.
- Added unit tests that cover all added functionality.

# Additional comments

- [Relevant tendermint2
issue](tendermint/tendermint2#2)
- Resolves gnolang#275 

EDIT:
After comments from @jaekwon, this `Indexer` functionality has been
renamed to `EventStore`, and work on an independent indexer process
(process that can read from the event store) will begin soon that will
offer indexing functionality

cc @ilgooz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@moul @zivkovicmilos