Building a blockchain monitor plugin

[ranyefet]

Hi,

In our company we’re building a wallet service and we use bcoin as our node.
We need the ability to index coins and transactions that are related to our wallets addresses.

I was thinking of writing a plug-in that will listen for new block events and will check if the tx in the block are relevant for any of the addresses we need to monitor and if so, it will update our Postgres database with the relevant tx and coins.

Does it make sense?
One concern that I have is,what if the node crashes and I missed an event? How can I get the information I need in an efficient way?

[tuxcanfly]

Yeah that's a perfect use case for a plugin. Re: node crashes etc. I am working on a PR which moves indexing out of the chain and adds a interface which allows one to write custom indexers. Ref #424

https://github.com/tuxcanfly/bcoin/blob/index-plugin/lib/indexer/addrindexer.js

If you would like to checkout the branch and give feedback on the new implementation, that would be great.

[nodech]

👍 writing as indexer plugin

You can just implement indexBlock and unindexBlock which handles your logic (Check addrindexer and txindexer)..

After last discussion of indexers with @tuxcanfly , it should be possible to run indexer separately from node, which will give you even more flexibility. But this is not implemented yet, still work in progress.

If your node goes down or indexer goes down (or both), after starting they will eventually come to the correct state as node syncs and sends this info to indexer.

Even in case of block reorganization you should see unindexBlock, indexBlock, indexblock, which should get you to the correct state.

[ranyefet]

That looks great!
One thing I noticed about is that the parent Indexer class depends on LevelDB. Is that required for all plugins?

In my case I plan to use Postgres to have better querying capabilities and support connecting to DB from different application.

Another question I have, is it possible to index mempool transactions?
So that I will be able to show unconfirmed balance for the wallet immediately and not wait for block event.

[nodech]

On indexer and leveldb.

Main indexer class uses leveldb to keep track of chain state, e.g. reorganization happens, indexer was offline and it reconnected and so on.. So it keeps track of bare minimum to simplify API to index and unindex methods. So in your case you can leave those in leveldb and move main logic (which is basically "businuess" logic for indexers) using different DB.

Mempool already has indexer enabled, currently transactions that are in mempool should show up in indexer endpoints, currently I am working on it to possibly move it outside of the mempool and run it separately, but even if that fails Mempool will have its own endpoints which expose the same info..
On the other hand you can handle mempool events by yourself, mempool provides all necessary events to those transactions. Unfortunately you will have to handle logic additional logic as it is.

Hope to get better results.

[pinheadmz]

@ranyefet do you still need an open issue for this? All the decs are available on our slack as well if you have more questions: http://bcoin.io/slack-signup.html

[ranyefet]

@nodar-chkuaselidze thank you for taking the time to answer my questions!