refactor indexers #424

tuxcanfly · 2018-03-01T17:28:22Z

This PR refactors the indexers into a lib with it's own db.

Handle sync, tip catch up
Add migration to remove existing indices
Cleanup old indexing code

Refs #422

tuxcanfly · 2018-03-13T04:18:18Z

Moved the indexer itself to:

https://github.com/tuxcanfly/bindex

This PR will be limited to the plugin integration, migration etc.

bucko13

Looks great! Nice to have the plugin separated out. I guess this is probably similar to the approach we'll want to do at some point with bwallet.

bucko13 · 2018-03-13T18:31:28Z

lib/node/fullnode.js

    const chain = await this.chain.getMetaByAddress(addrs);
-    return chain.concat(mempool);
+    return mempool.concat(chain);


Why reverse this?

This case would occur if indexes are disabled, so I think it should just return mempool actually. Will check that.

bucko13 · 2018-03-13T18:42:53Z

lib/node/node.js

@@ -81,6 +81,11 @@ class Node extends EventEmitter {
    if (config.has('logger'))
      logger = config.obj('logger');

+    const plugins = this.config.array('plugins', []);


Does this mean we add bindex by default if indexing is turned on? Does it make a difference that it's in a separate module now?

By implicitly enabling bindex plugin we can ease the migration. No config files need be changed.

bucko13 · 2018-03-13T18:49:43Z

lib/node/rpc.js

@@ -968,7 +968,7 @@ class RPC extends RPCBase {
    if (hash) {
      block = await this.chain.getBlock(hash);
    } else if (this.chain.options.indexTX) {
-      const tx = await this.chain.getMeta(last);
+      const tx = await this.node.getMeta(last);


I guess this is the difference with having indexing as a plugin on the node rather than built into the chain huh? 😄

Yup :) Since indexing happens outside the chain.

bucko13 · 2018-03-13T19:04:37Z

migrate/chaindb4to5.js

@@ -0,0 +1,82 @@
+'use strict';


Should we have an explicit note somewhere that this basically just removes your old db and does a fresh sync with the new indexing?

Yeah we can add that to the CHANGELOG

Added to CHANGELOG

tuxcanfly · 2018-03-20T21:53:04Z

Thinking this through, I think the indexing interface and nested plugins add unnecessary complexity. Also, splitting out tx, addr as plugins etc would make it hard to query them back, since we would need to know which keys the plugins are writing to.

To keep things simple, I propose that the core indexers be a part of the indexing plugin directly. Alternate implementations can fork the index plugin for specific use cases (utxo set, colored coins) etc. Thanks @bucko13 for the discussion.

bucko13 · 2018-03-26T20:39:41Z

lib/node/fullnode.js

@@ -147,6 +144,14 @@ class FullNode extends Node {
      noAuth: this.config.bool('no-auth')
    });

+    // Index needs access to the node.


Is this needed for some of the events?

Yup, we need it to listen to the chain events. This specific comment can be improved though, will update it.

tuxcanfly · 2018-03-28T23:53:06Z

There's an issue with the way re-orgs are handled. Working to fix that and add more tests.

chjj · 2018-03-30T05:26:03Z

Needs rebase on top of master.

tuxcanfly · 2018-03-31T04:23:38Z

@chjj I realized I need to store all the blockhashes similar to wallet, because a re-org will leave my client out of sync and I wouldn't know where to rescan from. I might need to borrow some code from the wallet, specifically WalletDB. Is that fine or do you think this part should be moved to a common class? FWIW @nodar-chkuaselidze will need it too for bmultisig.

nodech · 2018-03-31T04:58:36Z

My case is simpler, because bmultisig is plugin for wallet I can watch wallet events and check coins in db on load. Wallet will be handling that for me.

tuxcanfly · 2018-04-04T03:11:05Z

Making bindex a part of lib to prevent dependency issues. fullnode's dependency on bindex for RPC calls etc makes it difficult to manage as an external module. Most of the chain syncing ideas were inspired from wallet so keeping it in lib is similar in that regard.

tuxcanfly · 2018-04-10T01:57:11Z

Needs some updates to handle different databases for different indexers.

braydonf · 2018-08-17T20:25:34Z

lib/indexer/addrindexer.js

+   */
+
+  async getHashesByAddress(addrs) {
+    const set = new BufferSet();


You probably could use an array here

As a side note, if there are a lot of txids for an address this will likely have performance issues. The largest number of transactions for an address right now is over 3 million.

braydonf · 2018-08-23T21:34:51Z

Worth considering as this code, and database, is being changed. There is an issue with the existing txindex. The tx index is keeping an entire copy of the transaction https://github.com/bcoin-org/bcoin/blob/master/lib/primitives/txmeta.js#L239 so there is a duplicate of the transaction on disk.

I've documented this more at #585

Edit: The existing issues with txindex and addrindex can be addressed in another PR, this PR I think is mainly around having those indexes be separated into another database (one for each index).

module indexer introduces a extensible architecture for indexing the chain. It provides a base class which handles syncing with the chain, handling re-orgs, interruptions, dynamic toggling, etc. TXIndexer and AddrIndexer are provided for indexing transactions and addresses, using the same flags as before i.e --index-tx and --index-address. Indexes are stored in a different database and can be maintained independently of the chain.

also rebase on top of master, fix conflicts

tuxcanfly · 2018-10-28T09:19:28Z

Similar to #532 need to decide how we want to handle re-index in case of spv, pruned nodes. We might want to ask the user to reset the chain.

tuxcanfly · 2018-12-06T18:05:00Z

Added an error when re-indexing when pruned, the only case where it's not possible to re-index.

braydonf · 2018-12-10T22:08:18Z

Indexing, in this case, I think really only applies to full nodes, not pruned or spv. Aside from anything that can be indexed headers, or select transactions such as the wallet. It's currently technically possible for the txindex to be enabled on a pruned node, as the transaction data is duplicated, but this should likely change to reduce total disk space usage (see #585).

tuxcanfly · 2018-12-11T16:57:51Z

Agreed, but we already support indexing when pruned, so disabling this would probably need a deprecation notice. Re-indexing was never supported, but with this PR, it is supported for full nodes, but not for pruned nodes. The error applies to the latter case.

braydonf · 2019-03-27T01:03:40Z

Some continued work on this is at an indexer branch that is being merged with changes from file block store in #703 and some of the additional addrindex fixes.

lib/indexer/indexer.js

tuxcanfly · 2019-04-13T08:20:39Z

Closing this in favor of #758 reworked to support the new flat file backend.

bucko13 added the WIP label Mar 1, 2018

tuxcanfly force-pushed the index-plugin branch from ff0f466 to fb3abfb Compare March 5, 2018 07:02

tuxcanfly force-pushed the index-plugin branch from 75fd2bd to e47cabe Compare March 13, 2018 03:27

bucko13 added enhancement Improving a current feature and removed WIP labels Mar 13, 2018

bucko13 reviewed Mar 13, 2018

View reviewed changes

tuxcanfly force-pushed the index-plugin branch from 420b2cf to 430b07c Compare March 16, 2018 11:08

tuxcanfly changed the title ~~refactor indexers into plugin based system~~ refactor indexers into a plugin Mar 20, 2018

tuxcanfly force-pushed the index-plugin branch from fdc4b44 to 17cf297 Compare March 21, 2018 23:25

bucko13 reviewed Mar 26, 2018

View reviewed changes

chjj changed the base branch from wallet-rewrite to master March 30, 2018 05:23

chjj changed the base branch from master to wallet-rewrite March 30, 2018 05:25

tuxcanfly changed the base branch from wallet-rewrite to master March 30, 2018 20:25

tuxcanfly force-pushed the index-plugin branch from 0b71cd3 to 62d405e Compare March 30, 2018 20:26

tuxcanfly force-pushed the index-plugin branch 5 times, most recently from 0867be8 to 08b38b3 Compare April 9, 2018 22:19

tuxcanfly force-pushed the index-plugin branch from 1dc0eaa to 6846fe7 Compare April 11, 2018 22:57

tuxcanfly requested a review from chjj July 12, 2018 16:10

tuxcanfly force-pushed the index-plugin branch from 1eec93b to c064d7e Compare July 19, 2018 09:39

tuxcanfly force-pushed the index-plugin branch 2 times, most recently from c6cd319 to 3287508 Compare August 13, 2018 14:23

braydonf reviewed Aug 17, 2018

View reviewed changes

braydonf mentioned this pull request Aug 28, 2018

Disk I/O efficiency improvements #585

Closed

tuxcanfly added 6 commits October 25, 2018 00:11

indexer: refactor chain and integrate indexers

9ce19d5

indexer: refactor layout, tests

949a8bf

indexer: update docs

16a9bef

indexer: add getSpentView, remove from chain

107697b

also rebase on top of master, fix conflicts

chaindb: fix typo - deprecation warning

2abc3e2

tuxcanfly force-pushed the index-plugin branch from 3287508 to 2abc3e2 Compare October 24, 2018 18:45

tuxcanfly mentioned this pull request Oct 27, 2018

Add support for neutrino #370

Closed

3 tasks

tuxcanfly mentioned this pull request Nov 23, 2018

Building a blockchain monitor plugin #640

Closed

fullnode: error when re-indexing when pruned

4decdcf

tuxcanfly force-pushed the index-plugin branch from cfc50f6 to 4decdcf Compare December 6, 2018 18:03

pinheadmz mentioned this pull request Jan 22, 2019

Alternate database backend. #275

Closed

tuxcanfly mentioned this pull request Mar 13, 2019

blockstore: index transactions braydonf/bcoin#3

Closed

braydonf mentioned this pull request Mar 25, 2019

Add file block storage #703

Merged

braydonf added the indexer label Mar 26, 2019

bucko13 reviewed Apr 2, 2019

View reviewed changes

lib/indexer/indexer.js Show resolved Hide resolved

tuxcanfly closed this Apr 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor indexers #424

refactor indexers #424

tuxcanfly commented Mar 1, 2018 •

edited

Loading

tuxcanfly commented Mar 13, 2018

bucko13 left a comment

bucko13 Mar 13, 2018

tuxcanfly Mar 13, 2018

bucko13 Mar 13, 2018

tuxcanfly Mar 13, 2018

bucko13 Mar 13, 2018

tuxcanfly Mar 13, 2018

bucko13 Mar 13, 2018

tuxcanfly Mar 13, 2018

tuxcanfly Apr 19, 2018

tuxcanfly commented Mar 20, 2018

bucko13 Mar 26, 2018

tuxcanfly Mar 26, 2018

tuxcanfly commented Mar 28, 2018

chjj commented Mar 30, 2018

tuxcanfly commented Mar 31, 2018

nodech commented Mar 31, 2018

tuxcanfly commented Apr 4, 2018

tuxcanfly commented Apr 10, 2018

braydonf Aug 17, 2018

braydonf Aug 17, 2018

braydonf commented Aug 23, 2018 •

edited

Loading

tuxcanfly commented Oct 28, 2018

tuxcanfly commented Dec 6, 2018

braydonf commented Dec 10, 2018 •

edited

Loading

tuxcanfly commented Dec 11, 2018

braydonf commented Mar 27, 2019 •

edited

Loading

tuxcanfly commented Apr 13, 2019

refactor indexers #424

refactor indexers #424

Conversation

tuxcanfly commented Mar 1, 2018 • edited Loading

tuxcanfly commented Mar 13, 2018

bucko13 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tuxcanfly commented Mar 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tuxcanfly commented Mar 28, 2018

chjj commented Mar 30, 2018

tuxcanfly commented Mar 31, 2018

nodech commented Mar 31, 2018

tuxcanfly commented Apr 4, 2018

tuxcanfly commented Apr 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

braydonf commented Aug 23, 2018 • edited Loading

tuxcanfly commented Oct 28, 2018

tuxcanfly commented Dec 6, 2018

braydonf commented Dec 10, 2018 • edited Loading

tuxcanfly commented Dec 11, 2018

braydonf commented Mar 27, 2019 • edited Loading

tuxcanfly commented Apr 13, 2019

tuxcanfly commented Mar 1, 2018 •

edited

Loading

braydonf commented Aug 23, 2018 •

edited

Loading

braydonf commented Dec 10, 2018 •

edited

Loading

braydonf commented Mar 27, 2019 •

edited

Loading