Indexer fixes and improvements

CHANGELOG.md

@@ -5,13 +5,28 @@
 ### How to upgrade
 
 The way that block data is stored has changed for greater performance,
-efficiency, reliability and portability. To upgrade to the new disk layout
-it's necessary to move block data from LevelDB (e.g. `~/.bcoin/chain`) to
-a new file based block storage (e.g. `~./.bcoin/blocks`).
+efficiency, reliability and portability.
+
+- Block and undo block data has been moved from LevelDB into flat files.
+- The transaction and address indexes have been moved into separate
+  LevelDB databases.
+- The transaction has been de-duplicated, and will reduce disk usage by
+  half for those running with `txindex` enabled.
+- The `txindex` and `addrindex` can now be enabled after the initial
+  block download.
+- The `addrindex` has been sorted to support querying for large sets
+  of results, and will no longer cause CPU and memory exhaustion issues.
+- The `addrindex` will correctly distinguish between `p2pkh` and
+  `p2wpkh` addresses.
+
+To upgrade to the new disk layout it's necessary to move block data
+from LevelDB (e.g. `~/.bcoin/chain`) to a new file based block
+storage (e.g. `~./.bcoin/blocks`), and remove `txindex` and `addrindex`
+data from the chain database, for those that have that feature enabled.
 
 To do this you can run:
 ```
-node ./migrate/chaindb4to5.js /path/to/bcoin/chain
+node ./migrate/chaindb4to6.js /path/to/bcoin/chain
 ```
 
 The migration will take 1-3 hours, depending on hardware. The block data
@@ -22,6 +37,12 @@ Alternatively, you can also sync the chain again, however the above
 migration will be faster as additional network bandwidth won't be used
 for downloading the blocks again.
 
+For those with `txindex` and `addrindex` enabled, the indexes will be
+regenerated by rescanning the chain on next startup, this process can
+take multiple hours (e.g. 8 hours) depending on hardware and the
+index. Please take the potential downtime in re-indexing into account
+before upgrading.
+
 ### Wallet API changes
 
 #### HTTP
@@ -50,6 +71,31 @@ for downloading the blocks again.
   `iswitness`, `witness_version` and `witness_program`.
   (a28ffa272a3c4d90d0273d9aa223a23becc08e0e)
 
+### Node API changes
+
+#### HTTP
+
+Several CPU and memory exhaustion issues have been resolved with some
+additional arguments for querying multiple sets of results for addresses
+that have many transactions.
+
+- `GET /tx/address/:address` has several new arguments: `after`, `reverse`
+  and `limit`. The `after` argument is a txid, for querying additional results
+  after a previous result. The `reverse` argument will change the order that
+  results are returned, the default order is oldest to latest. The `limit`
+  argument can be used to give results back in smaller sets if necessary.
+- `POST /tx/address` This has been deprecated, instead query for each address
+  individually with `GET /tx/address/:address` with the expectation that
+  there could be _many_ results that would additionally need to be queried
+  in a subsequent query using the `after` argument to request the next set.
+- `POST /coin/address` and `GET /coin/address/:address` are deprecated as
+  coins can be generated using results from `/tx/address/:address` and
+  querying by only a range of the latest transactions to stay synchronized.
+  Coins could otherwise be removed from results at any point, and thus the
+  entire set of results would need to be queried every time to discover
+  which coins have been spent and are currently available.
+- `GET /` has new fields `.indexes.{addr,tx}` for the status of indexers.
+
 ### Network changes
 
 - Regtest params have been updated to correspond with other bitcoin
@@ -76,6 +122,9 @@ for downloading the blocks again.
 - The option for `coin-cache` has been removed, this setting was causing
   issues during the sync with out-of-memory errors and was making performance
   worse instead of better.
+- The database location for indexes can be configured via the
+  `--index-prefix` option. Default locations are `prefix` + `/index`
+  (e.g. `~/.bcoin/testnet/index/tx` and `~/.bcoin/testnet/index/addr`).
 
 ### Script changes
 
@@ -97,6 +146,32 @@ for downloading the blocks again.
 - Config file `wallet.conf` won't be read during test runs that was
   causing issues with some testing environments.
 
+### Chain changes
+
+- The transaction index methods are now implemented at `node.txindex`:
+  - `getMeta(hash)`
+  - `getTX(hash)`
+  - `hasTX(hash)`
+  - `getSpentView(tx)`
+- The address index method `getHashesByAddress` is now implemented
+  at `node.addrindex`:
+  - `getHashesByAddress(addr)` It now accepts `Address` instances
+    rather than `Address|String` and the results are now sorted in
+    order of appearance in the blockchain.
+  - `getHashesByAddress(addr, options)` A new options argument has
+    been added with the fields:
+    - `after` - A transaction hash for results to begin after.
+    - `limit` - The total number of results to return at maximum.
+    - `reverse` - Will give results in order of latest to oldest.
+- The following methods require `node.addrindex.getHashesByAddress`
+  in conjunction with `node.txindex.getTX` and `node.txindex.getMeta`
+  respectively, and now includes a new options argument as described
+  above for `getHashesByAddress`:
+  - `node.getMetaByAddress(addr, options)`
+  - `node.getTXByAddress(addr, options)`
+- The following method has been deprecated:
+  - `getCoinsByAddress(addr)`
+
 ### Other changes
 
 - A new module for storing block data in files.

lib/bcoin-browser.js

@@ -55,6 +55,12 @@ bcoin.HDPrivateKey = require('./hd/private');
 bcoin.HDPublicKey = require('./hd/public');
 bcoin.Mnemonic = require('./hd/mnemonic');
 
+// Index
+bcoin.indexer = require('./indexer');
+bcoin.Indexer = require('./indexer/indexer');
+bcoin.TXIndexer = require('./indexer/txindexer');
+bcoin.AddrIndexer = require('./indexer/addrindexer');
+
 // Mempool
 bcoin.mempool = require('./mempool');
 bcoin.Fees = require('./mempool/fees');

lib/bcoin.js

@@ -76,6 +76,12 @@ bcoin.define('HDPrivateKey', './hd/private');
 bcoin.define('HDPublicKey', './hd/public');
 bcoin.define('Mnemonic', './hd/mnemonic');
 
+// Index
+bcoin.define('indexer', './indexer');
+bcoin.define('Indexer', './indexer/indexer');
+bcoin.define('TXIndexer', './indexer/txindexer');
+bcoin.define('AddrIndexer', './indexer/addrindexer');
+
 // Mempool
 bcoin.define('mempool', './mempool');
 bcoin.define('Fees', './mempool/fees');

lib/blockchain/chain.js

@@ -1948,75 +1948,6 @@ class Chain extends AsyncEmitter {
     return this.db.getBlockView(block);
   }
 
-  /**
-   * Get a transaction with metadata.
-   * @param {Hash} hash
-   * @returns {Promise} - Returns {@link TXMeta}.
-   */
-
-  getMeta(hash) {
-    return this.db.getMeta(hash);
-  }
-
-  /**
-   * Retrieve a transaction.
-   * @param {Hash} hash
-   * @returns {Promise} - Returns {@link TX}.
-   */
-
-  getTX(hash) {
-    return this.db.getTX(hash);
-  }
-
-  /**
-   * @param {Hash} hash
-   * @returns {Promise} - Returns Boolean.
-   */
-
-  hasTX(hash) {
-    return this.db.hasTX(hash);
-  }
-
-  /**
-   * Get all coins pertinent to an address.
-   * @param {Address[]} addrs
-   * @returns {Promise} - Returns {@link Coin}[].
-   */
-
-  getCoinsByAddress(addrs) {
-    return this.db.getCoinsByAddress(addrs);
-  }
-
-  /**
-   * Get all transaction hashes to an address.
-   * @param {Address[]} addrs
-   * @returns {Promise} - Returns {@link Hash}[].
-   */
-
-  getHashesByAddress(addrs) {
-    return this.db.getHashesByAddress(addrs);
-  }
-
-  /**
-   * Get all transactions pertinent to an address.
-   * @param {Address[]} addrs
-   * @returns {Promise} - Returns {@link TX}[].
-   */
-
-  getTXByAddress(addrs) {
-    return this.db.getTXByAddress(addrs);
-  }
-
-  /**
-   * Get all transactions pertinent to an address.
-   * @param {Address[]} addrs
-   * @returns {Promise} - Returns {@link TXMeta}[].
-   */
-
-  getMetaByAddress(addrs) {
-    return this.db.getMetaByAddress(addrs);
-  }
-
   /**
    * Get an orphan block.
    * @param {Hash} hash
@@ -2057,21 +1988,6 @@ class Chain extends AsyncEmitter {
     return this.db.getCoinView(tx);
   }
 
-  /**
-   * Get coin viewpoint (spent).
-   * @param {TX} tx
-   * @returns {Promise} - Returns {@link CoinView}.
-   */
-
-  async getSpentView(tx) {
-    const unlock = await this.locker.lock();
-    try {
-      return await this.db.getSpentView(tx);
-    } finally {
-      unlock();
-    }
-  }
-
   /**
    * Test the chain to see if it is synced.
    * @returns {Boolean}
@@ -2687,8 +2603,6 @@ class ChainOptions {
     this.bip91 = false;
     this.bip148 = false;
     this.prune = false;
-    this.indexTX = false;
-    this.indexAddress = false;
     this.forceFlags = false;
 
     this.entryCache = 5000;
@@ -2771,16 +2685,6 @@ class ChainOptions {
       this.prune = options.prune;
     }
 
-    if (options.indexTX != null) {
-      assert(typeof options.indexTX === 'boolean');
-      this.indexTX = options.indexTX;
-    }
-
-    if (options.indexAddress != null) {
-      assert(typeof options.indexAddress === 'boolean');
-      this.indexAddress = options.indexAddress;
-    }
-
     if (options.forceFlags != null) {
       assert(typeof options.forceFlags === 'boolean');
       this.forceFlags = options.forceFlags;