CHANGELOG.md

Bcoin release notes & changelog

Important: Any time you upgrade bcoin, you must execute npm rebuild before restarting.

unreleased

Wallet changes

• A new u bucket is added to txdb that stores only spendable coins indexed by account and value.
• A new multi-algorithm coin selection. (Lowest Larger, Branch and Bound, and Single Random Draw)

How to upgrade

To upgrade wallet, we need to copy coins from c bucket to u bucket in txdb.

To do this you can run:

node ./migrate/walletdb7to8.js /path/to/bcoin/wallet

Node API changes

RPC

• Added getnodeaddresses which returns entries from the hostlist.

Indexer changes

• Support for multiple compact filters added as per BIP158 specification. To enable the support of multiple filters the data previously in ~/.bcoin/index/filter has been moved to ~/.bcoin/index/filter/BASIC.
• The filter indexer now also stores the filter hashes corresponding to the blockfilters.

How to upgrade

To upgrade to the multiple compact filter support we need to move filter data from ~/.bcoin/index/filter to ~/.bcoin/index/filter/BASIC and generate filter hashes for the existing filters.

To do this you can run:

node ./migrate/indexerdb0to1.js /path/to/bcoin/index

v2.2.0

• Support for bech32m has been added. bcoin can now validate and send BTC to addresses for witness programs with versions > 0. The address indexer has also been updated to retrieve the new addresses. A bug was fixed where the indexer may return data for a witness version 0 address even if a version 1 address was queried, if the two addresses had the same data (hash). After upgrading bcoin, a full rescan may be necessary to re-index these collisions correctly. To accomplish this:

  • Stop bcoin (bcoin-cli rpc stop or ctrl-C)
  • Delete the existing addr indexer data (rm -rf ~/.bcoin/index/addr)
  • Restart bcoin
  • The address indexer will automatically begin re-indexing the chain. It may take up to 24 hours.

• The logging module blgr has been updated. Log files will now be rolled over at around 20 MB and timestamped. Only the last 10 log files will be kept on disk and older log files will be purged. These values can be configured by passing --log-max-file-size (in MB) and --log-max-files.

v2.1.2

• Fixed wallet RPC method importprunedfunds.

v2.1.1

• Fixed a bug with the wallet 'conflict' event.

v2.1.0

• Added an endpoint to the wallet HTTP API at GET / for an aliveness check. It returns the current version and network type.

v2.0.0

How to upgrade

The way that block data is stored has changed for greater performance, efficiency, reliability and portability.

• Block and undo block data has been moved from LevelDB into flat files.
• The transaction and address indexes have been moved into separate LevelDB databases.
• The transaction index has been de-duplicated, and will reduce disk usage by half for those running with txindex enabled.
• The txindex and addrindex can now be enabled after the initial block download.
• The addrindex has been sorted to support querying for large sets of results, and will no longer cause CPU and memory exhaustion issues.
• The addrindex will correctly distinguish between p2pkh and p2wpkh addresses.
• BIP158 compact block filters can be computed and indexed (filterindex).

To upgrade to the new disk layout it's necessary to move block data from LevelDB (e.g. ~/.bcoin/chain) to a new file-based block storage (e.g. ~/.bcoin/blocks), and remove txindex and addrindex data from the chain database, for those that have that feature enabled.

To do this you can run:

node ./migrate/chaindb4to6.js /path/to/bcoin/chain

The migration will take 1-3 hours, depending on hardware. The block data will now be stored at /path/to/bcoin/blocks. After the data has been moved the chain database will be compacted to free disk space.

Alternatively, you can also sync the chain again, however the above migration will be faster as additional network bandwidth won't be used for re-downloading the blocks.

For those with txindex and addrindex enabled, the indexes will be regenerated by rescanning the chain on next startup. This process can take multiple hours (e.g. 8 hours) depending on hardware and the index. Please take the potential downtime in re-indexing into account before upgrading.

Client API changes

• Client module /lib/client was created from the external dependency bclient.
• NodeClient method estimateFee now returns JSON instead of a number.
• Added fee command to bcoin-cli.
• Added getBlockHeader to NodeClient and corresponding header command to CLI.
• Added getFilter to NodeClient and corresponding filter command to CLI.

Wallet API changes

HTTP

• PUT /wallet/:id Creating a watch-only wallet now requires an accountKey argument. This is to prevent bcoin from generating keys and addresses the user can not spend from.
• POST /wallet/:id/create
  • Now has a sign argument for optional signing of transactions.
  • Now has a template option, that will skip templating inputs when sign = false, but you can enable it if necessary. It does not have an effect when sign = true.
  • Exposes blocks, which will be used for fee estimation if rate is not set.
  • Exposes sort (Default true), that can be used to disable BIP69 sorting.

RPC

• Bug fix addresses for the getnewaddress command with various networks.
• Deprecate the ismine and iswatchonly fields from the validateaddress command and add isscript, iswitness, ischange, witness_version and witness_program to partially match the v0.18.0 Bitcoin Core release (26a2000b0177fd2668b7d82e5aa52829cf2bfdf6)
• Add wallet RPC getaddressinfo to return ismine and iswatchonly with the correct values instead of their previous values which were hard-coded to false. Also returns address, scriptPubKey, isscript, iswitness, witness_version and witness_program. (a28ffa272a3c4d90d0273d9aa223a23becc08e0e)

Node API changes

HTTP

Several CPU and memory exhaustion issues have been resolved with some additional arguments for querying multiple sets of results for addresses that have many transactions.

• GET /tx/address/:address has several new arguments: after, reverse and limit. The after argument is a txid, for querying additional results after a previous result. The reverse argument will change the order that results are returned, the default order is oldest to latest. The limit argument can be used to give results back in smaller sets if necessary.
• POST /tx/address has been deprecated. Instead, query for each address individually with GET /tx/address/:address with the expectation that there could be many results that would additionally need to be queried in a subsequent query using the after argument to request the next set.
• POST /coin/address and GET /coin/address/:address are deprecated as coins can be generated using results from /tx/address/:address and querying by only a range of the latest transactions to stay synchronized. Coins could otherwise be removed from results at any point, and thus the entire set of results would need to be queried every time to discover which coins have been spent and are currently available.
• GET / has new fields .indexes.{addr,tx,filter} for the status of indexers.

RPC

• Both getblock and getblockheader return bits in base 16 as a string instead of base 10 and a number.

Network changes

• Regtest params have been updated to correspond with other bitcoin implementations for cross-testing. Key prefixes have been updated and SegWit is always active (39684df6051b599da9ed1383cb7e3f8605806a74). The regtest seeds have also been cleared so that the node will not attempt to connect to itself.
• Testnet seeds have been updated (8b6eba165b090edfc1ff5dbc9c422758d60b6342)

Logging changes

• Wallet and node HTTP and RPC is now more clearly identified. The wallet HTTP server will log with the context wallet-http and the RPC as wallet-rpc. The node HTTP will log with the context node-http and the RPC as node-rpc. There was also a reduction in the duplicate logs in the case of an error for the node RPC, and the addition of method logging for the wallet RPC.
• There is now also additional logging during wallet rescans.

Configuration changes

• Pool options now has a --discover option exposed, and the --only node option will disable discovery of new nodes.
• The option for coin-cache has been removed. This setting was causing issues during sync with out-of-memory errors and was making performance worse instead of better.
• The database location for indexes can be configured via the --index-prefix option. Default locations are prefix + /index (e.g. ~/.bcoin/testnet/index/tx and ~/.bcoin/testnet/index/addr).

Script changes

• Script has been updated for policy flags. For example VERIFY_CONST_SCRIPTCODE to disable OP_CODESEPARATOR in non-SegWit scripts and make a positive findAndDelete result invalid.
• SegWit scripts that terminate with a stack size not equal to one now have an error of CLEANSTACK instead of EVAL_FALSE.
• VERIFY_UPGRADABLE_NOPS now does not apply to OP_CHECKLOCKTIMEVERIFY and OP_CHECKSEQUENCEVERIFY.

Testing changes

• Switched to use a security-focused rewrite of mocha called bmocha, for further details see: https://github.com/bcoin-org/bmocha
• Data has been updated to be in sync with other implementations around policy-only script validation.
• Tests now cleanly close with all timers being cleared.
• Configuration file wallet.conf won't be read during test runs, which was causing issues with some testing environments.

Chain changes

• The transaction index methods are now implemented at node.txindex:
  • getMeta(hash)
  • getTX(hash)
  • hasTX(hash)
  • getSpentView(tx)
• The address index method getHashesByAddress is now implemented at node.addrindex:
  • getHashesByAddress(addr) It now accepts Address instances rather than Address|String and the results are now sorted in order of appearance in the blockchain.
  • getHashesByAddress(addr, options) A new options argument has been added with the fields:
    • after - A transaction hash for results to begin after.
    • limit - The total number of results to return at maximum.
    • reverse - Will give results in order of latest to oldest.
• The following methods require node.addrindex.getHashesByAddress in conjunction with node.txindex.getTX and node.txindex.getMeta respectively, and now includes a new options argument as described above for getHashesByAddress:
  • node.getMetaByAddress(addr, options)
  • node.getTXByAddress(addr, options)
• The following method has been deprecated:
  • getCoinsByAddress(addr)

Other changes

• A new module for storing block data in files.
• Use of buffer-map for storing hashes (see bcoin-org#533).
• Use of bsert for assertions.
• SIGINT handling will close the full node, spvnode and wallet.
• Using the bcoin library in a REPL now has auto-completion by pressing tab.
• Various documentation updates.
• Mempool fix to add non-standard (non-SegWit) to reject cache.
• Dependencies are committed to the repository and bcoin does not depend on npm.
• Updates to dependencies including bcrypto to version > 5.
• Various small fixes to run bcoin in a browser.

v1.0.0

Migration

The latest bcoin release contains almost a total rewrite of the wallet. A migration is required.

Bcoin has also been modularized quite a bit. Users installing globally from npm should be sure to run:

$ npm update -g bcoin

For users who cloned the respository themselves, it might be wise to simply remove the node_modules directory and do a fresh npm install.

$ cd bcoin/
$ git pull origin master
$ rm -rf node_modules/
$ npm install
$ npm test

---

Once upgrading is complete, a migration script must be executed. The migration script resides in ./migrate/latest as an executable file.

A bcoin prefix directory must be passed as the first argument.

Example:

$ ./migrate/latest ~/.bcoin

What this does:

• Renames chain.ldb to chain.
• Renames spvchain.ldb to spvchain.
• Renames walletdb.ldb to wallet.
• Runs a migration (chaindb3to4.js) on chain.
• Runs a migration (chaindb3to4.js) on spvchain.
• Runs a migration (walletdb6to7.js) on wallet.

The chain migration should be minor, and only taxing if you have --index-address enabled.

If you have a testnet, regtest, or simnet directory, the migration must also be run on these, e.g.

$ ./migrate/latest ~/.bcoin/testnet

Also note that the --prefix argument now always creates a fully-fledged prefix directory. For example --prefix ~/.bcoin_whatever --network testnet will create a directory structure of ~/.bcoin_whatever/testnet/ instead of ~/.bcoin_whatever. Please update your directory structure accordingly.

Configuration changes

Wallet and Node are now separated and use different persistent configuration files. Some configuration options have moved to wallet.conf, which is stored in the prefix directory next to the existing bcoin.conf.

All configuration options in wallet.conf will have the wallet- prefix added implicitly. The prefix is still needed when using CLI/ENV configuration methods. The wallet now has it's own HTTP server, so options such as wallet-http-host must also be specified along with http-host if you require this setting (required for Docker networking).

Wallet-specific settings such as api-key and wallet-auth have moved to wallet.conf. If using CLI/ENV options, these are prefixed with wallet-. Various configuration method examples are shown below. Note some config options (eg. network) are automatically passed to wallet plugin if specified through CLI/ENV, but should be specified in wallet.conf for bwallet-cli.

Example using wallet.conf:

network: testnet
wallet-auth: true
api-key: hunter2
http-host: 0.0.0.0

Example using CLI options: ./bin/node --network=testnet --http-host=0.0.0.0 --wallet-http-host=0.0.0.0 --wallet-api-key=hunter2 --wallet-wallet-auth=true

Example using ENV: BCOIN_NETWORK=simnet BCOIN_HTTP_HOST=0.0.0.0 BCOIN_WALLET_HTTP_HOST=0.0.0.0 BCOIN_WALLET_API_KEY=hunter2 BCOIN_WALLET_WALLET_AUTH=true ./bin/node

Bcoin client changes

The bcoin cli interface has been deprecated and is now replaced with a separate tool: bcoin-cli. bcoin wallet has also been deprecated and is replaced with bwallet-cli. Both bcoin-cli and bwallet-cli can be installed with the bclient package in npm.

Wallet API changes

The main wallet API changes have to do with changes in the structure of the HTTP server itself. The bcoin wallet HTTP server will now always listen on it's own port. Standard ports are:

main: 8334
testnet: 18334
regtest: 48334
simnet: 18558

The default account object is no longer present on the wallet. To retrieve it, request /wallet/:id/account/default instead. More minor changes to the JSON output include the removal of id and wid properties on a number of objects. Please update your code accordingly.

For socket.io events, all wallet prefixes have been removed. For example, wallet join should be replaced with simply join. All wallet-related events (tx, confirmed, etc) now receive the wallet ID as the first argument. The tx event will receive [id, details] instead of [details].

The _admin endpoints have been moved up a level. Instead of requesting /wallet/_admin/backup (for example), request /backup instead. If --wallet-auth is enabled, a new flag --admin-token is accepted. Admin token is a 64 character hex string and allows access to all admin routes. For example, GET /backup?token=xxx.... It also allows access to any wallet on the HTTP or websocket layer (websockets who use the admin token can join a wallet of *, which notifies them of events on all wallets).

Notable changes

• wallet/http - A conflict event will be correctly emitted for all double spends.
• wallet/http - Account balances are now indexed and will appear on account objects via the API.
• bcoin - Bcoin has been rewritten using class syntax. Objects will not be inheritable with legacy prototypal inheritance.
• pkg - Bcoin is once again buildable with browserify.
• pkg - Numerous modules have been moved out to separate dependencies. For example, the lib/crypto module has been moved to the bcrypto package (and also heavily optimized even further).
• bcoin - The main bcoin module has been rewritten in terms of what it exposes and how. Conventions have changed (Address instead of address). Due to the use of class syntax, something like bcoin.Address(str) is no longer possible. Please take a look at lib/bcoin.js to get a better idea of what has changed.
• net - Support for BIP151 and BIP150 has been dropped.

v1.0.0-beta.15

Notable changes

v1.0.0-beta.14

Notable changes

• pkg - Ignored bcoin* files in npmignore have been removed. This fixes the npm install.

v1.0.0-beta.13

Notable changes

• config - Options using megabyte units are now calculated properly again (6182df044228f9215938e7d314435f3f2640acca, a630d23a97b68f189a85105856fedc4e9e515754, 7728a0047053d4c368e60426e5fc7cc812d54caf).

• address - Bech32 addresses are now supported (6acef06cbc87a3051ba238a2fb640562e718135e). This changes the semantics of the Address object: to support bech32, Address.fromBase58 calls should be replaced with Address.fromString. Likewise, addr.toBase58 calls should be replaced with addr.toString

• rpc - getblockbyheight is now exposed via JSON-RPC. It takes the same parameters as the getblock call, requiring a height instead of block hash (12d3ee6f9582afa9a3ba8984c63dcbc27b8db57e).

• bin - bcoin --version and bcoin --help now exit with appropriate messages (f2f94a800e37c5dbdda6920fa6b85fbd485c212a).

• net - The p2p pool now exposes an --only option (a1d0948f2e528c5d77d6502659fafd064b1e693b).

• mempool - The mempool indexAddress option should now work correctly (fba9b46d253c19bbf8e662d9d75ab03dc9e20a78).

• rpc - JSON-RPC calls now properly cast booleans a la bitcoin core (dd49ee076196d2353783e3044185165dbac4aeb9).

• rpc - Various RPC calls have been improved and updated (c78707ed6a71ce46c41c4e43ecb505e78a84b380, c1e684dc12d0a86573a905d54d4f81fce921987a, 5bde338a53117b1bd0fd92df0abc23d95180ab32).

• rpc - Retroactive pruning is now available via the pruneblockchain call (f0bc6d5925419ba4a8289fa7828efc48ecc152d4).

• http - Getting block by height via the Rest API now works again (df4c8cc68c965bd818a5004354d2652751d4a702).

• net - Peers who serve invalid orphans are now punished properly (0ceca23cb5a3d724c79c6bf587ede5d473df8486).

• utils - An implementation of GCS filters is now supported (b994c278f25408886c3095d0c24123baaf07f78f).

• http - The witness option is now properly exposed on the Rest API for segwit wallets (f04ad612b216becd35765c6e231f7820c7eee358).

• deps - Node.js >=v7.6.0 is now a required dependency (a0bd9680fed07c5eb37c227d160b0868f8adaf31).

• build - The browser build has switched from browserify to webpack (19f236f74072d473123d20282d2119f6d9130458).

• bcoin - The codebase has been rewritten to use all ES6 and ES7 features supported by node.js (aa05bb5df79d9a3af53060a4c0c066226f6e9e4c, 78d62c73b82e1953999d1cf80c90ed2035d4996e, e00472891df5934d8fc3aa63662f852816aa86b0, c53f4cf89e46d9de8ab7f65430310567558fe03f, 8c7279518f5341a2482a79ac98f0574468541edc).

• workers - The worker pool has been refactored to solve the dissonance between the browser and node.js (27c60ce76e57af1695d78f912227d93194812c88).

• net - DNS requests should now timeout sooner (647b6909c6d527eb82f6d789c88a23b2f8a60126).

• http - Satoshi values are now required for all JSON serialization, as opposed to BTC strings which were used in the past (2f51fd1c5066f194a5a52383f4dd45497b882706).

• bin - The --no-wallet flag is now exposed for nodes who want to run without a wallet.

• chain - BIP91 support is now exposed via the --bip91 option. The segwit and segsignal rules will be automatically added when calling getblocktemplate. To enable bip91 on an existing database --force-flags must be passed. This will force bip91 to be enforced, but will not invalidate any previous blocks (bcoin cli reset [height] may need to be used).

• chain - BIP148 support is now exposed via the --bip148 option. This likewise must be enabled with --force-flags on existing chains. This has the same potential "reset" implications as the --bip91 option.

Migrating

This release contains a few non-backward-compatible changes.

Bcoin now requires node.js >=7.6.0 due to the use of ES6/7 features, and for the sake of more stability.

Bcoin's rest API now assumes satoshi values for in all JSON serialization. This is a breaking change for code that is not aware of it. All code which hits the rest API must be updated to use only satoshi values as opposed to BTC strings.

In other words, this:

{
  "unconfirmed": "1.12",
  "confirmed": "1.12"
}

Becomes this:

{
  "unconfirmed": 112000000,
  "confirmed": 112000000
}

v1.0.0-beta.12

Notable changes

• networks - Fixed simnet wpkh prefix.
• http - wallet join without wallet auth has been fixed for responses. This was causing a hanging issue with the client.

v1.0.0-beta.11

Notable changes

• networks - Simnet params have been fixed.
• cli - Chain reset call has been fixed.

v1.0.0-beta.10

Notable changes

• wallet/http - Create wallet route modified (POST /wallet/:id? changed to PUT /wallet/:id).
• wallet/http - Create account route modified (POST /wallet/:id/account/:account? changed to PUT /wallet/:id/account/:account).
• wallet/http - auth socket.io event name for wallet auth changed to wallet auth.
• config - payout-address option was changed to coinbase-address.
• node - Plugin system is now exposed. See bcoin-org#156.
• config - The internal API for the config object has been rewritten and is now more reusable, particularly by node plugins.
• http/rpc - Both the HTTPBase and RPCBase objects now allow "mounting" by other rpc and http servers.
• wallet - The wallet code has been completely removed from the node, and now resides entirely within one module. The wallet is exposed only as a plugin or a separate server.
• rpc - prioritisetransaction is now exposed properly (deltaFees are now tracked on mempool entries).
• rpc - Proper id and error codes are now implemented.
• rpc - Several getblocktemplate improvements have been implemented for more accuracy. e.g. curtime will now be updated each call.
• mining - The internal miner API has been rewritten, and now mimics stratum in a sense.
• chain - Faster verification with checkpoints.
• net - Fixed a potential block stalling issue.
• net - Hardcoded seeds for main added. Makes for better shipping with browsers.
• wsproxy/proxysocket - DNS resolution is no longer exposed.
• logger - Log files now trim to 20mb on boot.
• hostlist - A persistent hosts file is now written by default.

v1.0.0-beta.9

Notable changes

• mempool - Trimming now removes dependency chains by cumulative fee rate.
• mempool - Cumulative descendant fees are now updated properly when removing a transaction.
• net - Preliminary upnp support for adding port mappings.
• chain/mempool/miner - Various atomicity fixes and extra sanity checking.
• pool/peer - Peer height is now tracked and exposed on the RPC as bestheight.

v1.0.0-beta.8

Notable changes

• mempool - Fixed critical fee estimator bug causing throwing in the mempool.

v1.0.0-beta.7

Notable changes

• http - Always display spent coins in tx routes (e.g. /tx/[txid]).
• mempool - An on-disk mempool is now exposed via --persistent-mempool (also makes fee data persistent).
• chain - chain.add now takes a flags parameter to avoid POW and non-contextual checks if necessary.
• net - HostList is now potentially persistent with a hostLocation option.
• net - Smarter stall behavior and timeouts.

v1.0.0-beta.6

Notable changes

• http - Better bitcoind compatability for JSON-RPC.

v1.0.0-beta.5

Notable changes

• miner - Better fee rate comparisons.
• deps - Upgrade deps, fix build on arm and windows.

v1.0.0-beta.4

Notable changes

• miner - Optimized TX sorting.
• rpc - Improved getblocktemplate to provide more accurate results to bitcoind.

v1.0.0-beta.3

Notable changes

• miner - Improved fee rate sorting.
• rpc - Fix incompatibilities in rpc api (getblocktemplate & submitblock).

v1.0.0-beta.2

Notable changes

• pool - Increase max header chain failures to 500 (prevents the initial sync from reverting to getblocks so frequently).

v1.0.0-beta.1

Notable changes

• wsproxy: Fixed proof of work handling in websocket proxy (43c491b).
• chain: Optimized MTP and network target calculations (1e07d1b).
• wallet: Implemented "smart" coin selection (304f0e7e).
• protocol: Increased default network fees for main (09c2357).
• http: Fix for handling DELETE http method (393dd5d).
• miner: Improved handling of default reserved size and sigops (f2964e0 and 7104e4c).

v1.0.0-beta

Notable changes

• Initial tagged release.