channeldb: specify freelist bbolt options by default

[Crypt-iQ]

Fixes #3241.

These options could really be specified anywhere that bbolt is opened, so if that's of interest I can also do that. But the only other large database I had on my machine was wallet.db which is the btcwallet repo. Perhaps watchtowers might use a lot of space, or there could be some design parameters that change in the future where the other databases start using more space (macaroons.db etc). Should I add these options to all the tests so they reflect the new behavior?

Updated bbolt dependency to v1.3.3 per etcd-io/bbolt#153
The only downside is that if the db txn has to be rolled back in case of failure, the freelist must be built again temporarily which can be heavy with big databases. I think this is an acceptable tradeoff since this shouldn't be happening too often.

[cfromknecht]

. But the only other large database I had on my machine was wallet.db which is the btcwallet repo. Perhaps watchtowers might use a lot of space, or there could be some design parameters that change in the future where the other databases start using more space (macaroons.db etc).

Indeed btcwallet and towers would also be good candidates for these settings! For towers we can do this on the client and tower side dbs.

Updated bbolt dependency to v1.3.3 per

Perfect, yeah i hit this panic in testing with the current version

[cfromknecht]

small nit: swap the order of the commits so bbolt is updated first

[Crypt-iQ]

@cfromknecht Addressed comment and added the options to watchtower client + server. Can make a PR for btcwallet next. For testing, I'm wondering if the options should also be specified in unit/itests? If so, in order to reduce code duplication, I suppose I could add the options to channeldb/options.go similar to the way the cache sizes are setup?

[wpaulino]

For testing, I'm wondering if the options should also be specified in unit/itests? If so, in order to reduce code duplication, I suppose I could add the options to channeldb/options.go similar to the way the cache sizes are setup?

I'd say yes in order to exercise what would be used in production. Adding the options wouldn't remove the duplication in btcwallet, but seems fine to do for the other databases within lnd.

[cfromknecht]

@Crypt-iQ since the options are set inside the Open method, wouldn't these already be used in testing as well? Which cases do you think are still missing?

[Crypt-iQ]

@Crypt-iQ since the options are set inside the Open method, wouldn't these already be used in testing as well? Which cases do you think are still missing?

Hey you're right, I meant the other databases (sphinxreplay, macaroons) and some of the temporary test databases. I didn't look at the results of my search very closely, mb

[cfromknecht]

Hey you're right, I meant the other databases (sphinxreplay, macaroons) and some of the temporary test databases. I didn't look at the results of my search very closely, mb

okay cool then i think we're good. sphinxreplay and macaroon dbs are less of a priority as they're much smaller, but not opposed to adding them as well

[Crypt-iQ]

Hey you're right, I meant the other databases (sphinxreplay, macaroons) and some of the temporary test databases. I didn't look at the results of my search very closely, mb

okay cool then i think we're good. sphinxreplay and macaroon dbs are less of a priority as they're much smaller, but not opposed to adding them as well

The freelist on my 8MB sphinxreplay db has 5 free pages on it... so I don't think I'll include it here. And my macaroons.db is tiny.

[cfromknecht]

LGTM 🚀will start running this version to ensure there aren't any other regressions

[cfromknecht]

@Crypt-iQ decided we should push this off to 0.8, since it's a pretty big change for a point release and want to leave ample time for testing

[Crypt-iQ]

@cfromknecht no problem, I'm looking into the InitialMmapSize option as well, so might be good to hold off until I can determine if it's worth adding

[cfromknecht]

@Crypt-iQ sounds good, yeah we haven't done much exploration wrt bbolt options, would be interested to hear what you find!

[Crypt-iQ]

Looked at InitialMmapSize and some of the related bbolt code. Bbolt uses mmap to map the db file and sometimes it needs to be remapped during write tx's via munmap and mmap. In these cases, the write transaction that is referencing the old mmap needs to copy these references to the heap. So IMO it seems like we should just set a high InitialMmapSize and never have to remap. Certain write transactions could cause high memory allocations on the heap depending on how many nodes they access during the txn.

It is also the case that long-running read transactions can block certain writes from happening. The read txns hold a lock on the mmap and writes that need to remap also need the lock, thus they are blocked. See: boltdb/bolt#472

The creator of boltdb suggested an mmap size of 1TB (boltdb/bolt#489 (comment)) but I think we could make do with much less given the size of the channel graph (my channel.db is only ~2G). It is virtual memory after all so we could set the InitialMmapSize to 50GB or so. I will test it out and report back.

Edit: Just remembered 32-bit systems exist and can't have memory-mapped files > 2G. Could just disable for 32-bit systems? On another note, I see no performance difference with a mmap size of 1TB and my regular mmap size of ~500MB.

[cfromknecht]

LGTM 🌋

We should get this in early on in the road to 0.8 so that we can get adequate testing in master. Been running on my node for the past week or so without issue.

[Crypt-iQ]

I'm kind of indifferent on the InitialMmapSize parameter. I tested and couldn't, in any reasonable (non-test) circumstance, produce any huge memory allocations when resizing the mmap. It is also not resized very often so it's not likely to occur that long-running reads (which there aren't that many of) block special write txn's.

Just a note. Looking at the boltdb code and issues, it seems that the mmap size cannot be greater than 2GB for 32-bit machines which makes sense. If channel.db files or other boltdb-backed db files start regularly growing larger than this, support for those machines may need to be dropped.

[Roasbeef]

I'm kind of indifferent on the InitialMmapSize parameter.

I don't think we need to set this value in this PR, and can defer tuning it to a later PR.

But the only other large database I had on my machine was wallet.db which is the btcwallet repo.

Once this goes in, we can follow up with the change on the btcwallet side as well.

[Roasbeef]

LGTM 🐝