Add ARC caching and bloom filter for blockstorage

[Kubuxu]

Resolves #2850
License: MIT
Signed-off-by: Jakub Sztandera kubuxu@protonmail.ch

[whyrusleeping]

blocked on #2601

[Kubuxu]

@kevina In other issues you said that AllKeysChan my be hard on performance, what is the alternative?

[kevina]

@Kubuxu, unfortunately if you use the datastore there are not any. I decided that for the filestore it would make more sense to just use the LevelDB directly.

If I where you I try hacking go-datastore and increase the channel buffer size here, https://github.com/ipfs/go-datastore/blob/master/query/query.go#L185 and see if that speeds things up (try a buffer size of say 16, than maybe something huge like 1024 or larger).

To see how much overhead is really involved you could hack go-datastore to be able to get the underlying LevelDB involved and query it directly.

[Kubuxu]

Rebased on to master, tests are running.

Should I implement block cache in this PR or in a different one?

[whyrusleeping]

@Kubuxu do it separately if its not too much trouble.

[whyrusleeping]

rebase on master so we dont get the docker failures?

[Kubuxu]

I should also squash everything as first commits are failing.

[Kubuxu]

Squashed.

[whyrusleeping]

lets make this a constant

[Kubuxu]

I am introducing CacheOpts in the ARC caching of whole blocks I am working on right now. Would it be ok if I added it in that PR?

[whyrusleeping]

sure

[Kubuxu]

@whyrusleeping I will add the context passing in the next PR too. I will have to rework all places where ctor is used either way.

[whyrusleeping]

@Kubuxu i'm still not sure this is ready. How does it fare on long running nodes? What happens when the filter gets full, or there are lots of deletes, etc

[Kubuxu]

Main problem would be if there is a lot of deletes, then the false positive ratio would rise proportionally with ratio of blocks deleted, In worst case scenario (all block deleted) the performance would be where it was before the bloom filter (+1ms of latency).

Other case would be if the repo was much bigger in comparison with the bloom filter preselected size. Our current settings should provide optimal operation (<1% of false positives) up to 30k blocks in storage, at about 100k blocks it will rise to about 15% of false positives (but still it reduces number of Has requests by roughly 73%, 2% of real positives and 15% false positives).

I plan to add auto rebuild (and expand) but I would need concrete metrics for that of real operation. That is why I plan working this week on the metrics interface I mentioned on sprint.

[whyrusleeping]

@Kubuxu alright, that sounds good to me. Lets get this in and get some mileage on it

[jbenet]

it would be great to see benchmark graphs on a large typical workload with and without this PR.

We need to get into the habit of benchmarking and graphing with large batteries of tests.

[Kubuxu]

In this case the aim of introducing Bloom filtering and ARC caching on Has requests wasn't purely about performance but reducing load on the underlying file system, and thus hard drives.

We are not caching blocks (we are trying that in #2942, the performance benefits isn't clear), but only if block is available in the datastore or not.

Best metric for this change set will be reduction of disk hits when caching is available, on which I was supposed to work quite a time ago by improving metric collection.