[Benchmark] - Store pins in a datastore instead of a DAG

[obo20]

Recently, JS-IPFS recently merged in a PR that switched the pinset to being stored in a datastore instead of a DAG (ipfs/js-ipfs#2771).

Go-IPFS could benefit from the same update being made to how it stores pinsets as pinning becomes drastically slower when working with larger pinsets.

There are a few graphs in the PR linked above that show how vastly this improved pinning / unpinning times for js-ipfs. In addition, below are some statistics from just a few of the nodes Pinata is running around the world. As can be seen, this impacts both flatFS and BadgerDB.

Node Name	FileSystem	Number of root CIDs Pinned	Average ms to Pin of < 100KB file	Average ms to Unpin Time of < 100KB file
Fresh Local Node	FlatFS	5	20	12
Node-1	FlatFS	112300	444	425
Node-2	FlatFS	304738	1311	1301
Node-3	FlatFS	465270	1696	1779
Node-4	FlatFS	4686	37	38
Node-5	FlatFS	5836	79	67
Node-6	FlatFS	5609	82	54
BadgerNode-1	BadgerDB	18558	151	145
BadgerNode-2	BadgerDB	16667	113	108
BadgerNode-1 (2 days later)	BadgerDB	28627	180	177
BadgerNode-2 (2 days later)	BadgerDB	26806	222	193

In addition to this, it appears that if there are multiple pins being added to the DAG pinset at the same time, each of these update will have to wait in line to be updated sequentially, which has the potential to create very long backups if a node gets a lot of traffic at once.

Related Issues:

• Pinning is slow when there are many pins #5221

[aschmahmann]

closed by #7750