WIP: Add support for multiple blockstores #3257

kevina · 2016-09-25T01:00:31Z

Closes #3119.

Required for #2634.

whyrusleeping

What is the reasoning for doing this at the blockstore level? I thought you were doing this with a multi-datastore type thing?

whyrusleeping · 2016-09-26T09:33:51Z

blocks/blockstore/blockstore.go

+}
+
+func NewBlockstoreWPrefix(d ds.Batching, prefix string) *blockstore {
+	if prefix == "" {


I dont like this, lets force the user to give valid input, in the function above this we can just pass the default value.

whyrusleeping · 2016-09-26T09:36:24Z

blocks/blockstore/blockstore.go

-	if err == nil && exists {
-		return nil // already stored.
-	}
+	// Note: The Has Check is now done by the MultiBlockstore


I agree its probably okay to remove this and do it intentionally in a certain place (instead of at every layer). But this gets a bit weird... For example, if someone just wants to use the blockstore on its own, they don't get this nice optimization.

Are you sure about that? MultiBlockstore just calls Has method of Blockstore but it doesn't mean that Blockstore (on Datastore) will check the Has of datastore.

UPDATE: ahh, you call an explicit Has.

So can I take it that this is resolved?

whyrusleeping · 2016-09-26T09:38:57Z

blocks/blockstore/bloom_cache_test.go

@@ -104,11 +104,11 @@ func TestHasIsBloomCached(t *testing.T) {
 	block := blocks.NewBlock([]byte("newBlock"))

 	cachedbs.PutMany([]blocks.Block{block})
-	if cacheFails != 2 {
-		t.Fatalf("expected two datastore hits: %d", cacheFails)
+	if cacheFails != 1 {


why does this change?

I verified this was correct, the change was due to an implementation detail. Unfortually, I can't remember why. If it is important enough I will spend an hour or two looking into it.

It always weirds me out when i see tests changing. Maybe @Kubuxu is more familiar with this and can check it?

It was to because PutMany on default blockstore was accessing datastore two times, once checking Has for a key and then calling Put.

This means that now we don't check the datastore with Has we just Put the value.

https://github.com/ipfs/go-ipfs/pull/3257/files#r80438802

whyrusleeping · 2016-10-03T22:18:17Z

repo/fsrepo/fsrepo.go

@@ -557,6 +565,27 @@ func (r *FSRepo) Datastore() repo.Datastore {
 	return d
 }

+func (r *FSRepo) DirectMount(prefix string) ds.Datastore {


Maybe call this DatastoreAt or GetMount. DirectMount just leaves me confused about what this is

kevina · 2016-10-03T22:35:23Z

@whyrusleeping

What is the reasoning for doing this at the blockstore level? I thought you were doing this with a multi-datastore type thing?

To better interact with caching. In particular the bloom filter.

Kubuxu · 2016-10-03T23:50:49Z

blocks/blockstore/multi.go

+func (bs *multiblockstore) Locate(key key.Key) []LocateInfo {
+	res := make([]LocateInfo, 0, len(bs.mounts))
+	for _, m := range bs.mounts {
+		_, err := m.Blocks.Get(key)


Wouldn't it be better to use Has here?

Not necessary, the problem is with the filestore simply calling Has() will not guarantee the block is available as the backing file might have changed or no longer be available.

This should be on filestore to figure out, IMO.
Why should we load X MiB into memory from slow harddrive if other blockstores can figure it out without loading whole content or even from Has caches.

Allright. I agree will fix this and the Filestore's Has() method.

The other thing is that Get() will return more information than Has() for the filestore. In particular it will return if the key doesn't exist, or if the key exists but there was a problem reconstructing the block.

I am holding off on this change for a bit while I think about it.

Kubuxu · 2016-10-03T23:55:57Z

blocks/blockstore/multi.go

+}
+
+func (bs *multiblockstore) Put(blk blocks.Block) error {
+	// Has is cheaper than Put, so see if we already have it


This comment is misleading. You don't check Has before Put because it is cheaper..
If you don't do it, you can end up with data being duplicated in two blockstores.

Your right, I will fix it to be clearer.

Kubuxu · 2016-10-04T00:00:54Z

blocks/blockstore/multi.go

+				out <- key
+			}
+		}
+	}()


If @whyrusleeping agrees with me I would open a goroutine per BS and make them pipe from one AllKeysChan to one external. This way if first BS is slow and second is fast, the first won't slow down whole process.

Kubuxu · 2016-10-04T00:02:56Z

blocks/blockstore/multi.go

+		if err == nil {
+			return blk, nil
+		}
+		if firstErr == nil || firstErr == ErrNotFound {


I would replace it with firstErr == nil && err != ErrNotFound

If I do that than firstErr could end up being nil even if the block was not found. The idea is to return ErrNotFound only if that is the case for all the blockstore's. If not than return the more serious error.

whyrusleeping

This just feels very complicated to me. not sure how best to make it less so. It might help if i look at exactly what is needed by the filestore here

whyrusleeping · 2016-11-02T23:50:54Z

blocks/blockstore/multi.go

+func (bs *multiblockstore) Locate(c *cid.Cid) []LocateInfo {
+	res := make([]LocateInfo, 0, len(bs.mounts))
+	for _, m := range bs.mounts {
+		_, err := m.Blocks.Get(c)


this should use Has

I am fine with that. Not 100% happy with it as filestore Has doesn't really guarantee the block is available (and changing it so it does will make the Has() call really expensive), but its not a major problem.

I partly take the Has() comment back. The filestore Has() needs fixing or the Put below needs fixing.

whyrusleeping · 2016-11-02T23:56:01Z

blocks/blockstore/multi.go

+	return res
+}
+
+func (bs *multiblockstore) Put(blk blocks.Block) error {


So if i add a file with the filestore, then i add another file normally that overlaps a block with it. one block from my 'normal' add will be referencing a file on disk?

That feels odd to me.

Yeah, but rather difficult to avoid.

whyrusleeping · 2016-11-02T23:56:20Z

blocks/blockstore/multi.go

+				out <- key
+			}
+		}
+	}()


whyrusleeping · 2016-11-03T00:06:17Z

blocks/blockstore/util/remove.go

+func RmBlocks(mbs bs.MultiBlockstore, pins pin.Pinner, out chan<- interface{}, cids []*cid.Cid, opts RmBlocksOpts) error {
+	prefix := opts.Prefix
+	if prefix == "" {
+		prefix = mbs.Mounts()[0]


does this only remove from one blockstore?

whyrusleeping · 2016-11-03T00:07:25Z

core/builder.go

 	if err != nil {
 		return err
 	}

+	mounts := []bstore.Mount{{fsrepo.CacheMount, cbs}}
+
+	if n.Repo.DirectMount(fsrepo.FilestoreMount) != nil {


Filestore mount? does this do anything yet?

Not in this P.R. Looks like I missed that one when factoring out the code.

whyrusleeping · 2016-11-03T00:13:28Z

core/commands/add.go

@@ -78,6 +80,7 @@ You can now refer to the added file in a gateway, like so:
 		cmds.BoolOption(hiddenOptionName, "H", "Include files that are hidden. Only takes effect on recursive add.").Default(false),
 		cmds.StringOption(chunkerOptionName, "s", "Chunking algorithm to use."),
 		cmds.BoolOption(pinOptionName, "Pin this object when adding.").Default(true),
+		cmds.BoolOption(allowDupName, "Add even if blocks are in non-cache blockstore.").Default(false),


This really complicates things... thoughts @Kubuxu ?

i brought this up in IRC.

whyrusleeping · 2016-11-03T00:14:55Z

core/core.go

-	DAG        merkledag.DAGService // the merkle dag service, get/add objects.
-	Resolver   *path.Resolver       // the path resolution system
+	Peerstore  pstore.Peerstore       // storage for other Peer instances
+	Blockstore bstore.MultiBlockstore // the block store (lower level)


should be an interface

what should be an interface?

the blockstore should just be of type Blockstore. That way its easy to swap out with any other blockstore implementation.

@whyrusleeping I did not design the MultiBlockstore to be able to be swapped out. (As I see it it, it was designed to allow use of other Blockstores so why would you want to swap it out, but I can understand your viewpoint) If that is something you want it is likely possible. I image I can have something done in a few days.

whyrusleeping · 2016-11-03T00:15:14Z

repo/fsrepo/defaultds.go

-func openDefaultDatastore(r *FSRepo) (repo.Datastore, error) {
+const (
+	RootMount      = "/"
+	CacheMount     = "/blocks" // needs to be the same as blockstore.DefaultPrefix


why is this called the cache mount?

Please see #3119.

Factored out of ipfs#3257 (Add support for multiple blockstores). License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Each datastore is mounted under a different mount point and a multi-blockstore is used to check each mount point for the block. The first mount checked of the multi-blockstore is considered the "cache", all others are considered read-only. This implies that the garbage collector only removes block from the first mount. This change also factors out the pinlock from the blockstore into its own structure. Only the multi-datastore now implements the GCBlockstore interface. In the future this could be separated out from the blockstore completely. For now caching is only done on the first mount, in the future this could be reworked. The bloom filter is the most problematic as the read-only mounts are not necessary immutable and can be changed by methods outside of the blockstore. Right now there is only one mount, but that will soon change once support for the filestore is added. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

This option adds a files the the primary blockstore even if the block is in another blockstore such as the filestore. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

whyrusleeping · 2017-03-06T19:46:56Z

This is no longer needed

kevina · 2017-03-06T19:54:37Z

@whyrusleeping I beg to differ. Something like this might still be needed. For example if you want multiple ipfs-packs in a single node.

whyrusleeping · 2017-03-06T19:56:14Z

Alright, I'll keep the branch around (its also archived on the branch archive repo). But keeping the PR closed until we have an actionable usecase need for it.

Kubuxu added the status/in-progress In progress label Sep 25, 2016

kevina mentioned this pull request Sep 25, 2016

[WIP] Filestore Implementation #2634

Closed

6 tasks

kevina changed the title ~~WIP: Add basic support for multiple blockstores~~ WIP: Add support for multiple blockstores Sep 25, 2016

kevina force-pushed the kevina/multiblockstore branch from 67da189 to 3cf5695 Compare September 25, 2016 05:18

whyrusleeping requested changes Oct 3, 2016

View reviewed changes

Kubuxu reviewed Oct 3, 2016

View reviewed changes

Kubuxu reviewed Oct 4, 2016

View reviewed changes

kevina mentioned this pull request Oct 7, 2016

Sprint: Oct 10 → Oct 17 ipfs/team-mgmt#217

Closed

kevina force-pushed the kevina/multiblockstore branch 2 times, most recently from 1be4ba9 to 145180a Compare October 16, 2016 16:36

kevina added this to the Filestore implementation milestone Oct 19, 2016

whyrusleeping requested changes Nov 3, 2016

View reviewed changes

kevina added a commit to ipfs-filestore/go-ipfs that referenced this pull request Nov 3, 2016

Separate out the G.C. Locking from the Blockstore interface.

ffe9d7d

Factored out of ipfs#3257 (Add support for multiple blockstores). License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

This was referenced Nov 3, 2016

Separate out the G.C. Locking from the Blockstore interface. #3348

Merged

builder: add blockstoreOption for customizing the blockstore #3349

Closed

kevina added 7 commits November 3, 2016 00:05

Implement multiblockstore.AllKeysChan(), GC now uses the first mount.

a93b152

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Add "block locate" command.

d31cb4b

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

"block/filestore rm": Allow removal of pinned but duplicate blocks.

e32702e

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

"add": add "--allow-dup" option.

9c8159b

This option adds a files the the primary blockstore even if the block is in another blockstore such as the filestore. License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

In NewBlockstoreWPrefix don't assume "" is the default value.

58cc20e

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Minor refactor of "block rm" command.

008de00

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

kevina force-pushed the kevina/multiblockstore branch from 145180a to 008de00 Compare November 3, 2016 04:06

Kubuxu added status/ready Ready to be worked and removed status/in-progress In progress labels Dec 19, 2016

kevina mentioned this pull request Jan 20, 2017

Review Filestore for 300TB Challenge. Update Stories + Specs ipfs-inactive/archives#85

Closed

whyrusleeping closed this Mar 6, 2017

whyrusleeping deleted the kevina/multiblockstore branch March 6, 2017 19:46

whyrusleeping removed the status/ready Ready to be worked label Mar 6, 2017

kevina restored the kevina/multiblockstore branch March 6, 2017 19:50

Kubuxu deleted the kevina/multiblockstore branch August 10, 2017 17:10

WIP: Add support for multiple blockstores #3257

WIP: Add support for multiple blockstores #3257

Conversation

kevina commented Sep 25, 2016 • edited Loading

whyrusleeping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kubuxu Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

Kubuxu Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevina commented Oct 3, 2016

Kubuxu Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kubuxu Oct 4, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Kubuxu Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whyrusleeping commented Mar 6, 2017

kevina commented Mar 6, 2017

whyrusleeping commented Mar 6, 2017

kevina commented Sep 25, 2016 •

edited

Loading

Kubuxu Oct 3, 2016 •

edited

Loading

whyrusleeping Oct 3, 2016 •

edited

Loading

Kubuxu Oct 3, 2016 •

edited

Loading

Kubuxu Oct 3, 2016 •

edited

Loading

Kubuxu Oct 4, 2016 •

edited

Loading

Kubuxu Oct 3, 2016 •

edited

Loading