[WIP] S3 Datastore

[hinshun]

Based on #5516
Related: #5235, ipfs/notes#214

This is some initial work to get S3 datastore working with IPFS.
Uses fork of go-ds-s3: ipfs/go-ds-s3#7

I have tested this with an AWS S3 bucket. But BloomFilterSize must be set to a high number otherwise the daemon will send too many Get requests to S3.

~/.ipfs/config example
{
    // ...
    "Datastore": {
    "StorageMax": "10GB",
    "StorageGCWatermark": 90,
    "GCPeriod": "1h",
    "Spec": {
      "mounts": [
        {
          "child": {
            "bucket": "<bucket-name>",
            "region": "us-west-2",
            "regionEndpoint": "s3.us-west-2.amazonaws.com",
            "rootDirectory": "",
            "type": "s3ds"
          },
          "mountpoint": "/blocks",
          "name": "s3ds",
          "type": "log"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "HashOnRead": false,
    "BloomFilterSize": 10000000
  }, // ...

[magik6k]

Need #5187 first

[magik6k]

There already is a logging datastore: https://github.com/ipfs/go-ipfs/blob/master/repo/fsrepo/datastores.go#L66

[hinshun]

Thanks, just starting to learn the codebase.

[hinshun]

I attempted to gx install go-ipfs through a ipfs daemon with s3 datastore and it was very slow and got OOM-killed. For smaller loads it seems to work fine but I think we'll need to properly implement go-ds-s3 with aws-sdk-go.

[MichaelMure]

Any idea if this would allow to run multiple IPFS nodes with a shared s3 datastore ?

[magik6k]

If you only use this for /blocks/ and disable GC on all nodes then it may be possible

[hinshun]

As an update, I rewrote ipfs/go-ds-s3 with aws-sdk-go here and was successful in gx installing all of go-ipfs's dependencies.

[bonedaddy]

I've been trying something similar to this and have noticed a critical issue, which is that if you pin some content, and run ipfs pin ls you can see the content as being pinned. If you shutdown the node, start it up and run ipfs pin ls again, nothing shows up.

[georgyo]

Should this ticket be closed in favor of making ipfs/go-ds-s3 in the default plugin list?