'FetchBlocks' gateway option #4150

Voker57 · 2017-08-17T23:15:45Z

If FetchBlocks=false, don't fetch blocks which are not already present in storage.

Also add Gateway.FetchBlocks option

This mode can be useful for creating gateways which are supposed to serve only chosen content.

Kubuxu · 2017-08-18T00:05:22Z

There is already ipfs daemon --offline. Is it not good enough?

Voker57 · 2017-08-18T00:17:17Z

@Kubuxu --offline will also disable fetching via API. I want only gateway to be restricted.

magik6k · 2017-08-18T00:26:23Z

I think it might be better to implement this as a config option

whyrusleeping · 2017-08-28T16:30:09Z

cmd/ipfs/daemon.go

@@ -148,6 +149,7 @@ Headers.
 		cmds.BoolOption(initOptionKwd, "Initialize ipfs with default settings if not already initialized").Default(false),
 		cmds.StringOption(routingOptionKwd, "Overrides the routing option").Default("dht"),
 		cmds.BoolOption(mountKwd, "Mounts IPFS to the filesystem").Default(false),
+		cmds.BoolOption(offlineGatewayKwd, "Do not fetch blocks because of gateway requests").Default(false),


s/because of/for/

whyrusleeping

This looks pretty reasonable to me. My only concern is that we don't have a solid general way of setting up configuration for the gateway. I hesitate to add more flags to ipfs daemon without having a good plan for future gateway configuration options, every flag we add is really hard to remove (has to go through deprecation period, documentation, etc) and flags suddenly being gone makes future documentation people might read confusing.

@Voker57 @Kubuxu @magik6k @lgierth can we try and put together some ideas for this?

magik6k · 2017-08-28T22:10:24Z

IMO daemon already has too many flags (--enable-gc as one example that could be in config). Gateway already has it's config scope, this could be done in config as Gateway.Offline. I'd say that all options that make sense in live deployments should be in config.

Stebalien · 2017-08-29T00:46:33Z

How about something like: #4180

ghost · 2017-08-29T01:55:13Z

Phew I recovered it -- deleting my two comments above.

Good idea 👍

Agreed that a config option is desirable over a command flag, and we'll also want to refactor the gateway a bit. It should get a GatewayOpts struct, similar to the HostOpts struct I implemented in libp2p/go-libp2p#197

I'll put up to debate whether "offline" is a good name though. It already means a couple of different things in go-ipfs. This option seems spiritiually related to the ipfs add --local and ipfs name resolve --local flags, in that it only operates locally (apart form IPNS lookups I guess). I'm not ready to say one is better of the other, just marking this as an important bit. The reason for it's importance is probably outside of this PR and more in the already existing meanings of "offline". It's important the proposed feature fits in nicely with the existing modes of connectedness and disconnectedness and doesn't unneccessarily complicate them.

ghost · 2017-08-29T02:13:04Z

Btw, mentioning the --local flag, I noticed that its description in ipfs --help is completely inaccurate. See #4182.

Voker57 · 2017-08-29T10:26:28Z

@lgierth Isn't GatewayConfig already doing same job as proposed GatewayOpts? https://github.com/ipfs/go-ipfs/blob/master/core/corehttp/gateway.go#L14

ghost · 2017-08-29T14:09:06Z

Yeah you're right that's already useful

whyrusleeping · 2017-09-06T22:07:26Z

Maybe the better term would be NoFetch as a field inside the gateway options?

Voker57 · 2017-09-09T10:11:39Z

Renamed option to FetchBlocks and moved to config.

nicola · 2017-11-14T11:51:32Z

I want this so badly :(

I want to host an IPFS node on my machine, but I don't want people to get content from hashes that I don't allow.. :/ Does this fix that?

Voker57 · 2017-12-31T21:28:38Z

@nicola yes.

Kubuxu · 2018-10-10T14:50:34Z

Where is the option handled in the gateway? I see it only added to GatewayOptions.

Voker57 · 2018-10-10T15:08:47Z

Sorry, I screwed up a rebase. Will update.

Kubuxu · 2018-10-10T15:11:26Z

Also:

+ ./bin/gx deps dupes

+ test -z package go-ipfs-config imported as both QmSoYrBMibm2T3LupaLuez7LPGnyrJwdRxvTfPUyCp691u and QmQwsBSJXHs3nHBLPJKd1iBNpW9uJ9acSpqbbjafvNJQc3

and as tests passed on circleci and travis, they seem to be too weak. I would recommend using IPTB to setup 2 node cluster.

Voker57 · 2018-10-11T10:46:39Z

Fixed the PR and improved the tests.

Stebalien

@magik6k or @schomatis, would one of you see this to completion (i.e., handle reviews)? This will need a bit more work to be a bit less invasive but it would be really nice to have it.

Stebalien · 2018-10-25T16:27:26Z

core/coreapi/coreapi.go

 }

 // NewCoreAPI creates new instance of IPFS CoreAPI backed by go-ipfs Node.
-func NewCoreAPI(n *core.IpfsNode) coreiface.CoreAPI {
-	api := &CoreAPI{n}
+func NewCoreAPI(n *core.IpfsNode, offlineMode bool) coreiface.CoreAPI {


This is really not the right place to configure this.

why not? I don't see how @magik6k solution is better, except for modifying less code.

In retrospect, it's probably a reasonable place to do this (although I would have used functional options). Really, I was reacting to the fact that the CoreAPI needed to store a separate DAGService (I don't want it to turn into yet another "IpfsNode"). However, it turns out that we have to do that for sessions (no way around that as far as I can tell given the current sessions design) so this really isn't an issue.

CoreAPI is what will eventually be exposed as the interface to go-ipfs-*, so we want to make sure it's designed fairly well. The offline mode thing could be interpreted as a 'global option', and in CoreAPI we want to:

Hide all options and use sane defaults (mostly done with functional options)

Have a standard way to set them on the interface (for global options applying to all commands this is probably going to be in form of context hints)

We could also define a set of functional options for use with CoreAPI constructors, but I don't really like that as some implementations such as RPC will likely need to establish a new connection if user needs a global option for one command but not others

Stebalien · 2018-10-25T16:30:08Z

core/coreapi/coreapi.go

 )

 var log = logging.Logger("core/coreapi")

 type CoreAPI struct {
 	node *core.IpfsNode
+	dag  ipld.DAGService


This is going to be a footgun. Now the CoreAPI needs to make sure to use it's dag instead of the global one.

Maybe #4009? This keeps on haunting us and I've yet to see a better proposal.

magik6k · 2018-10-25T23:48:36Z

I'll handle this

Voker57 · 2019-01-04T12:55:30Z

Rebased and updated to use new API. It is now similar to #5649 but more limited in scope: will still resolve IPNS fully, which is my desired behavior. Not sure how to cover this by tests, ones included in this PR would still pass but IPNS will not be queried fully with #5649.

License: MIT Signed-off-by: Iaroslav Gridin <voker57@gmail.com>

magik6k

Right, this option shouldn't prevent IPNS from working.

The Api.FetchBlocks option also needs to be set for commands as some are accessible through the gateway port. There are also some other fixes in the other PR that would need to be ported here.

I'll import the FetchBlocks api option to #5649

magik6k · 2019-01-04T14:11:56Z

core/coreapi/coreapi.go

@@ -217,6 +216,10 @@ func (api *CoreAPI) WithOptions(opts ...options.ApiOption) (coreiface.CoreAPI, e

 	}

+	if !settings.FetchBlocks {
+		subApi.dag = dag.NewDAGService(bserv.New(subApi.blockstore, offlinexch.Exchange(subApi.blockstore)))


settings.FetchBlocks needs to set exchange/blocks too, otherwise it still will be possible to use gateway to fetch stuff from the network with things like /api/v0/block/get

magik6k · 2019-01-04T14:20:12Z

core/corehttp/gateway.go


 	id "gx/ipfs/QmRBaUEQEeFWywfrZJ64QgsmvcqgLSK3VbvGMR2NM2Edpf/go-libp2p/p2p/protocol/identify"
 )

 type GatewayConfig struct {
 	Headers      map[string][]string
 	Writable     bool
+	FetchBlocks  bool


Is this used anywhere?

no, leftover.

magik6k · 2019-01-04T15:06:39Z

IPNS should now work in #5649, can you check if it works for you?

Voker57 · 2019-01-04T16:12:27Z

IPNS should now work in #5649, can you check if it works for you?

yes, now this PR works like I need it. I'm closing this one.

Voker57 force-pushed the feat/offline-gateway branch from 0cb7e7f to dc0db30 Compare August 17, 2017 23:20

whyrusleeping reviewed Aug 28, 2017

View reviewed changes

Voker57 force-pushed the feat/offline-gateway branch from dc0db30 to 30ac2b7 Compare August 28, 2017 20:27

whyrusleeping reviewed Aug 28, 2017

View reviewed changes

Voker57 force-pushed the feat/offline-gateway branch from 30ac2b7 to 9abdfd8 Compare September 9, 2017 10:10

Voker57 changed the title ~~'Offline' gateway option~~ 'FetchBlocks' gateway option Sep 9, 2017

Voker57 force-pushed the feat/offline-gateway branch from 9abdfd8 to 74bdfee Compare September 22, 2017 13:52

Voker57 force-pushed the feat/offline-gateway branch from 74bdfee to 8cc6abc Compare October 20, 2017 11:56

Voker57 force-pushed the feat/offline-gateway branch from 8cc6abc to 95187d3 Compare November 1, 2017 14:10

Voker57 force-pushed the feat/offline-gateway branch from 95187d3 to 2bb6f95 Compare December 16, 2017 04:45

whyrusleeping requested review from Stebalien, kevina and frrist December 31, 2017 21:29

Voker57 force-pushed the feat/offline-gateway branch from 2bb6f95 to bffa0e0 Compare January 19, 2018 20:17

Voker57 mentioned this pull request Oct 10, 2018

Add FetchBlocks config option ipfs/go-ipfs-config#14

Closed

Voker57 force-pushed the feat/offline-gateway branch from bffa0e0 to a00e0dd Compare October 10, 2018 11:28

Voker57 requested a review from Kubuxu as a code owner October 10, 2018 11:29

Voker57 force-pushed the feat/offline-gateway branch 2 times, most recently from 7d2dc12 to c83b869 Compare October 10, 2018 12:23

Voker57 mentioned this pull request Oct 11, 2018

Update go-ipfs-config ipfs/iptb#94

Closed

Voker57 force-pushed the feat/offline-gateway branch 3 times, most recently from 4de08dd to a1920ac Compare October 11, 2018 10:46

Stebalien requested changes Oct 25, 2018

View reviewed changes

Stebalien mentioned this pull request Oct 25, 2018

Option to configure IPFS gateway to only serve files on disk #5644

Closed

magik6k mentioned this pull request Oct 26, 2018

Local Gateway option #5649

Merged

Voker57 force-pushed the feat/offline-gateway branch 2 times, most recently from bb2d521 to e711316 Compare December 1, 2018 12:17

Voker57 force-pushed the feat/offline-gateway branch from e711316 to 02b80ab Compare January 4, 2019 12:17

Voker57 added 2 commits January 4, 2019 15:07

Add "FetchBlocks" option to make gateway not download data on demand

e1335af

License: MIT Signed-off-by: Iaroslav Gridin <voker57@gmail.com>

Add tests for FetchBlocks gateway option

34baed7

License: MIT Signed-off-by: Iaroslav Gridin <voker57@gmail.com>

Voker57 force-pushed the feat/offline-gateway branch from 02b80ab to 34baed7 Compare January 4, 2019 13:09

magik6k reviewed Jan 4, 2019

View reviewed changes

Voker57 closed this Jan 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'FetchBlocks' gateway option #4150

'FetchBlocks' gateway option #4150

Voker57 commented Aug 17, 2017 •

edited

Loading

Kubuxu commented Aug 18, 2017

Voker57 commented Aug 18, 2017

magik6k commented Aug 18, 2017

whyrusleeping Aug 28, 2017

whyrusleeping left a comment

magik6k commented Aug 28, 2017

Stebalien commented Aug 29, 2017

ghost commented Aug 29, 2017

ghost commented Aug 29, 2017

Voker57 commented Aug 29, 2017

ghost commented Aug 29, 2017

whyrusleeping commented Sep 6, 2017

Voker57 commented Sep 9, 2017

nicola commented Nov 14, 2017 •

edited

Loading

Voker57 commented Dec 31, 2017

Kubuxu commented Oct 10, 2018 •

edited

Loading

Voker57 commented Oct 10, 2018

Kubuxu commented Oct 10, 2018

Voker57 commented Oct 11, 2018

Stebalien left a comment

Stebalien Oct 25, 2018

Voker57 Oct 27, 2018

Stebalien Oct 27, 2018

magik6k Oct 27, 2018

magik6k Oct 27, 2018

Stebalien Oct 25, 2018

magik6k commented Oct 25, 2018

Voker57 commented Jan 4, 2019

magik6k left a comment

magik6k Jan 4, 2019

magik6k Jan 4, 2019

Voker57 Jan 4, 2019

magik6k commented Jan 4, 2019

Voker57 commented Jan 4, 2019

'FetchBlocks' gateway option #4150

'FetchBlocks' gateway option #4150

Conversation

Voker57 commented Aug 17, 2017 • edited Loading

Kubuxu commented Aug 18, 2017

Voker57 commented Aug 18, 2017

magik6k commented Aug 18, 2017

Choose a reason for hiding this comment

whyrusleeping left a comment

Choose a reason for hiding this comment

magik6k commented Aug 28, 2017

Stebalien commented Aug 29, 2017

ghost commented Aug 29, 2017

ghost commented Aug 29, 2017

Voker57 commented Aug 29, 2017

ghost commented Aug 29, 2017

whyrusleeping commented Sep 6, 2017

Voker57 commented Sep 9, 2017

nicola commented Nov 14, 2017 • edited Loading

Voker57 commented Dec 31, 2017

Kubuxu commented Oct 10, 2018 • edited Loading

Voker57 commented Oct 10, 2018

Kubuxu commented Oct 10, 2018

Voker57 commented Oct 11, 2018

Stebalien left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k commented Oct 25, 2018

Voker57 commented Jan 4, 2019

magik6k left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

magik6k commented Jan 4, 2019

Voker57 commented Jan 4, 2019

Voker57 commented Aug 17, 2017 •

edited

Loading

nicola commented Nov 14, 2017 •

edited

Loading

Kubuxu commented Oct 10, 2018 •

edited

Loading