serve command (http api) #24

verokarhu · 2014-08-06T17:59:21Z

This pull request adds a serve command which hosts a http server that listens for POST and GET requests. At the moment it's only possible to GET and POST individual blocks (blobs?).

I decided against implementing the API like it was described in issue #2 because

having a /ctrl/add path is not restful since / is the location where the POSTed files end up
distinguishing between uploaded blocks and files is unnecessary since a client can just chop the file up themselves if they want a chunked upload

whyrusleeping · 2014-08-06T22:22:02Z

cmd/ipfs/serve.go

+}
+
+func init() {
+	cmdIpfsServe.Flag.Uint("port", 80, "Port number")


I dont know how i feel about defaulting to 80, since that requires super user priveleges on most systems

A fair point, forgot that ports below 1024 require root. 8080 makes more
sense as a default.

On Thu, Aug 7, 2014 at 1:22 AM, Jeromy Johnson notifications@github.com
wrote:

In cmd/ipfs/serve.go:

"github.com/gonuts/flag"

"github.com/jbenet/commander"

h "github.com/jbenet/go-ipfs/http"

"strconv"
+)

+var cmdIpfsServe = &commander.Command{

UsageLine: "serve",

Short: "Serve an HTTP API",

Long: ipfs serve - Serve an http gateway into ipfs.,

Run: serveCmd,

Flag: *flag.NewFlagSet("ipfs-serve", flag.ExitOnError),
+}

+func init() {

cmdIpfsServe.Flag.Uint("port", 80, "Port number")

I dont know how i feel about defaulting to 80, since that requires super
user priveleges on most systems

—
Reply to this email directly or view it on GitHub
https://github.com/jbenet/go-ipfs/pull/24/files#r15907722.

jbenet · 2014-08-07T12:19:20Z

@verokarhu great!

having a /ctrl/add path is not restful since / is the location where the POSTed files end up

Makes sense.

distinguishing between uploaded blocks and files is unnecessary since a client can just chop the file up themselves if they want a chunked upload

Yep, this is fine. Eventually we will want more complicated things like, rabin fingerprint chunking done on the server and perhaps even support multipart upload. Not important anytime soon.

I'll CR now.

jbenet · 2014-08-07T12:20:57Z

README.md

@@ -38,6 +38,7 @@ Tool commands:
 Advanced Commands:

    mount         Mount an ipfs read-only mountpoint.
+    serve         Serve an http gateway into ipfs.


let's make this command either serve-http or just do serve http (subcommand), as serve is not protocol ambiguous. (I can imagine ipfs serve git and so on in the future)

jbenet · 2014-08-07T12:34:03Z

Okay, made some comments. Everything else LGTM. I haven't tested this-- if you can create a test file, would be great. Don't worry about query params, we can make that a future issue.

verokarhu · 2014-08-07T16:35:39Z

I left the multiaddr change and a test file out for now. Not familiar with how http testing in go is done but I'll look into it for a future pull request if/when I get around to doing more features for the http api.

Hopefully my assurances that it has been thoroughly tested on Win7 is enough for now. :)

As a sidenote, I wonder if the windows version of the mount command could use this API. @whyrusleeping mentioned that he had worked on an FTP server in issue #9 but I haven't thought about it enough to know if this API would be more suitable.

whyrusleeping · 2014-09-10T21:10:05Z

@verokarhu http testing that ive done in the past has used the napping library to query a server you start up within the test to ensure that the data is consistent with the you expect. I would love to get this code polished and merged in soon. It will be very nice to have as an alternative to fuse (since fuse is being a little problematic for people to easily set up at the moment)

whyrusleeping · 2014-09-10T21:11:29Z

also, take a look at how the mount command works with the daemon code (on the daemon branch) and make serve do the same thing.

verokarhu · 2014-09-12T13:23:01Z

I'm no git master but wouldn't it be simplest to get the daemon branch merged before using those features in this pull request?

whyrusleeping · 2014-09-12T16:20:26Z

Yeah, hold off on the daemon stuff for a little bit until we get that finalized and merged in, but if you could write some sort of unit tests for the http interface, that would be extremely helpful!

verokarhu · 2014-09-12T16:23:53Z

All right, I'll do that. Thanks for the tip on napping btw, I'll look into it. Have been meaning to figure out http testing for another project (and ipfs of course =) ). There seems to have been some changes to the master that this pull request isn't compatible with so I'll have to fix that as well.

whyrusleeping · 2014-09-14T05:22:35Z

Just a ping to let you know that the daemon code is merged in!

verokarhu · 2014-09-17T18:32:21Z

There is now tests for posthandler and servehttp. I suppose one could refactor it a bit and enable testing the routes as well but that seems overkill ATM.

The biggest problem now is getting gorilla vendored. Try as I may, godep seems to remove the entire workspace when attempting to add gorilla/mux...

I'm curious, why is both godep and vendoring inside the repo used simultaneously? This hybrid approach between vendoring and revision locking seems odd, though I'm not exactly familiar with godep.

whyrusleeping · 2014-09-17T19:31:49Z

We use godep for vendoring, the make file should do it for you.

btc · 2014-09-17T20:33:34Z

godep is used to copy all dependencies into Godeps/_workspace. Throughout the codebase, import statements are rewritten to point to this local workspace.

The command make vendor in package root performs this task.

verokarhu · 2014-09-17T21:08:05Z

I don't have make (i'm not sure if that's available on windows) but I ran the command contained in the makefile. This resulted in godep deleting the workspace (?) and the following error message:

C:\ipfs\src\github.com\jbenet\go-ipfs>godep save -r ./...
godep: rename C:\IPFS\src\github.com\jbenet\go-ipfs\server\http\http.go.temp C:\IPFS\src\github.com\jbenet\go-ipfs\server\http\http.go: Cannot create a file when that file already exists.

Anyways, I was able to manually merge the half-done state godep left things in and it seems to work. Hopefully the build completes succesfully.

verokarhu · 2014-09-17T21:38:18Z

Looks like the build still fails. Am I correct in assuming that the builds were broken already?

The tests in dht_test.go look peculiar, is it making an actual node that it contacts? If so, that should really be mocked out.

whyrusleeping · 2014-09-18T18:17:50Z

Yeah, the tests that are failing in your code dont seem to be related to the work youve done. Ill take a little further look at them to make sure, see if i can repro

whyrusleeping · 2014-09-18T18:29:54Z

@verokarhu Yeap, confirmed that the tests fail on master. Your stuff looks good, once we get a go ahead from @jbenet and @perfmode we can get this merged in!

whyrusleeping · 2014-09-18T18:46:07Z

server/http/ipfs.go

+	u "github.com/jbenet/go-ipfs/util"
+)
+
+type ipfs interface {


Im not sure i agree with the need for an interface here.

I did it with an interface because it makes it easy to mock out the ipfs methods by swapping in a test interface. Another way of doing it would be to have function variables but that felt less go-ey and more like javascript. If there is a third, nicer way of mocking them out I'm of course open to suggestions.

No, that makes good sense. Youll just need to add a function to get a reader for a node (via merkledag/dagreader)

👍 to this interface. We should probably have something similar in /core/interface.go later on, or something.

mildred · 2014-09-26T18:46:45Z

I haven't looked at everything, but why is HTTP POST being used to upload files and not HTTP PUT. I believe the PUT method is better suited for that purpose (being idempotent is a good thing, and the client can know the hash in advance).

A description can be found on stackoverflow. Basically:

Do you name your URL objects you create explicitly, or let the server decide? If you name them then use PUT. If you let the server decide then use POST.
PUT is idempotent, so if you PUT an object twice, it has no effect. This is a nice property, so I would use PUT when possible.
You can update or create a resource with PUT with the same object URL
With POST you can have 2 requests coming in at the same time making modifications to a URL, and they may update different parts of the object.

verokarhu · 2014-09-30T20:10:13Z

I think it's good to go. Some of the things you mentioned are unfixed like returning a 404 error since it seemed like that might need work in other parts of ipfs as well (the error returned was context deadline exceeded). The setup part in serve.go is also still manual.

jbenet · 2014-09-30T20:13:01Z

@verokarhu travis build failed, and failure is in TestServeHTTP:
https://travis-ci.org/jbenet/go-ipfs/jobs/36706143

--- FAIL: TestServeHTTP (0.00 seconds)
panic: runtime error: slice bounds out of range [recovered]
    panic: runtime error: slice bounds out of range
goroutine 3 [running]:
runtime.panic(0x6b5dc0, 0xaa5b6a)
    /home/travis/.gvm/gos/go1.2.2/src/pkg/runtime/panic.c:266 +0xb6
testing.func·005()
    /home/travis/.gvm/gos/go1.2.2/src/pkg/testing/testing.go:385 +0xe8
runtime.panic(0x6b5dc0, 0xaa5b6a)
    /home/travis/.gvm/gos/go1.2.2/src/pkg/runtime/panic.c:248 +0x106
github.com/jbenet/go-ipfs/server/http.(*handler).ServeHTTP(0xc21001eaf0, 0x7fc23411f638, 0xc21000a560, 0xc210049820)
    /home/travis/gopath/src/github.com/jbenet/go-ipfs/server/http/http.go:36 +0x2f4
github.com/jbenet/go-ipfs/server/http.TestServeHTTP(0xc21004e120)
    /home/travis/gopath/src/github.com/jbenet/go-ipfs/server/http/http_test.go:35 +0x26d
testing.tRunner(0xc21004e120, 0xaa0480)
    /home/travis/.gvm/gos/go1.2.2/src/pkg/testing/testing.go:391 +0x8b
created by testing.RunTests
    /home/travis/.gvm/gos/go1.2.2/src/pkg/testing/testing.go:471 +0x8b2
goroutine 1 [chan receive]:
testing.RunTests(0x79c5d8, 0xaa0480, 0x2, 0x2, 0x1)
    /home/travis/.gvm/gos/go1.2.2/src/pkg/testing/testing.go:472 +0x8d5
testing.Main(0x79c5d8, 0xaa0480, 0x2, 0x2, 0xaa9580, ...)
    /home/travis/.gvm/gos/go1.2.2/src/pkg/testing/testing.go:403 +0x84
main.main()
    github.com/jbenet/go-ipfs/server/http/_test/_testmain.go:49 +0x9c
FAIL    github.com/jbenet/go-ipfs/server/http   0.012s

verokarhu · 2014-09-30T20:21:46Z

Oh man, that's what I get for forgetting to run the tests before the last changes. Should really start using an IDE so they are run automatically. Should be ok now.

jbenet · 2014-09-30T20:27:46Z

Ok, tests pass, LGTM! thanks @verokarhu 👍

serve command (http api)

whyrusleeping · 2014-10-01T19:47:33Z

Im so happy about this.

Fix example in Option.DebugAssertSingleGoroutine docs

warn when loggable key is overridden fixes #23

whyrusleeping reviewed Aug 6, 2014
View reviewed changes

jbenet reviewed Aug 7, 2014
View reviewed changes

jbenet force-pushed the master branch 2 times, most recently from ecf49fe to 2a7dafd Compare August 21, 2014 03:44

whyrusleeping reviewed Sep 18, 2014
View reviewed changes

whyrusleeping force-pushed the master branch from 609a71b to 5fafaa4 Compare September 18, 2014 21:02

whyrusleeping mentioned this pull request Sep 26, 2014

Implemented basic read-only webserver #129

Closed

verokarhu added 19 commits September 30, 2014 22:59

added comment to exported function

6338cac

add serve to readme.md

356ee03

get rid of fmt dependency in http

a82f814

add todo

866ea42

implemented suggestions by whyrusleeping

0a25b64

moved http to server/http and implemented serve http subcommand

8eb95fa

daemon listener to http serve code

d894d63

fix compile errors

c4d55c7

test for serveHTTP

436f14b

rewrote test to use separate test interface

65f9087

test for posthandler

f320023

vendor github.com/gorilla dependencies

cf0100c

fix imports on mux

4510977

fix localNode call

600829b

fix compile error after sync

7b22bdc

use dagreader in servehttp

8382c79

return the read bytes when EOF is reached

0beff4b

switch to multiaddr and use /ipfs/ path in http api

0b0bee6

fix compile error after rebase

c6c758d

fix http test

f40fbb2

jbenet added a commit that referenced this pull request Sep 30, 2014

Merge pull request #24 from verokarhu/master

0d84af5

serve command (http api)

jbenet merged commit 0d84af5 into ipfs:master Sep 30, 2014

ribasushi pushed a commit that referenced this pull request Jul 4, 2021

Merge pull request #24 from opentracing/bg/doc-fix

76881bf

Fix example in Option.DebugAssertSingleGoroutine docs

ribasushi pushed a commit that referenced this pull request Jul 4, 2021

Merge pull request #24 from ipfs/fix/warnOveride

4b54e7d

warn when loggable key is overridden fixes #23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

serve command (http api) #24

serve command (http api) #24

verokarhu commented Aug 6, 2014

whyrusleeping Aug 6, 2014

verokarhu Aug 7, 2014

jbenet commented Aug 7, 2014

jbenet Aug 7, 2014

jbenet commented Aug 7, 2014

verokarhu commented Aug 7, 2014

whyrusleeping commented Sep 10, 2014

whyrusleeping commented Sep 10, 2014

verokarhu commented Sep 12, 2014

whyrusleeping commented Sep 12, 2014

verokarhu commented Sep 12, 2014

whyrusleeping commented Sep 14, 2014

verokarhu commented Sep 17, 2014

whyrusleeping commented Sep 17, 2014

btc commented Sep 17, 2014

verokarhu commented Sep 17, 2014

verokarhu commented Sep 17, 2014

whyrusleeping commented Sep 18, 2014

whyrusleeping commented Sep 18, 2014

whyrusleeping Sep 18, 2014

verokarhu Sep 18, 2014

whyrusleeping Sep 18, 2014

jbenet Sep 29, 2014

mildred commented Sep 26, 2014

verokarhu commented Sep 30, 2014

jbenet commented Sep 30, 2014

verokarhu commented Sep 30, 2014

jbenet commented Sep 30, 2014

whyrusleeping commented Oct 1, 2014

serve command (http api) #24

serve command (http api) #24

Conversation

verokarhu commented Aug 6, 2014

whyrusleeping Aug 6, 2014

Choose a reason for hiding this comment

verokarhu Aug 7, 2014

Choose a reason for hiding this comment

jbenet commented Aug 7, 2014

jbenet Aug 7, 2014

Choose a reason for hiding this comment

jbenet commented Aug 7, 2014

verokarhu commented Aug 7, 2014

whyrusleeping commented Sep 10, 2014

whyrusleeping commented Sep 10, 2014

verokarhu commented Sep 12, 2014

whyrusleeping commented Sep 12, 2014

verokarhu commented Sep 12, 2014

whyrusleeping commented Sep 14, 2014

verokarhu commented Sep 17, 2014

whyrusleeping commented Sep 17, 2014

btc commented Sep 17, 2014

verokarhu commented Sep 17, 2014

verokarhu commented Sep 17, 2014

whyrusleeping commented Sep 18, 2014

whyrusleeping commented Sep 18, 2014

whyrusleeping Sep 18, 2014

Choose a reason for hiding this comment

verokarhu Sep 18, 2014

Choose a reason for hiding this comment

whyrusleeping Sep 18, 2014

Choose a reason for hiding this comment

jbenet Sep 29, 2014

Choose a reason for hiding this comment

mildred commented Sep 26, 2014

verokarhu commented Sep 30, 2014

jbenet commented Sep 30, 2014

verokarhu commented Sep 30, 2014

jbenet commented Sep 30, 2014

whyrusleeping commented Oct 1, 2014