Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bundle of changes #112

Merged
merged 83 commits into from
Sep 19, 2018
Merged

Bundle of changes #112

merged 83 commits into from
Sep 19, 2018

Conversation

keks
Copy link
Contributor

@keks keks commented May 24, 2018

This branch bundles a couple of changes so we can test against all of them at once in go-ipfs

@ghost ghost assigned keks May 24, 2018
@ghost ghost added the status/in-progress In progress label May 24, 2018
@keks keks force-pushed the refactor/bundle branch from 63232bf to f144d9c Compare May 25, 2018 12:01
@keks
Copy link
Contributor Author

keks commented May 25, 2018

@Stebalien

in one of the PRs above you said you wanted the PR bundled and the changes to go-ipfs ready. This is now done! I'm pretty pleased with the result, especially since it removes some odd stuff like that function HandleError or the method RawNext on ResponseEmitter.

The changes to go-ipfs are in ipfs/kubo#5035.

@keks keks requested a review from Stebalien May 25, 2018 12:02
@@ -70,26 +74,52 @@ func (re *responseEmitter) SetEncoder(enc func(io.Writer) cmds.Encoder) {
re.enc = enc(re.stdout)
}

func (re *responseEmitter) SetError(v interface{}, errType cmdkit.ErrorType) {
func (re *responseEmitter) CloseWithError(err error) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be really great if this method didnt return an error. I can't think of any situation where the caller wants to do something with the error, or even can. I think it's likely better to just log an error and move on.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I let made this return an error is that CloseWithError(error) error is idiomatic and is used throughout the entire Go ecosystem. I don't see a reason to diverge - maybe we don't really need it in ipfs, but possible somewhere else. And now that the command's Run function just returns an error we don't call this so often that it's too much of a pain for the caller to do the logging themselves.

@keks
Copy link
Contributor Author

keks commented May 31, 2018

Hey @whyrusleeping judging by Stebalien's contribution activity graph it seems they are currently not avalailable, so could you maybe review this PR or delegate this? Thanks!

@keks
Copy link
Contributor Author

keks commented May 31, 2018

Also if you have questions on the structure of this bundle PR, shoot!

@whyrusleeping
Copy link
Member

@keks yeah, he's just now getting back from a vacation. I'll still want his review, but i'll take a pass here :)

chan.go Outdated
r.err = val
return nil, ErrRcvdError
// TODO keks remove logging
log.Error("unexpected error value:", val)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rm ?

chan.go Outdated

func (re *chanResponseEmitter) Emit(v interface{}) error {
if e, ok := v.(*cmdkit.Error); ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is because we shouldnt be emitting errors ever anymore, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. Should I remove this or are we not confident enough this doesn't happen? Or should I return that as an error, so it will definitely fail?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we should keep it, and maybe even add a check for the non-pointerized version too just in case. Then also add a comment explaining why this bit of code is here.

@keks
Copy link
Contributor Author

keks commented Jun 9, 2018

Hey @Stebalien, do you think you'll find some time to CR this PR and the one in go-ipfs? Would love to merge this soon

chan.go Outdated
@@ -70,6 +82,20 @@ func (r *chanResponse) Length() uint64 {
return r.length
}

func (re *chanResponse) Head() Head {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whats this method supposed to do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not new, I just moved it from here: https://github.com/ipfs/go-ipfs-cmds/pull/112/files/15c1508126b5934ffae976e4a09c4353fa90956e#diff-d1319de8df19ba21c67a62fa1fde5f89L201

But I think it's needed for HEAD requests, it's even in the ResponseEmitter interface; but it looks like it's not called from anywhere in go-ipfs-cmds/..., so maybe we can drop it. I'd suggest doing that in a new PR though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to do it in this PR.

http/response.go Outdated
@@ -73,7 +76,24 @@ func (res *Response) RawNext() (interface{}, error) {
err = io.EOF
}

return m.Get(), err
if err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe put this inside the same if err != nil as above? might make it a bit clearer as to whats going on here.

Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass.

@@ -57,7 +61,15 @@ func (r *chanResponse) Error() *cmdkit.Error {
return nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we get rid of both of these nil tests? I'd rathe panic if something is wrong (and we'll panic in the second case anyways).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well the r == nil check obviously doesn't help because we already access r.wait before. But what's wrong with the r.err == nil check?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We call r.err.Error() below, so r.err better not be nil.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I was refering to both of the r == nil checks (in this function and the one above). Unfortunately, git hides that part of the patch.

chan.go Outdated
func (re *chanResponse) Head() Head {
<-re.wait

err, ok := re.err.(*cmdkit.Error)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will treat EOF as an error. Maybe we should just have a separate closed flag if we need that and not write EOF to the error field.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, we can just check <-re.closed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that this is not correct. However, there can be an error even if re.closed already is closed, so I don't think it's a good indicator. I think I fixed this locally and will push the changes once I'm through you first pass.

chan.go Outdated
case v, ok := <-r.ch:
if !ok {
r.ch = nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure this isn't racy? Also, this'll cause this method to hang forever if called again.

chan.go Outdated
return nil, io.EOF
}

if err, ok := v.(cmdkit.Error); ok {
if err, ok := v.(*cmdkit.Error); ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure we just want to remove this entirely.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure, here whyrusleeping argued we should check both all the time. There's an issue on fixing this for cmdkit.Error in general, but there was little feedback.

Also, that line is wrong, should be without the dereference. I'll fix that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're already checking if something is an error before putting it into the channel. If we leave this here, we'll need to leve the cmdkit.Error case in the switch statement below. However, IMO, we should just remove both (there shouldn't be any errors here).

chan.go Outdated
}
if e, ok := v.(*cmdkit.Error); ok {
log.Errorf("unexpected error value emitted: %s at\n%s", e.Error(), debug.Stack())
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this call CloseWithError and return?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, we don't emit errors anymore. Doing so is a programmer error, so we want them to know. But panicking because of this feels too strong, so I opted for logging an error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That won't cause a panic, it'll just "do the right thing" (redirect the error). As it stands, we'll end up sending the error on the value channel.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, I meant to return. Anyway, I change it to panic now. I hope we can then in the near future remove those panics because we are more confident we don't do that mistake anymore.

chan.go Outdated
})

re.l.Lock()
defer re.l.Unlock()

if re.ch == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a valid "closed" check anymore. Probably want to check:

select {
  case <-re.closed:
    return ...
  default:
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right it's not a valid close check anymore, but I think your proposal is also racy: It might occur that re.closed is not yet closed but re.ch is, which would result in a write to closed channel error. Thinking this through...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be if we only close the channel while holding the lock.

This is actually a good case for removing the closeOnce sync.Once (a user may think that closeOnce is protecting the close while really the lock is).


return nil

log.Errorf("got an error: %v\n%s", err, debug.Stack())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, we should probably forward these errors to CloseWithError for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should do away with emitting errors and make sure doing so is an error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should. However, we should either panic in cases like this or "do the right thing". Simply logging and dropping the error on the floor is not going to end well.

case io.Reader:
_, err = io.Copy(re.stdout, t)
if err != nil {
re.SetError(err, cmdkit.ErrNormal)
err = nil
return err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if this should also call CloseWithError. We could also say that it's the callers responsibility to return the error from Run. Thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling CloseWithError here would close the emitter and only print io.Copy's error. I think it's better to allow the caller to specify a more intelligible error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough.

//"io"
"testing"

"github.com/ipfs/go-ipfs-cmdkit"
//"github.com/ipfs/go-ipfs-cmdkit"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably remove before final merge.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yup.

command.go Outdated
defer re.Close()
var err error

defer func() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about putting all the logic in Call in a separate call function that turns an error. That way, we don't need this defer? The shared err variable makes be uncomfortable.

Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second round.

executor.go Outdated
return nil
err = cmd.Run(req, re, env)
log.Debugf("cmds.Execute: Run returned %q with response emitter of type %T", err, re)
if err == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can just call CloseWithError (with a nil error).


re, res = NewChanResponsePair(req)

go func() {

This comment was marked as resolved.

http/client.go Outdated
return re.CloseWithError(err)
}

return re.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, can probably just call CloseWithError.

http/response.go Outdated
e := res.err
res.err = nil
return e
if res.err == io.EOF {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impossible condition.

http/response.go Outdated
}

return m.Get(), err
return v, err
}

func (res *Response) Next() (interface{}, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis function checks if the value is an error, I assume this isn't possible given RawNext?

e = &cmdkit.Error{Message: err.Error()}
}

if re.streaming || e.Code == cmdkit.ErrFatal {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't all errors "fatal" now? That is, can't we just always use the trailer now?

writer.go Outdated
case Single:
return val.Value, nil
case *cmdkit.Error:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, I assume this case is impossible.

@Stebalien
Copy link
Member

Nice! This is starting to look a lot more robust.

@keks
Copy link
Contributor Author

keks commented Jun 29, 2018

@Stebalien thank you for the thorough feedback! I tried to incorporate all of it and made some additional changes:

  • remove the Header interface and the Head() methods from all Response types. It wasn't being used anywhere.
  • remove the RawNext methods from all Response types
  • remove the HandleError function
  • unexport WriterResponseEmitter
  • add a new global error variable ErrClosedEmitter that is meant to be returned by Emit and Close{,WithError}.
  • work a little bit on testing. More to come in a different PR.

Right now Travis complains that writer.go is not gofmt'd. However, when doing go fmt ./..., no file is changed. I'm not sure what's going on here.

Also, in my branch of go-ipfs that has the changes for this library there is a test (this one) that doesn't complete currently. Investigating that now. Also my branch fell behind a bit, so I'll still need to rebase that on master -- but I first want to get that test to pass.

@keks keks force-pushed the refactor/bundle branch from fab2a3a to 8522cfc Compare June 30, 2018 14:32
@keks
Copy link
Contributor Author

keks commented Jun 30, 2018

Okay, I fixed all the tests in the go-ipfs branch and rebased it is it's en par with master.

I still have no clue with Travis let's this PR fail though. It thinks that writer.go is not gofmt'd, even though it is -- at least for me calling go fmt doesn't change anything. Rebuilding doesn't change anything.
Has anyone of you stumbled over something like this in the past? My go installation is up to date.

@keks
Copy link
Contributor Author

keks commented Jun 30, 2018

cc @Stebalien

chan.go Outdated
defer re.wl.Unlock()

if _, ok := v.(Single); ok {
defer re.closeWithError(nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll close without error even if we fail somewhere down the line. It can also panic if we emit a single value after closing.

chan.go Outdated

func (r *chanResponse) Request() *Request {
return r.req
}

func (r *chanResponse) Error() *cmdkit.Error {
<-r.wait
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems kind of funny for this method to wait for at least one value to be emitted. I'd expect it to either wait until the end or always return immediately (preferably the latter to reduce the chance of a deadlock).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the concern about deadlocks, but returning immediately raises the question of what to return when the command is still running and we didn't see an error yet. So I went for wait-until-close, I'm currently testing this in go-ipfs to see if it makes problems.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, if we block on error until the stream is closed and change nothing else, we get a deadlock. Now I just return nil if the connection is not closed yet.

chan.go Outdated
return r.req
}

func (r *chanResponse) Error() *cmdkit.Error {
<-r.wait

if r == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is racy as we set err under a lock (and can set it at any time).

Err: re.err,
}
}

func (re *responseEmitter) Emit(v interface{}) error {
// unwrap
if val, ok := v.(cmds.Single); ok {
v = val.Value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we not need to close in this case?

executor.go Outdated

err := cmd.PostRun[typer.Type()](res, lower)
closeErr = lower.CloseWithError(err)
errCh <- closeErr

This comment was marked as resolved.

// if they are errors
if e, ok := v.(error); ok {
if err, ok := v.(error); ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is throwing the error away, not returning it. It's creating a new error variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it's creating a new error variable, and I use it as the argument when calling CloseWithError. I don't think I'm throwing it away?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Got it. I thought you were trying to overwrite the named return value err.

executor.go Outdated
@@ -105,6 +132,12 @@ func (x *executor) Execute(req *Request, re ResponseEmitter, env Environment) (e
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, Run panics, we won't wait for PostRun to exit. Not much of an issue but we may want to consider waiting on errCh in here instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd still want to return the error from errCh, but yes the defer should also wait for it. So let's wait in both places.

ch, isChan = value.(chan interface{})
if single, ok := value.(cmds.Single); ok {
value = single.Value
defer re.closeWithError(nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably needs to go after the lock.

if _, ok := value.(cmdkit.Error); ok {
log.Warning("fixme: got Error not *Error: ", value)
value = &value
if err, ok := value.(cmdkit.Error); ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the debug error assertion?

writer.go Outdated
return errors.New("provided closer does not support CloseWithError")
}

func (re *writerResponseEmitter) SetError(v interface{}, errType cmdkit.ErrorType) {
err := re.Emit(&cmdkit.Error{Message: fmt.Sprint(v), Code: errType})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we remove this function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't just remove it, because then it wouldn't satisfy the interface anymore. But I made it simply log an error.

Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to consider the story around calling Close and CloseWithError. We could tell users not to do that and to simply return the errors from Run or we could say the first one wins.

If we say the first one wins, we'll need to fix several of the CloseWithError implementations so that they don't barf when called twice. We'll also want to avoid logging to noisily when we accidentally close twice.

If we do say "don't close, well do it for you", then we'd need to figure out how to handle Single as that does close.

// if they are errors
if e, ok := v.(error); ok {
if err, ok := v.(error); ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. Got it. I thought you were trying to overwrite the named return value err.

chan.go Outdated
@@ -155,6 +151,10 @@ func (re *chanResponseEmitter) Emit(v interface{}) error {

select {
case re.ch <- v:
if _, ok := v.(Single); ok {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: We could unwrap this before sending to simplify the Next() implementation. However, that's up to you (and may not save any code).

chan.go Outdated
// waitErr is closed when the stream is closed.
// Error checks if the stream has been closed by checking if this channes is closed.
// Its closing is protected by wl.
waitErr chan struct{}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'd just call this closedCh or something like that. waitErr implies some relation to the error.

case Single:
return val.Value, nil
default:
return v, nil
}
case <-ctx.Done():
close(r.done)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, there's a subtle but probably harmless issue here: Next() can return an error while re.Error() won't. We may want to close here (with ctx.Err()?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure. That would mean that if one call to Next() gets cancelled, the stream will end. Intuitively I would say that the stream should live on and the caller should be able to do another call to Next() later. Don't you think?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, no, you're totally right.

chan.go Outdated

if re.ch == nil {
return fmt.Errorf("emitter closed")
// unblock Length() and Error()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't unblock Error().

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should still fix this comment.

return re.Close()
}

e, ok := err.(*cmdkit.Error)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably be consistent with the cmdkit.Error -> *cmdkit.Error dance. We should either do that everywhere or nowhere.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still looks like an issue. We should probably have a "if cmdkit.Error, convert to *cmdkit.Error" check here.

@@ -200,6 +211,10 @@ func (re *responseEmitter) Emit(v interface{}) error {
}
}

if isSingle {
return re.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This'll return success to the user if, e.g., encoding fails. Should probably call re.CloseWithErr(err).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still looks like an issue.

writer.go Outdated
// old error emitting semantics.
// Also errors may occur both as pointers and as plain values, so we need to
// check both.
debug.AssertNotError(v)

if _, ok := v.(Single); ok {
defer re.Close()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will close without error if encoding fails.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I would send the encoding error down the stream. If I try to send an unencodable value through a writer, the stream should live on and I should get an error so I can send something else.

However, we should return the Close() error, if there is one.

writer.go Outdated
func (re *WriterResponseEmitter) SetLength(length uint64) {
func (re *writerResponseEmitter) SetError(v interface{}, errType cmdkit.ErrorType) {
err := &cmdkit.Error{Message: fmt.Sprint(v), Code: errType}
log.Errorf("error setting error %q: cannot set error with writerResponseEmitter", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can't do the right thing here, maybe we should just remove this from the interface. We really don't want the user to try calling this and then return a "success" to the user.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, hasn't this been removed from the interface?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, looks like I forgot deleting it. done.


if single, ok := value.(cmds.Single); ok {
value = single.Value
defer re.closeWithError(nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will throw away encoding errors.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO encoding errors should be passed to the caller instead of be used to close the stream. And closeWithError always returns nil so I figured we don't need it to handle the error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is that we're closing the response stream as if the command succeeded when, really, the command failed.

@Stebalien
Copy link
Member

FYI, if I take more than a week to get to a review, the notification has been buried in my inbox. At that point, just bug me on IRC (really, feel free to bug me on IRC repeatedly as soon as you have something for me to review).

@keks
Copy link
Contributor Author

keks commented Jul 27, 2018

Thank you for the review. I was traveling and will work on these over the next week. And okay, next time I'll be more stubborn :)

@keks
Copy link
Contributor Author

keks commented Aug 8, 2018

Hey @Stebalien I now commented or acted on everything except the thoughts on Single. I'm still pondering on that one, especially whether it's possible to only set the headers that this is a single value, but still leave the closing to the caller. Otherwise making Close() idempotent, i.e. closing a closed channel doesn't do anything, also seems like a good idea. Giving that some thought right now.

@keks
Copy link
Contributor Author

keks commented Aug 8, 2018

I think the most idiomatic way is to ignore double close errors when getting an error on Close or CloseWithError, at least in places where we expect these to occur.
For this I'd introduce a new error variable ErrClosingClosedEmitter and return that on double close. On the caller side I'd always check errors like

err = re.Close()
if err != ErrClosingClosedEmitter && err != nil {
  // handle error
}

Progress:

  • add variable
  • use variable in
    • chan
    • writer
    • cli
    • http
  • ignore this error in
    • Call()
    • Execute()
    • where else?
      • somewhere in go-ipfs?
  • add tests for
    • chan
    • cli
    • http

@Stebalien
Copy link
Member

ignore this error in

  • Call
  • Execute

And possibly some places in go-ipfs.


Warning, @whyrusleeping is grumbling about this being a massive breaking change. My plan is:

  1. Finish up this PR but don't merge it:
  2. Try patching go-ipfs.
  3. If the patch is too hard to write or is unreviewable:
    1. Make the Run function an interface{}.
    2. Type switch on error/no error.
    3. Add back a SetError function that just calls CloseWithError under the hood.

@keks keks force-pushed the refactor/bundle branch from 635c6f2 to 029db85 Compare August 23, 2018 17:08
@Stebalien
Copy link
Member

So, it turns out I was wrong about how we currently handle HTTP headers/errors. We always handle trailing errors using the responseReader helper (so we don't need to do that twice). My comment was incorrect: #112 (comment)

Given this, I believe we can switch to sending errors-in-trailers only for all 200 responses (don't additionally include the value). Thoughts? This should cover both the old client talking to a new server and a new server talking to an old client cases.

I'd like to eventually move to:

  1. Sending a non-200 response on parse failure or command not found (returning the error as a plain string in the response).
  2. Sending a 200 response in all other cases, returning any errors in a trailer.

But we can tackle this later.

@keks keks force-pushed the refactor/bundle branch from ba50fd9 to e488a34 Compare August 31, 2018 10:39
@keks
Copy link
Contributor Author

keks commented Aug 31, 2018

@Stebalien done! Want to give it another round of review?

http/parse.go Outdated
@@ -218,7 +218,10 @@ func parseResponse(httpRes *http.Response, req *cmds.Request) (cmds.Response, er
return nil, fmt.Errorf("unknown error content type: %s", contentType)
default:
// handle errors from headers
e.Message = httpRes.Header.Get(StreamErrHeader)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not be returning an error in this case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it stands now the error will be returned on the first call to Next(), but I can also make it return immediately. I believe that made the test case more complex, but maybe that's okay.

err = re.closeWithError(err)
// always close singles with nil error.
// encoding errors go to caller.
err = re.closeWithError(nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we return the error to the "caller", expecting them to do something, we can't close. This should be something like isSingle && err != nil. Am I missing something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should have been isSingle && err == nil. If an encoding or copy error occurs, we didn't send the value so it doesn't make sense to close the emitter but we should return it. If encoding went fine, gracefully close the emitter.

@Stebalien
Copy link
Member

You know, I think this is pretty much ready (modulo those two questions/issues). This is going to be awesome!

@keks
Copy link
Contributor Author

keks commented Sep 10, 2018

Hey @Stebalien, please CR :)

@Stebalien Stebalien changed the title [WIP] Bundle of changes Bundle of changes Sep 12, 2018
@Stebalien
Copy link
Member

@keks all ready to go! I'll merge this as soon as ipfs/kubo#5035 is ready.

@Stebalien Stebalien merged commit c1f1b23 into master Sep 19, 2018
@ghost ghost removed the status/in-progress In progress label Sep 19, 2018
@Stebalien Stebalien deleted the refactor/bundle branch September 19, 2018 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants