api: ChainNotify sometimes freeze, and force close makes the node panic

[jsign]

I've been experiencing this for some time but got away with it since isn't deterministic.

I took a moment and created this test case that could be used in /cli or somewhere to reproduce the problem.

func TestClientChainNotifyBlock(t *testing.T) {
	api, closer, err := client.NewFullNodeRPC("ws://127.0.0.1:1234/rpc/v0", http.Header{"Authorization": []string{"Bearer <putyourauthtokenhere>"}})
	if err != nil {
		panic(err)
	}
	defer closer()

	numchans := 5
	chans := make([]<-chan []*store.HeadChange, 0)
	for i := 0; i < numchans; i++ {
		ch, err := api.ChainNotify(context.Background())
		if err != nil {
			panic(err)
		}
		chans = append(chans, ch)
	}
	for _, c := range chans {
		<-c
	}
}

Run with: go test ./cli -run TestClientChainNotifyBlock -count 1000 (or in whatever package it is)
This can show two problems:

1. If the command is run multiple times, it eventually blocks on some run. When gets blocked, the lotus node start seeing that things get queued:

2020-01-20T14:05:18.402-0300    WARN    chainstore      store/store.go:174      head change sub is slow, has 1 buffered entries
2020-01-20T14:05:18.402-0300    WARN    chainstore      store/store.go:174      head change sub is slow, has 1 buffered entries
2020-01-20T14:05:18.402-0300    WARN    chainstore      store/store.go:174      head change sub is slow, has 2 buffered entries

2. If you Ctrl+C, it makes the Lotus node panic (may need some retries to popup):

panic: send on closed channel

goroutine 16547 [running]:
github.com/filecoin-project/lotus/lib/jsonrpc.(*wsConn).handleChanOut(0xc0259acb40, 0x19a8f00, 0xc028947db8, 0x92, 0x16, 0x8)
        /home/ignacio/code/lotus/lib/jsonrpc/websocket.go:209 +0xa6
github.com/filecoin-project/lotus/lib/jsonrpc.handlers.handle.func1(0x7fe29c3f39f0, 0xc028a1b650)
        /home/ignacio/code/lotus/lib/jsonrpc/handler.go:232 +0xde
github.com/filecoin-project/lotus/lib/jsonrpc.(*wsConn).nextWriter(0xc0259acb40, 0xc02773fa40)
        /home/ignacio/code/lotus/lib/jsonrpc/websocket.go:111 +0x1ba
github.com/filecoin-project/lotus/lib/jsonrpc.handlers.handle(0xc02be35c50, 0x1f10a80, 0xc028b3c340, 0xc028965980, 0x3, 0xc028965988, 0xc028906e20, 0x14, 0x2d52878, 0x0, ...)
        /home/ignacio/code/lotus/lib/jsonrpc/handler.go:226 +0xb3d
created by github.com/filecoin-project/lotus/lib/jsonrpc.(*wsConn).handleCall
        /home/ignacio/code/lotus/lib/jsonrpc/websocket.go:362 +0x295

[jsign]

@magik6k, tried again with this extra fix and no more blockings.