pin cmd: stream recursive pins

[magik6k]

This is api breaking change.

When doing pin ls on large repos nothing will happen until all pins have been gathered, with this we stream them as we traverse the recursive pin children

[Stebalien]

I take it this now returns the pins as separate objects instead of as {keys: [pins...]}? That definitely sounds like an improvement but breaking things is... unfortunate.

One solution is to add a --stream flag and turn it on by default (on the client). I wish we had a nice, standard way to add features like this.

Thoughts @diasdavid? This change will affect JS as well.

[daviddias]

I like this proposal. Before merging, can we test it with js-ipfs-api and make sure that the X-Stream/X-Chunked headers are being well set and that we do get a proper ndjson stream?

[whyrusleeping]

@diasdavid for sure.

Before making this change we should also get buyin from all users of this api. which is a pretty strong task.

It may also be worth identifying other places where a similar change could make sense to do, and maybe do them all at once

[kevina]

One solution is to add a --stream flag and turn it on by default (on the client). I wish we had a nice, standard way to add features like this.

Yes I would something similar. I have done similar things with other commands I believe.

[whyrusleeping]

@diasdavid is there a js-ipfs-api test for this that we can easily run?

[daviddias]

@whyrusleeping running js-ipfs-api tests will run you through these -- https://github.com/ipfs/interface-ipfs-core/blob/master/js/src/pin.js -- which has a test for pin ls with recursive type.

[magik6k]

So I've tried to move this, but I'm not that familiar with cmds lib:

• the --stream flag works with curl just fine
• the Type depends on whether or not the flag is set so I had to leave it out, which is a bit problematic for the Encoder.
  • Should I just treat the data I get there as a unmarshalled json (i.e. try casting to map[string]interface{})
  • I could just emit multiple RefKeyList objects, but the json output would be rather weird.
• X-Chunked-Output: 1 is now set in headers regardless of whether or not the --stream flag is set. I'm not sure if this is caused by rewrite to the new cmd lib, lack of the Type field or something else.

[Stebalien]

New version in #6493.