⚡️ v0.27.0 RELEASE 🚀

[daviddias]

🚀 This release brings a new transport to the libp2p family, perf improvements and stability (add files with 10x the size!), Windows support, bug fixes and more!
🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟🌟

🗺 What's left to release

• Bring back PubSub interop (with tests!) PubSub Interop Tests and CLI+HTTP-API Implementation #1081
• Finish the deployment of the new rendezvous endpoints fix: disable WebRTC by default and add how to support it + websocket-star to the FAQ #1111
• Fix multiaddrs for *-star transports fix: update *-star multiaddrs to explicity say that they need tcp and a port #1117
  • Needs to support addrs with /tcp/port too libp2p/js-libp2p-webrtc-star#129
  • Support multiaddrs with tcp/port/{ws, wss} as well libp2p/js-libp2p-websocket-star#37
• Implement Crypto Puzzle for js-libp2p-webrtc-star - Implement the same Crypto Puzzle that websocket-star has libp2p/js-libp2p-webrtc-star#128 (next release)
  • Fix peer-id unmarsheling of keys Catch error when unmarshaling instead of crashing libp2p/js-peer-id#72
    • Fix libp2p-crypto unmarsheling of keys Catch error when unmarshaling instead of crashing libp2p/js-libp2p-crypto#113
• Deploy a websocket-star-rendezvous with CryptoPuzzle enabled

🔦 Highlights

🗂 The Files API has been upgraded with a new Streaming + Buffered interface

Now you can add and fetch Files from IPFS using your favorite streams library, Readable Streams (aka Node.js Streams) or Pull Stream, you pick! Also, if you don't need to handle large files or if you are ok with buffering the whole thing, you can also use the Callback or Promises API for simplicity.

See function signatures, descriptions and examples at: https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md

Ref:

• New Streaming and Buffered Interfaces ipfs-inactive/interface-js-ipfs-core#162
• Exposed Interfaces (Callbacks & Promises, node.js streams and pull-streams) #557

🌟 New transport! websocket-star

js-ipfs has a new WebSockets relayed transport, a similar transport to webrtc-star but more stable (less memory hungry). With this addition, we disable webrtc by default and added instructions on how to enabled it to our FAQ so that you can pick the best transport (webrtc-star vs websocket-star) for your needs.

Big thank you to @mkg20001 for creating websocket-star in the first place! :)

📦 You are now able to add any directory size and depth to IPFS with the new multipart stream builder

js-ipfs-api got a big update with ipfs-inactive/js-ipfs-http-client#629, upgrading the way we constructed multipart streams to send large amounts of Files to an IPFS daemon. Thanks to this, now js-ipfs CLI can any any directory size to IPFS.

Thanks to @pgte for shipping it in a flash!

🏡 Windows Support

Yeeaaas! js-ipfs now supports Windows -- #1017 (comment) --!! Please send a huge hi5 to @richardschneider for slaying all the dragons on this one.

🦁 Safari Support (both Mobile and Desktop)

This is not necessarily a 0.27 update, but it is something you can do now! What this means is that now there are two mobile browsers that can run your Distributed Applications! (Chrome on Android and Safari on iOS)

🚥 More interop tests, including PubSub

Out set of interoperability tests is growing to make sure that JS and Go IPFS stay compatible at all times. The biggest new addition were PubSub tests, see these and more at https://github.com/ipfs/js-ipfs/tree/master/test/interop

👢 Bootstraper tests

Quick bootstraper tests were added -- #899 -- so that our Infrastructure team can always check that js-ipfs is able to contact the Bootstraper and Gateways after a configuration change. This should mitigate downtime issues we observer in the past.

✨ ipfs.ls is now implemented!

This was a long time coming. Thank you to @pgte for shipping this one.

💫 ipfs file ls is now implemented too!

You know when they say that an implementation of S3 is bug by bug compatible? Well, in our case, js-ipfs is getting command by command compatible with go-ipfs so that any app uses go-ipfs as a daemon can use js-ipfs too. This one was shipped by @richardschneider, thank you!

📶 Progress bar!

You can now get progress reports while adding a file to IPFS, both through the CLI (progress bar) or through the API, see how here: https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md#add

Thank you @dryajov for shipping this!

🎢 Performance and Memory Improvements with pull-block

@Beanow improved the performance of pull-block, one of the js-ipfs-unixfs-engine dependencies, reducing vastly its memory fingerprint. What this translates too is to a faster and less memory hungry ipfs.add. I've successfully managed to added multiple files of 750Mb to a browser application using js-ipfs, this is more than a 10x improvement over previous reports!!

✅ Release Checklist

• Robustness and quality
  • Ensure that all tests are passing, this includes:
    • unit
    • interop
    • sharness
  • Run tests of the following projects with the new release:
    • ipfs-pubsub-room
    • peerpad
• Documentation
  • Ensure that README.md is up to date
  • Ensure that all the examples run
• Communication
  • Create the release issue
  • Announcements (both pre-release and post-release)
    • Twitter
    • IRC
    • Reddit
  • Blog post

🙌🏽 Want to contribute?

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

• Check the issues with the help wanted label at the Ready column in our waffle board - https://waffle.io/ipfs/js-ipfs?label=help%20wanted
• Join an IPFS All Hands, introduce yourself and let us know where you would like to contribute - https://github.com/ipfs/pm/#all-hands-call
• Hack with IPFS and show us what you made! The All Hands call is also the perfect venue for demos, join in and show us what you built
• Join the discussion at http://discuss.ipfs.io/ and help users finding their answers.

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode.

[haadcode]

@diasdavid, I ran tests from master against fresh version of ipfs-log and soon-to-be-released new version or OrbitDB and the tests are running all fine! 👍 🎉 There are a bunch of replication and pubsub tests as well as file and object API tests, so it looks solid. There's couple of API breaking changes that I found that I think should be highlighted:

• ipfs.cat() --> ipfs.catReadableStream() (when previously used with readable-stream)
• pubsub message topicCIDs --> topicIDs

Feel free to use the above PRs for testing the before final release and let me know if you need any help.

[pgte]

@diasdavid ran the tests for ipfs-pubsub-room, and released version v0.6 of it that supports the new pubsub message format. All tests pass.

Updated y-ipfs-connector (released version 1.6.0 of that) and tested peerpad with it. Also, all looking good! 👍

[daviddias]

@pgte @haadcode mind testing again with latest master?

[daviddias]

Question: Shall we disable WebRTC by default?

I'm leaning towards yes. Seems that we are not yet ready to use WebRTC without causing confusion to the users (see: #1089). #1088 is one of our top priorities and I would be more comfortable enabling WebRTC-Star or WebSockets-Star once we make more progress.

This doesn't mean that WebRTC won't work, you just need to add the multiaddr manually so that it gets activated.

Thoughts?

[Beanow]

Question: Shall we disable WebRTC by default?

Yes for me as well.

Regardless of how you use WebRTC there's always the risk of using one too many resources and crashing. Though the primary reasons to be using it is having out-of-the-box peer discovery. Which you either want to go "true P2P" the easy way, or to have pubsub.

For both I think we should add clear configuration examples. As I would expect that to top the FAQs when WebRTC is not available by default.

[interfect]

Without WebRTC enabled by default, how will browser nodes be able to communicate with each other? Will WebSockets allow that?

Will turning on WebRTC manually just bring back the browser crashes?

[daviddias]

@interfect, it will prevent the crashes and avoid causing confusion to users that do not require to connect to every browser node.

The proper solution for the problem will be #1088

[daviddias]

Update: PR here #1111

[daviddias]

I'm working towards libp2p/js-libp2p#122 to give another option for Browser connectivity which in theory and from tests run, will be way more stable than the current WebRTC situation.

[daviddias]

I believe that this PR -- #1111 -- will make a lot of people happy :)

[Beanow]

@diasdavid yay! 🎉

As a sidenote this may be a good moment to review how we should warn users about the unsafe rendezvous. As this is the first time it should work out-of-the-box. The WebRTC crashes should have been a big enough deterrent for most production use until now.

Ideally it would be clear during development and runtime, but at the same time allow for upgrading to requiring safe variants without config changes later (provided they didn't manually configure an unsafe rendezvous server).

[daviddias]

@Beanow I realized today that we can actually have their pseudo safe version :) (I wouldn't call anything safe or secure until properly audited). See https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/libp2p-browser.js#L15

[Beanow]

Ah that's right. It's only if you want to add the module yourself it becomes an issue.
Perhaps then we should add something like: new WebSocketStar({allowUnsafe: true}) as an explicit requirement before attempting to connect without ID challenge. For those that wish to pass the module.

[daviddias]

@Beanow The way it is built right now is that the crypto puzzle validator is the rendezvous point, if someone wants to "pass in the module" they will have to use another rendezvous point.

[Beanow]

@diasdavid which would be OK if there is a distinction in addresses between the two modes and the client can validate the expected behavior of the rendezvous. Like http vs https.

In other words I'm suggesting WebSocketStar should be the https example and downgrading security properties should be explicit. Similar to how you would need to allow self-signed certs explicitly in an https client.

[daviddias]

Understood and agreed, @Beanow, although I believe it is not clear what the "unsafe" wording being used in these rendezvous points means.

To clarify, libp2p and therefore IPFS as well, will always validate that they are trying to dial to the right node through the SECIO handshake.

That the crypto puzzle that the rendezvous does is to enable that point to also be able to check that it is being dialed by the right now, avoiding someone to pollute the routing table.

This is a known issue and our proposal to fix goes in two stages:

• a) enable the security guard that @mkg20001 implemented in both webrtc-star and websocket-star (already done).
• b) Upgrade libp2p module loading structure so that transports can use other components already loaded (as per initial design but then it didn't get into the implementation) so that instead of having to use rendezvous points, they can dial to any IPFS node and make any IPFS node a rendezvous point. I've discussed this proposal in length with multiple people and now @pgte is taking the lead for it, track here dependency injection during boot time libp2p/js-libp2p#130

[mkg20001]

implemented in both webrtc-star and websocket-star

@diasdavid webrtc-star only has type checking to prevent server crashes. crypto challenge isn't added.

[daviddias]

@mkg20001 opened a issue to track that libp2p/js-libp2p-webrtc-star#128 :)

[Beanow]

@diasdavid true, safe / unsafe is pretty polarizing for a subject like security in general, we'd need an unambiguous term for the flag.

While DI and SECIO are great, those depend on the transport client, it doesn't make directly instantiating the transport client more explicit in it's behavior.

Though in spite of SECIO, being able to trivially proxy traffic still lets you collect metadata and gives you handshakes and cyphertext to store, replay or attempt any other kind of attack on. A situation that proved pretty detrimental for WEP and WPA.

[daviddias]

@Beanow absolutely correct :) Just so you know, we are indeed working towards getting IPFS audited. Both known and unknown issues will get a priority tag so that we can work on patching them.

[mkg20001]

A good solution would be to add an "allowJoinWithDisabledChallenge" (let's make the name long and annoying to type so nobody uses it :D ) parameter and throw listening errors if a user tries to listen on an insecure server and the flag is false.

[daviddias]

Let's make sure to continue those discussions, especially in the issues for js-libp2p and the transports.

In another news, just went through the items on the Release checklist:

• docs are updated
• tests are passing
• pubsub-room and peerpad work with new js-ipfs and also with the new transport!

It is time to hit the drums and announce that the release is ready and invite everyone to try it out before publishing to npm! 🚀

[Beanow]

@diasdavid hate to be nitting over this buuuut. I like @mkg20001's solution, but it's the very definition of an API breaking change.

Which means: libp2p-websocket-star goes from ^0.5.1 to ^0.6.0 in the direct dependencies and doesn't automatically end up in this release. Which is why I'm bringing it up now and on this issue.

[daviddias]

@Beanow I'm confused by your statement. libp2p-websocket-star latest is v0.5.1, it has the crypto puzzle and there is nothing in the pipeline for that one now. What were you trying to convey?

Update: If you were trying to say that adding the Crypto Puzzle to webrtc-star will be a breaking change, I do not necessarily agree or disagree because it depends on the migration path we set up for users. We could:

• Start by allowing both to work (as it is now) and in a future release of js-ipfs disabling the handshake without the Crypto Puzzle. This way we mitigate the effects and leave space for smooth migration.
• Just ship it right now and encourage all users to upgrade. This would result in delaying this release or just shipping another version once it is done.

I'm more inclined to the smooth transition path.

[mkg20001]

@diasdavid @Beanow Added the requested changes to libp2p-websocket-star

Should I ship it as 0.5.2 or 0.6.0?

[Beanow]

@diasdavid adding a constructor argument such as allowJoinWithDisabledChallenge is an API change. And if we release js-ipfs@0.27.0 with ^0.5.1 for the dependency, it won't satisfy this API change when it lands.

Anyone using the built-in libp2p-websocket-star@0.5.1 from js-ipfs@0.27.0 would be vulnerable to silent downgrades. For example by changing the rendezvous server address.

[daviddias]

We do not need to set up this flag allowJoinWithDisabledChallenge at all, the transport has access to the PeerId, it can always execute the Crypto Puzzle. See https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/libp2p-nodejs.js#L16

Also, thanks for catching the ^0.5.1, that is a typo. We don't release pre 1.0.0 deps with ^, only ~. That said:

If in the future a peer agrees to join without the Crypto Puzzle (because its users set it up like that), a previous node that doesn't support that mode won't get downgraded as it will just try to do the Puzzle with the other peer and then fail.

I might be missing something really obvious here. My apologies for making this convo long and possibly frustrating. I'm sure we can get to the bottom of it :)

[mkg20001]

@diasdavid No if the id is set it can still join to a server without the puzzle! (Fixed in 0.6.0)

[daviddias]

@mkg20001 ah, it was a bug on js-libp2p-websocket-star then, got it. Glad to you already fixed it. Let's make sure to track these things in issues when they pop up.

[ghost]

The /p2p-websocket-star addresses need fixing, they lack the /tcp part which is essential for TCP-based addresses, see libp2p/js-libp2p-websocket-star#35

[daviddias]

@lgierth the transport actually supports multiple variations https://github.com/whyrusleeping/js-mafmt/blob/master/test/index.spec.js#L87-L92. We have been doing the same for p2p-webrtc-star for while too https://github.com/whyrusleeping/js-mafmt/blob/master/test/index.spec.js#L75-L80

Are you proposing that we should fail on a multiaddr that doesn't explicitly call the tcp+443+wss?

[ghost]

Are you proposing that we should fail on a multiaddr that doesn't explicitly call the tcp+443+wss?

Yes that would be good -- they're incorrect multiaddrs: /p2p-websocket-star requires a stream underneath, and /ip4 and /ip6 don't provide that capability. (In the case of /p2p-webrtc-star it's okay since there's backward-compat concerns, but the old-style addresses should be removed at some point too.)

It's not critical to remove the support right now, but at least the docs and tests should be updated before release, so that nobody starts to use these addresses in the first place.

[mitra42]

is pubsub intended to replace YJS, or is YJS using ipfs-connector running over it, or is the best practice to configure YJS differently to use pubsub ?

Obviously there are parts of YJS crucial to pubsub type applications - which dont appear to be in the API - especially the ability to get the history of a channel in order for a newly joining peer to build the current state. Is it intended that YJS should be the way to do this going forward ?

[haadcode]

@diasdavid tests in orbit-db running fine with the latest master 👍

[daviddias]

Hi all, I created an issue for us to track the path moving out of the centralized rendezvous points which also brings a lot of context to why they exist and why they still exist -- libp2p/js-libp2p#134.

It's not critical to remove the support right now, but at least the docs and tests should be updated before release, so that nobody starts to use these addresses in the first place.

Alright, I'll update it everywhere! Thanks for reviewing, this why I like to create an announcement first and wait a bit to release :)

Update: Track: #1117

@diasdavid tests in orbit-db running fine with the latest master 👍

Woot! Thanks @haadcode! Does that mean that there will be a new orbit.chat soon with latest js-ipfs?

is pubsub intended to replace YJS, or is YJS using ipfs-connector running over it, or is the best practice to configure YJS differently to use pubsub ?

Good question. y.js using it's ipfs-connector uses PubSub already to propagate it's events (see the code: https://github.com/ipfs-shipyard/y-ipfs-connector/blob/master/src/index.js#L33).

One is not designed to replace the other, in fact, one enables the other. What you see happening is:

• PubSub is innovation that has made its way into the IPFS Core Protocol
• a sync'able KeyValue Store such as the one powered by y.js is still in IPFS Userland. That is why you have to attach it

We do have the intention to have a KeyValue Store inside IPFS but that is still at its early stage -- ipfs-inactive/interface-js-ipfs-core#157 --. The first step to that will be getting the API and a KeyValue store that works only locally (like MFS) and then design its syncing layer. Again, that is still very early stage, until then continue to check out things like y.js.

[mitra42]

@diasdavid - thanks for the clarification. To be clear, we don't use YJS for KeyValue store, we use it for Append-Only-Logs, a critical building block for a lot of apps, and very hard to do in IPFS by any other means.

[haadcode]

Woot! Thanks @haadcode! Does that mean that there will be a new orbit.chat soon with latest js-ipfs?

@diasdavid I won't have time to update orbit.chat anytime soon, but it is on my TODO list (we have API breaking changes in the new orbitdb, so orbit.chat needs a bit of work too).

we don't use YJS for KeyValue store, we use it for Append-Only-Logs

@mitra42 you might find https://github.com/orbitdb/orbit-db and its Log Database helpful for out-of-the-box append-only logs "natively" on IPFS. We just released a major new version that fixes many of the problems you might've had with previous versions.

[mitra42]

@haadcode - the semantics of orbit-db append only logs, dont quite match, but I'll send you an email since its off-topic for release v0.27.0

[daviddias]

Thanks @alexsicart for trying out this pre-release \o/ Quoting: https://twitter.com/alexsicart/status/937305396263473152

"Thanks! I've tried and the performance improvement is hugee!! Also less memory usage."

[daviddias]

Notes on upgrading from js-ipfs 0.26 to 0.27

• The Files API got an upgrade, make sure to check it out at https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md (and see update on the first post of this issue)
• The PubSub API got a small update that changes the key topicCID to topicID to match go-ipfs API
• WebRTC is now not enabled by default, please check the FAQ https://github.com/ipfs/js-ipfs#faq

[daviddias]

js-ipfs 0.27 released!

Tweet: https://twitter.com/daviddias/status/937565486912606208