Sprint January 11th

[RichardLitt]

Sprint January 11

Sprint Goals

TBD

Sprint Discussions

Schedule

Please take notes in a separate pad, if you can, and link it here.

Endeavour	Lead	Time (PDT - UTC/Z - CET)	Pad
apps on ipfs	Juan Benet	10:30PDT 18:30Z 19:30CET	https://public.etherpad-mozilla.org/p/ipfs-jan-11-apps-on-ipfs
infrastructure	Lars Gierth	11:00PDT 19:00Z 20:00CET	https://public.etherpad-mozilla.org/p/ipfs-jan-11-infrastructure

Please add the Agenda to the Pad before the endeavour sprint starts.

Sprint Deliverables

• Add your tasks below in a comment, that way we only have people listed who are really in the sprint
• Add links to issues down here. Only add things you can finish this sprint.

[daviddias]

I've pushed libp2p to the end of the queue, 21:00Z today, so that me and @jbenet can make it

[RichardLitt]

@diasdavid Sounds good, thanks for taking the initiative.

[dignifiedquire]

@diasdavid does that mean that you and @jbenet are not around for the others before?

[jbenet]

there's a meeting @diasdavid and i are attending (about libp2p related things actually) during both the infra + libp2p hangouts. we decided to move libp2p to the end. i'll just catch up with the infra notes later (unless im needed, in which case maybe move to end too?)

[dignifiedquire]

Moved from the Etherpad:

Apps On IPFS

Lead: @dignifiedquire

Participants

• @RichardLitt
• @noffle
• @Kubuxu
• @diasdavid
• @whyrusleeping
• @jbenet

Agenda

• Webui
• Distributions
• Station status
• Github repos ipfs-improve thing
• Package Managers in general
  • registry-mirror lessons
  • nailing npm and pacman
  • pypi, apt, nix after
• ipget - solid + packaged (npm, apt, brew, etc)
  • current issues: https://github.com/noffle/ipget/issues

Notes

[ghost]

Infrastructure

Lead: @lgierth

Participants

• @dignifiedquire
• @whyrusleeping
• @lgierth
• @RichardLitt

Agenda

• dev040 migration
• ipv6
• ansible/provsn
• more storage
• multiple ipfs instances to avoid data loss, e.g. restart before files api flush

Notes

• storage
  • wikipedia is a bit weird to mirror
  • storage in the u.s. would be really useful
  • @whyrusleeping is going to build a massive array at home -- 72TB and hopefully 1Gb/s
• dev040 migration
  • all the bootstrap/gateway hosts will run both v03x and v04x
    • both instances share the private key
    • either the v03x or v04x instance gets to claim the default ports 4001, 5001, 8080
    • right now it 5 * v03x and 3 * v04x, over time we'll gradually phase out v03x
• ipv6
  • people have been asking for this for a while IPv6 for gateway and swarm infra#88
  • bootstrap needs to deal with multiple addresses for the same node (@whyrusleeping)
  • possible addresses per node:
    • /ip4/tcp
    • /ip4/utp
    • /ip6/tcp
    • /ip6/utp
    • /ip6/tcp with fc00::/8 address
    • /ip6/utp with fc00::/8 address
• ansible
  • all of the big three (chef, puppet, ansible) are really annoying
    • code tends to become a mess if you don't pay really close attention
    • layers of indirection make it hard to test and debug stuff
    • sloooow
  • provsn solves these problems by using plain shell scripts
    • includes file templating and variable inheritance
    • will be published as an independent tools next week or so
    • less than 100 lines of bash at the moment
    • testing with ipfs deployment, will port other stuff too (cjdns, nginx, metrics, pinbot)

[ghost]

@lgierth TODO

• dev040
  • have ipfs.io resolve via both v03x and v04x
  • discuss migration plan with users
  • automatic docker hub build of ipfs/go-ipfs image
• infrastructure
  • fix ownership of sirius and castor directories Fix ownership of sirius and castor directories ipfs-inactive/ops-requests#11
• go-ipfs
  • harden path prefix gateway: harden path prefix kubo#1988
  • rebase: remove dead jenkins code ci: remove dead jenkins code kubo#2114
  • CR resolve _dnslink.example.com Make dns resolve paths under _dnslink.example.com kubo#2184
  • CR combine multiple bootstrap addrs combine multiple bootstrap addrs into single peer info kubo#2199
• archives
  • start mirroring cdn.media.ccc.de
  • debug wikispecies archive with @eminence Wikipedia ipfs-inactive/archives#20 (comment)
• whislist
  • convert ipfs playbooks to using provsn
  • multiaddr /dns support Support DNS  multiformats/go-multiaddr#7
  • convert all of infrastructure.git to using provsn

[whyrusleeping]

@whyrusleeping TODO

• work on ipfs files read hanging when blocks don't exist
• fix bootstrapping to allow multiple addresses
• private networks
• ship 0.3.11
• merge 0.4.0 into master

[jbenet]

@diasdavid @whyrusleeping reminder post your etherpad's notes here

[dignifiedquire]

@dignifiedquire TODO

• PR updated webui for 0.3.11
• Bug fixes for the webui
• New deployment setup for the webui
• Ship distributions

[daviddias]

David Dias TODO

• js-ipfs
  • js-ipfs (cli + http api)
    • block
      • get
      • put
      • del
      • stat
    • config
      • show (core + cli + tests)
      • replace (core + tests)
    • init
    • CR+M Add config command to cli js-ipfs#53 (thank you @xicombd for the PR :) )
  • js-ipfs-merkle-dag
    • Make it conform with IPFS JS guidelines WIP: PR js-ipfs-merkle-dag#1
    • Move it to use IPFS Repo Use IPFS Repo js-ipfs-merkle-dag#3
  • js-ipfs-repo
    • feature: remove block
    • feature: block exists
    • es6'ify plus fix lil things es6ify and fix certain err conditions that were not being returned js-ipfs-repo#23
  • js-ipfs-data-importing
    • Create a protobuf version of the layouting (similar to what go-ipfs does) (a good test is creating it, adding to the repo and then let go-ipfs use it <- this will be so sweet)
    • Extrapolate learnings and merge them into the spec - DEX specs#57
  • bitswap
    • work towards feature parity with go-ipfs first (protobufs style)
  • update project status, make it cleaner to understand (hopefully helping other language implementations) update project roadmap, make it more explicit, help organize js-ipfs#47
• api documentation
• libp2p
  • utp
  • interop go-js swarm (with tests -> libp2p-ci)
• npm on ipfs
  • debug and solve with @whyrusleeping latests issues on files API to get it completely ready
  • update install process with 0.4.0 rc-1
• specs
• extra

[whyrusleeping]

go-ipfs

Lead: @whyrusleeping

Participants

Agenda

• release things
• release test repo
• prev agenda: https://public.etherpad-mozilla.org/p/ipfs-jan-5-go-ipfs
• CRing open PRs

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Notes

TODOS:

• ship 0.3.11 (Tuesday) (@whyrusleeping)
• merge 0.4.0 into master (Tuesday) (@whyrusleeping)
• ship distributions (for update) (@dignifiedquire)
• cleanup PRs and issues (@whyrusleeping)
• go-ipfs-release repo (to test webui + api) (@whyrusleeping)
• solid tests for fs-repo-migrations 2-to-3 (@chriscool)
• document release process (@whyrusleeping)
• document what go-ipfs inits in the repo in specs/repo/fs-repo (welcome docs, etc) (@whyrusleeping)
• blog post about 0.4.0 (@kyledrake)

Future:

• IPLD
• Perf improvements
• sharding

After sprint meeting is finished, create the respective action items on the Github sprint issue

[RichardLitt]

I have added an IPFS Weekly roundup for last week. ipfs/newsletter#12 Feedback due by this time tomorrow.

[RichardLitt]

TODO

• Sprint Management
• Write the weekly roundup
• IPFS API
  • Split into current functions master branch and expected spec branch.
    • Add a link to the api from the specs repo
  • Add curl commands to all method descriptions
  • Check all responses have maximal entries
    • Log any errors in go-ipfs
  • Standardize formatting for ipfs --help entries, basing on 0.4
  • Open an issue about sharness testing.
  • Open a feature issue about having multiple language implementations for curl requests
  • Hope Get sharness testing working
• Hope Research PATENTS.md strategies

[daviddias]

js-ipfs

Lead: @diasdavid

Participants

Agenda

• js-ipfs update
• Data Importing
• The Spec
• The Implementation
• State of IPLD (jbenet?)
• API specs - WIP: IPFS API (core+http+cli) specs#65
• Core
• HTTP (weird mix of RPC vs REST)
• CLI
• Bitswap
• WIP: exchange spec specs#53
• Other stuff that can use review
• init js-ipfs#42

[daviddias]

libp2p

Lead: @diasdavid

Participants

Agenda

• update
• js and go libp2p interop
  • (priority) 1st milestone - level up go-libp2p swarm and have tests between both
  • 2nd milestone - Peer Routing interop
  • 3rd milestone - Record Store interop
• transports
  • utp

You have 30 minutes for this agenda, 5 minutes before the meeting ends, consider moving the remaining items to a github discussion thread so that all the other sprint meetings can start in time.

Notes

https://github.com/nickdesaulniers/node-rtc-peer-connection

[jbenet]

@jbenet TODO

• make decisions for IPLD spec
• dht discussion with other p2p systems
• discuss uPort, ipld, keychain, and more with chris at ConsenSys
• review + publish weekly 2
• distributions site feedback user testing feedback distributions#24
• backlog
• merge Fix appveyor kubo#2137 -- windows CI now works. doesn't pass yet though. (:star:)

(will add more)

[jbenet]

@diasdavid what work do you need from me the most? i know there's some importer/chunker stuff to spec out. what else? what's highest prio?

@whyrusleeping @lgierth same question

[hackergrrl]

TODO

• ramping up on ipfs pm and logistics
• tackle some small go-ipfs issues to ramp up + help with 0.4.0
  • Removing a pin of a hash that does not exist should fail immediately kubo#2155
  • adding hidden files directly should work without flag kubo#2145
  • Mount only /ipfs/ kubo#2187

[daviddias]

@jbenet other than the chunker spec, is getting the exchange spec to a point where it is clear what to do to support both protobufs and IPLD

[rht]

• Add http(s) scheme, e.g. ipget /https/ipfs.com; rht/go-ipfs@2c5eedc (which hashes the file in the intermediate step, but this is not shown)

[whyrusleeping]

• debug random multistream goroutines...
• fix ipfs files 'deadlocks'
  • thought we were good, actually not
• [~] private networks
  • implement 'encrypted transport' thing
• thin out go-libp2p deps
  • go-net needs to die
• [~] directory sharding pls

[RichardLitt]

This Week

This week I spent a lot of time working on the Weekly roundup contributor code. It's not done, but it is very close, and I had a ton of fun doing it. Now that I am with @diasdavid and @whyrusleeping and @noffle in Svalbard for arcticjs, I hope to make really significant inroads on the API this week in person with them.

• Sprint Management
• Write the weekly roundup
• IPFS API
  • Split into current functions master branch and expected spec branch.
    • Add a link to the api from the specs repo
  • Add curl commands to all method descriptions
  • Check all responses have maximal entries
    • Log any errors in go-ipfs
  • Standardize formatting for ipfs --help entries, basing on 0.4
  • Open an issue about sharness testing.
  • Open a feature issue about having multiple language implementations for curl requests
  • Hope Get sharness testing working
• Hope Research PATENTS.md strategies
  • Linux Patents DCO - how do they do it? Docker, Git. Is the DCO all you need? Apache. Does GPL have a patents section? CLAHub

Extra

• CR
  • CR+M node-ipfs was renamed to js-ipfs + fix url for station ipfs-inactive/support#14
  • CR https://github.com/ipfs/fs-repo-migrations/pull/19/files
  • CR Fixes and release of a new version for 0.3.11 ipfs-webui#188
  • CR+M add roadmap to README ipfs-shipyard/py-ipfs#22
  • CR+M Update link to hasteIPFS to link global gateway awesome-ipfs#41
• Closed all open issues and PRs on awesome-ipfs, added note about usable products only. PR awesome-ipfs/pull/40
• Edited pm templates slightly. PR pm/pull/81
• Worked on the weekly blog. PR blog/pull/16, blog/pull/17, blog/pull/18
• Worked on the shutdown notice for go-ipfs. PR go-ipfs/pull/2180
• Authored IPFS Weekly Sprint 1 #2, improved workflow and docs considerable. Sprint 1 #2
• Created tinyletter.com/ipfsweekly.
• Edited dist descriptions. PR distributions/pull/25
• Reversed order of blog. PR blog/pull/20
• Tooling modules:
  • RichardLitt/ignore-github-users for ignoring @greenkeeperio-bot
  • RichardLitt/gh-upstream for going to a GitHub upstream repository
  • RichardLitt/git-remote-upstream-url to get remote upstream url
  • PRed sindresorhus/gh-home adding gh-upstream check
  • Trivial PR to novaugust/top-gh-contribs.
  • RichardLitt/generator-nms as a yeoman generator for node modules using standard
• Added note to go-ipfs/contribute about squashing commits. https://github.com/ipfs/go-ipfs/compare/feature/squash-note?expand=1
• Opened up-for-grabs label community#94

  ipfs/api

• Opened: api#22, api#21, go-ipfs#2190, go-ipfs#2189
• Added awesome_bot to ipfs/weekly

[hackergrrl]

Up at ArcticJS with @RichardLitt, @diasdavid, and @whyrusleeping! Lots of ramping up this week, and took on two issues: one merged, one needing another small revision.

• ramping up on ipfs pm and logistics
• tackle some small go-ipfs issues to ramp up + help with 0.4.0
  • adding hidden files directly should work without flag kubo#2145
  • [~] Removing a pin of a hash that does not exist should fail immediately kubo#2155
  • Mount only /ipfs/ kubo#2187

[daviddias]

• js-ipfs
  • Clean up and Organize the Project Status, make it easy to understand (hopefully helping other
  • js-ipfs impl
    • block
      • get
    • config
      • show (core + cli + tests)
      • replace (core + tests)
    • CR+M Add config command to cli js-ipfs#53 (thank you @xicombd for the PR :) )
  • js-ipfs-merkle-dag
    • Make it conform with IPFS JS guidelines WIP: PR js-ipfs-merkle-dag#1
    • Move it to use IPFS Repo Use IPFS Repo js-ipfs-merkle-dag#3
    • more stuff on Use IPFS Repo js-ipfs-merkle-dag#3
  • js-ipfs-repo
    • feature: remove block
    • feature: block exists
    • es6'ify plus fix lil things es6ify and fix certain err conditions that were not being returned js-ipfs-repo#23
  • js-ipfs-data-importing
    language implementations) update project roadmap, make it more explicit, help organize js-ipfs#47
• npm on ipfs
  • get registry-mirror up again (hit another deadlock, still working through that with @whyrusleeping )

[jbenet]

@jbenet TODO

• make decisions for IPLD spec
• dht discussion with other p2p systems
• discuss uPort, ipld, keychain, and more with chris at ConsenSys
• review + publish weekly 2
• distributions site feedback user testing feedback distributions#24
• [~] backlog
• published 0.3.11 to npm-go-ipfs
• improved npm-go-ipfs publish.sh
• merge Fix appveyor kubo#2137 -- windows CI now works. doesn't pass yet though. (:star:)
• CR + Merges
  • CR + M Build ipfs cmd inside docker container kubo#2213 container bug
  • CR + M fix bootstrapping bug and add real test for bootstrapping kubo#2210 bootstrapping bug
  • CR + M make files stat print out node type kubo#2209 files stat print node type
  • CR Add arguments to 'ipfs pin ls' kubo#2208
  • CR + M test-lib: improve on previous fix kubo#2206 test-lib improvement
  • CR + M Fix test str contains kubo#2204 test-lib improvement
  • CR + M Lets 'ipfs add' include top-level hidden files kubo#2202 ipfs add hidden files directly
  • CR + M combine multiple bootstrap addrs into single peer info kubo#2199 multiple bootstrap addrs per peer (caused bootstrapping bug)
  • CR + M do resolve operations concurrently kubo#2197 resolve ops concurrently
  • CR + M Added env var info to init, dameon help kubo#2195 env var info to init, daemon help
  • CR Remove old DND TXT schema kubo#2193 remove old dns txt schema
  • CR + M cbor & json: add HeaderPath variable multiformats/go-multicodec#9 cbor & json: add HeaderPath variable
  • CR + M Added IPFS Weekly 2 ipfs-inactive/blog#19 blog: weekly 2
  • CR + M fix locking in syncbatch go-datastore#35 datastore: sync batch
  • CR + M Add new webui infra#144 infra pinned webui

[dignifiedquire]

@dignifiedquire Sync Update

Webui

• Ship latest webui with 0.3.11(feat: Update to the latest version of the webui kubo#2188) (:star:)
• Misc bug fixes (Fixes and release of a new version for 0.3.11 ipfs-webui#188)
• [~] CR and help with ES6/7 refactor, Remove jQuery ipfs-webui#195

Distributions

• Improvements based on feedback Improve on the UI distributions#26 (close to shipping)

Misc

• Get appveyor tests for go-ipfs finally merged (Fix appveyor kubo#2137) (:star:)

[ghost]

@lgierth sync update

• dev040
  • [~] have ipfs.io resolve via both v03x and v04x
  • [~] discuss migration plan with users
  • automatic docker hub build of ipfs/go-ipfs image
• infrastructure
  • fix ownership of sirius and castor directories Fix ownership of sirius and castor directories ipfs-inactive/ops-requests#11
• go-ipfs
  • harden path prefix gateway: harden path prefix kubo#1988
  • rebase: remove dead jenkins code ci: remove dead jenkins code kubo#2114
  • CR resolve _dnslink.example.com Make dns resolve paths under _dnslink.example.com kubo#2184
  • CR combine multiple bootstrap addrs combine multiple bootstrap addrs into single peer info kubo#2199
  • discuss recognizing gateways on websites Recognize IPFS gateways on arbitrary websites notes#92
• archives
  • start mirroring cdn.media.ccc.de
  • debug wikispecies archive with @eminence Wikipedia ipfs-inactive/archives#20 (comment)
• whislist
  • [~] convert ipfs playbooks to using provsn
  • multiaddr /dns support Support DNS  multiformats/go-multiaddr#7
  • convert all of infrastructure.git to using provsn