2018 Q4 OKR Planning #5474

daviddias · 2018-09-15T19:45:57Z

Urls:

2018 Q3 Retrospective Document. Please complete this ASAP.
2018 Q4 OKR Spreadsheet

It's time to do the OKR Planning for Q4 \o/. This is the first time that the go-ipfs team is going to do it this way, you can find a lot of information at ipfs/team-mgmt#698. Please make sure to read it to get the full context in how we are going to do this (Retrospective + Open OKR Planning).

OKR.md

keks · 2018-09-26T17:05:38Z

OKR.md

+
+**It is a joy to use go-ipfs programatically**
+- `PX` @magik6k - The Core API is finalized and released. Make it easier to import go-ipfs as a package
+- `PX` - go-ipfs-api exposes the new Core API


@diasdavid did you intend to let this also include the remote API protocol in this? Because I'm not sure it's covered in this document?

My plan for this was to start with http api implementation and when the new rpc api becomes a thing support both as some users will likely need http is some setups

Do you have a proposal for an updated KR that effectively captures that work? Taking PRs! =]

@keks - Do you think the RPC API should be included (even partially) in this document? I don't know anything about that work but is it something that you think should be started in this next quarter?

@magik6k is owner for this whole objective

making go-ipfs-api expose is lower priority (blocked by a lot of go-ipfs changes)

To clarify, I was basically talking about this issue. We'll need to coordinate with the other implementors to find out what we really need, but the issue is that the current HTTP API has a lot of weird edge cases, which is painful for everyone who implements it.
Again, I'm not talking about the RPC methods we expose for others to call (these should mostly be the core api), but about the wire format used to make these calls. If we get that one right, a lot of stuff is going to be easier.

OKR.md

momack2 · 2018-10-01T17:19:35Z

OKR.md

+- `PX` - It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps)
+- `PX` - It takes less than 12 hours to transfer 200GB sharded dataset over Fast Ethernet (100Mbps)
+- `PX` - There is a prototype implementation of GraphSync for go-ipfs
+- `PX` - There is a better and more performant datastore module (e.g Badger or better)


momack2 · 2018-10-01T17:20:08Z

OKR.md

+**go-ipfs handles large datasets (1TB++) without a sweat**
+- `PX` - It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps)
+- `PX` - It takes less than 12 hours to transfer 200GB sharded dataset over Fast Ethernet (100Mbps)
+- `PX` - There is a prototype implementation of GraphSync for go-ipfs


somewhat dependent on js-ipfs graphsync

@hannahhoward already thinking about speeding up directories and may have knowledge of how to implement (but needs a partner for spec work)

momack2 · 2018-10-01T17:26:14Z

OKR.md

+- `P1` - Rewrite pinning data structures to support large data sets / many files performantly
+
+**The bandwidth usage is reduced significantly and is well kept under control**
+- `PX` - Users can opt out of providing every IPLD node (and only provide root hashes)


Includes modifications to the DHT. Need an owner to coordinate with @Stebalien on design. @magik6k started on this, but need to flesh out the design and spec.

Need to rephrase this to be aiming for a spec and early implementation (.5 is a spec but no implementation started)

momack2 · 2018-10-01T17:28:56Z

OKR.md

+
+**The bandwidth usage is reduced significantly and is well kept under control**
+- `PX` - Users can opt out of providing every IPLD node (and only provide root hashes)
+- `PX` - "Bitswap improvements around reducing chattiness, decreasing bandwidth usage (fewer dupe blocks), and increasing throughput"


measure: looking at the number of duplicate blocks and the number of parties we send a want list to when we don't need to
goal: don't want to upload as much as we download to find the content. Don't want to download as many duplicate blocks.

can make larger improvements by changing protocol, but current proposal is internal to feature to improve where we look for data
"reduce number of duplicate blocks by 75%"

momack2 · 2018-10-01T17:31:07Z

OKR.md

+- `PX` - "Bitswap improvements around reducing chattiness, decreasing bandwidth usage (fewer dupe blocks), and increasing throughput"
+
+**It is a joy to use go-ipfs programatically**
+- `PX` @magik6k - The Core API is finalized and released. Make it easier to import go-ipfs as a package


what is a good priority for this?

momack2 · 2018-10-01T18:32:03Z

OKR.md

+- `P0` @hannahhoward - List a sharded directory with 1M entries over LAN in under 1 minute, with less than a second to the first entry.
+- `PX` - There is a prototype implementation of GraphSync for go-ipfs
+- `P0` @magik6k - There is a better and more performant datastore module (e.g Badger or better)
+- `P1` - Rewrite pinning data structure to support large pinsets and flexible pin modes


@kevina interested in contributing to this - not sure about owning per say yet =]

I spent a very small amount of time working on something related to pinning and so I'd be interested in contributing to this objective. I think with enough support maybe I can own it, but I'd need to discuss that with someone who knows more about the system and the OKR to determine whether or not that is reasonable. And I don't want to step on your toes @kevina, if you are motivated to own this work.

Link to the previous discussion: #5474 (comment)

To clarify my interest.

I think this task will be made a lot easier with some architectural changes to the blockstore. I wrote up my first draft of such changes in #5528. I also think that we should use a snapshot based approach to G.C. and completely avoid the need for read and write locks (but with perhaps mutexes that will be held for short durations). In order for that to happen we will need some additional changes. Once I write up this proposal I would like it to be given some serious thought as I think it can solve a lot of problems we are having.

If we agree to these changes I will gladly take lead to help push it though, although completing it in Q4 may be to aggressive a strategy.

I get the general sense that others would generally like to avoid any architectural changes and instead attempt to solve this with an independent data structures. I personally see this as creating additional complexity so I not sure I can agree with this approach. Thus, if we go down that path I probably not a good person to lead this.

So, there are two issues here:

The KR is really about the pin datastructures itself. That is, we need to be able to (a) support flexible pin modes and (b) support many pins.

GC needs to get faster. A generational approach may work, personally, I'd go with a recounting approach. Regardless, I consider that to be an orthogonal issue.

momack2 · 2018-10-01T18:49:02Z

OKR.md

+
+**The bandwidth usage is reduced significantly and is well kept under control**
+- `PX` - Spec and draft implementation of allowing users to opt out of providing every IPLD node (and only provide root hashes)
+- `PX` - Bitswap improvements reduce number of duplicate blocks downloaded by 75%


I think these are P0?

michaelavila · 2018-10-01T22:44:59Z

OKR.md

+- `P0` @magik6k - There is a better and more performant datastore module (e.g Badger or better)
+- `P1` - Rewrite pinning data structure to support large pinsets and flexible pin modes
+
+**The bandwidth usage is reduced significantly and is well kept under control**


I like the idea of contributing to this objective, but I'm probably not the best person to own it.

@Stebalien who do you suspect should own these OKRs? In the absence of a more qualified person (and in an effort to move important goals forward), I'm willing to take on some or all ownership. If there's a better owner, then I can work with them towards this objective.

schomatis · 2018-10-02T14:51:59Z

OKR.md

+- `P0` - It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps)
+- `P0` @hannahhoward - List a sharded directory with 1M entries over LAN in under 1 minute, with less than a second to the first entry.
+- `PX` - There is a prototype implementation of GraphSync for go-ipfs
+- `P0` @magik6k - There is a better and more performant datastore module (e.g Badger or better)


@magik6k I can help with the Badger transition (or take ownership of this one if you want).

Oh, sure, go ahead

Thanks @schomatis, I'll update.

schomatis · 2018-10-02T14:56:09Z

Two objectives I could add to Q4 are:

As suggested by @Stebalien, design a new MFS interface (Review the MFS interface go-mfs#3).
Keep making the code in go-unixfs more understandable (comments, refactoring, bug fixes, high level docs), particularly the HAMT package.

@momack2 Could you help me phrase these ones with a KR-oriented approach?

Kubuxu · 2018-10-02T15:10:35Z

If you think it could be useful, I could re-establish metric tracking based on github's webhooks. We had it set up in the past but it wasn't maintained. It would allow us to track things like PR review time, issue response and others.
It would also give us metrics for the future.

schomatis · 2018-10-02T15:20:25Z

@Kubuxu Yes please! I think those metrisc would be very useful in general, I'm not so sure they would help measure this particular KR (which will impact those times but I think the signal will get lost in the general noise).

That said, I can't think of any useful metric for "making the code easier to reason about" except asking new developers like @hannahhoward if they could provide an estimate of how much time it took them to understand a particular code component (e.g., HAMT) for a particular PR they were working on and keep track of that for the future.

hannahhoward · 2018-10-02T21:18:50Z

I noticed there are no OKR's around Unix-FS V2 -- which seems at minimum related to some of the existing OKR's -- like sharded directory speed -- since a format change might provide order of magnitude improvements.

I'm not sure where UnixFS V2 is in the middle of all this, but it feels relevant to some of these and wouldn't want it to get lost. Also, that seems tied to OKRs around go-unixfs readability per @schomatis -- cause maybe the solution is make UnixFS V2 and do a good job commenting it!

hannahhoward · 2018-10-02T21:55:06Z

^^^ Sorry. Did not see UnixFS V2 is under outstanding Q3 priorities. Still I think a conversation about prioritization and where this fits with the new Q4 OKRs would be useful.

eingenito · 2018-10-02T22:40:47Z

OKR.md

+- `P2` - Every package has tests and tests+code coverage are running on Jenkins
+- `P2` - There is an up-to-date Architecture Diagram of the Go implementation of IPFS that links packages to subsystems to workflows
+
+**gx becomes a beloved tool by the Go Core Contributors**


Who has made major changes to gx in the past? Is it only @whyrusleeping? Are these actually appropriate for someone else to pick up?

@travisperson will probably take these (and @schomatis has been doing some gx work). We have a long discussion here whyrusleeping/gx#179 culminating with an offline discussion between @whyrusleeping, @travisperson and I (key points here whyrusleeping/gx#179 (comment)).

Thanks. @travisperson are you comfortable with my adding your name to these OKRs? Are they pretty accurate as stated, and might they belong in another project?

@eingenito ya, these looks good.

eingenito · 2018-10-02T23:33:29Z

OKR.md

+## 2018 Q4
+
+**go-ipfs handles large datasets (1TB++) without a sweat**
+- `P0` - It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps)


Is there any existing description of the path to achieving this OKR? Is it related to or independent of the effort to minimize duplicate blocks? And I'm assuming it's apart from any DHT traffic. So is it smarter sessions, or improved I/O path, or datastore speed? Or is it a protocol change like graph sync?

... and, some of these OKRs are challenging to assign (or take ownership of) because they seem like a collection of features and improvements that may well be worked on by multiple people.

kevina · 2018-10-04T04:05:01Z

OKR.md

+- `P0` - It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps)
+- `P0` @hannahhoward - List a sharded directory with 1M entries over LAN in under 1 minute, with less than a second to the first entry.
+- `PX` - There is a prototype implementation of GraphSync for go-ipfs
+- `P0` @schomatis - There is a better and more performant datastore module (e.g Badger or better)


@kevina would also like to be somewhat involved with this.

eingenito · 2018-11-26T20:06:46Z

Content has moved to https://docs.google.com/spreadsheets/d/139lROP7-Ee4M4S7A_IO4iIgSgugYm7dct620LYnalII/edit#gid=1720716278

docs: create OKR.md

8fafcf2

daviddias requested a review from Kubuxu as a code owner September 15, 2018 19:45

ghost assigned daviddias Sep 15, 2018

ghost added the status/in-progress In progress label Sep 15, 2018

daviddias mentioned this pull request Sep 15, 2018

2018 Q4 OKR Planning - It's that time of the quarter again ✨ ipfs/team-mgmt#698

Closed

daviddias assigned keks, warpfork, kevina, Kubuxu, magik6k, schomatis, djdv, whyrusleeping, Stebalien and daviddias and unassigned keks, warpfork, daviddias, kevina, Kubuxu, magik6k, schomatis and djdv Sep 15, 2018

Stebalien reviewed Sep 26, 2018

View reviewed changes

OKR.md Show resolved Hide resolved

keks reviewed Sep 26, 2018

View reviewed changes

momack2 reviewed Sep 28, 2018

View reviewed changes

OKR.md Outdated Show resolved Hide resolved

momack2 reviewed Oct 1, 2018

View reviewed changes

momack2 added 2 commits October 1, 2018 11:18

Update with priorities, owners, and new phrasing

b974a06

update pinning KR

4a9cb1b

momack2 reviewed Oct 1, 2018

View reviewed changes

Update OKR.md

918d28b

michaelavila reviewed Oct 1, 2018

View reviewed changes

schomatis reviewed Oct 2, 2018

View reviewed changes

eingenito reviewed Oct 2, 2018

View reviewed changes

Update OKR.md

fb20b0f

eingenito reviewed Oct 2, 2018

View reviewed changes

momack2 mentioned this pull request Oct 3, 2018

Create 2018-10-01--go-core-dev-team-weekly.md ipfs/team-mgmt#711

Merged

Updates to gx and new for @schomatis

aad9e7f

kevina reviewed Oct 4, 2018

View reviewed changes

eingenito closed this Nov 26, 2018

ghost removed the status/in-progress In progress label Nov 26, 2018

daviddias deleted the 2018-Q4-OKRs branch November 26, 2018 20:24

2018 Q4 OKR Planning #5474

2018 Q4 OKR Planning #5474

Conversation

daviddias commented Sep 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevina Oct 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schomatis commented Oct 2, 2018

Kubuxu commented Oct 2, 2018

schomatis commented Oct 2, 2018

hannahhoward commented Oct 2, 2018

hannahhoward commented Oct 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eingenito commented Nov 26, 2018

kevina Oct 2, 2018 •

edited

Loading