OKRs - 2019 Q1 Go Core Dev #794
Conversation
ref #792 - [ ] [Score 2018 Q4 OKRs](https://docs.google.com/spreadsheets/d/139lROP7-Ee4M4S7A_IO4iIgSgugYm7dct620LYnalII/edit#gid=1720716278) - [ ] [Async Retrospective](https://docs.google.com/document/d/1gdxLWcFZPQN5921Br2kSEJqMjHx-jO5JTDEbGExMSuE/edit) - [ ] 2019 Q1 OKRs Open Planning (this thread) - [ ] Move 2019 Q1 OKRs to [2019 Q1 OKRs Spreadsheet](https://docs.google.com/spreadsheets/d/1BtOfd7s9oYO5iKsIorCpsm4QuQoIsoZzSz7GItE-9ys/edit#gid=1720716278)
ghost
assigned 
eingenito
assigned michaelavila, keks, travisperson, Stebalien, kevina, hannahhoward, Kubuxu, magik6k and schomatis
|
I've just assigned this issue to the IPFS core dev team. @warpfork I'm mentioning you here; I'm not sure if IPLD has separate OKRs. First if you had OKRs assigned to you for Q4 2018 please review and update the Q4 column in the Q4 2018 OKRs Spreadsheet to make sure you agree with the completion status of the OKRs. I copied the predicted value that we did mid-quarter into this column. |
|
I'm going to seed this discussion with the leftover OKRs from Q4 2018. I've crossed ones out that I don't understand or that I think are no longer (or never really were) a priority. This is just opinion and some of it uninformed so feel free to champion and of these OKRs that you think are still important. Data transfer is fast and efficient
Adding and providing large data sets works efficiently
Datastore works efficiently for large data sets
go-ipfs becomes a well maintained project
gx becomes a beloved tool by the Go Core Contributors Travis are you still working on this?
Complete outstanding endeavours that are still high priorities from Q3
|
OKR/GO_CORE.md
Outdated
| * Implement provider strategy such that a user can add (and provide) npm or tr-wikipedia without turning off providing and without significantly impacting finding content | ||
|
|
||
| ### Datastore works efficiently for large data sets | ||
| * Final go/no-go decision on Badger (full design comparison vs other options + final decision) |
There was a problem hiding this comment.
Maybe this should be something like:
A fast, stable datastore capable of supporting large data sets > (?)TB has been selected and released as experimental
There was a problem hiding this comment.
That's pretty much where badger is right now
There was a problem hiding this comment.
Right - good point. Maybe this OKR should be about a new default datastore that has the right characteristics for large data sets.
OKR/GO_CORE.md
Outdated
| * A work-tracking process is adopted that allows the team to easily track the highest priority unclaimed issues, see the status of in progress work, and know which PRs need review, easily. | ||
|
|
||
| ### gx becomes a beloved tool by the Go Core Contributors Travis are you still working on this? | ||
| * go-ipfs doesn't have checked-in gx paths |
There was a problem hiding this comment.
@travisperson is this still a goal of yours? I don't know if you're still working on gx udpates or not.
There was a problem hiding this comment.
I think a goal should be to either onboard a contributor/full-time maintainer excited about pushing gx forward (@warpfork - any ideas from recent conferences?) and really double down on making this a powerful proof of concept that is friendly and useful for go developers, or adopt @Stebalien's proposal to optimize for developer velocity in the short/mid term.
There was a problem hiding this comment.
Yah - absolutely agree; one of those is going to be an OKR. And possibly even both.
There was a problem hiding this comment.
"Optimizing developer velocity" could be the objective. The KR would be "go-ipfs developers are no longer adversly affected by gx" (with some significant rewording). That way, either getting a maintainer or moving away from gx are valid solutions.
|
@eingenito while I don't feel able to move on graphsync until there's a spec (still none), what I would like to do is apply "DAG smarts" to driving bitswap. Meaning, if I'm walking the DAG, instead of simply driving Bitswap with a session, I might use one of Bitswap lower level APIs in concert with smarts about the DAG -- namely, a peer that provides a parent is likely ot have a child, and probably some siblings. This also entails completing the seperation/refactor of bitswap into lower and higher level components -- "Just as for blocks from specific peers" low-level bitswap, and sessions/block getting interface high-level bitswap. If I had to express this as an OKR, I'd say: "Transfer of Merkle DAGs is faster using Bitswap than asking for blocks individually" As an aside, this is also important for supporting providing strategies which are not "provide every block in a tree" |
|
@momack2 would absolutely love some collaboration on the IPTB work i'm doing. particularly bringing IPTB to a clustered environment! |
|
@hannahhoward Being the one working on Graphsync from the JS side, I can only say that this sounds like a great plan. Indeed the won't be a fully fleshed out Graphsync spec soon. I will also rather concentrate on making small steps (i.e. being able to replace Bitswap with some other system), rather than thinking top down. The existing Graphsync proposals will be a guidance for me though. |
|
@hannahhoward and @vmx that sounds like a fine OKR. I'm starting to wonder how we'll figure out if a DAG smart bitswap is worth implementing? I guess we need to know how often IRL bitswap stalls after broadcasting and waits around for provider queries to complete? Or maybe how often bitswap stalls at all? I wonder if we can get this information off logs from our gateway. I wonder if a good OKR might be to add better logging to bitswap to get better real world data off the nodes that we're running. |
|
I'd like to be able to continue / focus more on CoreAPI work, mostly around implementing http client, extracting interface and creating standard tests. Also, there is still a lot of work in CoreAPI land (in order of impact):
As for other OKRs, should we have unixfsv2 on our radar or is this more of IPLD thing? |
|
@bigs -
@hsanjuan would that be a better fit for the cluster wg or are there go-ipfs dependencies? @hannahhoward and @vmx - agree that a solid/fleshed out graphsync spec is still outstanding, but curious if we have enough to get started on selectors in Q1 or if that's something that's on the IPLD OKRs to finalize first. |
|
High priority items from our roadmap and 2019 Project Epics that I don't see covered in the proposals so far:
There are also a few goals about the contributor experience (documentation, extensible interfaces, resolving in-flight refactors) that will accrue value the sooner we can allocate time to them. Do we have bandwidth to take on any of these items this quarter? Given what we're putting on our Q1 OKRs are we on track to achieve these ambitious and impressive goals? |
@momack2 I'd postpone selectors. For Graphsync proposal (C) you'd only need the "Path Selector" (the "CID Selector" really is a subset). Not sure if it's worth starting to implement the full selectors thing. |
|
CoreAPI isn't really a refactor, but it may be used to eventually replace / organize a good chunk of the 'glue-code' in go-ipfs. The main focus is providing a good external API interface which can support many backends, like directly using You can see it's current state in ipfs/kubo#4498. I think something like 'Extract CoreAPI interface from go-ipfs' and 'Implement CoreAPI with HTTP backed' would be more accurate. |
You're right - this is actually a Libp2p request for faster DHT, which I think is something on their roadmap (@mgoelzer and @raulk to keep me honest) |
OKR/GO_CORE.md
Outdated
| * A work-tracking process is adopted that allows the team to easily track the highest priority unclaimed issues, see the status of in progress work, and know which PRs need review, easily. | ||
|
|
||
| ### go-ipfs is easy to integrate | ||
| * A new flexible and performant RPC API is designed |
There was a problem hiding this comment.
I think CoreAPI and Base32 actually fall under this objective.
OKR/GO_CORE.md
Outdated
| * A work-tracking process is adopted that allows the team to easily track the highest priority unclaimed issues, see the status of in progress work, and know which PRs need review, easily. | ||
|
|
||
| ### gx becomes a beloved tool by the Go Core Contributors Travis are you still working on this? | ||
| * go-ipfs doesn't have checked-in gx paths |
There was a problem hiding this comment.
"Optimizing developer velocity" could be the objective. The KR would be "go-ipfs developers are no longer adversly affected by gx" (with some significant rewording). That way, either getting a maintainer or moving away from gx are valid solutions.
I'd like to make this push after we get the CoreAPI in place and after we get the new RPC system. Basically, I'd like to avoid spending too much time documenting interfaces that are going to be deprecated.
In service of this:
|
OKR/GO_CORE.md
Outdated
|
|
||
| ### go-ipfs becomes a well maintained project | ||
| * Every non-trivial PR is first reviewed by someone other than @Stebalien. | ||
| * A work-tracking process is adopted that allows the team to easily track the highest priority unclaimed issues, see the status of in progress work, and know which PRs need review, easily. |
There was a problem hiding this comment.
I'd like to make this push after we get the CoreAPI in place and after we get the new RPC system. Basically, I'd like to avoid spending too much time documenting interfaces that are going to be deprecated.
@momack2 convinced me that ignoring documentation is a bad idea (ok, fine...) and suggested
- Every file with a non-trivial change gets fully documented.
Yes, this is going to be hard, but I think that's the only way we're going to make this happen.
There was a problem hiding this comment.
🙌 this allows us to improve and update our documentation incrementally as we touch/grok the code in service of our Q1 efforts. To measure this, we can pull all changed files in Q1 and check documentation coverage (and maybe spot-check documentation quality to ensure it's above par). Our goal with updating the docs is to make it easier for other contributors (new or old) to quickly onboard on modules and help move them forward.
OKR/GO_CORE.md
Outdated
|
|
||
| ### Complete outstanding endeavours that are still high priorities from Q3 | ||
| * base32 is supported and enabled by default | ||
| * CoreAPI refactor is completed |
There was a problem hiding this comment.
I've added the following to the spreadsheet:
- New fuse mount implementation replaces the existing one.
It really depends on how smoothly things go, I kind of wanted to leave this for Q2, and use Q1 for smaller, but related cleanups all around go-ipfs (things like ipfspath (go-path), which are used a fair bit outside go-ipfs and will very likely take time to complete (go-path is just an example, not necessarily blocking constructor refactor)). |
|
@eingenito in response to @momack2 's query I would support "a testing and benchmarking of data transfer speeds at a higher level than bitswap" OKR. We have gotten a lot of instrumentation into Bitswap actually, but after the speed improvements that are likely to be complete this quarter, we really need to move up to the level of IPFS benchmarking as a whole to measure real world transfer speeds. It's hard to know what exactly this would look like in as a repeatable, runnable test but I feel like we should explore it. I'm putting this in my retrospective which I'm of course late delivering so you don't have this information but I think it's relevant. If I were to express this as an OKR, I'd list as something like this: I'd actually rather not get more specific cause I think figuring out what that would look like is part of the OKR. What do you think? |
I agree . I think that would be great and it dovetails with @bigs desire to have collaboration on extending iptb @Stebalien you probably know these OKRs from last quarter:
We can put them in this quarter's OKRs as well. I'm trying to be spartan with our Q1 OKRs so far because we overestimated last quarter and staffing seems uncertain. |
That works for me |
You're right... but cluster really needs this. I'll think about this a bit. There may be a simpler approach to unblock them. |
I think so. Although depending on the solution, the pubsub subsystem might also be involved. See https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipns-pubsub. |
@Stebalien and @momack2 After talking it over with @michaelavila yesterday I'm going to add these for real and make him the owner. We sort of have to decide how far we want to take the providing work before switching focus to pinning/GC. We're intrigued by the possibility that the provider work, if it does expand to include tracking roots, and even counting references to blocks from providing roots, might have a lot of commonality with a lockless GC implementation. And we've touched on the possibility of coupling (re)providing and pinning more closely so there might be a thing emerging there. That's obviously not a simpler approach so by all means, it would be great if there were a quicker option. But even without one I think we'd have a good shot at getting them something in Q1. |
|
I had a long chat with @lanzafame yesterday about the cluster need for node+children pinning. Sounds like an alternate way to solve this problem for them would be to make pinning a single node much much faster - so they could make N pin calls (for each shard + the root) instead of needing a 1 layer recursive pin call to pin all shard roots from the master root. Graphsync selectors are intended to apply to both getting and pinning - so we should be careful to not to redesign a different approach. @lanzafame, if you could partner with @michaelavila on this pin/provider management work I think it could be really useful to drive this forward while keeping all constraints in mind. |
|
Hello go-ipfs folks - It's time for us to finalize our OKRs for Q1 2019! Goal is to have this done by EOWeek, so let's get on it. =] There's a quick tutorial at ipfs/roadmap#17 for a simple Roadmap timeline exercise to help us identify and reach consensus on the most important Q1 priorities. Knowing this working group, I suggest we do this asynchronously and if needed check in quickly at our next weekly meeting. As a reminder, this is how we divvied up responsibilities for drafting our 2019 Roadmap (ipfs/roadmap#4):
Everyone is also encouraged to jump in wherever they have ideas or feedback, but folks in each area should take responsibility for making sure all milestones are allocated a quarter. Once we do the exercise, we should use the "Q1" section of our roadmap as a feedback loop on whether our quarterly OKRs are putting us on the right track. I'm available to answer questions, take part in the timeline exercise, and give feedback on iterations - so let me know if anything is unclear! I'll go do a few examples that people can improve on, but you can also look at the Project WG Roadmap for the ideal end state. =] |
Some of this is already inflight, however we need an owner for benchmarking efforts
OKR/GO_CORE.md
Outdated
| * It takes less than 48 hours to transfer 1TB dataset over Fast Ethernet (100Mbps) | ||
| * Bitswap refactor + improvements reduce number of duplicate blocks downloaded by 75% | ||
| * Transfers of sparsely provided data are roughly as efficient as fully provided content | ||
| * Awesome go-ipfs benchmark test suite exists comparing ipfs performance and transfer size relative to bittorrent |
There was a problem hiding this comment.
| * Awesome go-ipfs benchmark test suite exists comparing ipfs performance and transfer size relative to bittorrent | |
| * Awesome go-ipfs benchmark test suite exists comparing ipfs performance and transfer size relative to bittorrent | |
| * go-ipfs data transfer is thoroughly tested as part of the CI pipeline |
This is in the spreadsheet right now, so adding it here for clarity
|
Hi go-ipfs team! We still have a few rows in our sheet without owners - especially a POC to help drive Base32 support over the finish line and someone to define and implement benchmarking to direct our performance improvement efforts and highlight the positive changes we make. Jump in on the sheet and volunteer before we freeze these tomorrow! |
Go IPFS/IPLD Core Dev OKRs
It's time for us to work on our OKRs for Q1 2019. We should keep in mind our WG goals as expressed in our roadmap. We'll use this PR for proposals and discussions of OKRs the next quarter.
ref #792