Improvement request: Milestones and Estimation

[drew2a]

Introduction

Below will be described thoughts on how we can implement "Epic Sprint" mentioned here #143 (comment) and have fun with it.

When I was thinking about it, I imagined small team (3 people) working on a relatively short feature.

I start from the core idea, described in the issue above: "to move forward step by step and achieve the goal after 6 months".

It means we should move forward by small iterations (in scrum methodology it is called sprints). 
To start doing this in a more predicted way I propose the following:

1. Use milestones as the main point for project tracking
2. Use labels to rough estimation of issues

Why milestones

GitHub provides two native tools to keep the track of a project: Projects and Milestones.
Since we are moving from one date to another, and don't have a heavy workflow, Milestones suit our needs better than Projects.

https://docs.github.com/en/github/managing-your-work-on-github/about-milestones

Good explanation "Projects vs Milestones" from stackoverflow:

• Milestones is a tool for Scrum methodology. Milestones are good for timeboxed iterations and working in sprints with batches of issues.
• Projects is a tool for Kanban methodology. Projects are good for continuous delivery and steady flow of work.

Note

• Projects (and probably notes) are better for research-related tasks.
• Milestones (and probably issues) are better for feature-related tasks.

Milestones naming convention

To reduce entropy I'll suggest the following milestone's name convention:

Team MAJOR.MINOR.PATCH: description, where:

• Team is a team name
• MAJOR.MINOR.PATCH it is SemVer notation
• Description is a goal of the milestone or it's core concept, or even description :)

for example, first milestone may looks like: Search 0.1.0: plots in Jenkins

Why labels for rough estimation

If we want to deliver the product by the scheduled date, we should estimate all tasks needed to accomplish the work.

Without estimation it looks like https://github.com/Tribler/tribler/milestone/35 or https://github.com/Tribler/tribler/milestone/31.
Of course it is much better than nothing.
But the question is "How can we be sure that all these tasks will be completed?"

Very often estimation (or planning) means to discuss tasks in detail and to make a sufficiently accurate estimation.
Not for us at this point, I think.

Instead of "accurate estimation" I propose to use "rough estimation".

When making a rough estimation, the developer has to check the issue briefly and set "rough estimation" based on his feelings. This estimation can be changed during the work on the task.
No magic.

There is a well-known practice "estimation in story points", see effort-vs-time. And again, I think, it is not for our team, because it too heavy to implement and hard to maintain 
and hard to understand
and not fun

Instead of story points and even hours, I propose to use:

• less than day
• 1 day
• 2 days
• 3 days
• 1 week
• 2 weeks
• 1 month
• more than month
• unknown

Conclusion

It is my opinion the project could benefit from a bit of organisation.
I tried to offer "minimal effort" to "achieve maximum benefits".

Please, feel free to comment and discuss it.

My main concern is to make our work easier and more predictable.
Not the opposite.

[ichorid]

I agree that we should split our big milestones into smaller ones. But then, how do we do release planning?

[kozlovsky]

I agree that we should split our big milestones into smaller ones. But then, how do we do release planning?

I think there are two ways to do releases:

1. Releases with some massive changes. Like, MyProgram 2.0 with completely redesigned architecture.
2. Releases tied to a specific schedule. Like, two releases each year on November 1st and May 1st.

The first way has a tendency to include too many features into the release, which then turns out to be delayed. The second way looks more manageable.

So, how about the following approach:

1. Schedule some future release at a specific date and create a corresponding milestone
2. Add this milestone label to all feature we want to include to this release
3. Estimate (roughly) how much time each feature should take
4. For each developer sum up all estimation of features assigned to him
5. For each developer do check is it possible to implement all these features to the release date. If not, prioritize features, and move some of them to another developer or to the next milestone.
6. Periodically refresh estimations for the remaining tasks. Instead of delaying the release, move the less important tasks to the next release. Keep the release date fixed.

What do you think, does it look real?

[ichorid]

@kozlovsky , I like your idea. However, GitHub does not allow to assign an issue to more than a single milestone. Milestones have dues dates. So we should probably use milestones as you suggested (for time-boxed releases), but use some other GitHub feature (tags? projects?) to track progress on sub-tasks as required by methodology suggested by @drew2a

[kozlovsky]

At first glance, I do not see significant contradictions with the @drew2a approach, as he does not mention sub-tasks. I think we can add dedicated projects for different Tribler components, like "Tribler-Search", "Tribler-UI", "Tribler-Channels" and organize sub-tasks within these projects. In each such project, subtasks can belong to different milestones.

[synctext]

Great in-depth discussion people!
How shall we organise our release cycles and the entire lab? Deeper connected discussion is how to do "industrial scale science and engineering". We have the luxury of numerous developers. Since 2008 we have running code for reputation and trust functions. We iterated a lot, it improved a lot, and all knowledge got wiped when the phd student graduated. The codebase is now 15 years and 5 months old.

My proposal would be to do time-boxed releases. We are often too ambitious with changes. They rarely finished on time. Bugs take considerable amount of time to find in huge code changes. To ensure we keeps bugs more manageable and keep matters rock stable we will do monthly releases. We used to require at least 6 RC releases, 4 Beta release and lots of manual testing before we could ship code. Invest a lot of time in automated unit test, integration tests and multi-platform install testers. This means all "big epic changes" need to be broken up into manageable chunks. That might feel like extra effort.. Big plans need to be broken up in parts. That is hard for tasks like moving to ASync or replaceing database. thoughts?

How do we use Github in a good manner? What are good policies for commits to "devel"? Is there a volunteer to act as "Stability Chief" role or we rotate that responsibility; depending on whose commit broke matters?

Fully agree with the "rough estimation" and starting to use well-known practice "estimation in story points" (rough in days/weeks)

[drew2a]

@kozlovsky I'm thinking more about "every month" release than "every half a year" release :) but the rest is correct (from my point of view).

I'm not sure about projects. Of course we can use it, but let's try to keep it simple :) do we really need it?

On the other hand, we can use Projects instead of Milestones.
It is also a good option.

My point here: organizational tool should be helpful and make life easier (not the opposite).

@ichorid what a particular case do you imagine, when thinking about assigning issue to multiple milestones?

GitHub doesn't have a sub-tasks. Only issues. You can just emulate subtasks by linking one issue to another.
But what a case?

[synctext]

I think we can add dedicated projects for different Tribler components, like "Tribler-Search", "Tribler-UI", "Tribler-Channels" and organize sub-tasks within these projects. In each such project, subtasks can belong to different milestones.

Each SCRUM cycle we usually need to divide our attention to: invest (test) infrastructure, address users features, try to reproduce reported bugs, improve PopularityCommunity Vn to Vn+1. Or we can opt to dedicate our cycles to 1 out of 4.

On the other hand, we can use Projects instead of Milestones.

Lets try one for a while and evaluate. No preference from my side.

[devos50]

We are often too ambitious with changes.

We did quite some refactoring efforts this year (e.g., moving to Python 3 and asyncio). We also did a full channels redesign. These changes, unfortunately, could not have been applied incrementally. Since I do not think that there are any major refactoring efforts soon, I'm highly in favour of the incremental changes approach.

They rarely finished on time.

Sadly yes. Our 7.5.2 milestone is due to by two months.

We used to require at least 6 RC releases, 4 Beta release and lots of manual testing before we could ship code.

We still require this. Our first release with asyncio required more than 6 RC versions if I remember correctly.

Is there a volunteer to act as "Stability Chief" role or we rotate that responsibility; depending on whose commit broke matters?

Ideally, we should all be part of the "Stability Task Force" :). Not sure how this role would be fulfilled by an individual.

[ichorid]

There are two distinct things we need our project management system to do for us:

1. We want to use milestones to indicate the next release target date.
2. We want to break down sub-projects (like Channels or Tunnels) into smaller parts and track the progress to predict what release the particular feature should go into.

The reason we want (2) is because it is the only way to track the liveliness of some sub-project. And we need that if we want to tolerate half-complete inactive code in the main codebase. This is the only alternative to merge/rebase hell.

Let's illustrate the problem with an example:
we're going to release 7.6 in a month or so. The 7.6 branch is going to be forked from devel at some point. But currently, I'm working on a new community that is going to serve Channels previews. There is high probability that this feature will not be complete by the moment we fork for 7.6. So, I a facing a dilemma:

• should I develop in a personal branch and only merge my changes when these are ready?
• or should I do make changes incrementally, but then undo them on 7.6 branch?

The first choice is going to bring rebase hell to me and PR review hell to my colleagues (the change is going to be much more than 500 lines). The second is going to bring a lot of unnecessary work and instability to 7.6 requiring to undo the unfinished changes on 7.6 branch.

There is an alternative way to do things, though. We can start to tolerate unfinished features on devel, but these features then must be committed in a disabled state. We'll have to tolerate 7.6 shipping with some disabled features though. Marking progress of subprojects is necessary to track these unfinished features, so these will not turn into dead code.

[drew2a]

@ichorid probably, you are talking about Feature Flags?

[ichorid]

@ichorid probably, you are talking about Feature Flags?

Yes.

Everyone in the team should read the link you provided. That's an excellent write-up on how to safely and gradually introduce big changes into a codebase.

[synctext]

But currently, I'm working on a new community that is going to serve Channels previews. There is high probability that this feature will not be complete by the moment we fork for 7.6.

Please document the existing .mdblob format, existing protocol, and specify your ideas. I will reserve some time improving this first design and simplifying it with the whole development team. Each improvement to our work needs to fit in the release cycle, otherwise the design will need to be altered to take smaller steps. Channels are now the key part of Tribler that is not working (got 3x Traceback out of it today and crash of the core).

[ichorid]

Please document the existing .mdblob format ...

@synctext , that was a hypothetical example 🤦

I'm not planning to work on any Channels-related changes right now, except for some smallier GUI-related changes, bug fixes and minor performance improvements. Also, not every design can both fit into release cycle and be immediately useful. That's why I'm talking about Feature Flags.

[drew2a]

@synctext

Deeper connected discussion is how to do "industrial scale science and engineering"

As I see, we can use GitHub in a fairly efficient manner.

Scientists and Developers have different goals and different workflows.

We can separate two flows inside Tribler:

• Developers will use milestones (mostly because releases are linked to dates)
• Scientists will use projects (mostly because projects have states (columns) and it easy to organise and track tasks inside a project)

Rule:

• If scientist want to contribute to Tribler codebase, it should add corresponding issue to a milestone.

Thus we will have two trackable flows:

• Developers: https://github.com/Tribler/tribler/milestones
• Scientist (and students?): https://github.com/Tribler/tribler/projects

Good side-effect from using projects: scientists can use "notes" instead of "issues".
This will keep the repository clean.

[devos50]

Currently, our scientific process takes place in Github issues. Master and PhD students get a new issue assigned or take over an existing issue. For example, this issue was my main focus during the first years of my PhD. You can find an example of a master thesis project here. Meeting notes are included as separate comments in these issues, as well as progress updates. Do you think it would be a good idea to move students thoughts/progress updates/meeting notes to projects?

[synctext]

@drew2a Please only focus on what the development team needs. Master students are no longer allowed to contribute code to Tribler. Code quality was always too poor. Phd students and also master students have their dedicated "Epic issues". This is listed in this table, I try to keep it always up-to-date. The Phd students/scientists currently don't need more of Github I think.

Rule: If scientist want to contribute to Tribler codebase, it should add corresponding issue to a milestone.

Sounds good.

[drew2a]

@devos50

Do you think it would be a good idea to move students thoughts/progress updates/meeting notes to projects?

I think yes. But my point of view is just a developer's point of view.
I see 312 issues, 60 of which are bugs. Part of them — ideas and discussions. Part of them — enhancements.
I am drowning, drowning, drowning.

Ok, let's focus on dev team needs, as @synctext said.

In an attempt to keep a balance between theory and practice:

1. Let's try to use milestones as a main point of release tracking. For month or two, then will evaluate it
2. The last release is 7.5.3, thus let's add Milestone 7.5.4 with date Sep 28
3. Add to 7.5.4 milestone most important issues that we can finish before the end of the month

Or another way.

We will wait until all team members can gather in the office (next week) and then will make plans for October.

[ichorid]

The upcoming bugfix release is 7.5.3 BTW

[synctext]

I am drowning, drowning, drowning.

We have a complete lack of top-level documentation:

• Our download stats: https://githubdownloads.com/?username=tribler&repository=tribler
• Our dashboard draft: release.tribler.org (Beta)
• Our latest builds:
• Our unit tests: https://jenkins-ci.tribler.org/job/Test_tribler_devel/
• Our nightly stress tester:
• Our mutation tester (IPv8 only for now):
• Our documentation:

[qstokkink]

We do have a ReadTheDocs https://tribler.readthedocs.io/en/devel/introduction.html. But, it's not linked on the GitHub frontpage and I doubt it's fully up-to-date.

Also, a mutation test run for IPv8's 5-second unit test suite takes 5 hours. If we extrapolate that for Tribler, one mutation test run would take a bit over a week. Probably best to put that all the way down on the priority list.

[ichorid]

@synctext

We have a complete lack of top-level documentation:

So, your proposed solution to this is to dedicate some issue at GitHub, create a table in it, and maintain the table by hand?

[synctext]

So, your proposed solution to this is to dedicate some issue at GitHub

True, that is not the way to do it. Draft priorities of the lab for discussion:

1. Until October: get our new developers up to speed on Tribler Complexities (dive into PopularityCommunity)
2. Until November: We focus on Tribler stability, bug fixes, and fully automated test infrastructure
3. Until we have monthly release cycle: Freeze on new features and focus on stability of unit test, integration test, stress tester, and dashboard for deployment monitoring
4. Update developers documentation with top-level pointers of all test & monitoring infrastructure

[drew2a]

I've added labels for estimation: https://github.com/Tribler/tribler/labels?q=estimation