A proposal for bug and issue handling, and roadmap management #902
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,179 @@ | ||
| Galaxy Issue Management | ||
| ======================= | ||
|
|
||
| The purpose of this document is to formalize how we manage the full cycle | ||
| (reporting, tracking progress, resolution) of both feature/enhancement ideas | ||
| and bugs, as well as providing a few general guidelines for longer term | ||
| planning for various Galaxy related projects. Significant inspiration taken | ||
| from the way the [Docker](https://github.com/docker/docker) project manages | ||
| issues. | ||
|
|
||
| We've tried several different approaches over the course of the project and two | ||
| common problems we've tried to address in this approach are: | ||
|
|
||
| * The issue graveyard -- where, once off the first 'page' of issues or below | ||
| the 'fold' in Trello, things sometimes do not resurface and are lost to | ||
| history. | ||
|
|
||
| * Difficulty clearly presenting and maintaining a high level project roadmap | ||
| and associated meta-issues. | ||
|
|
||
|
|
||
| Milestones | ||
| ========== | ||
|
|
||
| Every pull request should, prior to merge, be assigned to the milestone | ||
| corresponding to the Galaxy release it will first appear in (15.10, 16.01, and | ||
| so on). This, along with the tags applied, will be used to generate the | ||
| release notes. | ||
|
|
||
| Any non-PR issue assigned to a milestone *must* be resolved or reassigned prior | ||
| to publishing that release -- that is, this is the new way to create and track | ||
| "Issues Blocking Release". This will be a primary way to report bugs and force | ||
|
There was a problem hiding this comment. I assume that once merged this document becomes active. Thus I would use 'This is the primary way to force bugs reconciliation' rather than 'This will be.' There was a problem hiding this comment. Yep, good catch. This was more editorialization on my part that I've formalized as it would be once active. |
||
| them to be reconciled (fixed, closed, or intentionally postponed) prior to | ||
| release. In practice, bugs should almost always be tagged with a milestone | ||
| which forces the reconciliation date. Issues *may* be, but they don't | ||
| necessarily have to be -- this is subjective and it depends on if the submitter | ||
| (or any other contributor) wants to force the issue to be revisited at the next | ||
| release. | ||
|
There was a problem hiding this comment. This creates a lot of drama and doesn't really set about how to resolve any of it. Lets adapt a non-fictitious example from Trello where Developer A is assigned some issue and Developer B decides it is an "Issue Blocking the Release" but doesn't want to work on it his or her self. Can Developer A just push it back a release, and can Developer B then re-push it forward? And what if it has no assignee, can any developer just delay a release indefinitely this way? I'm not sure what the solution is, but my preference it is probably to be conservative and explicitly just say that any committer can push back the milestone on an issue and no other committer is allowed to move it forward again except through issuing a pull request that resolves the issue in the older release. There was a problem hiding this comment. @jmchilton Hrmm, didn't really anticipate this as a drama-causing rule at all. In the context of non-bug issues I was mainly thinking 'Ok, this is a TODO that someone (any contributor) can add to indicate that we don't want to forget to at least think about (enough to fix, close, or punt) before the next release'. |
||
|
|
||
| Effective use of milestones should prevent bugs from falling through the | ||
| cracks, and will provide a mechanism for forcing the revisitation(and thus | ||
| progress or even potential discard) of ideas for enhancements or features. | ||
|
|
||
|
|
||
| Labeling Structure | ||
| ================== | ||
|
|
||
| To allow for easy search, filtering, and general issue management every issue | ||
| or PR (not tagged `Procedures` or `Planning`) is *required* to have three | ||
|
|
||
| labels which indicate the type, status, and focus area of the issue. Any issue | ||
| without these three tags will be automatically have a `triage` label applied | ||
| indicating that it needs human intervention to be correctly tagged. These | ||
| `triage` tagged issues will be regularly reviewed and tagged as appropriate. | ||
|
|
||
| Kind Labels | ||
| ----------- | ||
|
|
||
| The 'kind' label set is used for classifying the type of contribution or | ||
| request/report to separate enhancements and new features from bugs, etc. | ||
|
|
||
| * `kind/bug` - something is broken, and it needs fixing | ||
| * `kind/documentation` - documentation is unclear or can be improved | ||
|
There was a problem hiding this comment. The documentation is part of the code and the code is part of the documentation, this seems like a harmful or arbitrary distinction to me. I think There was a problem hiding this comment. agreed, not a useful distinction There was a problem hiding this comment. What about tasks that really are purely documentation and don't touch code? Not everything we do is code. Do we manage those issues somewhere else? There was a problem hiding this comment. @jxtx Leaving them in Trello is an option. Same as 'management', 'main', 'deployment' etc. tasks. There was a problem hiding this comment. @jxtx Documentation in repository or out? I assume you mean out, so like on the wiki. I think we should have a separate Galaxy project project on github for discussing project wide things via issues and laying out procedures for things that aren't directly related to this repository (like how we manage There was a problem hiding this comment. This comparison doesn't work for me. Of course programs must be written primarily for people to read. That doesn't mean there are not tasks that are primarily writing documentation in nature. Still feels like a kind to me. I read area as an area of the application impacted, while kind refers to the kind of work that needs to be done. Of course, there are some things that are less clear to me like UI. There was a problem hiding this comment. @martenson A goal here, at least for @nekrut and I, is to gradually move everything out of Trello. So, this would start with "Galaxy Development" becoming read-only and moving everything here. Then the rest of our public boards, either here or to another repo. ToolShed is an interesting case. When does it get its own repository? There was a problem hiding this comment. @martenson since my alternative "blue sky" toolshed is unlikely to catch on any time soon... in the short term would you have any interest in doing this with me? Clone galaxy into a separate TS repository, and slowly remove everything but TS code, then work toward adding/removing features/cleaner API, internal refactoring, etc? There was a problem hiding this comment. Agreed, thanks. That was slightly off topic, you're right. Just wanted to respond to @jxtx's comment about separate TS trello and have a plan for that. |
||
| * `kind/enhancement` - polish to an existing feature or interface | ||
| * `kind/feature` - something brand new | ||
|
There was a problem hiding this comment. I don't know that the distinction between Update: I'd almost think the better distinction is There was a problem hiding this comment. I think feature, enhancement and There was a problem hiding this comment. @jxtx As the man who has had to decide for a few releases now what is a bug and what is an enhancement - I really don't think the lines are clear at all. It is possible more granularity will help, or it might hurt I am not sure. Update: ... and given that I am not sure, I would tend toward simplicity I guess. Occasionally more complexity helps though, I can buy that. There was a problem hiding this comment. @jmchilton Would it help you in making this call if the person who resolved the issue tagged it with their best guess though? There was a problem hiding this comment. +1 to an additional There was a problem hiding this comment. The submitter can only add tags if they are on the project right? I think the triage person or the closer will need to do this. Certainly, any of those three people are better than me doing it in a rush weeks later though - we can all agree on that 😄. There was a problem hiding this comment. Also - I'll argue that technical debt is a better than refactoring. Because either tag means it is of minimal importance to anyone after the fact, should be in tiny print on the release notes, etc.... But this applies both to refactorings and to the addition of new test cases to the repository, the same people should care about both - essentially no one :). technicaldebt then captures both kinds of contributions better than refactoring. There was a problem hiding this comment. I went ahead and added the 'kind/refactoring', but hadn't seen this last comment. Can swap it if people actually prefer technicaldebt, of course. |
||
|
|
||
| Status Labels | ||
| ------------- | ||
|
|
||
| The `status` of an issue or PR should be tracked using the following labels: | ||
|
|
||
| * `status/new` - a brand new issue | ||
| * `status/planning` - the issue planning phase, this issue may potentially need | ||
| more information (or just more thinking) to proceed to a work in progress | ||
|
There was a problem hiding this comment. Again - I'm not sure it is worth clarifying There was a problem hiding this comment. I like the three states, but "status/new" is probably redundant with triage. Once triages an issue is either closed or either planning or in progress. There was a problem hiding this comment. Yeah, I agree, on |
||
| * `status/WIP` - this issue or PR is currently being worked on and in the case | ||
| of a PR, it should not be merged until this tag is removed | ||
| * `status/review` - PR is complete and ready for review, or when applied to an | ||
| issue it is thought to be resolved but needs verification | ||
|
|
||
| We use the same set of status tags for PRs and issues to keep things simple, | ||
| but not every PR or issue needs to go through every state. For example, it'll | ||
| be common for a PR to be submitted with the label 'status/review', and get | ||
| merged without needing to go through the rest of the states. | ||
|
|
||
| Note that there are no `status/complete`, `status/wontfix`, `status/duplicate`, | ||
| or other terminal status indicators. This is intentional to keep the tail end | ||
| of bookkeeping from getting onerous. These sorts of terminal states *and their | ||
| justifications* (e.g. the reason why it's a wontfix, or a reference to the | ||
| duplicate issue) should be indicated in the closing comment by the issue | ||
| closer. | ||
|
|
||
|
|
||
| Area Labels | ||
| ------------ | ||
|
|
||
| The 'area' label is used for tagging issues and pull requests to a particular | ||
| focus area. This allows for easy searching within that particular domain, as | ||
| well as more organized release notes. Some examples, not-exhaustive, are here: | ||
|
|
||
| * `area/API` | ||
| * `area/cleanup` | ||
| * `area/jobs` | ||
| * `area/tests` | ||
| * `area/GIEs` | ||
| * `area/toolshed` | ||
| * `area/UI-UX` | ||
| * `area/workflows` | ||
|
|
||
| This list will definitely grow over time. | ||
|
|
||
| Other Useful Labels | ||
| ------------------- | ||
|
|
||
| While the three labels sets indicating kind, status, and area are required | ||
| there are several other labels that are be useful and/or have special purpose. | ||
|
|
||
| * `procedures` is a special tag that indicates that the issue is related to | ||
| project governance, and it overrides the need for the trio of | ||
| kind/status/area tags, and these are never auto-flagged for triage. | ||
|
There was a problem hiding this comment. Maybe add a link to |
||
|
|
||
| * `planning` is also a special tag that indicates the issue is related to | ||
| larger-scale issue planning. These issues are typically meta-issues | ||
| containing checklists and references to other issues which are subcomponents | ||
| and stepping stones necessary for issue resolution. These *can* utilize the | ||
| `focus/*` tags but are not required to. Status and type make little sense | ||
|
There was a problem hiding this comment. focus -> There was a problem hiding this comment. Yep, missed one when I swapped them back, good catch. |
||
| here. | ||
|
|
||
| * `roadmap` is a reserved tag for the primary project roadmap. This is a | ||
| meta-issue that is not expected to be completed, but rather serves as an | ||
| entry point to the high level development of the project. | ||
|
|
||
| * `beginner-friendly` can be used to indicate a nice entry-level issue that | ||
| only requires limited understanding of the larger Galaxy framework and | ||
| ecosystem. This is useful for encouraging new contributors. | ||
|
|
||
|
|
||
| The Roadmap | ||
| =========== | ||
|
|
||
| We will maintain a single `Roadmap` tagged meta-issue which will describe (at a | ||
| very high level) the *current* major areas of focus for the project. This is | ||
|
There was a problem hiding this comment. I would amend this to say NIH grant funded team members of the project or maybe the committers group. Obviously the projects grows in completely unexpected directions and the focus of the project as a whole is very much affected by non-committers and the priorities they pursue. There was a problem hiding this comment. I'm not sure NIH funded or not is the right distinction. However this is one of the most challenging things to figure out. Is this just for the Taylor and Nekrutenko labs? Is it for the committers group? Do we need to solve this now or wait until there are other organizations with more than a few people contributing regularly? There was a problem hiding this comment. You can say it is just a couple people, but we have people contributing to the project from the "outside" that are forces of nature. I can think of a few that have managed to add whole new dimensions to the project and really refine and set priorities for the project as a whole. I would say roadmap should be for the committers group or shouldn't exist. If the Nekrutenko and Taylor labs want to maintain their own independent priorities (not something I am poo-pooing at all, I like to eat and you guys feed people and are responsible for feeding people so you need to have the freedom to set priorities and ensure things get done) then it should be in Trello or another repo or somewhere else. There was a problem hiding this comment. @jmchilton My goal is to find a process that works for all contributors. I was simply posing the question. I strongly agree it should be for the committers group at this stage, and would add that members of that group should be clear about their own motivations and priorities. My point regarding organizations has nothing to do with amount of contribution as much has process. As more organizations with more people get involved, how do we have a unified roadmap while also capturing organizational priorities? I'm not diminishing anyones contributions, I'm trying to be flexible in how different contributors contribute to the overall roadmap. (Is this a problem anyone has solved well?) |
||
| similar to our PRIORITIES 2014/15 cards on Trello. Using [Task | ||
| Lists](https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments), | ||
| this issue will link to sub-issues which will go into much more detail, might | ||
| have its own checklists to even more subcomponent cards, and so on. | ||
|
|
||
| This `Roadmap` issue will be assigned to every release milestone, forcing | ||
| periodic review of the roadmap. | ||
|
|
||
| To prevent the roadmap from being tied completely to GitHub, and to facilitate | ||
| portable change tracking over time, we will also maintain the file | ||
| project/ROADMAP.md within the repository. Whenever the ROADMAP issue text is | ||
| changed, ROADMAP.md should be updated correspondingly. | ||
|
|
||
| This document won't have the organizational integration that a live GitHub | ||
| issue does, but this way we're be able to have a ROADMAP.md permanently | ||
| attached to the code regardless of what issue tracking or organizational | ||
| software we use in the future. | ||
|
There was a problem hiding this comment. I'm -0.5 on maintaining this file in the repository - we won't maintain it at all, so in order to encourage us to maintain it at all we should make it as easy as possible to edit. An issue is less onerous to edit given our project procedures. There was a problem hiding this comment. I'll go further with "-1 but you can try to change my mind". All redundancy must die. There was a problem hiding this comment. I think the advantage of having it in the repository would be that it is kept clean and trimmed. If it's an issue, either we're all editing the main issue text (which GH doesn't track yet), or we're commenting on a mile long thread of which only a portion may be relevant to the current roadmap. There was a problem hiding this comment. @jmchilton I wasn't expecting any actual 'maintenance' of the file via the repository, just keeping it as a historical record -- all edits will be in the issue itself, which is (maybe even automatically, instead of copy/paste?) replicated in ROADMAP.md to preserve the roadmap. It's just markdown, whether in the issue or file. If people don't care about having the roadmap in the repository and want the only authoritative copy (and history of it) on GitHub, then this can definitely go. There was a problem hiding this comment. @erasche What about commenting urself with what you updated in the main issue text? Or having P4 post diffs of main text periodically. There was a problem hiding this comment. @jxtx ah, wasn't clear that the "overall project roadmap" would be tossed out every few months. That's a very different beast. In that case, then yes, I'm fine with a mutable GH issue for a roadmap, maybe automatically mirrored into docs. There was a problem hiding this comment. @jxtx Let me merge two threads here - I feel like the problem with redoing the roadmap every few months is that we never seem to cut it down. I feel like every time we tried to adjust the one on Trello we would just add new stuff to it (planemo, IEs, etc...) and it never got smaller. Also to your comment about organizations, the organizational structure for better or worse for this repository is an individual focused one and not an organizational focused one - which I think was an interesting choice and very different than what most projects do. So what if we leverage the fact we are different and use it to ensure the roadmap stays trim. We have a roadmap issue and every committer gets one line to discuss what they intend to work on for the next three months. We meet as a committer group once every three months to discuss and update. Obviously, Anton is going to have a tremendous amount of say in what I and Martin work on for instance, and James will have a tremendous amount of say in what Dannon and Carl work on but ultimately that is reflected through the individual - not through an organization. I think this solves both problems nicely and leverages the unique structure of our project. There was a problem hiding this comment.
This is the motivation beyond redoing the roadmap and not updating it. There has to be an accompanying social piece here in which we actually consider each item on the roadmap and figure our whether it is still alive and where it should go. Roadmap is a bit more than just priorities, we should also put timelines on things (even if the granualarity, is "next release", "this year", "soon", and "the distant future"). Will this work? I don't know. Is it worth trying? I think so.
Is this a feature or a bug?
I think what you are proposing just reinforces what we currently have. I think the goal here is to push for a little bit of change. We routinely hear complaints that we lack a transparent high-level roadmap, and that people outside (and sometimes inside) the team aren't clear on the big picture priorities.
Is this really where we want to be? Even now, @nekrut and @jxtx make prioritization decisions together, with input from everyone on their teams. Here we have the potential to open this up, and make it more transparent and inclusive. Sure, we can just have a list of what each person thinks they are working on, but that loses all of the planning behind things. And it is inherently isolating. There was a problem hiding this comment. @jxtx I don't disagree with anything you said per se. This team has too many optimists and too much ambition - its reach exceeds its grasp - so I don't feel like a list of priorities without an assignment of resources to them is useful - even if that is what people request. Assigning people seems like it would make things much more realistic at the release level. Also, it is isolating but we are trying to do 45 things with 15 people, these tasks will be isolated for the most part and the practice reflects the theory on that I think. I'll drop the idea, it is... odd for sure - I just thought it meshed well with the reality of how things will proceed. There was a problem hiding this comment.
As you know, I don't consider these to be bad things ;)
I totally agree that people should be assigned to anything on the roadmap with a concrete timeframe (though I don't want to lost sight of "the distant future"). I just want to get there through a group discussion. I agree we are pushed toward isolation in practice based on the nature of the work, but I'd prefer to actively push back rather than just accept it. |
||
|
|
||
|
|
||
| Voting | ||
| ====== | ||
|
|
||
| Users can vote for issues by commenting with a +1. It's possible to sort the | ||
| issue list by 'most commented' which would be a good indicator of what issues | ||
| are 'hot', though this doesn't necessarily indicate a high vote. It's possible | ||
| that that this is good enough good enough and in some ways potentially more | ||
|
There was a problem hiding this comment.
|
||
| useful to find 'hot' issues than a flat vote count. | ||
|
|
||
|
There was a problem hiding this comment. We could automate a There was a problem hiding this comment. Yep, I could definitely add this feature. Would be nice to raising priorities. There was a problem hiding this comment. Oh my... I actually like this. But please not "hotness". There was a problem hiding this comment. We could do reddit style and have a There was a problem hiding this comment. 100% do not do hotness - that was a joke. Maybe There was a problem hiding this comment. ... in retrospect that should've been obvious. |
||
|
|
||
| Automation | ||
| ========== | ||
|
|
||
| For now, we will rely on a few simple automation rules: | ||
|
|
||
| * All issues, unless tagged `Procedures` or `Planning` will automatically be | ||
|
|
||
| tagged `triage`, indicating that they require attention. | ||
|
|
||
| * All PRs that are not assigned to a milestone will be tagged `triage` to | ||
| indicate that they require attention prior to merge. | ||
|
There was a problem hiding this comment. Will add to p4. Update: galaxyproject/p4#2 |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| Galaxy Project Roadmap | ||
|
There was a problem hiding this comment. The contents of this file I would rather see as a combination of GH Issues and Milestones than an extra file we need to keep updating. There was a problem hiding this comment. @martenson Not sure what you mean, can you say a little more about how it'd look? As mentioned in the other discussion, I mainly just wanted it as a permanent historical record of the primary roadmap that was portable outside of github. There was a problem hiding this comment. So this file would disappear and its contents would be recorded similar to nodejs/node#2522 (using issues and milestones hyperlinking). This is similar to what @jmchilton and @jxtx mentioned above, just more poorly worded. Also see another node example of meta-planning: nodejs/node#3000 There was a problem hiding this comment. @martenson Now I'm really confused. Yes, what I was proposing was that the contents of this file mirror exactly what is in the github roadmap issue. It's just a stub for now. |
||
|
|
||
| This is a document that outlines the Galaxy Roadmap. It should cover, at a | ||
| high level, current development interests and plans. Including links to active | ||
| GitHub meta-issues is highly encouraged. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following three paragraphs are not necessary in this document. We stated what, and we are about to say how. I do not see a need to mention history or Trello UI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. They are helpful in framing this discussion, but should be removed for the final document.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, I definitely started to editorialize a bit in here, fully expecting it to be cleaned up.