[RFC 0124] Do not auto-close stale Nix issues and PRs, matching Nixpkgs's policy #124
Conversation
Ericson2314
force-pushed
the
stale-nix-no-close
branch
from
620dcec to
0c9732f
Compare
Ericson2314
force-pushed
the
stale-nix-no-close
branch
from
0c9732f to
724bcd8
Compare
Ericson2314
changed the title
Ericson2314
changed the title
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: https://discourse.nixos.org/t/stale-bot-for-nix-just-closed-a-bunch-of-stuff/18661/4 |
| Firstly, whatever the policy is, it should be clearly announced. | ||
|
|
||
| Without a big announcement of this decision, many of us were caught of guard when issues were closed the other day. | ||
| Anyone that just saw the previous stale bot "marking stale" messages probably assumed stale bot was configured like Nixpkgs. |
There was a problem hiding this comment.
Especially true given that the message was not adapted to differ from nixpkgs and explain what would happen!
lheckemann
added
the
status: open for nominations
|
To me, a stale bot that doesn't auto-close eventually is pretty useless and really just produces unnecessary GitHub comment spam. The whole point is to clean up the backlog of outdated issues/PRs, and so we ask contributors to confirm that their issue/PR is still relevant. I agree that the "stale" message should inform people that their issue/PR will be closed unless action is taken. |
|
IMO the main problem with stale bot (in general) is less than the closing itself, but that "closed fixed" vs "closed expired" is not differentiated". That's a lost of information. The second problem is that having to "reawaken" all legit longstanding issues is a chore. If one could tag something "never stale" or "goes stale N times as slowly", that would help. A stale bot that didn't close but also didn't post a message, would perhaps still provide some value (does github let one filter by last modified date?) but not be so spammy. |
It feels unfair to ask contributors to do this (sometimes multiple times in a row) when it's not them who's blocking the process. Most often, Nix PRs and issues I see are blocked waiting for review/merge, rather than waiting for the contributor to do anything. So putting the burden on the contributor to keep taking action to keep their PR open, with no end in sight, is going to make them feel unappreciated and resentful, and then burn them out. |
https://github.com/NixOS/nix/blob/fb5f13fb654e29915b26e49bffada522652bb403/.github/stale.yml#L5 I volunteer to shepherd this. My experience is that I authored the nixpkgs stale bot RFC. I feel I don't have the experience of the Nix development process though, so I hope some other person with more experience there is on the shepherd team. |
This is precisely my experience with the constant 'stale' notifications in nixpkgs already, and it's only barely acceptable (despite the notification spam) due to the issues at least not being closed automatically. Auto-closing is, frankly, not useful - the only thing it really achieves is that the "open issues" count goes down. A stale issue doesn't cost anything, and concluding that an issue is no longer relevant just because it didn't get updated is incorrect - the reporter may have stopped using NixOS, or gotten tired of being asked to reproduce multiple times, and so on. It does however drive away contributors (as already mentioned), and cause issues that occur rarely to disappear into the void. Such issues are not likely to be frequently re-reported, especially not if tracking down the issue requires a lot of effort that most users won't put in. Issue reports are contributions to the project; it is not reasonable to throw them away merely because a maintainer hasn't gotten around to reproducing and/or fixing it yet. And if the desire is to auto-close stale issues that are missing information, then auto-closing should be scoped specifically to issues tagged as such, not just "any issue that has been quiet for a while". Edit: Auto-closing of issues is why I stopped reporting issues to Fedora, by the way. |
|
To give some more insight on the effect the stalebot has, here is my inbox: (Note this shows Nix's stalebot alone, it does not include the 100s of other repositories in which I'm subscribed to 1000s of issues.)
On some days, like So I give up. I imagine it's the same for many others. With a stalebot that closes, the issue will then quite certainly be closed, even thout I that most of these weren't fixed. The project will then lose track of those well-reported problems. The open-issue search will no longer show them when people search for their problems.
Yeah. Auto-closing of issues is why I stopped contributing to Signal. It sucks to be working into the bin. |
|
Given:
Then probably the most honest thing to do is just saying "We don't have enough humans over here, so please don't ask for feature requests, we just accept bug/performance/documentation PRs, and flakes related feedback" That would reduce the spam, would save people time since they won't give feedback that produces no progress, and would remove such bot wall which feels just too cold to me The bot is just a cheap solution to the real problem: we don't have enough empowered and skilled humans over there as to handle the target audience size One should always try to solve the root of the problem |
It's good to have feature request tickets where people can discuss how stuff should work, add upvotes to indicate desire, and so on. There is no fundamental problem with having a high issue count number. It should correctly reflect the number of bugs and feature requests in the project. The only thing to prevent is to have open issues that are in fact solved. For this one needs a triaging system, to keep track of which issues contributers with free time should look at to weed those out. This was the original motivation of the non-closing stale bot in nixpkgs: Help with triage, and track the info that it is unknown whether the issue was solved by now. That's what "stale" means. It doen't mean "probably fixed". It means "maybe, possibly, unlikely, fixed, we don't know". (The chance of a bug fixing itself over time is small; my guess is that < 5% of issues get solved without the person fixing it being unaware of the corresponding issue. So I personally question the positive impact it can have. In A good system would be a triage queue where a
The above two are less bad when the bot just labels and explains, but doesn't auto-close: Then you can at least ignore the notifiactions in your email program with a filter (you'll still see the spam in the Github notifications page, but that's less bad). But if the bot closes, you cannot filter away the email. This is why the bot should not auto-close: It destroys this not-great-but-acceptable triage system. |
It would be great if it only posted a comment on the user's first stale issue/pr. |
|
Maybe we should stop making the bot post a message at all. You can attach a description to the label that explains its meaning. Until now, the
Maybe this is good enough? |
|
That's awesome. What happens when you try to add a link there? |
|
|
I have created a Github feature request: Allow links in issue label descriptions. In any case, I the hoverable description might be a better solution than what we have now, even without a clickable link. |
|
@nh2 Would you like to shepherd this too? |
OK! |
|
This is more about the The proof-of-concept is slow and not perfect, but it demonstrates that even a simple substring-based classification approach could be an effective way to get a better picture of outstanding issues and find potential duplicates:
Perhaps this could help in reducing the amount of stale issues, by making it less of a shapeless issue pile. The code is here: https://git.cryto.net/joepie91/nixpkgs-gh-issues |
Niklas Hambüchen noted in NixOS/rfcs#124 (comment) that we can add a label description for the stale label that appears when you hover over the label, or look on https://github.com/NixOS/nixpkgs/labels I set the description to be > https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md which is our page explaining the stale bot. The stale bot comments/emails are a significant burden on our most prodigious contributors, and the reason for their existence to orient new contributors. Since our stale bot's configuration is benign enough to ignore (it does not close), I believe it is good enough to satisfy the new contributor orientation with the label description. Therefore, this commit disables commenting when labeling an issue or PR stale.
|
|
||
| 3. Making the Nix policy not auto-close, in light of the special situation of how the backlog arose in the first place. | ||
|
|
||
| # Detailed design |
There was a problem hiding this comment.
As written I feel the detailed design is more specific than necessary. Maybe it could be just "Follow the stale policy of nixpkgs, except Nix may have different exempt labels."
If you do want to keep the details, we should make it match how nixpkgs is now:
- change
markCommentto false - add a bullet to add a good description for the stale label
There was a problem hiding this comment.
Updated, let me know if it's good.
|
@nh2 @infinisil I feel like this is pretty close to ready, and we aren't seeing much opposition to it. I don't feel like we need to have a synchronous meeting to push this through, but please let me know if you disagree and I can work on scheduling something. Could you please review the RFC text for any nits you have? |
Ericson2314
force-pushed
the
stale-nix-no-close
branch
from
6aba5df to
b298d1a
Compare
It's good by me. |
Thanks! Co-authored-by: Niklas Hambüchen <mail@nh2.me>
|
The Final Comment Period for RFC 0124 has started with disposition to merge and, barring any blocking issues, will be merged after 2022-06-14. Your opinions, comments, and approvals are welcome! |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
I mentioned this on the discourse thread when this came up (and elsewhere over the past few years--sorry if I'm a broken record), but I see a lot of back-and-forth over it here, so I'll link it (https://discourse.nixos.org/t/stale-bot-for-nix-just-closed-a-bunch-of-stuff/18661/3) and reiterate that I think the ~compromise exit scenario is a bot/workflow that lets human maintainers manually request feedback and queue issues for closure if feedback isn't provided. This gives maintainers a lever for reducing the human logistical work of keeping track of whether issues have enough information to be actionable, while not degrading users/contributors with bot-driven hoop-jumping. This is most of the up-side of auto-close without most of the down-side. The progress GH is making towards their roadmap suggests that the key bits for building a well-automated triage-queue view could be in place this year? |
edolstra
added
status: FCP
|
@ryantm Since FCP has passed, do you want to go ahead and merge this? |
|
@edolstra yes, please go ahead and merge it. I cannot. |
|
@NixOS/rfc-steering-committee the FCP has passed with no additional comments. Please merge this RFC. |
|
In the interests of implementing point 2 of the design of this RFC, I've generated a list of PRs that were closed by stalebot and have not yet been reopened. The listI've checked a random sample of 10 to check there were no false positives, so the list should be accurate. To reopen all the issues (if they felt that's the right way to go about implementing point 2), somebody with triage permissions on the Nix repo would need to run the following, with the issue list on stdin: |
|
Finally implemented with NixOS/nix#7817 (comment). |
|
Thanks again for @alyssais for prodding this to completion! |
infinisil
added
status: accepted
and removed
status: FCP
…gs's policy (NixOS#124) * nix-mark-stale-issues: Copy template * nix-mark-stale-issues: First draft * stale-nix-no-close: Update * stale-nix-no-close: Make the diff less normative * nix-mark-stale-issues: Fix typo Thanks! Co-authored-by: Niklas Hambüchen <mail@nh2.me> Co-authored-by: Niklas Hambüchen <mail@nh2.me>
Do not auto-close Nix issues and PRs.
Make the policy clear, and harmonize Nix's with Nixpkgs's, which was approved by RFC 51.
Rendered