-
Notifications
You must be signed in to change notification settings - Fork 677
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add remove-members helper tool implemented in go #2575
Conversation
Welcome @daemon1024! |
Hi @daemon1024. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: daemon1024 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/cc |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@daemon1024 thanks for creating the PR for this!
Left some suggestions, but looks great overall! 🎉
Can you also add a test for this?
hack/remove-members.go
Outdated
|
||
//readMemberList reads the list of members to be removed from the given filepath | ||
func readMemberList(path string) ([]string, error) { | ||
file, err := os.Open(path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mrbobbytables is the inactive member list a yaml file? how is that generated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's been a text file with 1 member per line. The list is generated from this sheet after some manual vetting: https://docs.google.com/spreadsheets/d/1jqxMOo9f1EG72i4sGE7g6RlNPvQ4_qfOx9K-UM4YsLw/edit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! One more follow up question - how does one generate this sheet?
Want to remove you as a single point of failure :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The big stuff to regen it are in the notes tab in the sheet 👍 I thought it was owned/under the sig contribex shared dir, but it looks like I'm the only owner atm. I'll move it over in a bit.
hack/remove-members.go
Outdated
for _, member := range memberList { | ||
var orgs, teams []string | ||
count := 0 | ||
fmt.Print(member) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would be better to log
here and a more descriptive log message
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should I change all instances of fmt
to log
?
Currently the output looks like
~/P/o/hack (remove-members-tool|…) ❯❯❯ go run remove-members.go -path=../config/kubernetes members.txt
nikhita
Orgs: [kubernetes]
Teams: [sig-contributor-experience sig-release]
Number of occurences: 3
Commit Command: "commit -m Remove nikhita from the kubernetes org -m Remove nikhita from sig-contributor-experience, sig-release teams"
daemon1024
Orgs: []
Teams: []
Number of occurences: 0
P.s. I don't really want to remove you 😝 just for showing the output 😹
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think fmt is also ok, but would be good to use a more descriptive message :)
hack/remove-members.go
Outdated
|
||
re := regexp.MustCompile(`(\s+)?- ` + member + `(.*)?`) | ||
|
||
if re.Match(content) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can probably count how many times member
occurs in the path
and return the count. This will ensure that count
is accurate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean to count the number of matches in the said file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mean to count the number of matches in the said file?
Yeah :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not a blocker for this, but potentially something in the future. We can marshal/unmarshal yaml while leaving the comments (go-yaml/yaml#132). This would also let us get team names directly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mrbobbytables -- I had a dig at this earlier by parsing the org files using k8s.io/test-infra/prow/config/org.Config, but there seemed to be something off there.
The processed org config when written back to disk had a lot of diff compared to the original file.
We can possibly dig deeper into it later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's been a bit since I looked at it , but potentially whitespace =/ since yaml supports some fuzzy inclusion:
foo:
- bar
is the same as:
foo:
- bar
Theres what people have written (both formats) and what go will want to format it as.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That as well. we may want to bite the bullet with a big initial diff as the subsequent updates should be deterministic .
One other thing was, in order to process comments, the structs in k8s.io/test-infra/prow/config/org.Config
should have fields with yaml.Node
as field type and then have post-processing helper method. I had taken a step back because of that complexity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, what I'd probably do at that point is get away from editing by hand, and then have an add-member function or have a make command that ensures proper formatting before commit. Out of scope for this one though^^;;;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mrbobbytables -- that's (adding members) what I was trying to automate. 😉
But, yes. Let's talk about this in a separate thread.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding to whitespaces and comments related changes, The substantial amount of diff is also created due to the difference in order of the yaml properties.
I was trying to parse the yaml files and removing members in a more structured way, I also faced this.
diff --git a/config/kubernetes-incubator/org.yaml b/config/kubernetes-incubator/org.yaml
index baf4ca76..e2da8a87 100644
--- a/config/kubernetes-incubator/org.yaml
+++ b/config/kubernetes-incubator/org.yaml
@@ -1,10 +1,3 @@
-name: Kubernetes Incubator
-description: Kubernetes Incubator
-default_repository_permission: read
-has_organization_projects: true
-has_repository_projects: true
-members_can_create_repositories: false
-billing_email: github@kubernetes.io
admins:
- cblecker
- fejta
@@ -15,3 +8,10 @@ admins:
- nikhita
- spiffxp
- thelinuxfoundation
+billing_email: github@kubernetes.io
+default_repository_permission: read
+description: Kubernetes Incubator
+has_organization_projects: true
+has_repository_projects: true
+members_can_create_repositories: false
+name: Kubernetes Incubator
It seems to be sorting the fields.
I currently implemented this using kubernetes-sigs/yaml
.
/uncc @idvoretskyi |
Would it be fine to add some test |
hack/remove-members.go
Outdated
|
||
func main() { | ||
flag.StringVar(&configPath, "path", ".", "Path to config directory/subdirectory") | ||
flag.BoolVar(&dryrun, "dryrun", true, "Enable Dryrun or not. This flag controls whether the changes are simulated or live. If the changes are simulated it will only print the removal details and the commit message") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mrbobbytables for feedback on the help text
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could probably be simplified down to something like: Simulates changes to be applied and prints removal details
hack/remove-members.go
Outdated
flag.Parse() | ||
|
||
flag.Usage = func() { | ||
fmt.Fprintf(os.Stderr, "Usage: remove-members [flags] <file-containing-members-list>\n") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would be good to list out the flags here. http://courses.cms.caltech.edu/cs11/material/general/usage.html has some details on the format
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usage: remove-members [-dryrun] [-path] member-file (file-containing-members-list)
Is this fine?
It would be fine to add test yaml files 👍 |
hack/remove-members.go
Outdated
func readMemberList(path string) ([]string, error) { | ||
file, err := os.Open(path) | ||
if err != nil { | ||
return nil, err | ||
} | ||
defer file.Close() | ||
|
||
var members []string | ||
scanner := bufio.NewScanner(file) | ||
for scanner.Scan() { | ||
members = append(members, scanner.Text()) | ||
} | ||
return members, scanner.Err() | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about simplifying this using ioutil.ReadFile
as used in removeMemberFromFile
?
Note: ioutil.ReadFile
has been deprecated in Go 1.16 although it shouldn't be a blocker since this repository still uses Go 1.13.
* use pflags * use iouyil instead of bufio for reading member list * preserve filemode after removals
Should I move this to a new directory? ( Since I added an external package it seems to adding it to the go.mod at the root, I haven't committed it yet cause I am not sure if this a part of that or should I initiate a separate go.mod for this. |
@daemon1024: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
//removeMembers walks through the config directory and removes the occurences of the given member name | ||
func removeMembers(memberList []string, o options) error { | ||
for _, member := range memberList { | ||
var orgs, teams []string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will redeclare orgs and teams with each iteration, better to move it outside of for loop.
|
||
matches := re.FindAllIndex(content, -1) | ||
|
||
//Making Sure count from parsed config and regex matches are same |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
//Making Sure count from parsed config and regex matches are same | |
// Making sure count from parsed config and regex matches are the same |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
/remove-lifecycle rotten |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
/remove-lifecycle rotten /cc @palnabarun |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Fixes #2052
This tool is implementation of the bash script in go with a few changes like using flags instead of environment variables
This currently resides in the hacks folder which I believe is not the right place, let me know where I should move this to and if it deserves it's own package