Add remove-members helper tool implemented in go #2575

daemon1024 · 2021-03-15T17:37:19Z

Fixes #2052

This tool is implementation of the bash script in go with a few changes like using flags instead of environment variables

This currently resides in the hacks folder which I believe is not the right place, let me know where I should move this to and if it deserves it's own package

k8s-ci-robot · 2021-03-15T17:37:27Z

Welcome @daemon1024!

It looks like this is your first PR to kubernetes/org 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/org has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-03-15T17:37:27Z

Hi @daemon1024. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2021-03-15T17:37:29Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: daemon1024
To complete the pull request process, please assign spiffxp after the PR has been reviewed.
You can assign the PR to them by writing /assign @spiffxp in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

palnabarun · 2021-03-16T05:01:26Z

/cc

nikhita

@daemon1024 thanks for creating the PR for this!

Left some suggestions, but looks great overall! 🎉

Can you also add a test for this?

hack/remove-members.go

nikhita · 2021-03-16T07:30:35Z

hack/remove-members.go

+
+//readMemberList reads the list of members to be removed from the given filepath
+func readMemberList(path string) ([]string, error) {
+	file, err := os.Open(path)


@mrbobbytables is the inactive member list a yaml file? how is that generated?

It's been a text file with 1 member per line. The list is generated from this sheet after some manual vetting: https://docs.google.com/spreadsheets/d/1jqxMOo9f1EG72i4sGE7g6RlNPvQ4_qfOx9K-UM4YsLw/edit

Thanks! One more follow up question - how does one generate this sheet?

Want to remove you as a single point of failure :)

The big stuff to regen it are in the notes tab in the sheet 👍 I thought it was owned/under the sig contribex shared dir, but it looks like I'm the only owner atm. I'll move it over in a bit.

hack/remove-members.go

nikhita · 2021-03-16T07:43:50Z

hack/remove-members.go

+	for _, member := range memberList {
+		var orgs, teams []string
+		count := 0
+		fmt.Print(member)


would be better to log here and a more descriptive log message

Should I change all instances of fmt to log?

Currently the output looks like

~/P/o/hack (remove-members-tool|…) ❯❯❯ go run remove-members.go -path=../config/kubernetes members.txt nikhita Orgs: [kubernetes] Teams: [sig-contributor-experience sig-release] Number of occurences: 3 Commit Command: "commit -m Remove nikhita from the kubernetes org -m Remove nikhita from sig-contributor-experience, sig-release teams" daemon1024 Orgs: [] Teams: [] Number of occurences: 0

P.s. I don't really want to remove you 😝 just for showing the output 😹

I think fmt is also ok, but would be good to use a more descriptive message :)

nikhita · 2021-03-16T07:45:59Z

hack/remove-members.go

+
+	re := regexp.MustCompile(`(\s+)?- ` + member + `(.*)?`)
+
+	if re.Match(content) {


we can probably count how many times member occurs in the path and return the count. This will ensure that count is accurate.

Do you mean to count the number of matches in the said file?

Do you mean to count the number of matches in the said file?

Yeah :)

Not a blocker for this, but potentially something in the future. We can marshal/unmarshal yaml while leaving the comments (go-yaml/yaml#132). This would also let us get team names directly.

@mrbobbytables -- I had a dig at this earlier by parsing the org files using k8s.io/test-infra/prow/config/org.Config, but there seemed to be something off there.

The processed org config when written back to disk had a lot of diff compared to the original file.

We can possibly dig deeper into it later.

It's been a bit since I looked at it , but potentially whitespace =/ since yaml supports some fuzzy inclusion:

foo: - bar

is the same as:

foo: - bar

Theres what people have written (both formats) and what go will want to format it as.

That as well. we may want to bite the bullet with a big initial diff as the subsequent updates should be deterministic .

One other thing was, in order to process comments, the structs in k8s.io/test-infra/prow/config/org.Config should have fields with yaml.Node as field type and then have post-processing helper method. I had taken a step back because of that complexity.

Yeah, what I'd probably do at that point is get away from editing by hand, and then have an add-member function or have a make command that ensures proper formatting before commit. Out of scope for this one though^^;;;

@mrbobbytables -- that's (adding members) what I was trying to automate. 😉

But, yes. Let's talk about this in a separate thread.

Adding to whitespaces and comments related changes, The substantial amount of diff is also created due to the difference in order of the yaml properties.
I was trying to parse the yaml files and removing members in a more structured way, I also faced this.

diff --git a/config/kubernetes-incubator/org.yaml b/config/kubernetes-incubator/org.yaml index baf4ca76..e2da8a87 100644 --- a/config/kubernetes-incubator/org.yaml +++ b/config/kubernetes-incubator/org.yaml @@ -1,10 +1,3 @@ -name: Kubernetes Incubator -description: Kubernetes Incubator -default_repository_permission: read -has_organization_projects: true -has_repository_projects: true -members_can_create_repositories: false -billing_email: github@kubernetes.io admins: - cblecker - fejta @@ -15,3 +8,10 @@ admins: - nikhita - spiffxp - thelinuxfoundation +billing_email: github@kubernetes.io +default_repository_permission: read +description: Kubernetes Incubator +has_organization_projects: true +has_repository_projects: true +members_can_create_repositories: false +name: Kubernetes Incubator

It seems to be sorting the fields.

I currently implemented this using kubernetes-sigs/yaml.

nikhita · 2021-03-16T07:51:29Z

/uncc @idvoretskyi
/cc @mrbobbytables @cblecker

daemon1024 · 2021-03-16T11:31:44Z

Can you also add a test for this?

Would it be fine to add some test .yaml files cause it wouldn't be right to perform tests on the actual files. Or should we work with TempFile

nikhita · 2021-03-16T11:33:56Z

hack/remove-members.go

+
+func main() {
+	flag.StringVar(&configPath, "path", ".", "Path to config directory/subdirectory")
+	flag.BoolVar(&dryrun, "dryrun", true, "Enable Dryrun or not. This flag controls whether the changes are simulated or live. If the changes are simulated it will only print the removal details and the commit message")


@mrbobbytables for feedback on the help text

It could probably be simplified down to something like: Simulates changes to be applied and prints removal details

nikhita · 2021-03-16T11:36:50Z

hack/remove-members.go

+	flag.Parse()
+
+	flag.Usage = func() {
+		fmt.Fprintf(os.Stderr, "Usage: remove-members [flags] <file-containing-members-list>\n")


would be good to list out the flags here. http://courses.cms.caltech.edu/cs11/material/general/usage.html has some details on the format

Usage: remove-members [-dryrun] [-path] member-file (file-containing-members-list)

Is this fine?

nikhita · 2021-03-16T11:39:13Z

Would it be fine to add some test .yaml files cause it wouldn't be right to perform tests on the actual files. Or should we work with TempFile

It would be fine to add test yaml files 👍

hack/remove-members.go

palnabarun · 2021-03-17T06:10:13Z

hack/remove-members.go

+func readMemberList(path string) ([]string, error) {
+	file, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer file.Close()
+
+	var members []string
+	scanner := bufio.NewScanner(file)
+	for scanner.Scan() {
+		members = append(members, scanner.Text())
+	}
+	return members, scanner.Err()
+}


How about simplifying this using ioutil.ReadFile as used in removeMemberFromFile?

Note: ioutil.ReadFile has been deprecated in Go 1.16 although it shouldn't be a blocker since this repository still uses Go 1.13.

hack/remove-members.go

* use pflags * use iouyil instead of bufio for reading member list * preserve filemode after removals

daemon1024 · 2021-03-17T10:36:16Z

Should I move this to a new directory? ( helpers/remove-members maybe? )

Since I added an external package it seems to adding it to the go.mod at the root, I haven't committed it yet cause I am not sure if this a part of that or should I initiate a separate go.mod for this.

k8s-ci-robot · 2021-03-24T15:17:07Z

@daemon1024: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-org-test-all	`686abd3`	link	`/test pull-org-test-all`
pull-org-verify-all	`686abd3`	link	`/test pull-org-verify-all`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

fejta-bot · 2021-06-23T20:29:26Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

shamsher31 · 2021-07-09T13:29:32Z

cmd/remove-members/main.go

+//removeMembers walks through the config directory and removes the occurences of the given member name
+func removeMembers(memberList []string, o options) error {
+	for _, member := range memberList {
+		var orgs, teams []string


This will redeclare orgs and teams with each iteration, better to move it outside of for loop.

shamsher31 · 2021-07-09T13:36:54Z

cmd/remove-members/main.go

+
+		matches := re.FindAllIndex(content, -1)
+
+		//Making Sure count from parsed config and regex matches are same


Suggested change

//Making Sure count from parsed config and regex matches are same

// Making sure count from parsed config and regex matches are the same

k8s-triage-robot · 2021-08-08T14:35:22Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

mrbobbytables · 2021-08-08T14:40:20Z

/remove-lifecycle rotten

k8s-triage-robot · 2021-11-06T14:44:35Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2021-12-06T15:30:51Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

mrbobbytables · 2021-12-06T15:43:17Z

/remove-lifecycle rotten

/cc @palnabarun
was this going to be integrated into your helper?

k8s-triage-robot · 2022-03-06T15:43:56Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2022-04-05T15:57:36Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2022-05-05T16:15:11Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen
Mark this issue or PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2022-05-05T16:15:28Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue or PR with /reopen

Mark this issue or PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

daemon1024 added 4 commits March 15, 2021 20:05

initial remove-members tool implementation in go

c76b933

execute git commands when not dry run

323261c

document the tool

1dab05f

Provide usage instructions if no args provided

9a1daf2

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 15, 2021

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. area/github-management Issues or PRs related to GitHub Management subproject labels Mar 15, 2021

k8s-ci-robot requested review from idvoretskyi and spiffxp March 15, 2021 17:37

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 15, 2021

k8s-ci-robot requested a review from palnabarun March 16, 2021 05:01

nikhita reviewed Mar 16, 2021

View reviewed changes

k8s-ci-robot requested review from cblecker and mrbobbytables and removed request for idvoretskyi March 16, 2021 07:51

daemon1024 added 2 commits March 16, 2021 15:20

combine iferr logs with function calls

c9a6810

update flags and usage instructions

73f657a

nikhita reviewed Mar 16, 2021

View reviewed changes

palnabarun requested changes Mar 17, 2021

View reviewed changes

daemon1024 added 2 commits March 17, 2021 15:50

update remove-members.go

6668a0f

* use pflags * use iouyil instead of bufio for reading member list * preserve filemode after removals

update usage details

9c68029

daemon1024 requested review from palnabarun, nikhita and spiffxp March 25, 2021 20:17

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 23, 2021

nikhita mentioned this pull request Jul 1, 2021

Automate org membership updates #2812

Closed

shamsher31 reviewed Jul 9, 2021

View reviewed changes

palnabarun mentioned this pull request Jul 13, 2021

Implement CLI to add members to org configurations #2822

Merged

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 8, 2021

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 8, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 6, 2021

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 6, 2021

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Dec 6, 2021

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 6, 2022

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 5, 2022

k8s-ci-robot closed this May 5, 2022


		re := regexp.MustCompile(`(\s+)?- ` + member + `(.*)?`)

		if re.Match(content) {


		matches := re.FindAllIndex(content, -1)

		//Making Sure count from parsed config and regex matches are same

	//Making Sure count from parsed config and regex matches are same
	// Making sure count from parsed config and regex matches are the same

Add remove-members helper tool implemented in go #2575

Add remove-members helper tool implemented in go #2575

Conversation

daemon1024 commented Mar 15, 2021 • edited Loading

k8s-ci-robot commented Mar 15, 2021

k8s-ci-robot commented Mar 15, 2021

k8s-ci-robot commented Mar 15, 2021

palnabarun commented Mar 16, 2021

nikhita left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daemon1024 Mar 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhita commented Mar 16, 2021

daemon1024 commented Mar 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikhita commented Mar 16, 2021

Choose a reason for hiding this comment

daemon1024 commented Mar 17, 2021 • edited Loading

k8s-ci-robot commented Mar 24, 2021

fejta-bot commented Jun 23, 2021

shamsher31 Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

shamsher31 Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

k8s-triage-robot commented Aug 8, 2021

mrbobbytables commented Aug 8, 2021

k8s-triage-robot commented Nov 6, 2021

k8s-triage-robot commented Dec 6, 2021

mrbobbytables commented Dec 6, 2021

k8s-triage-robot commented Mar 6, 2022

k8s-triage-robot commented Apr 5, 2022

k8s-triage-robot commented May 5, 2022

k8s-ci-robot commented May 5, 2022

daemon1024 commented Mar 15, 2021 •

edited

Loading

daemon1024 Mar 16, 2021 •

edited

Loading

daemon1024 commented Mar 17, 2021 •

edited

Loading

shamsher31 Jul 9, 2021 •

edited

Loading

shamsher31 Jul 9, 2021 •

edited

Loading