Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(backup): Duplicated backups & restores #1021

Merged
merged 4 commits into from
Jun 25, 2024

Conversation

DionJones615
Copy link

@DionJones615 DionJones615 commented Jun 20, 2024

Changes

Move bar.Client.Update() function beneath the bar.Client.Get() function to prevent a situation that causes the reconciliation to repeat itself at this point, causing two full restores to complete at every startup.

The same can be true of a backup if/when the operator restarts and bar.Client.Get() has not run yet.

Before Fix:

2024-06-14T21:25:03.409Z	DEBUG	controller-jenkins	Reconciling Jenkins	{"cr": "jenkins"}
2024-06-14T21:25:03.409Z	DEBUG	controller-jenkins	Operator credentials secret is present	{"cr": "jenkins"}
2024-06-14T21:25:03.435Z	DEBUG	controller-jenkins	Scripts config map is present	{"cr": "jenkins"}
2024-06-14T21:25:03.457Z	DEBUG	controller-jenkins	Init configuration config map is present	{"cr": "jenkins"}
2024-06-14T21:25:03.484Z	DEBUG	controller-jenkins	Base configuration config map is present	{"cr": "jenkins"}
2024-06-14T21:25:03.484Z	DEBUG	controller-jenkins	GroovyScripts Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-14T21:25:03.484Z	DEBUG	controller-jenkins	ConfigurationAsCode Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-14T21:25:03.484Z	DEBUG	controller-jenkins	createServiceAccount with annotations map[eks.amazonaws.com/role-arn:arn:aws:iam::REDACTED:role/REDACTED]	{"cr": "jenkins"}
2024-06-14T21:25:03.643Z	DEBUG	controller-jenkins	Service account, role and role binding are present	{"cr": "jenkins"}
2024-06-14T21:25:03.643Z	DEBUG	controller-jenkins	Extra role bindings are present	{"cr": "jenkins"}
2024-06-14T21:25:03.649Z	DEBUG	controller-jenkins	Jenkins HTTP Service is present	{"cr": "jenkins"}
2024-06-14T21:25:03.655Z	DEBUG	controller-jenkins	Jenkins slave Service is present	{"cr": "jenkins"}
2024-06-14T21:25:03.655Z	DEBUG	controller-jenkins	Kubernetes resources are present	{"cr": "jenkins"}
2024-06-14T21:25:03.655Z	DEBUG	controller-jenkins	Jenkins master pod is present	{"cr": "jenkins"}
2024-06-14T21:25:03.655Z	DEBUG	controller-jenkins	Jenkins master pod is ready	{"cr": "jenkins"}
2024-06-14T21:25:03.670Z	DEBUG	controller-jenkins	Jenkins API client set	{"cr": "jenkins"}
2024-06-14T21:25:03.683Z	DEBUG	controller-jenkins	Installed plugins 'REDACTED'	{"cr": "jenkins"}
2024-06-14T21:25:06.402Z	INFO	controller-jenkins	Restoring backup '597366'	{"cr": "jenkins"}
2024-06-14T22:16:03.885Z	DEBUG	controller-jenkins	Reconciling Jenkins	{"cr": "jenkins"}
2024-06-14T22:16:03.885Z	DEBUG	controller-jenkins	Operator credentials secret is present	{"cr": "jenkins"}
2024-06-14T22:16:03.912Z	DEBUG	controller-jenkins	Scripts config map is present	{"cr": "jenkins"}
2024-06-14T22:16:03.924Z	DEBUG	controller-jenkins	Init configuration config map is present	{"cr": "jenkins"}
2024-06-14T22:16:03.942Z	DEBUG	controller-jenkins	Base configuration config map is present	{"cr": "jenkins"}
2024-06-14T22:16:03.942Z	DEBUG	controller-jenkins	GroovyScripts Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-14T22:16:03.942Z	DEBUG	controller-jenkins	ConfigurationAsCode Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-14T22:16:03.942Z	DEBUG	controller-jenkins	createServiceAccount with annotations map[eks.amazonaws.com/role-arn:arn:aws:iam::REDACTED:role/REDACTED]	{"cr": "jenkins"}
2024-06-14T22:16:04.085Z	DEBUG	controller-jenkins	Service account, role and role binding are present	{"cr": "jenkins"}
2024-06-14T22:16:04.085Z	DEBUG	controller-jenkins	Extra role bindings are present	{"cr": "jenkins"}
2024-06-14T22:16:04.090Z	DEBUG	controller-jenkins	Jenkins HTTP Service is present	{"cr": "jenkins"}
2024-06-14T22:16:04.100Z	DEBUG	controller-jenkins	Jenkins slave Service is present	{"cr": "jenkins"}
2024-06-14T22:16:04.100Z	DEBUG	controller-jenkins	Kubernetes resources are present	{"cr": "jenkins"}
2024-06-14T22:16:04.100Z	DEBUG	controller-jenkins	Jenkins master pod is present	{"cr": "jenkins"}
2024-06-14T22:16:04.100Z	DEBUG	controller-jenkins	Jenkins master pod is ready	{"cr": "jenkins"}
2024-06-14T22:16:04.441Z	DEBUG	controller-jenkins	Jenkins API client set	{"cr": "jenkins"}
2024-06-14T22:16:04.609Z	DEBUG	controller-jenkins	Installed plugins 'REDACTED'	{"cr": "jenkins"}
2024-06-14T22:16:07.942Z	INFO	controller-jenkins	Restoring backup '597366'	{"cr": "jenkins"}

After Fix:

2024-06-20T16:46:53.264Z	DEBUG	controller-jenkins	Reconciling Jenkins	{"cr": "jenkins"}
2024-06-20T16:46:53.265Z	DEBUG	controller-jenkins	Operator credentials secret is present	{"cr": "jenkins"}
2024-06-20T16:46:53.281Z	DEBUG	controller-jenkins	Scripts config map is present	{"cr": "jenkins"}
2024-06-20T16:46:53.294Z	DEBUG	controller-jenkins	Init configuration config map is present	{"cr": "jenkins"}
2024-06-20T16:46:53.316Z	DEBUG	controller-jenkins	Base configuration config map is present	{"cr": "jenkins"}
2024-06-20T16:46:53.316Z	DEBUG	controller-jenkins	GroovyScripts Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-20T16:46:53.316Z	DEBUG	controller-jenkins	ConfigurationAsCode Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-20T16:46:53.316Z	DEBUG	controller-jenkins	createServiceAccount with annotations map[eks.amazonaws.com/role-arn:REDACTED]	{"cr": "jenkins"}
2024-06-20T16:46:53.438Z	DEBUG	controller-jenkins	Service account, role and role binding are present	{"cr": "jenkins"}
2024-06-20T16:46:53.438Z	DEBUG	controller-jenkins	Extra role bindings are present	{"cr": "jenkins"}
2024-06-20T16:46:53.444Z	DEBUG	controller-jenkins	Jenkins HTTP Service is present	{"cr": "jenkins"}
2024-06-20T16:46:53.449Z	DEBUG	controller-jenkins	Jenkins slave Service is present	{"cr": "jenkins"}
2024-06-20T16:46:53.449Z	DEBUG	controller-jenkins	Kubernetes resources are present	{"cr": "jenkins"}
2024-06-20T16:46:53.449Z	DEBUG	controller-jenkins	Jenkins master pod is present	{"cr": "jenkins"}
2024-06-20T16:46:53.449Z	DEBUG	controller-jenkins	Jenkins master pod is ready	{"cr": "jenkins"}
2024-06-20T16:46:53.464Z	DEBUG	controller-jenkins	Jenkins API client set	{"cr": "jenkins"}
2024-06-20T16:46:53.476Z	DEBUG	controller-jenkins	Installed plugins 'REDACTED'	{"cr": "jenkins"}
2024-06-20T16:46:54.106Z	INFO	controller-jenkins	Restoring backup '139533'	{"cr": "jenkins"}
2024-06-20T16:47:20.300Z	INFO	controller-jenkins	Waiting for Seed Job Agent `seed-job-agent`...	{"cr": "jenkins"}
2024-06-20T16:47:20.300Z	DEBUG	controller-jenkins	Reconciling Jenkins	{"cr": "jenkins"}
2024-06-20T16:47:20.300Z	DEBUG	controller-jenkins	Operator credentials secret is present	{"cr": "jenkins"}
2024-06-20T16:47:20.323Z	DEBUG	controller-jenkins	Scripts config map is present	{"cr": "jenkins"}
2024-06-20T16:47:20.337Z	DEBUG	controller-jenkins	Init configuration config map is present	{"cr": "jenkins"}
2024-06-20T16:47:20.357Z	DEBUG	controller-jenkins	Base configuration config map is present	{"cr": "jenkins"}
2024-06-20T16:47:20.357Z	DEBUG	controller-jenkins	GroovyScripts Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-20T16:47:20.357Z	DEBUG	controller-jenkins	ConfigurationAsCode Secret and ConfigMap added watched labels	{"cr": "jenkins"}
2024-06-20T16:47:20.357Z	DEBUG	controller-jenkins	createServiceAccount with annotations map[eks.amazonaws.com/role-arn:REDACTED]	{"cr": "jenkins"}
2024-06-20T16:47:20.530Z	DEBUG	controller-jenkins	Service account, role and role binding are present	{"cr": "jenkins"}
2024-06-20T16:47:20.530Z	DEBUG	controller-jenkins	Extra role bindings are present	{"cr": "jenkins"}
2024-06-20T16:47:20.541Z	DEBUG	controller-jenkins	Jenkins HTTP Service is present	{"cr": "jenkins"}
2024-06-20T16:47:20.552Z	DEBUG	controller-jenkins	Jenkins slave Service is present	{"cr": "jenkins"}
2024-06-20T16:47:20.552Z	DEBUG	controller-jenkins	Kubernetes resources are present	{"cr": "jenkins"}
2024-06-20T16:47:20.552Z	DEBUG	controller-jenkins	Jenkins master pod is present	{"cr": "jenkins"}
2024-06-20T16:47:20.552Z	DEBUG	controller-jenkins	Jenkins master pod is ready	{"cr": "jenkins"}
2024-06-20T16:47:20.745Z	DEBUG	controller-jenkins	Jenkins API client set	{"cr": "jenkins"}
2024-06-20T16:47:20.838Z	DEBUG	controller-jenkins	Installed plugins 'REDACTED'	{"cr": "jenkins"}
2024-06-20T16:47:20.839Z	DEBUG	controller-jenkins	Skipping restore backup, backup already restored	{"cr": "jenkins"}

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

See the contribution guide for more details.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS.

Release Notes

- Fixed bug that caused duplicate backups and restores after jenkins-master or operator pod restart

@DionJones615 DionJones615 changed the title fix: Duplicated restores fix: Duplicated backups & restores Jun 20, 2024
@DionJones615 DionJones615 changed the title fix: Duplicated backups & restores fix(backup): Duplicated backups & restores Jun 21, 2024
@brokenpip3 brokenpip3 self-requested a review June 22, 2024 08:11
@brokenpip3
Copy link
Collaborator

brokenpip3 commented Jun 22, 2024

thanks for your contribution!

Could you please merge the upstream master on your PR? I want to run the new tests with the backup image built on the fly and used before merging this PR.

About the multiple backup/restore issue I replied here: #962 (comment)

Copy link
Collaborator

@brokenpip3 brokenpip3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, great finding, ty!

@brokenpip3 brokenpip3 merged commit 5ef6c73 into jenkinsci:master Jun 25, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants