Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When starting up, ASM follower sometimes fails to sync with ASM Leader #2059

Closed
matt2e opened this issue Jul 12, 2024 · 5 comments
Closed

When starting up, ASM follower sometimes fails to sync with ASM Leader #2059

matt2e opened this issue Jul 12, 2024 · 5 comments
Assignees
@gak
Labels
P0 Work on this now

Comments

@matt2e
Copy link
Collaborator

matt2e commented Jul 12, 2024 •
edited by bradleydwyer

  • Retrying sometimes succeeds
  • This prevents any modules from deploying because the ModuleContext can't be created

Logs:
Unable to sync asm/follower/http://10.2.108.216:8892: error getting secrets list from leader: permission_denied: RBAC: access denied

[log links removed]
@github-actions github-actions bot added the triage Issue needs triaging label Jul 12, 2024
@ftl-robot ftl-robot mentioned this issue Jul 12, 2024
Open
@bradleydwyer
Copy link

Contact @bradleydwyer to get links to detailed logs.

@gak gak mentioned this issue Jul 13, 2024
Closed
@matt2e matt2e self-assigned this Jul 15, 2024
@github-actions github-actions bot removed the triage Issue needs triaging label Jul 15, 2024
@wesbillman wesbillman added the P0 Work on this now label Jul 15, 2024
@matt2e
Copy link
Collaborator Author

matt2e commented Jul 16, 2024

From @AlexSzlavik:
this error is definitely coming from Istio. I think Alec Brooks and I chased this briefly wrt to runners and controllers not being able to talk to each other during deployment. It feels like a race condition wrt to IP caching. I wonder if there is a way for us to choegraph the deployment roll out a bit more

@matt2e matt2e assigned gak and unassigned matt2e Jul 16, 2024
@matt2e
Copy link
Collaborator Author

matt2e commented Jul 16, 2024

@gak is going to add some settings to help debug this

@gak
Copy link
Contributor

gak commented Jul 16, 2024 •
edited

stuartwdouglas added a commit that referenced this issue Jul 23, 2024
@stuartwdouglas
fix: add ability to delay runner registration
16c3b0b 
We think there may be some issues with runners registering themselves before istio is fully ready, resulting in RBAC errors.

This workaround should allow for a delay that will prevent this.

Possible fix for #2059
@stuartwdouglas stuartwdouglas mentioned this issue Jul 23, 2024
Merged
stuartwdouglas added a commit that referenced this issue Jul 23, 2024
@stuartwdouglas
fix: add ability to delay runner registration
c046f03 
We think there may be some issues with runners registering themselves before istio is fully ready, resulting in RBAC errors.

This workaround should allow for a delay that will prevent this.

Possible fix for #2059
stuartwdouglas added a commit that referenced this issue Jul 23, 2024
@stuartwdouglas
fix: add ability to delay runner registration (#2139)
48ea4a6 
We think there may be some issues with runners registering themselves
before istio is fully ready, resulting in RBAC errors.

This workaround should allow for a delay that will prevent this.

Possible fix for #2059
@gak
Copy link
Contributor

gak commented Jul 24, 2024

Fixed in #2139

@gak gak closed this as completed Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gak gak

Labels
P0 Work on this now
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gak @wesbillman @bradleydwyer @matt2e