Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Non-deterministic issue with localtargets.* DNSEntrypoint population #38

Closed
ytsarev opened this issue Jan 29, 2020 · 0 comments · Fixed by #43
Closed

Non-deterministic issue with localtargets.* DNSEntrypoint population #38

ytsarev opened this issue Jan 29, 2020 · 0 comments · Fixed by #43
Assignees
@ytsarev
Labels
bug Something isn't working

Comments

@ytsarev
Copy link
Member

ytsarev commented Jan 29, 2020

It's somehow hard to reproduce but we periodically facing

time="2020-01-29T17:08:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.2;172.17.0.4;172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:08:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:08:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:08:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:08:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.5, Text=, TTL=30"
time="2020-01-29T17:09:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.2;172.17.0.4;172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:09:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:09:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:09:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:09:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.5, Text=, TTL=30"
time="2020-01-29T17:10:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""

Recreation of gslb CR fixes the issue

time="2020-01-29T17:10:08Z" level=info msg="Delete key /skydns/com/example/cloud/app3/6904e7b4"
time="2020-01-29T17:10:08Z" level=info msg="Delete key /skydns/com/example/cloud/app3"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6d89b98f to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/483cc44e to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/732d34f7 to Host=172.17.0.5, Text=, TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/03acc020 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/4b25072e to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/4fe76c50 to Host=172.17.0.5, Text=, TTL=30"

Looks like a race condition somewhere between external-dns / etcd backend population of local coredns
@ytsarev ytsarev self-assigned this Jan 29, 2020
@ytsarev ytsarev added the bug Something isn't working label Jan 29, 2020
@ytsarev ytsarev mentioned this issue Feb 10, 2020
Closed
ytsarev added a commit that referenced this issue Feb 10, 2020
@ytsarev
Disable external-dns ownership for local coredns
5b901b8 
* Disables ownership for local scenario
* Fixes #38
* Upstream issue details kubernetes-sigs/external-dns#1414
@ytsarev ytsarev mentioned this issue Feb 10, 2020
Merged
ytsarev added a commit that referenced this issue Feb 10, 2020
@ytsarev
Disable external-dns ownership for local coredns
0f05c83 
* Disables ownership for local scenario
* Fixes #38
* Upstream issue details kubernetes-sigs/external-dns#1414
ytsarev added a commit that referenced this issue Feb 11, 2020
@ytsarev
Disable external-dns ownership for local coredns (#43)
80e9a5d 
* Disables ownership for local scenario
* Fixes #38
* Upstream issue details kubernetes-sigs/external-dns#1414
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ytsarev ytsarev

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Disable external-dns ownership for local coredns k8gb-io/k8gb
1 participant
@ytsarev