-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
external-dns continuously deleting and adding the same A and TXT records #4059
Comments
Do I understand correctly that you have 1 zone that has some subdomains that delegate the subdomain to other NS servers? |
Hi, thanks for following up! There are no records missing in our case, and our availability has (so far) not been impacted. The primary symptom is that we see a lot of delete + add activity on A- and TXT records associated with zones that have externally managed NS records. Edit: The rest of this comment was based on an erroneous read of the logs on a particular environment. See my latest comment for an update.
|
We also see external-dns delete/recreate zone apex records on every reconciliation. I suspect that this is related to the new TXT registry format where the record name contains the record type: By default, the new TXT registry name is This happens even if you have a prefix defined for registry TXT records, since the record type is prepended to the DNS name of the A record, not the prefixed name of the TXT record. I think #3774 would fix this problem. |
Follow-up: Turns out the issue is still present on the simpler setups, I was just using the wrong log filters. We have several domains where there's only one zone and no nesting, and the root zone is affected by the TXT / A issue. |
Thanks!
@paul-at-cybr Can you confirm that's only on APEX records as @eyvind wrote? Maybe a workaround to try is to use a subdomain like |
@szuecs We've observed the same thing (Just the Google provider) affecting all records, not just apex domains E.g. Our zone is ( |
Looks like the regression was introduced between In our case, it is affecting any records that have the following provider configuration: Ingress:
CRD:
Arguably this config shouldn't be present on the CRD |
we have same issue
seem that there 2 same values CNAME and TXT being update by externalDNS,
|
Terribly sorry, I've struggled to find the time to properly follow up on this.
We're seeing the issue on subdomains as well, such as We have not observed the bug in any other subdomains. Only on apex records and on subdomains with nested managed zones. I've considered introducing a txt suffix to see if this fixes things, but this section from the txt registry readme indicates I might not want to do that:
|
Quick note that I found this bug while beginning to look into what seems to be a similar syndrome observed in our dev setup. Any pointers on how to debug would be appreciated. So far all I've seen is that in DO (where the zone is hosted) records appear and vanish at random. There's nothing obviously related in the external dns container logs (it isn't saying "I'm creating this record...", "I'm deleting this record..." although clearly it is. So presumably the default log level is not very informative? |
I've tracked the regression in this instance down to 5339c0c @johngmyers Hoping you might be able to advise on the changes in that commit. I see the key difference is previously the provider-specific logic:
Whereas now it has an additional logic:
This logic change does make sense on the surface, but feels like a breaking change from the old behaviour. Having multiple DNS entries that use different providers in the same resource (e.g. Ingress, CRD etc.) will cause constant deletes/recreates if that resource uses any provider-specific annotations cc @szuecs for your thoughts too Feels like with this change there needs to be a way to tie provider-specific config to providers, and not have them be considered for records that are ultimately handled by a different provider |
Signed-off-by: Sandor Szücs <sandor.szuecs@zalando.de>
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
This issue does not seem to be resolved for us as of 0.14.1. /remove-lifecycle stale |
@paul-at-cybr Which providers are you using and which are affected? Are the resources responsible for the records affected using any provider specific config? |
Seeing the same thing here too with either 0.13.6 or 0.14.1 on Azure Kubernetes using Azure Private DNS. Cloud dns provider: Azure
K8S platform: AKS |
same issue happening for me too. installed with the following flags:
you can check the DNS record here:
Let me know if you need any other hint or logs or want me to dig in any direction in the code.. this is kinda annoying-ish 😬 |
What happened:
After upgrading the external-dns image from 0.13.4 => 0.14.0, external-dns seems to have gotten stuck trying to continuously delete and recreate a subset of the records it manages.
A common thread among the domains external-dns misbehaves on, is that they all have NS records that are not managed by external-dns. In our case, these records are manually configured.
What you expected to happen:
A low rate of log output, allowing me to breathe easy in the knowledge that we are not hammering the Google Cloud DNS API.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
I've collected a series of debug-level log entries, washed them for public consumption, and attached them. Hope they prove useful.
Environment:
The text was updated successfully, but these errors were encountered: