-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
After update from 0.7.4 to 0.7.5 external-dns try to delete ns record #1895
Comments
Thanks for the report, we'll look into this issue. |
@Riley-sohn can you also try the official image mentioned in the release ( Can you also provide more info on when you see it and other possible information on how to fully reproduce the bug? |
We are also seeing this bug on AWS - using k8s.gcr.io/external-dns/external-dns:v0.7.5 after upgrading from k8s.gcr.io/external-dns/external-dns:v0.7.4. |
904d8e4 (#1813) seems to have included support for NS records being returned from the registry. In our case, we are managing a hosted zone for foo.bar.com. We also have an A record for the same domain name, pointing to an AWS Load Balancer. It looks like External DNS is seeing the NS record being returned from the registry, not finding any resources which map to it (since NS records are usually managed outside of external DNS), and tries to delete it since it thinks it's 'managed'. There doesn't appear to be enough metadata for External DNS to distinguish whether it's supposed to manage the NS record (which is a new feature) or if it is managed externally. (as would be the case for anyone previously using 0.7.4) |
@backjo I will take a deeper look at the problem tomorrow. We can think of feature-flagging it for now, but I'd be happy to hear suggestions before I analyze the problem. |
@Raffo haven't thought about it too deeply, but the first thought that comes to mind is that for this use case, there should probably be metadata in the registry that tracks which specific records for a domain name are being managed by external-dns and which records are not. |
Thank you for the context @backjo , that seems to be quite bad. I will try to reproduce it and fix it as soon as possible which means we'll likely rush 0.7.6 (till we have a better versioning convention, coming next year). |
@backjo are you also using google as provider? I don't have a GKE cluster to use to reproduce the bug, but I tried deploying to AWS and I was not able to reproduce the error. Can you please provide the full manifest that you use to deploy ExternalDNS and the full example of what you are doing? |
@Raffo unlike the original poster, we are using AWS / EKS , not GKE - it appears to be provider agnostic. Try the following setup:
|
@Raffo @backjo maybe we should provide a way to skip a specific DNS record type from plan? E.g. make plan condition list configurable, somewhere around https://github.com/kubernetes-sigs/external-dns/pull/1813/files#diff-0967da8a495108f825298b0bc67c95fe9d301a1bd86baadc65a7cb3623686cc3R245 |
WDYT about storing information in the registry around which specific record types are being managed for a given domain - I could see an eventual case where I might want to manage some domain NS records via external DNS, but others only maybe the A record. |
Ok, we took some time with @njuettner to try to reproduce the bug and we managed to reproduce it following the steps that @backjo provided. The issue is clearly created by #1813 and we discussed at length how we want to fix it. ExternalDNS was not designed to handle custom records as we are doing with the
Of course, a revert is to ensure our priority which is stability, not to say that features should not be added. If we ended up deciding for such a solution, we will start a discussion with the community to understand the need to manage custom DNS records with ExternalDNS. @backjo: I have one more question for you: given that a |
@Raffo In this case, these NS records are in the delegated zone, not the root zone - i.e. this zone is being delegated to from another, higher level hosted zone. foo.com Zone bar.foo.com Zone |
Thank you for the update @backjo , very appreciated. As I said, I will give an update tomorrow with a plan to move forward. |
We are surprised by this bug happening in all of our clusters in AWS Route53 as well! This is pretty scary that it tries to delete the NS records!
|
@universam1 I'm a bit late with a fix, but I'm going to be publishing an update today. |
Thank you for quick response! Guess what surprised us most that such a change came with a patch version only- we would expect that to be reflected by a feature or major version bump at least. Our review process is based on semantic versioning principle trust. Thank you for considering that. |
For sure we've been lacking a proper versioning strategy that could give better ideas of the change contained in a release. We did however take care of putting good information in the release notes and in fact the release for 0.7.5 contained information on this bug as soon as it was discovered. Again, that's a bug, not a breaking change designed on purpose to be one. I'm in the process of working around it. |
This is what I came up with to work around this issue: #1915 . We basically feature flag the NS records support, thus "protecting" the default functionality from ExternalDNS. |
All right, the fix was merged, I will start the release process later today, we will have an official 0.7.6 image released by the end of the week. |
I manually tested the change on AWS and I can confirm that the regression is fixed. We proceed with the release of |
What happened: After update external-dns image from 0.7.4-debian-10-r67 to 0.7.5-debian-10-r0, external-dns try to delete a record and ns record of domain-filter DNS name. After we change image from 0.7.5 to 0.7.4, it's ok.
How to reproduce it (as minimally and precisely as possible): make hostedzone with filtered-domain name and use external-dns to manage that hosted zone
Anything else we need to know?:
Environment:
external-dns --version
): 0.7.5logs:
The text was updated successfully, but these errors were encountered: