[external-dns] - Use IDs embedded in external_dns_route53_zone_arns as zone-id-filter

[egarbi]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

The current behaviour of external-dns presents challenges when working with AWS Route 53, especially when adding records to a few specific zones when you have lots of zones defined in your account. The default approach fetches all available zones, leading to:

1. Access Denied errors
   Multiple zones for the same domain cause "Access Denied" errors when adding records to a specific zone. (ie: example.com is defined both as private and public but you only need private)
2. Rate limiting
   Frequent calls to the AWS Route 53 API, despite only needing records for a subset of zones, result in rate-limiting issues.
   Example:

time="2023-12-15T14:19:16Z" level=info msg="Applying provider record filter for domains: [example1.com. 
.example1.com. example.es. .example.es. example1.es. .example1.es. example3.es. .example3.es. example.nl. 
.example.nl. example.in. .example.in. example2.com. .example2.com. example3.com. .example3.com. example4.com. 
.example4.com. example.be. .example.be. example.it. .example.it. example.fr. .example.fr. example2.nl. .example2.nl. 
example.info. .example.info. ...etc .]"

Describe the solution you would like

Utilize the existing variable external_dns_route53_zone_arns to filter out unnecessary zones. The solution involves using the zone-id-filter as a switch for the container, as demonstrated below:

❯ kubectl describe pods -n external-dns | grep zone
      --zone-id-filter=Z3XXXXXXXXT5
      --zone-id-filter=Z4HJKGXXXXT6

By parsing the zone IDs from external_dns_route53_zone_arns and passing them as zone-id-filters within the Helm values, we can optimize the module to allow only the necessary zone ARNs from IAM and at the same time have external-dns handling records exclusively within the specified zone.

Extra benefits

Enhanced reliability when adding records to specific zones.
Prevention of "Access Denied" errors by aligning zone retrieval with the intended zone for record addition.
Reduction in AWS Route 53 API calls, minimizing the risk of rate-limiting.

Describe alternatives you have considered

Additional context

Achieving the same goal by passing arbitrary switches to the container (e.g., domain_filters, etc) is possible. However, this requires users to possess a thorough understanding and analysis of their particular use case. The proposed approach eliminates the need for user intervention, providing a more user-friendly solution that works for most, if not all, use cases.

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[allamand]

@egarbi you can configure this with the external-dns helm values either in terraform or using GitOps-bridge with ArgoCD like in https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/argocd/bootstrap/control-plane/addons/aws/addons-aws-oss-external-dns-appset.yaml#L65

[egarbi]

@egarbi you can configure this with the external-dns helm values either in terraform or using GitOps-bridge with ArgoCD like in https://github.com/aws-samples/eks-blueprints-add-ons/blob/main/argocd/bootstrap/control-plane/addons/aws/addons-aws-oss-external-dns-appset.yaml#L65

I know, but it would be more convenient if the module had this feature readily available without requiring any extra effort from the user. I am planning to create a PR to showcase my idea, and I believe it would be beneficial to implement it. Ultimately, the decision is for the team to make.