Restore support for exclusion-only domain filters for AWS provider

[orirawlings]

Description

This aims to fix regression #2821.

I've tweaked endpoint.DomainFilter.IsConfigured() to pass for filters that only express exclusion rules (either regular expression-based exclusion or an explicit exclusion list). I've also tweaked some of the initialization to handle empty strings in explicit filter lists less ambiguously.

Fixes #2821

Checklist

• Unit tests updated
• End user documentation updated

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: orirawlings / name: Ori Rawlings (4ad5699)

[k8s-ci-robot]

Welcome @orirawlings!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[orirawlings]

/remove-lifecycle stale

[Raffo]

Added a few comments, but I am a bit confused by this PR. It is changing other providers as well as the AWS provider. Can you clarify the changes that you made here?

[Raffo]

Why did you modify IsConfigured below if you are removing it here?

[orirawlings]

IsConfigured is still consumed at other call sites.

[Raffo]

What is the purpose of modifying this method?

[orirawlings]

There are call sites in the code that need to determine if filtering rules have been defined. As far as I can tell, those call sites are generally not concerned with "are allow rules defined" separately from "are exclude rules defined". As such, I've normalized this method to align with its name, "are any rules defined".

[Raffo]

This changes the behavior completely. Was the previous one wrong? If so, why?

[orirawlings]

I think this behavior is confusing and unexpected. Logically a filter list composed entirely of ignored values should be equivalent to a filter list that was never configured at all. It is especially confusing when a list with a single ignored value is considered as not configured, but multiple ignored values are considered configured.

This PR normalizes IsConfigured so that it only returns true if valid values have been included.

[orirawlings]

Added a few comments, but I am a bit confused by this PR. It is changing other providers as well as the AWS provider. Can you clarify the changes that you made here?

Thanks for the review @Raffo

I tried to document some of the context in the linked issue (#2821), but let me try to clarify here. (It's been a while since I prepared this PR so I have to refresh my memory as well)

If I recall there were two surprises:

1. If domain filters are defined with only exclusions they were considered as "not configured". Different areas of the code checked configured state and handled differently. In some places, the domain filter would proceed to be used as is, in other areas of the code the domain filter was considered invalid and ignored completely. I attribute this variation to the evolution of the domain filter. Initially, domain filters were merely explicit allowlists of the domains that could be managed. But over time it was extended to support regular expressions and separately to support exclusions. Not all call sites were updated to accommodate these new assumptions (for example, some call sites inspect domainFilter.Filters slice directly to modulate format of log warnings). Some call sites don't use IsConfigured() at all because they instead need to prepare data structures based directly on explicit allow/exclude lists (ex. querying for only specific zones from DNS provider rather than querying all zones and passing it through the filter).
2. Behavior of the domain filter lists can be varied by passing multiple "" entries. In some parts of the code, we're checking the length of the domain filter list, but in other parts of the code we're checking if the first entry of the list is "". Thus we can force some unexpected results by adjusting the length of the filters while still using all ignored values. This is actually how I'm currently working around the issue. I think it is odd and surprising for users.

From the issue:

I'm currently able to work around this by setting multiple empty domain filter entries to trick the domain filter into passing the IsConfigured() check. Yes, surprising, I know.

It's because "" is an overloaded value in the configuration system. In one part of the code, it means "permit everything" and in another part of the code it means "this hasn't been configured yet, ignore".

So this PR attempts to normalize a few things.

1. "" values in allow/exclude lists should be filtered out during initialization. That simplifies checks across a handful of call sites. If len(list) == 0 we can infer that there aren't any valid allow or exclude values.
2. We include checks on the exclude list and regex when determining IsConfigured() state. That avoids ignoring the filters in cases where allow rules are empty (i.e. permit everything except what is explicitly excluded).

Changes have been made to some providers outside of the AWS provider to align with normalization number 1, above. Other providers have been changed to fix log warnings that might be misleading when only exclusion rules are configured. In those cases the call site was calling IsConfigured(), assuming that the method would only be checking the allow list and then only including the allow list in the log message, when a helpful message should expose information about the entire filter configuration (allow regex, exclusion list, exclusion regex).

[Raffo]

@orirawlings ACK that I've seen this, but I wasn't able to dig deep into this again. Thanks for your detailed explanation, I'll try to give feedback as soon as I manage.