Handle migration to the new TXT format: use ForceUpdate strategy

[mcharriere]

Description

This PR changes the way that missing records are handled from a specific logic to a force update strategy.
The idea basically is that instead of having a dedicated plan and a especial logic inside the planner to handle missing TXT records, external-dns can force the update of the records avoiding conflicts.

The base for this PR was introduced in #2811 and #2913

Fixes #3217

Checklist

• Unit tests updated
• End user documentation updated

[k8s-ci-robot]

Welcome @mcharriere!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: mcharriere / name: Matías Charrière (1bd3834, e938f5a, a0e291b, ed49e67, 44b5761, df5e9ae, 65e480d)

[szuecs]

/ok-to-test
/approve

[szuecs]

@mcharriere thanks makes sense and simplified it 😊

[mcharriere]

@szuecs thank you for the quick review 😄. the pr is not merging because it's missing the lgtm label. could you check that? thanks again!

[phajduk]

@mcharriere if I understand this correctly, using this feature brings us an ability to manually clear up all TXT entries, created by external-dns, and such will get recreated?

[mcharriere]

if I understand this correctly, using this feature brings us an ability to manually clear up all TXT entries, created by external-dns, and such will get recreated?

hi @phajduk. no, that's not correct. it still needs the at least 1 TXT record to be present in the registry (either the old format or the new one). This PR doesn't change the behavior, it only uses a different strategy.

[johngmyers]

The model here is slightly wrong, leading to test cases for situations that will never occur.

The new field will only ever be set on current Endpoints. Sources will not set it in the candidates. The new bit means "something about how this Endpoint is currently stored in the provider/registry is inconsistent". The only thing that has to happen is for the planner's Calculate() to schedule an update when the bit needs to change from set to unset.

We do not need tests for having ForceUpdate set in the candidates.

Instead of being a new field, this could go into ProviderSpecific.

[mcharriere]

The model here is slightly wrong, leading to test cases for situations that will never occur.

The new field will only ever be set on current Endpoints. Sources will not set it in the candidates. The new bit means "something about how this Endpoint is currently stored in the provider/registry is inconsistent". The only thing that has to happen is for the planner's Calculate() to schedule an update when the bit needs to change from set to unset.

That's the point of having a ForceUpdate, so the planner can queue endpoints for changes, independently of their real state. That's useful to solve registry inconsistencies as you said, but that's just one use case.
I know it's not ideal, but having registry specific logic (as before) on different places makes the migration to the new format really hard.

We do not need tests for having ForceUpdate set in the candidates.

Instead of being a new field, this could go into ProviderSpecific.

I don't think that's the appropriate place for it, because it's definitively not provider specific and on the other hand we don't need to keep track of the forced update.

The main reason behind this PR is to simplify the migration process. Currently upgrading from 0.11 to a newer version when working with AWS in a dual zone setup (public+private). I want also to create a follow up PR based on this one to remove the old format DNS record to finalize the migration.

[johngmyers]

Once you remove MissingRecords(), Provider and Registry are the same interface. ProviderSpecific can hold registry-specific data. The inconsistency and resulting need for an update is specific to the txt registry. The txt registry's PropertyValuesEqual() can use such registry-specific data in ProviderSpecific to cause the planner to send an update.

So I would say this definitely belongs in ProviderSpecific. ProviderSpecific (and PropertyValuesEqual) already provide the necessary mechanisms to cause the planner to send an update; there is no need for a new field.

Separately from this, there is no need to have tests where ForceUpdate is set in candidates. There isn't a use case for that.

[mcharriere]

hi there. as per @johngmyers suggestion, I reimplemented the PR using a ProviderSpecific property to force the update of the records. This is a bit cleaner than before, as it removes any logic inside the planner and it's located only on the TXT registry.

Thank you @johngmyers for your patience :)

@szuecs @johngmyers I'd appreciate another review. thank you!

[szuecs]

/approve cancel
Have to read it again, thanks @johngmyers

[szuecs]

/approve

[johngmyers]

Much better, thanks!
The one comment is non-blocking. I'm separately removing all special-case logic from PropertyValuesEqual and it'd be simpler if we don't add in something I'll just remove later. Cancel the hold if you disagree.

/lgtm
/hold to give author chance to address non-blocking comment

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: johngmyers, mcharriere, szuecs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [szuecs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[johngmyers]

/lgtm
/hold cancel