One invalid record in ChangeBatch stops all others from updating

[richstokes]

What happened:

time="2020-04-13T22:23:40Z" level=info msg="Desired change: CREATE demo.example.io A [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=info msg="Desired change: CREATE demo-us-west-1.example.io A [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=info msg="Desired change: CREATE demo-host-il.example.io A [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=info msg="Desired change: CREATE demo.example.io TXT [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=info msg="Desired change: CREATE demo-us-west-1.example.io TXT [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=info msg="Desired change: CREATE demo-host-il.example.io TXT [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=error msg="Failure in zone example.io. [Id: /hostedzone/ZVEABCZXYZ123]"
time="2020-04-13T22:23:40Z" level=error msg="InvalidChangeBatch: [RRSet of type A with DNS name demo.example.io. is not permitted because a conflicting RRSet of type  CNAME with the same DNS name already exists in zone example.io., RRSet of type TXT with DNS name demo.example.io. is not permitted because a conflicting RRSet of type  CNAME with the same DNS name already exists in zone example.io.]\n\tstatus code: 400, request id: ca31ed28-2fef-4429-b769-ae04d297da51"
time="2020-04-13T22:23:40Z" level=error msg="Failed to submit all changes for the following zones: [/hostedzone/ZVEABCZXYZ123]"

What you expected to happen:
Ignore the invalid record, process the others

The use case here is the record demo.example.io is created outside of the K8s cluster. But the K8s ingress still needs to be able to handle traffic for this host, since the CNAME is set up to failover between two K8s clusters.

In previous versions of external-dns (<= v0.5.17) everything worked, since it just ignored any records that already exist. Now its batching changes and failing everything, even when only one of the records is "invalid".

Perhaps we need an "ignore" configuration option that would tell external-dns to continue on failure of N records instead of trying to do bulk, atomic submissions?

Environment:
external-dns: v0.7.1
K8s v1.16.2

[szuecs]

Also similar problem #731

[szuecs]

I think as workaround you could try natch site 1 or 2

[richstokes]

Thanks @szuecs - adding --aws-batch-change-size=2 as an arg works. Am I missing something or is there no documentation for this?

[szuecs]

It’s not really meant to fix your issue.
It’s meant to reduce the bytes send, because of aws api limits.

[OmerKahani]

Starting to work on this - https://kubernetes.slack.com/archives/C771MKDKQ/p1592295222475600

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[seanmalloy]

Can someone confirm this is still an issue with the latest release(v0.7.3)?

[Blanko2]

/remove-lifecycle stale
this seems to still be pending - at least this is what the changelogs show

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[szuecs]

/remove-lifecycle stale

[funkypenguin]

I can confirm this is still an issue on 0.7.6:

time="2021-03-30T09:40:38Z" level=error msg="failed to submit all changes for the following zones: [/hostedzone/Z00-redacted--VC]"
time="2021-03-30T09:41:39Z" level=info msg="Desired change: CREATE lorem.elpenguino.net CNAME [Id: /hostedzone/Z00-redacted--VC]"
time="2021-03-30T09:41:39Z" level=info msg="Desired change: CREATE loremipsumdolorsitametconsecteturadipiscingelitloremipsumdolorsitametconsecteturadipiscingelit.elpenguino.net CNAME [Id: /hostedzone/Z00-redacted--VC]"
time="2021-03-30T09:41:39Z" level=info msg="Desired change: CREATE txtlorem.elpenguino.net TXT [Id: /hostedzone/Z00-redacted--VC]"
time="2021-03-30T09:41:39Z" level=info msg="Desired change: CREATE txtloremipsumdolorsitametconsecteturadipiscingelitloremipsumdolorsitametconsecteturadipiscingelit.elpenguino.net TXT [Id: /hostedzone/Z00-redacted--VC]"
time="2021-03-30T09:41:39Z" level=error msg="Failure in zone elpenguino.net. [Id: /hostedzone/Z00-redacted--VC]"
time="2021-03-30T09:41:39Z" level=error msg="InvalidChangeBatch: [FATAL problem: DomainLabelTooLong (Domain label is too long) encountered with 'loremipsumdolorsitametconsecteturadipiscingelitloremipsumdolorsi', FATAL problem: DomainLabelTooLong (Domain label is too long) encountered with 'txtloremipsumdolorsitametconsecteturadipiscingelitloremipsumdolo']\n\tstatus code: 400, request id: 065e58ea-8383-4987-8b99-ff2f76634d8c"
time="2021-03-30T09:41:39Z" level=error msg="failed to submit all changes for the following zones: [/hostedzone/Z00-redacted--VC]"

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[szuecs]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[szuecs]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[szuecs]

/remove-lifecycle stale

[allamand]

Do we have any inputs on this issue ?

[szuecs]

I am working on that I will have time to fix it. 😀
In general I would be happy to review a pr here, if it's using binary search to figure out the bad records. So always do a split batch, one hopefully works the other not and follow up with the failing one to split and try again. We need to be careful with aws api limits.

[jegeland]

This should have been resolved with the contribution @knackaron and myself (then knackjeff) made back in 2021 with #2127 and included in 0.9.0. Just change the batchsize back to 1 resorts back to the old style behavior and simply submits the DNS change requests one at a time and not the "number of bytes sent" as @szuecs stated above. Have tested this in production environments and it solved this issue for us.

[szuecs]

@jegeland I don't think it's a great solution but it works for us as well. I meant that reducing batch size is not an optimal solution because the user has to calculate max bytes herself and judge about how big the average dns record might be. So it's not great for the user and not reducing api calls to cloud providers. Having too many issues with number of api calls to aws in the past with several incidents, I want to fix it when I have a bit more time to invest into coding and testing it.

[dudicoco]

@jegeland how is that different than the workaround suggested in #1517 (comment)?

I must stress that this is a workaround and does not resolve the issue.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[paritoshparmar14]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[szuecs]

/reopen
/priority backlog

[k8s-ci-robot]

@szuecs: Reopened this issue.

In response to this:

/reopen
/priority backlog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jukie]

Isn't this resolved in later versions now? Failed records are lumped into their own batch and retried.

[szuecs]

@jukie yeah that's true, but not completely. IIRC it will split into two chunks and one will be applied and the other not and in the next iteration it will do the same, so it will fix it after some time. Maybe we can do better than that, maybe it's fine.
Let's close for now and see if we get issues that suggest we need to be better here.
/close

[k8s-ci-robot]

@szuecs: Closing this issue.

In response to this:

@jukie yeah that's true, but not completely. IIRC it will split into two chunks and one will be applied and the other not and in the next iteration it will do the same, so it will fix it after some time. Maybe we can do better than that, maybe it's fine.
Let's close for now and see if we get issues that suggest we need to be better here.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.