OpenStack Designate provider: failure to update existing recordset #603

dhague · 2018-06-18T13:22:24Z

When trying to update the IP address of an existing record, external-dns fails to do so and displays the log entries:

time="2018-06-15T15:04:19Z" level=info msg="Creating records: training.k8s-playground.c.eu-de-2.cloud.sap./A: 10.236.96.161"
time="2018-06-15T15:04:19Z" level=error msg="Expected HTTP response code [201 202] when accessing [POST https://dns-3.eu-de-2.cloud.sap/v2/zones/ed3d348c-bc97-46ad-a28e-03205587c1b9/recordsets], but got 409 instead\n{\"message\": \"Duplicate RecordSet\", \"code\": 409, \"type\": \"duplicate_recordset\", \"request_id\": \"req-b1bbe4d1-a2c5-4cff-ba39-bbfd9f213a8a\"}"

Both A and TXT records are correctly created in the first place, and the configured user can list the domain's recordsets using the openstack CLI.

The text was updated successfully, but these errors were encountered:

njuettner · 2018-06-22T09:09:29Z

I think this is related to the caching issue which we rolled out in the latest version:

Could you please try to see if it works with this image: registry.opensource.zalan.do/teapot/external-dns-test:pr-612-2

njuettner · 2018-06-28T13:47:07Z

New version is out which fixes the issue. https://github.com/kubernetes-incubator/external-dns/releases/tag/v0.5.4

dhague · 2018-06-28T15:47:21Z

Unfortunately 0.5.4 does not fix the issue (apologies for not testing the PR, I've been busy).
When updating the IP address of a Service monitored by external-dns I still see the same:

time="2018-06-28T15:37:57Z" level=info msg="Creating records: training.k8s-playground.c.eu-de-2.cloud.sap./A: 10.236.96.46"
time="2018-06-28T15:37:57Z" level=info msg="Creating records: training.k8s-playground.c.eu-de-2.cloud.sap./TXT: \"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/kube-system/ingress-nginx\"" 
time="2018-06-28T15:37:57Z" level=error msg="Expected HTTP response code [201 202] when accessing [POST https://dns-3.eu-de-2.cloud.sap/v2/zones/ed3d348c-bc97-46ad-a28e-03205587c1b9/recordsets], but got 409 instead\n{\"message\": \"Duplicate RecordSet\", \"code\": 409, \"type\": \"duplicate_recordset\", \"request_id\": \"req-6d370d58-81b5-4869-99df-ee55efa98192\"}"

This is using image registry.opensource.zalan.do/teapot/external-dns:v0.5.4

dhague · 2018-06-28T15:56:00Z

FWIW, I am starting the container with these args:

--log-level=info
--policy=upsert-only
--provider=designate
--registry=txt
--source=service

I am using Helm for deployment, using the stable/external-dns chart, updated with image.tag: 0.5.4 and adding nodes to the resources in its clusterrole.yaml template to fix an authz error I saw in the log.

njuettner · 2018-06-28T16:05:03Z

Could you try to enable debugging using--log-level=debug and comment the output again?

dhague · 2018-06-28T16:23:10Z

Here's the debug output after I updated the IP address on the service kube-system/ingress-nginx:

time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service default/kubernetes"
time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service kube-system/default-http-backend"
time="2018-06-28T16:20:18Z" level=debug msg="Endpoints generated from service: kube-system/ingress-nginx: [ingress-nginx.k8s-playground.c.eu-de-2.cloud.sap 0 IN A 10.236.98.191]"
time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service kube-system/kube-dns"
time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service kube-system/kube-system-external-dns-external-dns"
time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service kube-system/kubernetes-dashboard"
time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service kube-system/tiller-deploy"
time="2018-06-28T16:20:18Z" level=debug msg="No endpoints could be generated from service training-service/minio-nginx-default-backend"
time="2018-06-28T16:20:18Z" level=info msg="Creating records: ingress-nginx.k8s-playground.c.eu-de-2.cloud.sap./A: 10.236.98.191"
time="2018-06-28T16:20:18Z" level=error msg="Expected HTTP response code [201 202] when accessing [POST https://dns-3.eu-de-2.cloud.sap/v2/zones/ed3d348c-bc97-46ad-a28e-03205587c1b9/recordsets], but got 409 instead\n{\"message\": \"Duplicate RecordSet\", \"code\": 409, \"type\": \"duplicate_recordset\", \"request_id\": \"req-fbdf23ef-9fed-4e04-b420-fd4ada887c3a\"}"

sebastien-prudhomme · 2018-09-13T10:42:19Z

Issue still there in master branch.

Trying to debug: the endpoint labels defined for designateRecordSetID and designateZoneID in Records() function doesn't more exist when trying to read them in addEndpoint() function.

Seems the labels are reseted outside the designate.go source code.

sebastien-prudhomme · 2018-09-14T10:29:31Z

@njuettner In the Records function of the txt registry, endpoint labels of the record are replaced by the labels of the TXT record:

        for _, ep := range endpoints {
                if labels, ok := labelMap[ep.DNSName]; ok {
                        ep.Labels = labels

A merge of the labels should correct the bug.

sebastien-prudhomme · 2018-09-14T13:17:53Z

Merging the endpoints labels works for updating records. But there is still an issue with the TXT records. External-dns tries to create them instead of updating them:

That is because ApplyChanges() of the TXT registry creates TXT endpoints from scratch, instead of reusing what is returned as current records by the provider.

So the labels for designateRecordSetID and designateZoneID labels for the TXT records don't exist anymore.

@njuettner I've seen that the endpoint struct have a ProviderSpecific field. What is it used for?

njuettner · 2018-09-14T15:24:40Z

@sebastien-prudhomme When the PR was opened, there was already a use case for CloudFlare configuration which is not used by every provider, see #650

JrCs · 2019-03-05T15:32:17Z

The bug is always there.
This is because the endpoints return by the Records() function are not preserved when the ApplyChanges() function is call. In particular the endpoints Labels map was completely rewritten.
Perhaps an interface to the endPoints will be better with a specific structure for each providers.

fejta-bot · 2019-06-03T16:24:29Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Toriniasty · 2019-06-11T10:52:34Z

Hi,
any chance that this issue is going to be fixed soon?

Many thanks!

fejta-bot · 2019-07-11T11:14:35Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

multi-io · 2019-08-08T10:48:51Z

#1136 fixes TXTRegistry to retain the endpoint labels between Records() and ApplyChanges() calls on the provider, which fixes this for us for the most part (there's still one 409 per change happening for the TXT record, but the A record is updated correctly and all subsequent reconciliations go though without errors).

I'm sure this is not the right/best patch for this. This "remember stuff in labels" approach of the designate provider seems unusual; no other provider does this.

matthiasbernhardt · 2019-08-26T14:25:01Z

Does the presence of a new pull request rectify to revive the issue?
/remove-lifecycle rotten

fejta-bot · 2019-11-24T14:37:42Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

stefanandres · 2019-11-24T14:49:38Z

With https://github.com/kubernetes-sigs/external-dns/blob/master/CHANGELOG.md#v0517---2019-09-17 got #1136 merged which makes it at least usable. So I guess this issue here can be closed.

fejta-bot · 2019-12-24T15:21:30Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2020-01-23T16:06:56Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-01-23T16:07:05Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* Remove receipts migration * Remove receiptsmigration check from ci * Rename test * Remove oldenvironment * Code review changes

Fetch labels from existing TXT records to fix kubernetes-sigs#603

frittentheke · 2022-09-26T21:20:13Z

/reopen

k8s-ci-robot · 2022-09-26T21:20:16Z

@frittentheke: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Fetch labels from existing TXT records to fix kubernetes-sigs#603

frittentheke · 2023-01-20T14:54:45Z

@njuettner could you kindly reopen this issue as it's still broken and there is a PR fixing this #3049 (requires #3024 to be done first I believe)

njuettner added the kind/bug Categorizes issue or PR as related to a bug. label Jun 22, 2018

njuettner closed this as completed Jun 28, 2018

njuettner reopened this Jun 28, 2018

sebastien-prudhomme mentioned this issue Sep 19, 2018

Don't erase endpoints labels #713

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 3, 2019

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 11, 2019

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 26, 2019

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 24, 2019

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Dec 24, 2019

k8s-ci-robot closed this as completed Jan 23, 2020

lou-lan pushed a commit to lou-lan/external-dns that referenced this issue May 11, 2022

Remove receipts migration (kubernetes-sigs#603)

c23aea1

* Remove receipts migration * Remove receiptsmigration check from ci * Rename test * Remove oldenvironment * Code review changes

hikhvar pushed a commit to hikhvar/external-dns that referenced this issue Sep 26, 2022

Allow multiple records per A record

b22462c

Fetch labels from existing TXT records to fix kubernetes-sigs#603

hikhvar mentioned this issue Sep 26, 2022

Allow multiple records per A record in OpenStack Designate Provider #3049

Merged

2 tasks

hikhvar pushed a commit to hikhvar/external-dns that referenced this issue Sep 27, 2022

Allow multiple records per A record

2d2d19f

Fetch labels from existing TXT records to fix kubernetes-sigs#603

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenStack Designate provider: failure to update existing recordset #603

OpenStack Designate provider: failure to update existing recordset #603

dhague commented Jun 18, 2018 •

edited

Loading

njuettner commented Jun 22, 2018

njuettner commented Jun 28, 2018 •

edited

Loading

dhague commented Jun 28, 2018 •

edited

Loading

dhague commented Jun 28, 2018

njuettner commented Jun 28, 2018

dhague commented Jun 28, 2018

sebastien-prudhomme commented Sep 13, 2018 •

edited

Loading

sebastien-prudhomme commented Sep 14, 2018 •

edited

Loading

sebastien-prudhomme commented Sep 14, 2018

njuettner commented Sep 14, 2018 •

edited

Loading

JrCs commented Mar 5, 2019

fejta-bot commented Jun 3, 2019

Toriniasty commented Jun 11, 2019

fejta-bot commented Jul 11, 2019

multi-io commented Aug 8, 2019 •

edited

Loading

matthiasbernhardt commented Aug 26, 2019

fejta-bot commented Nov 24, 2019

stefanandres commented Nov 24, 2019

fejta-bot commented Dec 24, 2019

fejta-bot commented Jan 23, 2020

k8s-ci-robot commented Jan 23, 2020

frittentheke commented Sep 26, 2022

k8s-ci-robot commented Sep 26, 2022

frittentheke commented Jan 20, 2023

OpenStack Designate provider: failure to update existing recordset #603

OpenStack Designate provider: failure to update existing recordset #603

Comments

dhague commented Jun 18, 2018 • edited Loading

njuettner commented Jun 22, 2018

njuettner commented Jun 28, 2018 • edited Loading

dhague commented Jun 28, 2018 • edited Loading

dhague commented Jun 28, 2018

njuettner commented Jun 28, 2018

dhague commented Jun 28, 2018

sebastien-prudhomme commented Sep 13, 2018 • edited Loading

sebastien-prudhomme commented Sep 14, 2018 • edited Loading

sebastien-prudhomme commented Sep 14, 2018

njuettner commented Sep 14, 2018 • edited Loading

JrCs commented Mar 5, 2019

fejta-bot commented Jun 3, 2019

Toriniasty commented Jun 11, 2019

fejta-bot commented Jul 11, 2019

multi-io commented Aug 8, 2019 • edited Loading

matthiasbernhardt commented Aug 26, 2019

fejta-bot commented Nov 24, 2019

stefanandres commented Nov 24, 2019

fejta-bot commented Dec 24, 2019

fejta-bot commented Jan 23, 2020

k8s-ci-robot commented Jan 23, 2020

frittentheke commented Sep 26, 2022

k8s-ci-robot commented Sep 26, 2022

frittentheke commented Jan 20, 2023

dhague commented Jun 18, 2018 •

edited

Loading

njuettner commented Jun 28, 2018 •

edited

Loading

dhague commented Jun 28, 2018 •

edited

Loading

sebastien-prudhomme commented Sep 13, 2018 •

edited

Loading

sebastien-prudhomme commented Sep 14, 2018 •

edited

Loading

njuettner commented Sep 14, 2018 •

edited

Loading

multi-io commented Aug 8, 2019 •

edited

Loading