CRD source: add event-handler support

[ericrrath]

Description

When the --events flag is passed at startup, Source.AddEventHandler() is called
on each configured source. Most sources provide AddEventHandler()
implementations that invoke the reconciliation loop when the configured source
changes, but the CRD source had a no-op implementation. I.e. when a custom
resource was created, updated, or deleted, external-dns remained unware, and the
reconciliation loop would not fire until the configured interval had passed.

This change adds an informer (on the CRD specified by --crd-source-apiversion
and --crd-source-kind=DNSEndpoint), and a Source.AddEventHandler()
implementation that calls Informer.AddEventHandler(). Now when a custom
resource is created, updated, or deleted, the reconciliation loop is invoked.

Testing

I ran external-dns with the "inmemory" provider, the "noop" registry, and the --events flag with a CRD source, and observed normal startup:

host:~ user$ external-dns \
> --provider=inmemory \
> --inmemory-zone=example.com \
> --registry=noop \
> --source=crd \
> --crd-source-apiversion=example.com/v1 \
> --crd-source-kind=DNSEndpoint \
> --events \
> --interval=1800s
INFO[0000] config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[crd] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:inmemory GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatConfigFile:/etc/kubernetes/bluecat.json CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[example.com] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:noop TXTOwnerID:default TXTPrefix: TXTSuffix: Interval:30m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:true LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:example.com/v1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false} 
INFO[0000] Instantiating new Kubernetes client          
INFO[0000] Using kubeConfig                             
INFO[0000] Created Kubernetes client https://138.1.18.242:6443 
INFO[0008] All records are already up to date

Then I used the following files to create new instances of my custom resource, one in the default namespace, and one in the "foo" namespace (to verify that the event handler logic handles namespaces correctly):

host:~ user$ cat dnsendpoint-default.yaml 
apiVersion: example.com/v1
kind: DNSEndpoint
metadata:
  name: in-default-namespace
  namespace: default
spec:
  endpoints:
  - dnsName: 'in-default.example.com'
    recordType: A
    targets:
    - 192.0.2.1
host:~ user$ cat dnsendpoint-foo.yaml 
apiVersion: example.com/v1
kind: DNSEndpoint
metadata:
  name: in-foo-namespace
  namespace: foo
spec:
  endpoints:
  - dnsName: 'in-foo.example.com'
    recordType: A
    targets:
    - 192.0.2.2

I applied the files to create the resources. I added a sleep 30 in between to see if there was a corresponding period between external-dns detecting the two creations:

host:~ user$ kubectl apply -f dnsendpoint-default.yaml 
dnsendpoint.example.com/in-default-namespace created
host:~ user$ sleep 30
host:~ user$ kubectl apply -f dnsendpoint-foo.yaml 
dnsendpoint.example.com/in-foo-namespace created

Then I observed the expected external-dns logging indicating the resource creations were detected, and processed, with the expected interval:

INFO[0035] CREATE: in-default.example.com 0 IN A  192.0.2.1 [] 
INFO[0041] All records are already up to date           
INFO[0078] CREATE: in-foo.example.com 0 IN A  192.0.2.2 [] 
INFO[0084] All records are already up to date           

Then I deleted the resources:

host:~ user$ kubectl delete dnsendpoint in-default-namespace
dnsendpoint.example.com "in-default-namespace" deleted
host:~ user$ sleep 30
host:~ user$ kubectl -n foo delete dnsendpoint in-foo-namespace
dnsendpoint.example.com "in-foo-namespace" deleted

... and observed logging output indicating the resource deletions were detected and processed:

INFO[0119] DELETE: in-default.example.com 0 IN A  192.0.2.1 [] 
INFO[0165] DELETE: in-foo.example.com 0 IN A  192.0.2.2 [] 

Fixes #ISSUE

Checklist

• Unit tests updated - I couldn't think of a way to unit-test this yet
• End user documentation updated - will check

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k0da]

It looks like there is a way around that. kubernetes/kubernetes#95897

[k0da]

This is must have feature, can we have it in?

[k0da]

/remove-lifecycle stale

[seanmalloy]

/kind feature

[mgruener]

What remains to be done here? This would be quite an important feature to have if you can't have external-dns running with a very short sync-interval.

We are running external-dns on ~40 clusters against the same AWS account and with that setup we are running into the AWS API rate limiting if we set external-dns to sync every few minutes. Because of this, the normal reconcile only runs every 60 minutes, which means our users have to wait at least an hour till their deployments are fully up and reachable. If the external-dns CRD source would support event-handling, this could be massively improved.

[ericrrath]

I'm not aware of any more changes to the PR; I think this is ready to go once it's approved.

[jhoelzel]

I would also love to see this PR approved, its an amazing feature to have =)

[tanujd11]

Can we have this merge please if it is tested? We need to use this feature. Thanks

[sameeraksc]

We also need to use this feature. Thanks

[jhoelzel]

Guys this is already implemented in the helm chart it seems:
repoURL: https://charts.bitnami.com/bitnami
targetRevision: 6.5.3
chart: external-dns

      triggerLoopOnEvent: true
        policy: sync
        sources:
          - ingress
          - crd
          - service
        domainFilters:
          - xxy.test.com
        provider: pdns
        txtOwnerId: prodk3s
        txtPrefix: external-dns
        registry: txt
        crd:
          create: true

or if you are argoing (like you should):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-dns
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  destination:
    namespace: external-dns
    server: https://kubernetes.default.svc
  project: system
  source:
    repoURL: https://charts.bitnami.com/bitnami
    targetRevision: 6.5.3
    chart: external-dns
    helm:
       values: |
        triggerLoopOnEvent: true
        policy: sync
        sources:
          - ingress
          - crd
          - service
        domainFilters:
          - k8s.xxx.xxx
        provider: pdns
        txtOwnerId: prodk3s
        txtPrefix: external-dns
        registry: txt
        pdns:
          apiUrl: http://xxx
          apiKey: xxx
          apiPort: 80
        crd:
          create: true
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

and is also documented here:
https://github.com/kubernetes-sigs/external-dns/blob/master/docs/contributing/crd-source.md

I have been using it for a while now and it works like a charm

[metroshica]

@njuettner Is there anything else missing for this to be merged in?

[sameeraksc]

Not working for us @jhoelzel can you please check this if all good @njuettner ?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[Djelibeybi]

/remove-lifecycle rotten

[njuettner]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ericrrath, jlamillan, nachomillangarcia, njuettner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [njuettner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[njuettner]

/ok-to-test