Split horizon setup with target-net-filter may end up with endpoint without targets

[BadLiveware]

What happened:
Using net filter in order to update a split horizon DNS setup, one with include 10.0.0.0/8 and one with exclude 10.0.0.0/8 while having a HTTPRoute referencing two gateways, one with a "correct" address of 10.180.0.58 and one which has a hostname of istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local causes the internal external-dns instance to choose the hostname gateway but ends up with a record without a target.

Clipped logs to showcase the issue, full logs below

...
INFO[0000] config: { ... TargetNetFilter:[10.0.0.0/8] ...}
...
DEBU[0000] Endpoints generated from HTTPRoute gitlab/gitlab-ui: [gitlab.example.name 0 IN A  10.180.0.58 [] gitlab.example.name 0 IN CNAME  istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local []]
DEBU[0000] Endpoints generated from HTTPRoute echo-server/http: [gitlab.example.name 0 IN A  10.180.0.58 [] gitlab.example.name 0 IN CNAME  istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local [] echo.example.name 0 IN A  10.180.0.58 [] echo.example.name 0 IN CNAME  istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local []]
INFO[0000] CREATE: gitlab.example.name 0 IN CNAME   []
INFO[0000] CREATE: echo.example.name 0 IN CNAME   []
...

What you expected to happen:
net filter filters out the CNAME record and since the endpoint is without targets, the entire endpoint.

Thus the only record remaining is the correct one with a valid target.

How to reproduce it (as minimally and precisely as possible):
Use net-filter with two "competing" endpoints, one with a legit target, one with a hostname

See unit tests in 77c05ac in #3451 for failing tests

Anything else we need to know?:

Setup:
Gateways:

❯ kubectl get gateway
NAME                            CLASS   ADDRESS                                                                      PROGRAMMED   AGE
istio-ingressgateway-external   istio   istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local   True         20h
istio-ingressgateway-internal   istio   10.180.0.58

❯ kubectl tree gateway.v1beta1.gateway.networking.k8s.io istio-ingressgateway-external
NAMESPACE             NAME                                                                         READY  REASON  AGE
istio-ingressgateway  Gateway/istio-ingressgateway-external                                        True   Ready   20h
istio-ingressgateway  ├─Deployment/istio-ingressgateway-external-istio                             -              20h
istio-ingressgateway  │ ├─ReplicaSet/istio-ingressgateway-external-istio-567fd8fc9b                -              19h
istio-ingressgateway  │ │ ├─Pod/istio-ingressgateway-external-istio-567fd8fc9b-nptkw               True           18h
istio-ingressgateway  │ │ └─Pod/istio-ingressgateway-external-istio-567fd8fc9b-vd6sh               True           18h
istio-ingressgateway  │ ├─ReplicaSet/istio-ingressgateway-external-istio-59b4d4b7b5                -              20h
istio-ingressgateway  │ ├─ReplicaSet/istio-ingressgateway-external-istio-6f68d88bbf                -              20h
istio-ingressgateway  │ └─ReplicaSet/istio-ingressgateway-external-istio-f9cb6676d                 -              19h
istio-ingressgateway  └─Service/istio-ingressgateway-external-istio                                -              20h

❯ kubectl tree svc istio-ingressgateway-external-istio
NAMESPACE             NAME                                                               READY  REASON  AGE
istio-ingressgateway  Service/istio-ingressgateway-external-istio                        -              20h
istio-ingressgateway  ├─EndpointSlice/istio-ingressgateway-external-istio-9xk9s          -              20h
istio-ingressgateway  └─ServiceNetworkEndpointGroup/istio-ingressgateway-external-istio  -              19h

❯ kubectl tree gateway.v1beta1.gateway.networking.k8s.io istio-ingressgateway-internal
NAMESPACE             NAME                                                                         READY  REASON  AGE
istio-ingressgateway  Gateway/istio-ingressgateway-internal                                        True   Ready   20h
istio-ingressgateway  ├─Certificate/wildcard-example-name-tls                                      True   Ready   20h
istio-ingressgateway  ├─Deployment/istio-ingressgateway-internal-istio                             -              20h
istio-ingressgateway  │ ├─ReplicaSet/istio-ingressgateway-internal-istio-58dfdfc4c                 -              19h
istio-ingressgateway  │ ├─ReplicaSet/istio-ingressgateway-internal-istio-5cd5f4b865                -              20h
istio-ingressgateway  │ ├─ReplicaSet/istio-ingressgateway-internal-istio-6b747fcbff                -              20h
istio-ingressgateway  │ └─ReplicaSet/istio-ingressgateway-internal-istio-75945dd6ff                -              23m
istio-ingressgateway  │   ├─Pod/istio-ingressgateway-internal-istio-75945dd6ff-9k27v               True           23m
istio-ingressgateway  │   └─Pod/istio-ingressgateway-internal-istio-75945dd6ff-tl224               True           23m
istio-ingressgateway  └─Service/istio-ingressgateway-internal-istio                                -              20h
istio-ingressgateway    └─EndpointSlice/istio-ingressgateway-internal-istio-6cqw6                  -              20h

Logs:

╭─ ~/c/external-dns master
╰─❯ ./build/external-dns --source=gateway-httproute --provider=inmemory --once --kubeconfig=$KUBECONFIG --target-net-filter=10.0.0.0/8 --log-level debug
INFO[0000] config: {APIServerURL: KubeConfig:/home/dev/.kube/.switch_tmp/config.188358265.tmp RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[gateway-httproute] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false GatewayNamespace: GatewayLabelFilter: Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:inmemory GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] TargetNetFilter:[10.0.0.0/8] ExcludeTargetNets:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSAssumeRoleExternalID: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AWSSDServiceCleanup:false AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatDNSConfiguration: BluecatConfigFile:/etc/kubernetes/bluecat.json BluecatDNSView: BluecatGatewayHost: BluecatRootZone: BluecatDNSServerName: BluecatDNSDeployType:no-deploy BluecatSkipTLSVerify:false CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: InfobloxCreatePTR:false InfobloxCacheDuration:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: Interval:1m0s MinEventSyncInterval:5s Once:true DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false OCPRouterName: IBMCloudProxied:false IBMCloudConfigFile:/etc/kubernetes/ibmcloud.json TencentCloudConfigFile:/etc/kubernetes/tencent-cloud.json TencentCloudZoneType: PiholeServer: PiholePassword: PiholeTLSInsecureSkipVerify:false PluralCluster: PluralProvider:}
DEBU[0000] apiServerURL:
DEBU[0000] kubeConfig: /home/dev/.kube/.switch_tmp/config.188358265.tmp
INFO[0000] Using kubeConfig
INFO[0000] Created GatewayAPI client https://172.19.0.2
INFO[0000] Instantiating new Kubernetes client
DEBU[0000] apiServerURL:
DEBU[0000] kubeConfig: /home/dev/.kube/.switch_tmp/config.188358265.tmp
INFO[0000] Using kubeConfig
INFO[0000] Created Kubernetes client https://172.19.0.2
DEBU[0000] Endpoints generated from HTTPRoute gitlab/gitlab-ui: [gitlab.example.name 0 IN A  10.180.0.58 [] gitlab.example.name 0 IN CNAME  istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local []]
DEBU[0000] Endpoints generated from HTTPRoute echo-server/http: [gitlab.example.name 0 IN A  10.180.0.58 [] gitlab.example.name 0 IN CNAME  istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local [] echo.example.name 0 IN A  10.180.0.58 [] echo.example.name 0 IN CNAME  istio-ingressgateway-external-istio.istio-ingressgateway.svc.cluster.local []]
INFO[0000] CREATE: gitlab.example.name 0 IN CNAME   []
INFO[0000] CREATE: echo.example.name 0 IN CNAME   []
INFO[0000] CREATE: gitlab.example.name 0 IN TXT  "heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitlab/gitlab-ui" []
INFO[0000] CREATE: cname-gitlab.example.name 0 IN TXT  "heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/gitlab/gitlab-ui" []
INFO[0000] CREATE: echo.example.name 0 IN TXT  "heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/echo-server/http" []
INFO[0000] CREATE: cname-echo.example.name 0 IN TXT  "heritage=external-dns,external-dns/owner=default,external-dns/resource=httproute/echo-server/http" []

Environment:

• External-DNS version (use external-dns --version): external-dns-helm-chart-1.12.1-22-g93cb700f
• DNS provider: google cloud
• Others:

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[BadLiveware]

still relevant

/remove-lifecycle stale