Infoblox implementation will throw exceptions when there are over 999 hosted zones

[mmerrill3]

What happened:

External-dns is ran with the configuration to use infoblox as the DNS provider. If the size of the hosted zones from the GET API "zone_auth" is larger than 999 hosted zones, the infoblox API will throw an exception

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Run external-dns with infoblox as the DNS provider. If there are more than 1000 results come back from zone_auth, infoblox will return an exception

Here is a shell file to replicate the issue, fill in a GRID_HOST that has more than 1000 hosted zones.

export GRID_HOST=test.com
export WAPI_PORT=443
export WAPI_VERSION=2.10.3
export WAPI_USERNAME=<user>
export WAPI_PASSWORD=<password>
curl -kl \
      -X GET \
      -u ${WAPI_USERNAME}:${WAPI_PASSWORD} \
         https://${GRID_HOST}:${WAPI_PORT}/wapi/v${WAPI_VERSION}/zone_auth?_return_fields=extattrs,fqdn,view

The output will be :

{ "Error": "AdmConProtoError: Result set too large (> 1000)",
"code": "Client.Ibap.Proto",
"text": "Result set too large (> 1000)",
"trace": " File "ibapapachemod.py", line 470, in handlerequest\n File "/infoblox/common/lib/python/infoblox/one/admin_conn/wapibase.py", line 1485, in execute\n File "/infoblox/common/lib/python/infoblox/one/admin_conn/wapibase.py", line 1439, in _exec_handler\n File "/infoblox/common/lib/python/infoblox/one/admin_conn/wapihandler.py", line 887, in execute\n File "/infoblox/common/lib/python/infoblox/one/wapibaseobj.py", line 1377, in read\n"
Anything else we need to know?:

Environment:

• External-DNS version (use external-dns --version): 0.8.0
• DNS provider: Infoblox
• Others:
  A proposed solution may be to allow querying for the hosted zones with a regular expression. See https://www.infoblox.com/wp-content/uploads/infoblox-deployment-infoblox-rest-api.pdf, where "zone_auth" API can be called with a regular expression for a fqdn. For example, to find hosted zones that start with staging in test.com:

curl -k -u admin:infoblox -X GET "https://gridmaster/wapi/v2.10.3/zone_auth?fqdn~=^staging.*.test.com$&_return_fields=fqdn&_return_as_object=1"

[mmerrill3]

This is applicable for large organizations where there are thousands of hosted zones,... like mine.

[mmerrill3]

FYI @saileshgiri