Add --watchers flag to allow controller to respond automatically to Ingress or Service updates

[jlamillan]

This change is related to issue #484. It builds on this pull-request, and introduces a new flag called --events that when enabled, adds the controller's Run function as the event handler so that it's automatically called when a Service and/or Ingress are updated. Note that these "event" driven calls are complementary to the regular polling calls that occur every --interval. For example, we are experimenting with using --events as a way to increase our --interval while still being able to process updates to Services and Ingresses quickly and using the interval to handle of band changes every hour or so.

Some other notes:

• On start-up, the Informers perform an initial list operation. A BoundedFrequencyRunner is being used to wrap the handler to limit calls to RunOnce generated by events and to avoid a call to RunOnce for every Service/Ingress on initial startup.
• Similarly, the resync-period of the Informer is currently being set to 0 to further avoid any unnecessary call-backs from the SharedInformers unless a change has occurred. We can continue to rely on --interval to handle regular syncs since not all sources support events.
• When --events is specified, any k8s client timeout is disabled to avoid periodic warning messages of timeouts that are caused by the persistent HTTP connection mechanism the informers use.

[linki]

@jlamillan Thanks for the pull request. We'll look into it.

[jlamillan]

Thanks @linki.

Rebasing my your branch onto the latest...

[jlamillan]

Re-based again. @linki eagerly awaiting your thoughts and feedback here. We've been using this change for awhile and it's been working well for us.

[jlamillan]

/assign @Raffo

[Raffo]

I got no bandwidth for this right now, I have to push it to the new year (likely CW2, CW3). If someone wants to review this first, please go on.

[lbernail]

We'd be really interested in using informers too: in our mid-sized clusters (~1000 nodes, ~1500 services, ~1500 endpoints), generating endpoints takes ~5mn (the current code perform a lot of list pods calls, because we use headless services, which are heavy ones for the apiservers)

If I understand the PR correctly, it just uses Informers to trigger a sync loop on events. What would be even better is to use the informer cache to store services and pods (and, yes possibly trigger a sync on update).

In addition, using the pod objects is very inefficient compared to using endpoints (in which case we could native rely on pod Readiness instead of looking at the pod phase). If I understand correctly, we use pods for the publishHostIP case. Maybe we could use endpoints in general and fallback on pods for this specific use case?

[jlamillan]

What would be even better is to use the informer cache to store services and pods (and, yes possibly trigger a sync on update).

Good idea, I'll adjust the PR soon...

[lbernail]

Great, I can help if needed (and will definitely test)

Another idea would be to have a resync period (to avoid getting in an inconsistent state over time) but to use a BoundedFrequencyRunner for the main sync function to avoid running it if it was run less than X seconds ago (kube-proxy works like that for instance)

[jlamillan]

@lbernail, check out this branch, which updates ingress and service sources to use the informer cache instead of making API requests for ingresses and services/pods/nodes, respectively.

If that change looks good, I could possibly submit that as a separate pull-request and use this pull-request to subsequently add the ability to have informers to trigger a sync loop on events via --events flag.

i.e.

commit 1: Use k8s informer cache instead of active API server calls in ingress...
commit 2: Add --events flag to use k8s informers to automatically trigger sync loop…

[jlamillan]

/unassign @Raffo

[jlamillan]

/assign @njuettner

[njuettner]

@jlamillan do you mind fixing the conflicting files and then I'm happy to merge it.

[jlamillan]

Done @njuettner.

As a reminder - --events defaults to false. At some point, we should consider to having it default to true.

[SEJeff]

@jlamillan can you fix the conflicts?

[jlamillan]

@jlamillan can you fix the conflicts?

done.

[njuettner]

@jlamillan sorry again, do you mind fixing the conflicts again?

[FrederikNJS]

@njuettner: It seems that this has been rebased, and is ready to be merged. It would be a bit of a shame if @jlamillan would have to rebase again a 7th time.

[johanneswuerbach]

Sorry for the nudge @SEJeff / @njuettner , but we are also really looking forward to see this merged.

[jlamillan]

Looks like the RunOnce signature was updated the other day. Resolving conflicts, rebasing, and testing...

[njuettner]

lets merge it @jlamillan. thank you again for contributing, much appreciated!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlamillan, njuettner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [njuettner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment