How to Handle Lots of Watches / Informer Consistently

[ccravens]

This is more of a general question than a bug (sorry there wasn't an option for question, just bug).

I have a large number (hundreds to over a thousand) watchers / informers that will need to be created for various namespaces. I know that I can create an informer per namespace, or just have a single informer subscribe to all events for all namespaces (either more informers with receiving fewer events vs fewer informers with receiving all cluster events).

What would be more performant? What do I need to be concerned with when it comes to needing to handle so many events from so many different namespaces? Right now I also manage a timer for each informer that restarts the informer every 5 seconds if no events are received based off this conversation: #596. It seems that my application does not receive all events from the informer (sometimes events are received, sometimes they are not) and I believe it may be due to an HTTP timeout, which is why I'm resetting (I don't know if 5 seconds is too frequent?)

  setTimer() {
    this.timer = setTimeout(async () => {
      console.log('>>> RESTARTING INFORMER >>> ' + this.path);
      await this.informer.stop();
      await this.informer.start();
    }, this.timeout);
  }

  async clearTimer() {
    clearTimeout(this.timer);
    this.setTimer();
  }

    informer.on('add', (obj: T) => {
      informer.clearTimer();
      handler.handleAdd(obj);
    });
    informer.on('update', (obj: T) => {
      informer.clearTimer();
      handler.handleUpdate(obj);
    });
    informer.on('delete', (obj: T) => {
      informer.clearTimer();
      handler.handleDelete(obj);
    });
    informer.on('error', (err: T) => {
      informer.clearTimer();
      handler.handleError(err);
    });

I'm just looking for some feedback or guidance on the most performant way to handle lots of events and ensuring all events are received, thank you!!

[brendandburns]

restarting every 5 seconds is definitely way to often. Restarting maybe every 5 minutes is probably more sane.

In general: 1 watch == 1 TCP session, so as a result, in general, fewer watches is better. Though it also kind of depends on the frequency of changes. Also, sometimes you need more watches because of RBAC (e.g. not every namespace is available to every identity)

You should really be receiving all events, if you are missing events it's a bug somewhere (either your code or the informer code)

Please provide more details about your setup (number of objects, etc) and we can help debug.

Note that the informer implementation has one big drawback in it's current implementation in that it re-lists every time it resets which can cause a bunch of latency if there are lots and lots of resources in the system.

[ccravens]

Thanks @brendanburns! I've been looking into this past few days and wanted to provide example code of what I'm doing now, I haven't had enough time to fully test this yet, but hopefully we can get some samples of how to properly handle informers so that others can also get help (feel free to add these to the samples of the project if they are helpful), here is what I have so far that I am testing now:

import Timeout = NodeJS.Timeout;

const k8s = require('@kubernetes/client-node');

import { Informer, KubeConfig } from '@kubernetes/client-node';
import { ListPromise, ObjectCallback } from '@kubernetes/client-node/dist/informer';

export class InformerService<T> implements Informer<T> {
  private timeout: number = 300000;
  private timer: Timeout = null;
  private path: string = null;
  private informer: Informer<T> = null;

  constructor(config: KubeConfig, path: string, listFn: ListPromise<T>) {
    this.path = path;
    console.log('>>> CREATING INFORMER >>> ' + this.path);
    this.informer = k8s.makeInformer(config, this.path, listFn);
    this.setTimer();
  }

  setTimer() {
    this.timer = setTimeout(async () => {
      console.log('>>> RESTARTING INFORMER >>> ' + this.path);
      await this.informer.stop();
      await this.informer.start();
    }, this.timeout);
  }

  async clearTimer() {
    clearTimeout(this.timer);
    this.setTimer();
  }

  on(verb: string, fn: ObjectCallback<T>) {
    this.informer.on(verb, fn);
  }

  off(verb: string, fn: ObjectCallback<T>) {
    this.informer.off(verb, fn);
  }

  start(): Promise<void> {
    return this.informer.start();
  }

  stop(): Promise<void> {
    return this.informer.stop();
  }
}

and here's how I'm using it:

  public static createInformer<T, U>(
    path: string,
    handler: K8sInformerHandler<T>,
    listFn: any
  ): Informer<V1Pod | V1Job | V1Deployment | V1StatefulSet | V1ReplicaSet | V1DaemonSet> {
    const informer = new InformerService(K8sApi.config(), path, listFn);

    informer.on('add', (obj: T) => {
      informer.clearTimer();
      handler.handleAdd(obj);
    });
    informer.on('update', (obj: T) => {
      informer.clearTimer();
      handler.handleUpdate(obj);
    });
    informer.on('delete', (obj: T) => {
      informer.clearTimer();
      handler.handleDelete(obj);
    });
    informer.on('error', (err: T) => {
      informer.clearTimer();
    });

    informer.start();

    return informer;
  }

[ccravens]

@brendanburns after a couple of weeks of testing the above implementation, it does seem like it's much better but there are times where it seems I am not getting events by using the restart code above. Just wanted to see if the above code is a good implementation from your perspective? I'm concerned that I may be missing events randomly it seems? Any insight or thoughts on this would be much appreciated thanks!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.