Optimize auto-commit process

[mahendra]

The auto-commit thread is a bit heavy and there are points of optimization. I tried two attempts to do it.

• Spawn the commit thread only if necessary - If there are no messages being consumed, the timer keeps creating new threads at the specified intervals. This may not be necessary. We can control this behaviour such that the timer thread is started only when a message is consumed

• The previous approach optimized the commit thread such that the timer started only when there were messages to be consumed. This goes a step further and ensures the following:

  • Only one timer thread is created
  • The main app does not block on exit (waiting for timer thread to finish). In the current code, if you exit the main thread, the app would wait till the commit thread exits (typically 5 seconds).

  This is ensured by having a single thread blocking on an event and keeps calling a function. We use events instead of time.sleep() so as to prevent the python interpreter from running every 50ms checking if the timer has expired (logic copied from threading.Timer)

Also provide support for passing args/kwargs to the timer callback function. Maybe useful in some future scenario

Also, we keep the re-entrant nature of the timer code intact by using an threading.Event and passing it as an argument to the thread target.

Also, with this, the logic of commit() becomes simple for manual or timed operations. We can remove all the extra checks of stopping/restarting the timer now. Commit operation is done by acquiring a lock. By checking for the commit requirements inside the lock, we can avoid any extra steps of stopping/starting the timer (which now runs forever). We do the check for self.count_since_commit in two places

• Before the lock - so that we can avoid acquiring the lock if there is no need to commit
• After the lock - so that we can avoid committing if the other thread has done the commit before we could acquire the lock.

[mahendra]

@mumrah did you get a chance to look at this? I was working on a design/prototype for the MutliProcessConsumer. This patch in master would be helpful :-)

[mumrah]

Looks good, @mahendra thanks for the patch

[mahendra]

@mumrah yaaay! thanks for merging