Autoscale up/down workers

[schmidtw]

Ideally the outbound sender worker pools should auto-scale up or down based on the work load present in the queue.

Something along the lines of:

• Initially create 0 workers
• As requests show up in the channel/queue workers are created
• As workers no longer are needed (work isn't in the queue) they stop themselves

[schmidtw]

Based on some conversations with @johnabass it is probably best to not try to re-use workers, but spawn goroutines for them instead up to the specified limit (using a semaphore or atomic counter). Then as they complete their work, they decrement the semaphore/atomic counter & simply exit.

CPU costly things to be aware of & should probably be passed in to the new goroutine:

• HMAC Hash - https://github.com/Comcast/caduceus/blob/master/src/caduceus/outboundSender.go#L466
• Metric Counters - https://github.com/Comcast/caduceus/blob/master/src/caduceus/outboundSender.go#L500

Do check to make sure that the above can be passed safely as to not create synchronization issues.

Lastly, to help us understand if this is a better approach we should qualify the throughput (delivery rate & CPU usage) through testing before & after.

[schmidtw]

We restructured this code so we don't have a worker pool, but instead spawn 1 time use goroutines.

#119