Stopping task before the workflow finishes executing

[rashmigottipati]

The Athena team has experienced issues where the control in Swan stops the task before it gets to push its last metrics to the Cassandra database and thus lose a data point. The request is to have the ability to let the currently scheduled run of a task complete when we issue stop task request, such that no data points are lost.

Possible Solution:
The request to stop task would have occurred after incrementing the hitCount (which indicates the number of times the task has fired) and before firing the task. In this case, the hitCount = 1 (as an example) but the task hasn't fired, which implies that the workflow didn't execute as well. From the user's perspective, as the hitCount = 1, this leads to the assumption that the task has fired and the workflow finished executing but there are no metrics collected. Currently, Snap already provides graceful shutdown, which means that if a task if firing and hasn't finished executing the workflow, it won't preempt the task and unsubscribe the plugins immediately. It waits till the workflow finishes executing and then stops the task. To fix the problem that Athena team has experienced, we need to increment hitCount after the workflow finishes executing and successfully returns. This way, as a user, if I see that the task has hitCount = 1, it's guaranteed that there will be one data entry. In the case where stop task request is issued before even the workflow is executed, the hitCount will be 0 and the user wouldn't be expecting any data as well.