Flaky tests in scheduler/distributed_task_test.go

[IzabellaRaulin]

I received the following failures:

It looks like those tests reveal that for unsubscribing task dependencies happens in different moments for stopping task vs ended/disabled task.

a) Scenario for stop task:
stopTask() -> unsubscribePlugins -> emit TaskStoppedEvent
👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/scheduler.go#L530
(in case TaskStoppedEvent occurs, only taskWatcher handle that: https://github.com/intelsdi-x/snap/blob/master/scheduler/scheduler.go#L749)

b) Scenario for disabled task:
occurring X consecutive failures -> change state to core.TaskDisabled -> emit TaskDisabledEvent -> unsubscribePlugins
👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/task.go#L489
👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/scheduler.go#L768

c) Scenario for ended task (very similar to previous scenario in point b)
scheduler has Ended -> change state to core.TaskEnded -> emit TaskEndedEvent -> unsubscribePlugins

👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/task.go#L500
👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/scheduler.go#L757

According to tests in distributed_task_test.go relating to Ended state, it might happen that deps are not unsubscribed yet even "TaskEnded" event has been emitted. That is the reason why those two tests are flaky:
👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/distributed_task_test.go#L338
👉 https://github.com/intelsdi-x/snap/blob/master/scheduler/distributed_task_test.go#L383

Possible Solution:

Do unsubscribePlugins() before emit TaskEnded/TaskDisabled event