[bug]: ResourceWatcher stops working after leadership changes

[tomasfabian]

Describe the bug

When the leadership changes the ResourceWatcher throws an unobserved exception:

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. (Cannot access a disposed object.)
 ---> System.ObjectDisposedException: Cannot access a disposed object.
   at System.Reactive.Subjects.Subject`1.ThrowDisposed() in /_/Rx.NET/Source/src/System.Reactive/Subjects/Subject.cs:line 53
   at System.Reactive.Subjects.Subject`1.OnNext(T value) in /_/Rx.NET/Source/src/System.Reactive/Subjects/Subject.cs:line 141
   at KubeOps.Operator.Kubernetes.ResourceWatcher`1.OnException(Exception e)
   at k8s.Watcher`1.WatcherLoop(CancellationToken cancellationToken)
   at k8s.Watcher`1.<.ctor>b__9_0()

My suspicion is that this happens after calling the StartAsync again on the same instance after StopAsync.

The problem could be fixed by removing this line where the _reconnectHandler is disposed again outside of the previous if (!fromStop) clause.

To reproduce

Change the leadership from LeaderState.Leader to LeaderState.Candidate and back to LeaderState.Leader.

Expected behavior

The resource watcher should not stop watching after leadership changed.

Screenshots

No response

Additional Context

I can prepare a PR if you will confirm that the issue is legitimate. Could be related to #482.

[tomasfabian]

Shouldn't be the _reconnectSubscription.Dispose(); also inside the if (!fromStop) ? Or the subscription in the ctor should be recreated, because on the next StartAsync call it is not subscribed anymore. Do I understand it correctly, please?

The reconnect subscription should be either recreated on each StartAsync or not disposed during CloseAsync.

[buehler]

Hey @tomasfabian
Sorry for the late reply, let me check :-)

[tomasfabian]

Hi @buehler,
would it be also possible to release the fix for version 6, please?

Thanks.

[buehler]

Hey @tomasfabian
Hmm.... are you not able to upgrade to 7?
backporting is kind of a burden and I need more time to set up release/support branches for semantic release

[tomasfabian]

Ok no problem if it is a hassle, we will upgrade our operator to v7.