DSET-4080 fix recovery from error state

[zdaratom-s1]

Problem

• when DataSet returns Retryable error we try to retry with timeout. Once this time is over we stay in error mode since last error is never overriden, and new event handling is rejected.

Solution

• keep timestamp of last error and consider it while evaluate last error timestamp in order to recover from error mode.

[codecov-commenter]

Codecov Report

Merging #36 (d6a0ea3) into main (1295b6f) will increase coverage by 0.98%.
The diff coverage is 85.00%.

@@            Coverage Diff             @@
##             main      #36      +/-   ##
==========================================
+ Coverage   73.86%   74.84%   +0.98%     
==========================================
  Files          10       10              
  Lines        1377     1387      +10     
==========================================
+ Hits         1017     1038      +21     
+ Misses        298      290       -8     
+ Partials       62       59       -3     

Impacted Files	Coverage Δ
pkg/client/client.go	88.53% <85.00%> (+2.85%)	⬆️

... and 1 file with indirect coverage changes

[tomaz-s1]

For a change like that, I would feel more comfortable if we also had some end to end tests - I think there should be the mock Python server in place which we should be able to utilize for that.

[zdaratom-s1]

@tomaz-s1 I have more e2e test like unit (blackbox) test using waiting etc. but that test fails on pre commit hook despite the fact that in idea its passes. So after all I decided to mock the state (whitebox test), its also much faster.

[tomaz-s1]

@zdaratom Yeah, it actually looks like we don't have a setup for such tests in this repo, but we have it in the otel one (https://github.com/scalyr/dataset-exporter-opentelemetry/tree/main/dummy-server), so we should add an end to end test like this in that repo once the PR is merged.

[zdaratom-s1]

sounds reasonable, I will add TODO into the Jira