Stop retrying event tracker on context cancellation and timeout

[sergerad]

Changes

• Check for context completion in retry forever loops (in case anonymous func doesn't stop on context done)
• Check for context completion in anonymous funcs passed into retry forever loops where we want to avoid printing an error log in such cases

Background

When we shutdown, the loops that retry forever treat context cancellation as a retryable error. Instead, these loops should exit.

We also log these as ERRORs when they should not be logged at all.

These are the relevant log lines:

message:2023-07-05T23:40:04.443Z [ERROR] polygon.server.polybft.state-sync-manager.event_tracker: failed to sync: error="context canceled"

and

message:2023-07-05T23:39:49.465Z [ERROR] polygon.server.polybft.state-sync-manager.event_tracker: failed to start blocktracker: error="context canceled"

Local Tests

Running with fix, shutdown looks like:

[SIGNAL] Caught signal: terminated
Gracefully shutting down client...
...

Without fix (v1.0.1):

[SIGNAL] Caught signal: terminated
Gracefully shutting down client...
...
2023-07-06T21:34:05.747+1200 [ERROR] polygon.server.polybft.state-sync-manager.event_tracker: failed to start blocktracker: error="context canceled"
2023-07-06T21:34:05.747+1200 [ERROR] polygon.server.polybft.state-sync-manager.event_tracker: failed to sync: error="context canceled"

[sergerad]

@Stefan-Ethernal could you please review? Fix is not critical but the false ERROR log is a bit annoying for ops. Thanks

[igorcrevar]

I have approved the pr. However, I have also found that currently we have a bug if tt.Sync fails, next time there will be panic because of closing of already closed channel. Please add line (tt.ReadyCh = make(chan struct{})) in code before merge

// Sync concurrently, retrying indefinitely
go common.RetryForever(ctx, time.Second, func(ctx context.Context) error {
        tt.ReadyCh = make(chan struct{}) // re-initialize channel
       
        if err := tt.Sync(ctx); err != nil {
             if common.IsContextDone(err) {