[release-1.25] Add jitter to scheduled snapshots and retry harder on conflicts

[brandond]

Proposed Changes

Add jitter to scheduled snapshots and retry harder on conflicts

Also ensure that the snapshot job does not attempt to trigger multiple concurrent runs, as this is not supported.

Types of Changes

bugfix

Verification

See linked issue

Testing

Linked Issues

• [release-1.25] Etcd snapshots failing occasionally on ConfigMap update stage #6779

User-Facing Change

Scheduled etcd snapshots are now offset by a short random delay of up to several seconds. This should prevent multi-server clusters from executing pathological behavior when attempting to simultaneously update the snapshot list ConfigMap. The snapshot controller will also be more persistent in attempting to update the snapshot list.

Further Comments