Publish stable checkpoint periodically to Redis

[haydentherapper]

Currently, a log witness countersigns the latest checkpoint Rekor publishes. Rekor updates its checkpoint on every entry upload, which is extremely frequent. This means that two witnesses are very unlikely to countersign the same checkpoint. While gossiping, it will not be possible to reach a consensus on the same checkpoint, and therefore we can't mitigate split-view attacks.

This change publishes a checkpoint, a "stable checkpoint", every 5 minutes (configurable) to Redis. This runs as a goroutine, with a Redis key derived from the current time rounded to the nearest 5 minutes. We use set-if-not-exist for Redis, meaning you can run replicated instances of Rekor, with all instances writing to the same Redis key.

For a client that wants to gossip, this means waiting 5 minutes before a checkpoint is published that witnesses will countersign (Note that this is an area of active development and research too). The stable checkpoint can be accessed with a query parameter.

Fixes #1067. There is still value in batching in terms of reliablity, but stable checkpoints solve the gossiping issue without a breaking change.

Summary

Release Note

Documentation

[haydentherapper]

@bobcallaway @asraa @jiggoha @mhutchinson @AlCutter Looking for early feedback on the implementation while I'm writing tests

[codecov]

Codecov Report

Merging #1461 (2112494) into main (35f0081) will decrease coverage by 19.31%.
The diff coverage is 63.19%.

@@             Coverage Diff             @@
##             main    #1461       +/-   ##
===========================================
- Coverage   64.33%   45.03%   -19.31%     
===========================================
  Files          80       72        -8     
  Lines        7789     7208      -581     
===========================================
- Hits         5011     3246     -1765     
- Misses       2129     3622     +1493     
+ Partials      649      340      -309     

Flag	Coverage Δ
e2etests	?
unittests	45.03% <63.19%> (+0.37%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/api/api.go	0.00% <0.00%> (-68.24%)	⬇️
pkg/api/metrics.go	0.00% <ø> (-100.00%)	⬇️
pkg/api/tlog.go	0.00% <0.00%> (-39.03%)	⬇️
pkg/witness/publish_checkpoint.go	89.21% <89.21%> (ø)

... and 53 files with indirect coverage changes