Canary tweaks

[slim-bean]

A few follow up fixes after the PR yesterday mainly:

• Adding note to upgrade guide about metric name changes
• Instead of a metric recording the difference between the expected metrics result and actual result, expose two metrics one for actual and one for expected
• Adjusted the locking to be a bit tighter around some methods
• Added code to put a timeout on websocket read, as well as added a ping handler to increment a metric when the canary receives a websocket ping.
• Added a check to make sure we are receiving messages from Loki at a regular interval, if not restart the websocket connection.

[codecov-commenter]

Codecov Report

Merging #2350 into master will increase coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2350      +/-   ##
==========================================
+ Coverage   61.50%   61.52%   +0.02%     
==========================================
  Files         160      160              
  Lines       13536    13534       -2     
==========================================
+ Hits         8325     8327       +2     
+ Misses       4587     4583       -4     
  Partials      624      624              

Impacted Files	Coverage Δ
pkg/canary/comparator/comparator.go	77.73% <100.00%> (+0.64%)	⬆️
pkg/promtail/targets/file/tailer.go	76.13% <0.00%> (ø)
pkg/querier/queryrange/downstreamer.go	97.93% <0.00%> (+2.06%)	⬆️

[owen-d]

very confused on the double lock/unlock instead of the single lock for this function, but I don't see it being a problem either.

[owen-d]

lgtm