Reduce likelihood of data loss when remote endpoint has an outage

[rfratto]

PR Description

This PR increases the WAL truncation frequency default to 60 minutes to reduce the likelihood of samples being moved to a checkpoint when remote_writes are failing.

An extra measure has been added to the truncate loop where a WAL truncate will be skipped if the last remote_write timestamp hasn't changed. This applies after the min_wal_time and max_wal_time checks so data older than max_wal_time should continue to be deleted.

Which issue(s) this PR fixes

Fixes #400 (kind of, it's not perfect, but it's significantly better).

Notes to the Reviewer

PR Checklist

• CHANGELOG updated
• Documentation added
• Tests updated

[pracucci]

Thanks for working on a improvement. I agree it's not a proper fix, but should definitely improve it.

[pracucci]

Some thoughts

Switching the frequency from 1m to 60m it means in agent with low/moderate traffic you will have few segments over the last 60m.

Before this change you were creating a segment every 1m, while after this change a new segment is created once the previous one hit 128MB size or after 60m.

The Storage.Truncate() creates a checkpoint if there are at least 3 segments and the checkpoint will contain the low 1/3 of segments. Let's consider two opposite scenarios:

• High volume (1 segment / minute): every 60m we create a checkpoint. The checkpoint contains the oldest 20m and the WAL segments contain the newest 40m. The longest outage we can tolerate is 40m.
• Low volume (1 segment / hour): every 60m we create a checkpoint. Since we have 1 segment / hour, the checkpoint contains the oldest 1h and the WAL segments contain the newest 2h. The longest outage we can tolerate is 2h.

@rfratto @codesome is my analysis ☝️ correct? I'm wondering if:

1. the agent should reduce the max segment size or make it configurable
2. we should document the longest outage without data loss is two third of wal_truncate_frequency

[rfratto]

I am going to merge this, even though it only increases remote_write outage tolerance to 39 minutes or 80 minutes, depending on how long it's been since the last checkpoint. A more permanent solution is being discussed in a design doc.