Making ingester head compaction jitter Zone aware.

[alanprot]

What this PR does:
#5919 We introduced a jitter on the head compaction. This PR is proposing to change that jitter to take in consideration the AZ of the ingester.

Lets say we configure to have a head compaction interval set at 15 minutes across 3 Availability Zones (AZs). Prior to this change, the ingesters would determine a random jitter and perform the head compaction at any minute within each 15-minute interval. This change ensure that within every 15-minute interval, Ingesters in AZ1 execute head compaction from minute 0 to 5, those in AZ2 from minute 5 to 10, and those in AZ3 from minute 10 to 15.

I added a log on every tick to show the time of the tick and the AZ of the ingester on a 15 minutes period:

aws.az	tick_time	count(*)
us-east-1d	2024-05-06 18:59	1
us-east-1d	2024-05-06 18:58	9
us-east-1d	2024-05-06 18:57	7
us-east-1d	2024-05-06 18:56	11
us-east-1d	2024-05-06 18:55	6
us-east-1b	2024-05-06 18:54	11
us-east-1b	2024-05-06 18:53	5
us-east-1b	2024-05-06 18:52	7
us-east-1b	2024-05-06 18:51	8
us-east-1b	2024-05-06 18:50	11
us-east-1a	2024-05-06 18:49	5
us-east-1a	2024-05-06 18:48	10
us-east-1a	2024-05-06 18:47	8
us-east-1a	2024-05-06 18:46	8
us-east-1a	2024-05-06 18:45	8

---

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• [NA] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[alanprot]

@yeya24 Can you take a look? :D

I will update the changelog after if we go with it.

[yeya24]

I like this change. It looks great and I think it makes sense. Just some nits.

[CharlieTLe]

These tests seem to fail when it's experiencing cpu starvation from other tests running concurrently because of the usage of a wall clock and the timing could be off depending on how busy the CPU is. Could we use a mock clock or increase the duration in the tests to decrease the chance of race conditions?