*: Use modified time to detect partial uploads.

[bwplotka]

Problem:

Current flow for detecting partial blocks looks like this:

Partial aborted upload

Compactor	Producer (e.g. sidecar, or previous compactor run)
	Uploads block, fails before meta.json was finished
	Producer is being removed (e.g downscale, cluster shutdown)
	Partial block present on object storage (no / malformed meta.json)
Sync metas, sees partial block
What's the age of ULID? Is it older than PartialUploadThresholdAge? (2d) -> NO
(2d passes)
Sync metas, sees partial block
What's the age of ULID? Is it older than PartialUploadThresholdAge? (2d) -> YES
Block is deleted from object storage.

Our mechanism works generally but there are also cases it does not:

Producer down for 2d

Compactor	Producer (e.g. sidecar, or previous compactor run)
	Block was produced by Prometheus
	Uploads block, fails before meta.json was finished
	Partial block present on object storage (no / malformed meta.json)
	Sidecar is down for some reason / Object storage connection is down
Sync metas, sees partial block
What's the age of ULID? Is it older than PartialUploadThresholdAge? (2d) -> NO
(2d passes)
	All is back again, yay! Let's retry upload!
Sync metas, sees partial block
What's the age of ULID? Is it older than PartialUploadThresholdAge? (2d) -> YES
Block is deleted from object storage.
	Upload of missing meta.json succeeded, yay! Let's remove the local instance of this block.
WTH Block with just meta.json?	WTH Block with just meta.json?

It's super rare and theoretical, it's rare something important is down for 2d. And if it's fine we can increase the PartialUploadThresholdAge. There is
however more plausible scenario:

Producer uploads old block

Compactor	Producer (e.g. sidecar, or previous compactor run)
	Block was produced by Prometheus
	Uploads block, fails before meta.json was finished
	Partial block present on object storage (no / malformed meta.json)
	Sidecar is down for some reason / Object storage connection is down
Sync metas, sees partial block
What's the age of ULID? Is it older than PartialUploadThresholdAge? (2d) -> NO
(2d passes)
	All is back again, yay! Let's retry upload!
Sync metas, sees partial block
What's the age of ULID? Is it older than PartialUploadThresholdAge? (2d) -> YES
Block is deleted from object storage.
	Upload of missing meta.json succeeded, yay! Let's remove the local instance of this block.
WTH Block with just meta.json?	WTH Block with just meta.json?

Let's fix it (:

Ideas:

• Partial block mark JSON
• ModififedTime API call for object storages.

[darshanime]

When retrying upload (after connection/service is restored), how about checking if the ULID still exists in the bucket before attempting to upload the remaining files?
If it does not exist, we can conclude it has been removed by compactor and can attempt full reupload of block.

What do you think?

[stale]

Hello 👋 Looks like there was no activity on this issue for last 30 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for next week, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

@darshanime upload is not atomic, so ID can exists but not the full block.

Anyway, ModififedTime API call for object storages. looks like the way to go.

Thanks to @pracucci we now have this API so we can move to that 👍

[bwplotka]

AC:

• Move to Modified Time for partial delete detection

[bwplotka]

Duplicate of #2585