Layer deletion operation can be lost in case of a pageserver crash/restart

[LizardWizzard]

Steps to reproduce

1. Schedule layer deletion
2. Upload index_file.json change
3. Crash before actual deletion is done

Expected result

Layer should be deleted

Actual result

Layer wont be deleted. It is no longer in the index file and there is nothing that detects those. One option to solve it would be to persist delete intention in the index file first, then by looking at the index file after restart deletions can be retried.

Environment

Logs, links

• https://neondb.slack.com/archives/C04PSBP2SAF/p1684918340070659

[LizardWizzard]

The same can happen during upload, if we uploaded the layer successfully but index file wasnt updated. Likely after restart there upload operation will be retried, so in this case there is no actual leakage.

[shanyp]

#4378