SDR Pipeline and storage GC

[magik6k]

Work in progress:

• Pipeline cleanup logic for sealed sectors
• Pipeline cleanup logic for failed sectors
  • For now manual button in the UI
• GC marking removable files
• Second mechanism removing those files after approved (as an option also automatically)

[magik6k]

Remove button in the UI:

Pipeline Sector state indicator

[magik6k]

Also adds a cluster-wide storage use indicator and a GC-able file count

[magik6k]

GC webui page

[LexLuthr]

@magik6k Can you do small write on the working of this GC. We will need to document this in detail for users.

[magik6k]

There's a number of cleanup tasks:

• SDRPipelineGC periodically removes SDR-PoRep pipeline entries which have completed the sealing process and had their metadata stored in the long-term sector metadata table (sectors_meta)
  • Separately if a sector has failed sealing, it's entry can be removed in the webui
• StorageGCMark goes through all files, and if a sector file is:
  • Not "pinned" in the storage_gc_pins table (sector pin = don't remove this sector even after expired)
  • Not in the SDR-PoRep pipeline/table
    • "Failed" sectors must be removed from the pipeline table before having their data GCed
  • Not in the miner-actor Precommit table
  • Not in the miner-actor Live or Unproven sector sets
  • That sector will be "marked" in the storage_removal_marks table
• Notably, removal marks need a separate approval. Currently this can only be done through the WebUI, but later we can extend to auto-approval with some policies to select
• After a removal mark was approved, the periodic StorageGCSweep task will look at all approved removal marks, and remove the files which were approved to be removed

[LexLuthr]

@magik6k Am I correct to assume that we don't have a way to clean up failed sector yet even with this PR? I was hoping we could move "Terminate and Remove" and move it to GC along with cleaning up unsealed sectors.

I found the button and function to remove failed sector from pipeline. There is one edge case where we have done precommit but then sectors fails and we remove it from sdr-pipeline from UI. This sector will still not cleaned up because it would be part of precommit check if I am not wrong.

One remaining case would be hooking up "Terminate and Delete" to this process if possible.

[magik6k]

This sector will still not cleaned up because it would be part of precommit check if I am not wrong.

In that case the sector is marked as failed, and, for now will require that you click the button removing the sector from the pipeline (on the pipeline page). Eventually we can automate this, but right now I want the basic, conservative mechanism in.

One remaining case would be hooking up "Terminate and Delete" to this process if possible.

No need, terminated sectors pass all requirements to get marked for GC (not live and not unproven)

[LexLuthr]

This sector will still not cleaned up because it would be part of precommit check if I am not wrong.

In that case the sector is marked as failed, and, for now will require that you click the button removing the sector from the pipeline (on the pipeline page). Eventually we can automate this, but right now I want the basic, conservative mechanism in.

I think you misunderstood my concern. We are already removing this sector from sdr pipeline with the button. But since this sector has done precommit, it will be removed from GC list due to https://github.com/filecoin-project/curio/pull/38/files#diff-9b4dfc8d93411be23d2384e421666249165866d1d76f27f2b8b0494ff6311fecR161-R172

One remaining case would be hooking up "Terminate and Delete" to this process if possible.

No need, terminated sectors pass all requirements to get marked for GC (not live and not unproven)

Should we adjust "Terminate and Delete" to just terminate then?

[magik6k]

Should we adjust "Terminate and Delete" to just terminate then?

Fine to keep the immediate delete there imo

I think you misunderstood my concern. We are already removing this sector from sdr pipeline with the button. But since this sector has done precommit, it will be removed from GC list due to https://github.com/filecoin-project/curio/pull/38/files#diff-9b4dfc8d93411be23d2384e421666249165866d1d76f27f2b8b0494ff6311fecR161-R172

Ah yes, likely want to create a gc mark at the same time