SectorBuilder requires disk space on /tmp partition

[travisperson]

The SectorBuilder currently requires additional diskspace on the partition that /tmp is located on.

The amount of disk space requires is approximately 22x $SECTOR_SIZE for each sealing thread. The sector builder is currently configured with 2 worker threads (so total of 44x $SECTOR_SIZE).

For the alpha2 network (0.5.6) all miners are sealing 256MiB sectors, the total free disk space required on the /tmp partition is around 11GiB for each go-filecoin miner running on the host.

thread '<unnamed>' panicked at 'failed to write file: Os { code: 28, kind: Other, message: "No space left on device" }', src/libcore/result.rs:1084:5

A proposed workaround is to define the TMPDIR environment variable to a location with more diskspace for the go-filecoin daemon process.

Slack discussion: https://filecoinproject.slack.com/archives/CEGB67XJ8/p1568837802058400

[anorth]

Proofs people: could you please define how much disk space is required, in addition to the obvious space for staging and sealed sectors? @laser @schomatis

[schomatis]

I'm not sure how the full process operates, but normally replication takes 20 sector sizes (one MT per layer, each taking up 2 sectors).

[schomatis]

A proposed workaround is to define the TMPDIR environment variable to a location with more diskspace for the go-filecoin daemon process.

We can also add this as a configuration option in rust-fil-proofs if it works for you (to avoid adding yet another environment variable).

[laser]

@dignifiedquire (cc @anorth)

Proofs people: could you please define how much disk space is required, in addition to the obvious space for staging and sealed sectors?

This thread surprised me. It is the case that there's yet another place in rust-fil-proofs that writes to disk - but doesn't expose the choice of location to library consumers - which I didn't know about.

The environment variable-based approach is a hack, IMHO, and asking Filecoin operators to edit some TOML file (unrelated to the Filecoin node config they manipulate through the CLI) is scarcely better. Both of these approaches require that the node operator too much about the rust-fil-proofs system (3 Git repositories away from the Filecoin node) and provide, generally, a confusing/bad UX.

The existing approach of writing to /tmp is not working. I propose that rust-fil-proofs STOP writing to disk (e.g. using the tempfile crate) unless we expose the choice of write-location to rust-fil-proofs consumers (and the Filecoin node). The sector builder does this by accepting a metadata directory-path (and sealed and staged sector-paths) when it is instantiated. The rust-fil-proofs seal and unseal operations do this (by accepting an outfile path). It is not challenging to do this. It will require some coordination and API changes.

[anorth]

@laser do you know if this is now resolved?

[laser]

@anorth

@laser do you know if this is now resolved?

Yes, this has been resolved.