Automatically chunk files bigger than sector size

[magik6k]

Requiring users to do than manually is bad UX, and this should be reasonably simple

[whyrusleeping]

Yeah, we should have a whole thing around users adding their files. Encryption and splitting your data redundantly across multiple miners should be thought about too.

[laser]

@magik6k @whyrusleeping -

Requiring users to do than manually is bad UX, and this should be reasonably simple

We currently have a command which allows a user to import a file and get a single CID.

We also have a command which allows a user to propose a storage deal to a miner, providing a single CID.

What about adding a --split-boundary flag to the client import command?

10:06 $ ./lotus client import ~/.lotus/flarg
bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku

# assumes catpix.zip is 1028MiB
lotus client import --split-boundary=1GiB ~/.lotus/catpix.zip
(1of2) bafkreihdwdcefgh4dqkjv67uzcmw7ojee6detojuzjevtenxquvykuxedz
(2of2) bafkreih5ffbd63eh5s7wcy253h5pgw6cfjpk6vvn7tdkigsdfsk3bwiiza

We'd need to make sure that the file names (post-split) made sense to the user:

10:09 $ ./lotus client local
bafkreihdwdcefgh4dqkjv67uzcmw7ojee6xedzdetojuzjevtenxquvyku .lotus/catpix.zip_1of2 1GiB ok
bafkreih5ffbd63eh5s7wcy253h5pgw6cfjpk6vvn7tdkigsdfsk3bwiiza .lotus/catpix.zip_2of2 4MiB ok

If you have further ideas for how this should work, please post some example commands here. That would help me to understand what you had in mind.

[laser]

@arajasek - pinging you for some assistance rounding up some expected CLI interactions here :)

[jennijuju]

Should be able to define the sector size - 32 | 64 for mainnet

[frrist]

Picking up this issue for my Tribute week, I gather that the UX described above is sufficient.

One thing I'll note: lotus client import accepts a path to a CAR file or a "normal file". If a CAR file is provided, no attempt to chunk it will be made. However, "normal files" will be chunked (based on a flag from the user), and a list of CIDs will be returned as described in the above comments.

[laser]

@frrist - sounds good to me

[frrist]

Thinking about this more carefully, and after exploring the related bits in the lotus codebase, I think this issue needs more detail -- and therefore is a poor fit for Tribute work. The time to spec and implement this likely spans weeks. Details around the following should be discussed before anyone decides to pick this up:

• A more thought-full design on how large content should be split up/added to the network. Simply splitting data along some boundary will likely be insufficient and possibly lead to worse UX. leading to ->
• How will data (deals) span multiple sectors, and how should it be retrieved once split? I think this format needs to be agreed upon before work here can begin. e.g. what is the "root" CID of data that is split across many sectors, and how does a user go about retrieving it. Furthermore, if the data is split across multiple miners how does that request look?

@jennijuju I think the tribute tag should be removed from this and its effort rating adjusted to weeks.

[jennijuju]

@frrist thank you for looking into this. Agreeing "how large size data should be stored in the network" requires a dedicated project. That being said, can we keep the scope of this ticket small and just to be lotus client import will chuck the files to pieces that can be fit in one sector and returns a series of cids? then client can decide how they wanna make deals with those cids.

[TippyFlitsUK]

Hi 👋

The Legacy Lotus Markets sub-system reached EOL at the end of the 31st January 2023.

This ticket is being marked as won't fix and closed as the Lotus Team will no longer be making any further fixes or enhancements to the legacy markets subsystem.

Please feel free to re-open this ticket in the new Boost markets sub-system repository at https://github.com/filecoin-project/boost if you feel that it is still relevant.

Many thanks 🙏