Make PublishStorageDeals logic clever

[arajasek]

Some things we can do:

• Wait to submit a PublishStorageDeals message if the base fee "seems" high, and the deals don't start anytime soon
• Do not submit a PublishStorageDeals message if it seems like the cost of the message is (significantly?) greater than the total reward of the deal itself
• Combine multiple deals into one PublishStorageDeals message

[jennijuju]

can this be some config value where miner say: if the base fee is greater than x FIL, stop sending out type A, B, C message?

or the other way around, if the base fee is greater than xFIL, only send out type A,B,C messages?

[flyworker]

I can share something we do this morning for config the price with our miners

if gas fee < 0.1nano : price =0.000000001
if gas fee > 0.1 nano : price = 0.000000001*gas fee/ 0.1nano

those are based on 1GB avg file size

[jennijuju]

karalabe suggested: : I should be able to say I want X profit, and if the basefee skyrockets, then my deal fee should go along with it.

[f8-ptrk]

with the new control addresses in v1.2.3 there might be the possibility to at first give the pubSD messages their own control address - combined with the value in the config file this is already way cleverer than it was before. basically getting the messages out of the way to not block transfers, etc. then add config params for pubSD to fine grain the message values.

but the blocking issue does not disappear with all these funky param-set config values and a special lane control address. a pubSD that gets not on chain will still block a pubSD that could get on chain. and deadlines will expire...

there might be more needed to actually improve (financial, automated) deal success than just stopping the pubSD from causing financial harm due to individual (per message) message params.

karalabe suggested: : I should be able to say I want X profit, and if the basefee skyrockets, then my deal fee should go along with it.

lets assume two pubSD's - n0 and n1. n0 is already maxed out, no more profit - it will idle in the mpool. then n1 comes along having higher margins but expires before n0 and gets stuck in the pool behind n0. so now the most clever way would be to take the financial hit on n0 to get n1 on the chain. this will get absurdly complicated to implement and explain to people :D

problem with all this is: baseFee(now) will be used to calc the values at message creation and then baseFee(msg in pool) might throw all the nice calculations out of the window

[dineshshenoy]

It appears that some teams are already putting multiple deals in the same message. This could achieve significant cost savings especially when the base fee is high and worth doing. cc: @magik6k

[kaptinlin]

See message, it merges 30 deals in one message, the fee goes down a lot. It should be the top priority to do if we want to increase the number of deals currently. Give a wait time, and merge the deals into one in that time period.

[jennijuju]

We can add a configuration for the minimum amount of deals to publish in one message + minimum time duration to wait for new deals to be packed in one message

[f8-ptrk]

@jennijuju let's think about how we can make this smart in addition to working. at least that was the original intend of this issue.

no one is helped if we still have expired deals after this is implemented. so there should be the possibility to set a value to force send a publish message if a certain threshold is crossed:

• force push if (min. start epoch in message - current epoch) < seal duration + x

or even think about filling more than one message at a time. with a value to batch certain deal proposals together:

• time range deals start in.

this would allow to group deals that start, lets say, in a 12h window together while deals that start in the next 12h window will be filled into the next message. combined with the force push we should be able to avoid expiring deals.

especially online deals are not predictable. just grouping them will not solve the issue of being "not clever" :P

[dineshshenoy]

@PatrickDeuse , @kaptinlin - really good input! Do you have data you can share to understand what the impact of this change would be? The seal duration may be different for each miner so it could make calculating x complicated.

One easier change would be to set the time duration to batch deals periodically.

Can you share some information like:
How many deals do you get per day?
Are they spiky or evenly distributed?
Are deals typically started in 48 hours or 72 hours from when the request is received?

I'm trying to get a sense of how long we can wait to batch up deals and the overall cost savings for your operations.

[kaptinlin]

@dineshshenoy 50 deals i get per day. They are evenly distributed currently. The deals typically started in 72 hours, but it may reduce to 48 hours in future.
Thank you.

[f8-ptrk]

@dineshshenoy

currently online deals are rare and mostly rejected - but i could have taken around 250 yesterday, and i would have if the fees weren't that ugly.

so we take around 30-50 offline deals right now every day, we have some regulars and we have some non regulars we deal with offline. we could take up to 100 offline deals (roughly full miner capacity) if it weren't for strategic reasons that we do not.

we do min. 48h to start epoch. but we have clients that give up to 5 days.

me personally wouldn't need a lot of fancy stuff implemented into lotus itself if i get the possibility to outsource the logic. like this:

proposal arrives --> goes through the filtering (potentially outside of lotus already) --> data received,checked, etc --> pipe to outside program, (returning 0 or 1 if deal should be published or not) --> publishdeal routine in provider.go if external returned 0 (https://github.com/filecoin-project/lotus/blob/19d457ae5b1e6583089239852c962acba034a270/markets/storageadapter/provider.go#L73)

then everyone could just build their publishdeals messages outside if lotus if they want.

[f8-ptrk]

there are already messages on the chain that do deal proposal batching into publishdeals messages. i cannot find one right now - but i have seen them. up to 30 deals / message i think. cost reduction was around 60% i think.

[ribasushi]

i could have taken around 250 yesterday, and i would have if the fees weren't that ugly
...
so we take around 30-50 offline deals right now every day,

@PatrickDeuse just to make sure I understand: are you saying that without imrpovements to PublishDeal you spend less on offline deals? If yes - how?

( this is an aside from the batching being sorely needed, I was asked about it in the context of filecoin-discover as well )

[f8-ptrk]

@ribasushi

• the control over what order i import the data and therefore the order the PublishDeals messages get created allow me to lower the baseFee level i need to accept to get them on chain in time to not let deala expire.
• i can quite good control how they are packaged into sectors by accepting certain sizes and import them right (4x8GiB, 2x16GiB, 32GiB) - full sectors save sealing fees, sectors with deals inside expiring at the same time will save storage space later.
• knowing what deal "i" packaged into what sector allows keeping sectors in WaitDeals to get an order in how deals are sealed by manually push them into sealing
• deal duration right now is very important. the invested burnFee / month (precom + com messages) is stupid for 12 month deals (or even less) compared to CC sectors.
• ultimately: online deals fail, a lot. the time invested in just take what comes online and cleaning up after it might even outweigh the time invested to download and import offline deals

don't get me wrong. this is all kind of over the thumb on the fly calculations that go in my head when taking and processing offline deals. there are certain variables in my head:

• the basic facts about our sealing pipeline
• i know i can get a baseFee of 4nFIL or less, quite sure, once every 24h, every 48h for sure
• a 32Gb, 1 year deal at 0.0000002 will pay for the message fess AND the pledge at around 4nFIL
• a 16Gb, 1 year deals at 0.00000002 will pay for 25-35% of the sealing fees at 4nFIL (2 of them in the same sector, 50-... you get the picture)
• and so on

so deals that obviously fall out of the "at least pay some of the sealing fees at 4nFIL" scheme aren't even considered.

if you now take verified deals into the picture, a 3.2GiB, 1.5years, price 0, verified deal is as efficient in the sealing burnFee / power rate as a full CC sector, you can see how it becomes important that one con control what deals to prioritize.

stuff burns too much FIL. quite simple. and control over when and what for to burn it gives you a financial advantage.

[f8-ptrk]

to be honest:

i would prefer to get a pipe to listen to for deal proposals and then have API RPC calls to control how the deals are handled.

• manually building and triggering the pushdeals messages
• manually choosing the sector to package them in

same is true for the scheduler. having "fit all" mechanisms in the lotus is important but the option do do stuff manually will give people the possibility to get a competitive edge tailored to their style of operation.

[dineshshenoy]

Thanks folks - for this ticket, @dirkmc will be adding the ability to batch multiple deals into a single message. The configuration parameter will be a time-constant (i.e. 1hr, 4hr) that can be configured by the miner.

Lots of other good ideas here which need to be carved out into separate tickets and triaged accordingly. I think the batching concept will have significant benefits for reducing base fee costs, especially offline deals.

[f8-ptrk]

@dineshshenoy @dirkmc

if we go with this "fastest" solution - can we get at least a switch to disable the automatic creation of publishdeals messages by the lotus-miner? and if necessary an api call to tell the lotus-miner what message CID a deal is published in?