Exploring the Miner's Index and Market

[jsign]

Hey all! 👋🏼

As many of us know, miner selection in Filecoin is a challenging task. Having full visibility on what miners are offering and how reliable they might be is hard but very important to successfully use the Filecoin network.

Here at Textile, we're trying to approach this problem by creating a Miner's Index, which can aggregate information from different sources and develop useful distilled information to select miners. As we recognize this is a challenging problem, we won't come up with a perfect solution upfront, and also, we need your help to make this walk in the right direction.

For every miner, we'll be starting with the following information:

• On-chain information:
  • Sector-size.
  • Relative Power in the network.
• Off-chain information:
  • Storage price.
  • Verified storage price.
  • Min piece size.
  • Max piece size.
• Hosted Powergates telemetry:
  • Per region:
    • Total successful deals
    • Last successful deal
    • Total failed deals
    • Last failed deals
    • Last XX data-transfer times in deal-making (MiB/s)
    • Last XX sealing times in deal-making (Duration).

That seems like a good start! We know there're possibly more data that can be added, but we should remember that the idea is to come, which is useful for making a decision.

It's fair to say that this data should be interpreted cautiously since transfer-times, sealing times, and failures don't entirely depend on miners but clients, network conditions, temporary load, etc. That's why this problem is hard!.

Miner provided information

We should remember that miners are real persons or teams, so including their own metadata looks very interesting since closing the gap between clients and miners can open up many possibilities.

We're exploring letting miners fill in the following information:

• Location
• URL
• Email
• Twitter
• Interested in Curated Data
• Self-reported load or signaling how are you feeling now about receiving deals now.

If you're a miner reading this discussion, please feel free to propose what do you think would be useful from your's perspective. We see a big opportunity to have feedback and insights from miners as sources of information to solve this problem.

Miner Market

Letting the mind wander... we also believe that making deals in the network might need more interaction to succeed. We're considering extending the Miner Index to be a place where clients and miners can meet each other and coordinate future deals.

Allowing coordination between client and miners can mitigate many things that can go wrong and do better deals. If some client is planning to store a big dataset, it would really like to know that the data transfer won't fail, and most probably, the miner wants to be prepared upfront to have available resources. Moreover, depending on the plan, it could make sense to do price negotiation (different from the publicly available) and establish some long-term relationship. 🤝

Although the Filecoin network is trustless and verifiable, it's useful to build a social reputation score for existing miners. Filecoin users could mark miners as favorites, and as a community, we can discover which miners appear to be more reliable. We should be creative and think of more similar ideas! Big problems need the help of the community; we can't solve this alone!

Curated data and verified-clients

An example of why it would be useful to get clients and miners closer is facilitating storing curated or open datasets. These clients will become verified-clients by notaries and can leverage better prices from miners in the network. At the same time, miners in the network are interested in storing this data since storing it will account for 10x more power than an unverified deal.

This is a great example of two markets that need a place to meet each other and start talking. The Miner Market can be that place!

Ending thoughts

How do you feel about building the Miner Market? Do you think it is worth exploring? How about auctions to store data!? If we see some traction around this idea, we might consider building it!

We opened this Github Discussion because we want to hear from you, so don't be shy and share your thoughts! This is an exploratory phase, and we want to know more about your ideas to build something meaningful and useful. 🙌

[andrewxhill]

Oracle use-cases

I'm curious if any projects would make use of this if there was an oracle on ethereum (or another) with slices of the data?

[andrewxhill]

JSON API users

Most of our APIs are gRPC. The Miner Index // Marketplace APIs might be a great building block for projects that aren't using any of our other APIs. Perhaps a simple REST API with JSON is valuable to these projects, I'm curious if there are any out there to drive that need.

[rjan90]

I think this is definitely a market worth exploring, even though there are others exploring similar ideas (https://www.filswan.com/, https://preview.filhub.com/, https://storage.codefi.network/).

As a miner, signaling that you can take both offline & online deals, and maybe how many deals/day could be useful information for a potential client.

And as you have stated, the biggest challenge is probably making a "reputation score" of a miner, since a successful deal is dependent on a lot of factors. e.g a client trying to store 256 MB file with a miner, when the miner has a min size > 512MiB, will result in a failed deal.

[flyworker]

Filswan get some API listed for miner related information.

https://documenter.getpostman.com/view/13140808/TWDZJbzV

GET

List miners
https://api.filswan.com/miners?limit={{limit}}&offset={{offset}}&location={{location}}&offline_deal_available={{offline_deal_available}}&status={{status}}&sort_by={{sort_by}}&order={{order}}

This endpoint returns details about miners, default 10.

and

GET
Single miner
https://api.filswan.com/miners/{{miner_id}}

This endpoint returns details about the miner specified by the requested ID.

Maybe worth to take a look.

[andrewxhill]

Thanks! We've got a first version wired up... the main idea was to augment with some signals that aren't on chain. I think it'll pair well with what you've got above. More to come soon.