Record validation issue with multiple naming systems

[mildred]

I have been looking at record validation and saw that a record in the DHT had to be valid on the peer that stores it. If we want to allow different naming schemes, this is going to be a problem because a naming scheme has specific validation routines.

So, either we only define in a central place a list of all naming schemes with their validation functions that all peers must implement, or we change how we do validation or how we store our records.

Assuming we want to be flexible about the validation functions, we have two choices:

• we stop verifying records, which can lead to abuse
• or we manage somehow to only store and retrieve records of a defined type on peers that understand this record type

If the DHT implementation is somewhere near what the Kadmelia DHT does, the second option should be possible to implement, by essentially creating sub-networks for each supported validation scheme a peer has.

[mildred]

Let's change how a peer id works and give more than one id per peer. A peer would have one id per validation scheme it supports. The peer id would contain the id of the validation scheme associated with it such as two peers ids with the same validation scheme are considered closest than two id belonging to the same node.

This would have the effect of partitioning the hash table into slice of peers that understand the same schemes.

When looking for peers that supports a specific validation scheme, the id of the record should also include the validation scheme. GetClosestPeers would then find peers that share the same validation scheme as those are the closest to the data requiring the validation scheme.

Note that the validation scheme id used in the peer ids can match more than one validation scheme. For instance, you could have id=1 referring to the schemes defined by the IPFS organization at some point in time. id=2 could mean a second revision of the validation schemes and id=3 could implement a different naming system. Note that schemes ids must not necessarily be allocated this way. It could be a hash of the list of supported scheme names.

[mildred]

Another way would be to implement records completely differently with an extra level of indirection.

A peer providing a record, instead of putting the record value in the DHT, ii would tell the DHT that it provides the given record. Basically, it replaces a dht.PutValue(record_key, record_value) with dht.Provide(record_key).

A peer that wants to resolve the record would then ask the DHT where it can be found instead of asking its value directly. It replaces dht.GetValue(record_key) with dht.FindProviders(record_key). An extra step would be necessary to ask the providers the value of the record directly.

To implement that, the DHT object must expose a new method:

dht.GetValueFromPeer(peer, record_key) (which is very similar to the existing getValueOrPeers())

Also, it would be nice that a peer that notices that the record of another peer is out of date could notify it that it should update the record. It could be possible by exposing:

dht.PutValueToPeer(peer, record_key, record_value) (which exists but is private)

edit:

• libp2p implementation here: https://github.com/mildred/go-libp2p-kad-dht/tree/ipfs-objects
• application level implementation here: https://github.com/mildred/ipfs-objects/

[Kubuxu]

Assigning myself so I read it.

@whyrusleeping pinging you so it is higher on your list.

[Stebalien]

This is going to need a lot of design/spec work. There has been some recent work, but keeping this open isn't really going to help.