libp2p mesh testing #719

kumavis · 2016-10-10T19:43:55Z

kumavis · 2016-10-10T19:55:36Z

node.js (non-browser) intensive test suite: https://github.com/gavinmcdermott/js-libp2p-pstn

this is a great start but has been built with a few different design goals (non-browser, short and intensive, fully connected mesh)

metamask mesh tests would need to be:

browser based, running on webrtc
setup for a topology with many nodes (1-3k)
tests more lightweight, intermittent
focus on long-running tests and long-term health of the mesh

kumavis · 2016-10-10T20:06:58Z

cross-team interface/deployment will most likely happen in this way:
the extension runs a hidden iframe pointed at a domain (untrusted by the extension) controlled by the ipfs/libp2p team with some mesh test code that they can update at their will

kumavis · 2016-10-10T20:25:06Z

ping @diasdavid @gavinmcdermott

kumavis · 2016-10-10T20:31:54Z

marking as blocked until I have some concrete action items ( happy to work on some libp2p-side stuff if needed )

kumavis · 2016-10-20T17:30:53Z

This is blocked by this big crypto PR libp2p/js-libp2p-crypto#12

kumavis · 2016-11-07T19:52:57Z

next blockers:

kumavis · 2016-11-16T21:18:55Z

next blockers:

kumavis · 2016-12-05T19:32:12Z

next blockers:

kumavis · 2016-12-15T22:03:17Z

heres a new sketch of the explicit ethereum obj types ( 8 types in total )

[0x90] eth-block 
  ommers: eth-block-list
  txs: eth-tx-trie
  receipts: eth-tx-receipt-trie
  state: eth-state-trie

[0x91] eth-tx (local data only)
[0x92] eth-tx-receipt (local data only)
[0x93] eth-account-snapshot (local data only)

[0x94] eth-block-list (rlp array)
  [leaves]: eth-block

[0x95] eth-tx-trie (merkle trie)
  [leaves]: eth-tx

[0x96] eth-tx-receipt-trie (merkle trie)
  [leaves]: links to eth-tx-receipt

[0x97] eth-state-trie (secure merkle trie)
  [leaves]: links to eth-account-snapshot

[0x98] eth-storage-trie (secure merkle trie)
  [leaves]: links to raw binary

kumavis · 2016-12-16T05:25:05Z

here is an attempt at a concise-yet-hacky set ( 6 types )

[0x90] eth-block  (rlp)
  ommers: eth-rlp:eth-block
  txs: eth-trie:eth-tx
  receipts: eth-trie:eth-tx-receipt
  state: eth-trie:eth-account-snapshot

[0x91] eth-tx  (rlp) (local data only)
[0x92] eth-tx-receipt  (rlp) (local data only)
[0x93] eth-account-snapshot  (rlp)
  storage: eth-trie

[0x94] eth-trie[:type] (merkle trie)
  [leaves]: buffer or <type>

[0x95] eth-rlp[:type] (rlp)
  [leaves]: buffer or <type>

The hack works like this:
[ethBlock cid, 'state/<address>/balance']
would first resolve to
[<ethBlock.stateRoot cid>, '/<hash of address>:eth-account/balance']

since ethereum objects determine their leaf types from context, it doesnt fit neatly into a couple resolvers -- but if we cleverly use remainingPath we can specify the leaf type

kumavis · 2016-12-16T06:12:08Z

here is an ultra-concise ( with only 2 types! ), by maximizing path fanciness

[0x90] eth-rlp[:type]
  [leaves]: buffer or <type>

[0x91] eth-trie[:type]
  [leaves]: buffer or <type>

how to lookup:

eth-block: `<cid for block as eth-rlp>/:block/`
  ommers(index): `<cid for ommersRoot as eth-rlp>/<index>/:block/`
  txs(index): `<cid for transactionsTrie as eth-trie>/<rlp.encode(index)>/:tx/`
  receipts(index): `<cid for receiptTrie as eth-trie>/<rlp.encode(index)>/:txr/`
  state(address): `<cid for stateRoot as eth-trie>/<hash(address)>/:account/`

eth-tx: `<cid for tx as eth-rlp>/:tx/`
  (local data only)

eth-tx-receipt: `<cid for txr as eth-rlp>/:txr/`
  (local data only)

eth-account-snapshot: `<cid for accountSnapshot as eth-rlp>/:account/`
  storage(key): `<cid for storageRoot as eth-trie>/<hash(key)>/`

kumavis · 2016-12-16T06:19:25Z

theres one weird thing but i realize it would be true for the 6-type and 2-type version

resolving <cid for eth-block>/state/<address>
would not be the same as
resolving <cid for eth-block>/state/
then resolving <address>
bc <cid for eth-block>/state/<address> will hash the address after the first step, resolving to <cid for stateRoot as eth-trie>/<hash of address>

kumavis · 2016-12-16T06:19:46Z

@wanderer what do you think of the 2-type solution?

kumavis · 2016-12-16T06:41:09Z

zomg unified ethereum ipld proposal ( 1 type 😎 )

[0x90] eth-star[:type] ( basically rlp but we look for `trie@` prefix and `/:<type>` postfixes
  {leaves}: buffer
  {leaves}/:type: <type>

how to lookup:

eth-block: `<cid for block as eth-star>/:block/`
  ommers(index): `<cid for ommersRoot as eth-star>/<index>/:block/`
  txs(index): `<cid for transactionsTrie as eth-star>/trie@<rlp.encode(index)>/:tx/`
  receipts(index): `<cid for receiptTrie as eth-star>/trie@<rlp.encode(index)>/:txr/`
  state(address): `<cid for stateRoot as eth-star>/<hash(address)>/:account/`

eth-tx: `<cid for tx as eth-star>/:tx/`
  (local data only)

eth-tx-receipt: `<cid for txr as eth-star>/:txr/`
  (local data only)

eth-account-snapshot: `<cid for accountSnapshot as eth-star>/:account/`
  storage(key): `<cid for storageRoot as eth-star>/trie@<hash(key)>/`

note: special chars can be changed to maintain compliance with ipld-query spec. paths are ether pre-defined by the type or specified as hex, so avoiding collision is not a hard problem.

kumavis · 2016-12-16T07:06:21Z

that said, we have room for 16384 codes before we hit the 3 byte range, which is currently reserved for apps
so maybe we dont need to be so concise
so maybe taking up 8 types and having the prettiest paths is reasonable 😸

kumavis · 2016-12-16T07:25:32Z

tho im sure in the future ill be interested in adding this fancy type casting for some other reason, like converting eth contract storage values into cids or other formats

kumavis · 2016-12-16T09:17:17Z

@wanderer btw jbenet and daviddias said dont hack the paths, better to have more formats

Here are the Ethereum objects as I have mapped them so far. While there are only two underlying structures (RLP and Eth-(Secure-)Trie), they interpret the format of the leaf values based on context, which is lost when resolving at the IPLD level. I sketched out some versions that used fewer IPLD formats by employing some cleverness in the path naming conventions. (see here MetaMask/metamask-extension#719 (comment)). The consensus was to be explicit instead of clever, thusly the list of objects as I've proposed. I'd additionally like to reserve 0x99-0x9f for additional Ethereum objects I may have missed or may be introduced when Ethereum milestone Metropolis is released.

kumavis · 2016-12-23T22:19:57Z

Major TODOS:
( moved to top )

kumavis · 2017-01-02T21:27:10Z

who likes todo lists? you do!
( moved to top )

kumavis · 2017-01-02T22:07:24Z

TIL "identity hash fn" multiformats/multihash#13 (comment)

kumavis · 2017-01-13T19:21:28Z

consider using rlpx over libp2p transport

kumavis · 2017-01-13T22:15:15Z

proposal: blocks via ipfs-http-api on parity openethereum/parity-ethereum#4172
@wanderer @cdetrio this would be a nice way to skip syncing all together:

run parity client
wrap parity ipfs-http-api as block service
run public ipfs gateway on top of parity-backed block service

kumavis · 2017-02-16T05:30:44Z

identifying RPC method data/index requirements

clientConfig
  eth_protocolVersion
  eth_syncing
  eth_coinbase
  eth_mining
  eth_hashrate
  eth_gasPrice
  eth_accounts
  eth_sign
  eth_sendTransaction
  eth_sendRawTransaction
  eth_newFilter
  eth_newBlockFilter
  eth_newPendingTransactionFilter
  eth_uninstallFilter
  eth_getFilterChanges
  eth_getFilterLogs

ipfs
  eth_getBlockByHash
  eth_getUncleByBlockHashAndIndex

ipfs:selectors (selectors/transforms)
  eth_getBlockTransactionCountByHash
  eth_getUncleCountByBlockHash

index:txToBlock
  eth_getTransactionReceipt
  eth_getTransactionByHash
  eth_getTransactionByBlockHashAndIndex

blockTracker (head tracking)
  eth_blockNumber

index:CHT
  eth_call
  eth_estimateGas
  eth_getBalance
  eth_getStorageAt
  eth_getTransactionCount
  eth_getCode
  eth_getBlockByNumber
  eth_getTransactionByBlockNumberAndIndex (+index:txToBlock)
  eth_getBlockTransactionCountByNumber (+ipfs:selectors)
  eth_getUncleCountByBlockNumber (+ipfs:selectors)
  eth_getUncleByBlockNumberAndIndex
  eth_getLogs (+index:logToTx)


HEX String - an integer block number
String "earliest" for the earliest/genesis block
String "latest" - for the latest mined block
String "pending" - for the pending state/transactions

transaction
├─ blockHash { txToBlock }
├─ blockNumber { txToBlock }
├─ creates { txToBlock, compute, ... }
├─ from { ecrecover }
├─ gas {}
├─ gasPrice {}
├─ hash {}
├─ input {}
├─ nonce {}
├─ publicKey { ecrecover }
├─ r {}
├─ raw {}
├─ s {}
├─ to {}
├─ transactionIndex { txToBlock, blockToTxs }
├─ v {}
└─ value {}

kumavis · 2017-02-16T05:43:56Z

for those playing along at home, i made a thing: https://kumavis.github.io/metamask-mesh

kumavis · 2017-02-28T21:54:10Z

maybe relevant: https://github.com/bitcoin/bips/blob/master/bip-0122.mediawiki

kumavis · 2017-03-22T00:44:31Z

migrated js-ipld-eth-* modules to ipld org

kumavis · 2017-03-22T00:46:29Z

(a few weeks back) friends at Parity added an http server that serves parity data from CIDs. it got merged into master : )

kumavis · 2017-05-06T17:10:19Z

tracking parity integration here ipfs/js-ipfs#763

kumavis · 2017-05-09T21:38:40Z

ipfs routing improvements ipfs/js-ipfs#833

kumavis · 2017-05-09T21:39:44Z

here is the current custom rolled parity->ipfs server https://github.com/kumavis/ipfs-eth-bridge
it works if you can connect to it

kumavis · 2017-07-03T16:04:34Z

currently eth-ipfs bridge only serves blocks, transactions (no tries), state and storage tries, no tx receipts or no tx receipt tries. These are missing b/c parity does not index them by hash (tx receipt) or doesn't store them at all (tx trie, tx receipt tree).

the bridge is available over http as https://ipfs.lab.metamask.io

kumavis · 2017-07-04T04:06:14Z

what if: generic ipfs merkle tree heat map could show us what is happening on ethereum

kumavis · 2017-07-04T04:07:41Z

we will need to: create a special index for { tx tries, tx rxs, tx rx tries }

ghost · 2017-07-04T04:21:31Z

what if: (in the long run): a box that users can just "download and run". Will be like this https://github.com/kumavis/ipfs-eth-bridge

With the idea "Contribute to the light clients mesh, making it stronger and resilient by downloading this application".

kumavis · 2017-07-04T04:25:18Z

merkelize and serve the Canonical Hash Table over ipfs

ghost · 2017-07-04T04:36:53Z

serverless block explorer

To make it more sexy, people gets it from IPNS.

Pros: We get more people pulling IPLD links, making them available faster.

ghost · 2017-07-04T04:54:30Z

This thread is the sequel to this effort MetaMask/eth-ipfs-browser-client#1

kumavis · 2017-07-04T05:10:13Z

closing this thread in favor of the new repo

kumavis added this to the Public Release milestone Oct 10, 2016

kumavis self-assigned this Oct 10, 2016

kumavis added ready blocked labels Oct 10, 2016

danfinlay modified the milestones: Light Client Support, Public Release Oct 11, 2016

kumavis removed the ready label Nov 7, 2016

kumavis mentioned this issue Dec 23, 2016

Add eth-objects multiformats/multicodec#27

Merged

kumavis closed this as completed Jul 4, 2017

snyk-bot mentioned this issue Jan 21, 2020

[Snyk] Fix for 1 vulnerabilities villardez28/metamask-extension#7

Open

libp2p mesh testing #719

libp2p mesh testing #719

Comments

kumavis commented Oct 10, 2016 • edited Loading

Progress Tracker:

kumavis commented Oct 10, 2016 • edited Loading

kumavis commented Oct 10, 2016

kumavis commented Oct 10, 2016 • edited Loading

kumavis commented Oct 10, 2016

kumavis commented Oct 20, 2016

kumavis commented Nov 7, 2016 • edited Loading

kumavis commented Nov 16, 2016 • edited Loading

kumavis commented Dec 5, 2016 • edited Loading

kumavis commented Dec 15, 2016 • edited Loading

kumavis commented Dec 16, 2016 • edited Loading

kumavis commented Dec 16, 2016 • edited Loading

kumavis commented Dec 16, 2016

kumavis commented Dec 16, 2016

kumavis commented Dec 16, 2016 • edited Loading

kumavis commented Dec 16, 2016 • edited Loading

kumavis commented Dec 16, 2016

kumavis commented Dec 16, 2016

kumavis commented Dec 23, 2016 • edited Loading

kumavis commented Jan 2, 2017 • edited Loading

kumavis commented Jan 2, 2017

kumavis commented Jan 13, 2017

kumavis commented Jan 13, 2017 • edited Loading

kumavis commented Feb 16, 2017 • edited Loading

kumavis commented Feb 16, 2017 • edited Loading

kumavis commented Feb 28, 2017

kumavis commented Mar 22, 2017

kumavis commented Mar 22, 2017

kumavis commented May 6, 2017

kumavis commented May 9, 2017

kumavis commented May 9, 2017

kumavis commented Jul 3, 2017

kumavis commented Jul 4, 2017

kumavis commented Jul 4, 2017

ghost commented Jul 4, 2017

kumavis commented Jul 4, 2017

ghost commented Jul 4, 2017

ghost commented Jul 4, 2017

kumavis commented Jul 4, 2017

kumavis commented Oct 10, 2016 •

edited

Loading

kumavis commented Oct 10, 2016 •

edited

Loading

kumavis commented Oct 10, 2016 •

edited

Loading

kumavis commented Nov 7, 2016 •

edited

Loading

kumavis commented Nov 16, 2016 •

edited

Loading

kumavis commented Dec 5, 2016 •

edited

Loading

kumavis commented Dec 15, 2016 •

edited

Loading

kumavis commented Dec 16, 2016 •

edited

Loading

kumavis commented Dec 16, 2016 •

edited

Loading

kumavis commented Dec 16, 2016 •

edited

Loading

kumavis commented Dec 16, 2016 •

edited

Loading

kumavis commented Dec 23, 2016 •

edited

Loading

kumavis commented Jan 2, 2017 •

edited

Loading

kumavis commented Jan 13, 2017 •

edited

Loading

kumavis commented Feb 16, 2017 •

edited

Loading

kumavis commented Feb 16, 2017 •

edited

Loading