add_block_number gives exception and fails when 'NoneType' is encountered while calling the RPCs

[anurag-arjun]

First of all, thanks for the great tool! It is nice.

I have setup the tool alright, but am getting the following errors:

• It is giving an exception when it finds a None value returned from the RPC calls to Ethereum.

I am running a Parity node listening to Kovan chain.

Running it with this command:

It is extracting data properly before it fails midway..

[ankitchiplunkar]

Can you try rerunning the code python ether_sql.py scrape_data?
If it works then there was an issue in error handling part.
If not then I will try to recreate the error on my machine.

[anurag-arjun]

Hey - yes I did. It still gives similar errors. I tried with Infura as well - no luck.

[ankitchiplunkar]

Hmm,

I think there are two possible reasons that can cause this issue:

1. The local node does not have blocks after block 691. This is highly unlikely since the HTTP response gave code 200 as response.

2. There is some issue when connecting to Infura which gives bad status error Infura node gives badstatus error after some blocks #7.

To solve this issue we are switching to using https://github.com/ethereum/web3.py instead of https://github.com/ConsenSys/ethjsonrpc to connect to a node.

[anurag-arjun]

Hey - I changed the code to use web3.py instead of ethjsonrpc, and also further put the block fetch events in a Redis queue instead of a loop, but it still fails after sustained use. I think you run an analytics web app on top this - are you able to download the entire chain to the DB?

[ankitchiplunkar]

Hey - I changed the code to use web3.py instead of ethjsonrpc, and also further put the block fetch events in a Redis queue instead of a loop

Awesome can you create a pull request.

but it still fails after sustained use

Is the error same as above or some other error?

I think you run an analytics web app on top this

Not, yet but we intend to. The logic to push a new block when it is downloaded by the node is not integrated by the library.

are you able to download the entire chain to the DB?

I synced till 1.5M mainnet blocks on using a local Parity node (stopped because I wanted to download other tables) and ~20k blocks using Infura node (Infura node stops due to this #7), did not encounter Nonetype error.
I will now run a local parity node and try to scrape the Kovan testnet and update the results here.

[anurag-arjun]

Awesome can you create a pull request.

That won't make sense because it still doesnt work. Additionally, the web3.py library is missing functions for Uncles, and I think, Traces too.

To solve this issue we are switching to using https://github.com/ethereum/web3.py instead of https://github.com/ConsenSys/ethjsonrpc to connect to a node.

This will not make a difference. As far as I can understand, using the JSON-RPC interface directly for bulk calls is the problem. See blockchain-etl/ethereum-etl#2 (comment). The author is using the local node IPC via socket handling for reads - https://github.com/medvedev1088/ethereum-etl/blob/master/ethereumetl/ipc.py

I tried playing around with https://github.com/medvedev1088/ethereum-etl and was able to get it working to dump Block data into a PSQL DB without a problem. Will run the script for a larger block range today. Will keep you updated.

[ankitchiplunkar]

Additionally, the web3.py library is missing functions for Uncles, and I think, Traces too.

I have recently created pull requests to add Uncles ethereum/web3.py#857 and Traces ethereum/web3.py#850 in the web3.py. The Uncles pull request has been merged but the Traces one will take some time. I have started using web3.py in the branch https://github.com/analyseether/ether_sql/tree/develop/web3py to scrape the data, will merge this branch in master in a couple of days.

As far as I can understand, using the JSON-RPC interface directly for bulk calls is the problem.

Hmm, I didn't know that. I wanted to switch to web3.py because IPC's are faster and it is in active development. The code in branch https://github.com/analyseether/ether_sql/tree/develop/web3py uses IPC's to connect to the node.

Will run the script for a larger block range today. Will keep you updated.

Awesome!

Update on recreating errors on my end:

I pushed almost 3 million kovan testnet blocks in a psql yesterday, without an issue using the code in master branch master, did not encounter the NoneType error.