Missing entries in eth_getLogs response

[lastperson]

I'm running:
Parity version: 1.6.10
Operating system: Linux
And installed: from source
Starting params: --jsonrpc-interface all --jsonrpc-hosts=“all” --pruning archive --max-peers 100 --base-path /var/parity --geth --force-direct -l own_tx=trace --mode active --no-download --tx-queue-size 65536 --tx-queue-gas off --force-sealing --reseal-on-txs all --reseal-min-period 1000 --work-queue-size 1 --remove-solved --no-warp --tracing off --db-compaction ssd

We have 2 parity nodes, built and ran in the same way in archive mode. After initial sync is done, they start to miss events on the eth_getLogs requests(all the blocks prior to end of initial sync seem to be consistent). Interesting thing is that they are missing different logs. Previously we experienced the same behavior on 1.5.12.
Here is the query to receive events count from 0x60bf91ac87fee5a78c28f7b67701fbcfa79c18ec contract on block span 4309200-4310200.

curl 'http://localhost:8545' -H 'Content-Type: application/json' -H 'Accept: application/json' --data-binary '{"id":1,"jsonrpc":"2.0","params":[{"fromBlock":"0x41c0d0","toBlock":"0x41c4b8","address":"0x60bf91ac87fee5a78c28f7b67701fbcfa79c18ec"}],"method":"eth_getLogs"}' --compressed | grep -o logIndex | wc -l

On our nodes we receive 926 and 934 entries.
Etherscan API returns: 937 for the same request: https://api.etherscan.io/api?module=logs&action=getLogs&fromBlock=4309200&toBlock=4310200&address=0x60bf91ac87fee5a78c28f7b67701fbcfa79c18ec
If anyone has nodes in archive mode, I'd like to ask to run the same query and report the results.

Unfortunately this issue makes the filters feature in parity nodes absolutely unreliable for live environments.

[jo-tud]

I get 937 entries with Parity/v1.6.0-unstable-32f64e9-20170914/x86_64-linux-gnu/rustc1.14.0

[slothbag]

I'm getting some wierd responses from the latest Parity 1.7.2, previous version was working fine.. now half the event filters are returning empty or missing large numbers of entries.

[slothbag]

I just spent a few days getting my geth node back to a full sync and can confirm that geth is returning all logs correctly. Parity 1.7.2 appears to have an issue. I get 319 logs/events in geth, and 2 in Parity for my particular contract.

Edit: Still broken in 1.7.3

[slothbag]

Parity is pretty much unusable for live dapps because of this.. any chance we can bump the priority :)

[tjayrush]

I'm running Parity/v1.8.2-beta-1b6588c-20171025/x86_64-macos/rustc1.21.0 with --tracing on --pruning archive. I get 937 for the above curl request.

[slothbag]

Can you try this web3 code... Geth returns 237 logs, Parity returns zero.

var filter = web3.eth.filter({fromBlock: 0, toBlock: 'latest', address: "0x99d439455991f7f4885f20c634c9a31918d366e5", topics:["0x1cef2b4ec7f129e441c961eb4fae22fd3131868db6be990c6f065f35b6b53cf3"]});
    filter.get(function(err, result) {
        if (err || result.length == 0)
            throw 'No transfers found'
        console.log(result.length)
    })

Which probably translates to something like

curl 'http://127.0.0.1:8545' -H 'Content-Type: application/json' -H 'Accept: application/json' --data-binary '{"id":1,"jsonrpc":"2.0","params":[{"fromBlock":"0x1","toBlock":"latest","address":"0x99d439455991f7f4885f20c634c9a31918d366e5"}],"method":"eth_getLogs"}'

[slothbag]

could be --pruning archive, perhaps my node does not have the block history required

[tjayrush]

I added

X | jsonlint | grep 0x1cef2b4ec7f129e441c961eb4fae22fd3131868db6be990c6f065f35b6b53cf3 | wc

to X (if X is your curl URL). I got 234 topics with that value. It came back pretty quickly too--about 2 seconds.

[tjayrush]

Request the receipt for this transaction 0x3c587cbaf21529ac923880128dfff7213056a68c1ee926e83db95a26cfd870ea. It's the first transaction returned by the above CURL code. It's from block 0x3e90df. Here's the full returned receipt:

    {
      "address": "0x99d439455991f7f4885f20c634c9a31918d366e5",
      "blockHash": "0x58e81786372bf11fda923e38aa03926b5a46f44fa122fc060b64cd6e012945ed",
      "blockNumber": "0x3e90df",
      "data": "0x0000000000000000000000000000000000000000000000a2a15d09519be00000",
      "logIndex": "0x0",
      "topics": [
        "0x1cef2b4ec7f129e441c961eb4fae22fd3131868db6be990c6f065f35b6b53cf3",
        "0x00000000000000000000000083bad39f067ef72e354f2ea9d318eaad381a7f15"
      ],
      "transactionHash": "0x3c587cbaf21529ac923880128dfff7213056a68c1ee926e83db95a26cfd870ea",
      "transactionIndex": "0x5",
      "transactionLogIndex": "0x0",
      "type": "mined"
    },

[slothbag]

Thanks @tjayrush , i'm syncing with prunning archive now.. hopefully that gives me the logs

[tomusdrw]

@slothbag You don't really need --pruning archive for logs, just make sure you run with --no-warp

[bohendo]

@tomusdrw I've synced with the --no-warp option and I'm still missing some entries from eth_getLogs.

See below for two contradictory JSON RPC calls. The first gets a transaction receipt that lists 4 logs from some address at some block. The second gets logs from this address at this block and returns an empty array where at least 4 logs are expected.

{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0x74fd5d1312955003a23ac5d02f5dfb3627f22e7f0cb64c5ccc1a00faf80bcffb"],"id":1}
{"jsonrpc":"2.0","result":{"blockHash":"0xc63388cb3221a7351ce296c20262769caeb9a2e8e8cb0eff82ca480fd00131d4","blockNumber":"0x4b8a6d","contractAddress":null,"cumulativeGasUsed":"0x5be603","gasUsed":"0x4c277","logs":[{"address":"0x06012c8cf97bead5deae237070f9587f8e7a266d","blockHash":"0xc63388cb3221a7351ce296c20262769caeb9a2e8e8cb0eff82ca480fd00131d4","blockNumber":"0x4b8a6d","data":"0x0000000000000000000000007804d036274eb04352868775ca649b65cc2a48f700000000000000000000000000000000000000000000000000000000000743fc000000000000000000000000000000000000000000000000000000000006f291000000000000000000000000000000000000000000000000000000000007134200005a10b310a54982603c0f008a7010834354c30c8a08c25118e373d0a0b02b","logIndex":"0x3c","topics":["0x0a5311bd2a6608f08a180df2ee7c5946819a649b204b554bb8e39825b2c50ad5"],"transactionHash":"0x74fd5d1312955003a23ac5d02f5dfb3627f22e7f0cb64c5ccc1a00faf80bcffb","transactionIndex":"0xa8","transactionLogIndex":"0x0","type":"mined"},{"address":"0x06012c8cf97bead5deae237070f9587f8e7a266d","blockHash":"0xc63388cb3221a7351ce296c20262769caeb9a2e8e8cb0eff82ca480fd00131d4","blockNumber":"0x4b8a6d","data":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000007804d036274eb04352868775ca649b65cc2a48f700000000000000000000000000000000000000000000000000000000000743fc","logIndex":"0x3d","topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],"transactionHash":"0x74fd5d1312955003a23ac5d02f5dfb3627f22e7f0cb64c5ccc1a00faf80bcffb","transactionIndex":"0xa8","transactionLogIndex":"0x1","type":"mined"},{"address":"0x06012c8cf97bead5deae237070f9587f8e7a266d","blockHash":"0xc63388cb3221a7351ce296c20262769caeb9a2e8e8cb0eff82ca480fd00131d4","blockNumber":"0x4b8a6d","data":"0x000000000000000000000000ade7c04ded04e155aaaf9ff050e391858f4716e800000000000000000000000000000000000000000000000000000000000743fd00000000000000000000000000000000000000000000000000000000000715240000000000000000000000000000000000000000000000000000000000072c1c000042d08304e770dc35bd3463187084606214d7954029ccf3890539587785e1","logIndex":"0x3e","topics":["0x0a5311bd2a6608f08a180df2ee7c5946819a649b204b554bb8e39825b2c50ad5"],"transactionHash":"0x74fd5d1312955003a23ac5d02f5dfb3627f22e7f0cb64c5ccc1a00faf80bcffb","transactionIndex":"0xa8","transactionLogIndex":"0x2","type":"mined"},{"address":"0x06012c8cf97bead5deae237070f9587f8e7a266d","blockHash":"0xc63388cb3221a7351ce296c20262769caeb9a2e8e8cb0eff82ca480fd00131d4","blockNumber":"0x4b8a6d","data":"0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000ade7c04ded04e155aaaf9ff050e391858f4716e800000000000000000000000000000000000000000000000000000000000743fd","logIndex":"0x3f","topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef"],"transactionHash":"0x74fd5d1312955003a23ac5d02f5dfb3627f22e7f0cb64c5ccc1a00faf80bcffb","transactionIndex":"0xa8","transactionLogIndex":"0x3","type":"mined"}],"logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000008000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000002000000000010000000000000000000000000000000000000000000000000000100000000000000000000800000000000000000000000000000000000","root":null,"status":"0x1","transactionHash":"0x74fd5d1312955003a23ac5d02f5dfb3627f22e7f0cb64c5ccc1a00faf80bcffb","transactionIndex":"0xa8"},"id":1}


{"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x4B8A6D","toBlock":"0x4B8A6D","address":"0x06012c8cf97bead5deae237070f9587f8e7a266d"}],"id":1}
{"jsonrpc":"2.0","result":[],"id":1}

See my ethprovider script for all my parity options.

Interestingly: Running @slothbag's original query returns 937 logs as expected.

[0xPaladin]

My config:
Parity/v1.9.1-beta-aca9f13-20180201/x86_64-linux-gnu/rustc1.23.0

I'm using warp and on the kovan testnet and I can confirm this bug. I cannot pull any logs from:
0x786b5D8F272547EFB3e8754a7cB8ccc7150906f1

The optimal would be to keep warp functionality and pull logs as required - when we query. However if we have to use --no-warp to get log visibility/functionality this needs to be mentioned in the strart up documentation.