[BUG] Incorrect checkpoint: failed transactions are not retried

[byte-bandit]

This message was recorded on Paloma as successfully relayed (which it was, to be fair):
"publicAccessData": "GtYCiVZySEv+2XCPQ1dLaowuXpmSxUDikmsD9RVO5UM="

Checking the data gives us the TX hash:

vadmin@mainnet-validator:/root$ echo "GtYCiVZySEv+2XCPQ1dLaowuXpmSxUDikmsD9RVO5UM=" | base64 -d | od -t x1 -An
 1a d6 02 89 56 72 48 4b fe d9 70 8f 43 57 4b 6a
 8c 2e 5e 99 92 c5 40 e2 92 6b 03 f5 15 4e e5 43

We can look it up: https://etherscan.io/tx/0x1ad602895672484bfed9708f43574b6a8c2e5e9992c540e2926b03f5154ee543

And as we can see, the TX failed with incorrect checkpoint.

I assume that Pigeons still check the TX as successfully relayed since it could fail for a myriad of reasons on the target chain. However, if possible, I suggest we treat this one as a failure instead so it can be retried (as intended).

[verabehr]

@byte-bandit is this a duplicate of #846 or just one potential issue?

[byte-bandit]

@verabehr I'm not sure yet. #846 might be caused by this, but they're two different behaviours. This will affect SLC as well.

[byte-bandit]

I did however also encounter a message that had error data properly filled:

echo "ZXhlY3V0aW9uIHJldmVydGVkOiBJbmNvcnJlY3QgQ2hlY2twb2ludA==" | base64 -d
execution reverted: Incorrect Checkpoint%      

Both the original and this message were logic calls.

[verabehr]

@byte-bandit is this still open or was this addressed in one of your PRs?

[byte-bandit]

@verabehr This is still open. I have encountered it twice during my smoke testing this morning on private TN: https://bscscan.com/address/0x3cA59abe703f2B88Bc93846f4500D919Dc7ff105

The issue is that pigeons report a valid TX hash, assign it to the message as public access data, everyone attests (because the TX does exist) and that's it. So to Paloma, it looks like everything went through without an issue.

I have however seen this Incorrect Checkpoint error coming up in our error logs for messages that failed. So, sometimes Pigeons DO see this problem and act like they should be.

I expect that it's either something to do with different endpoint providers behaving differently (i.e. some reporting the error, others don't), or maybe with the way Pigeons perform the TX validation check. I didn't have the time to look into this yet, I expect it'll take another day or two to get to the bottom and iron out.

@taariq
Since this is going to keep causing headaches down the road, I'm going to pivot my priorities on this issue, then finish up the Pigeon work on ERC20 transfer. By then, we can re-evaluate if OP keys are still the priority (with the eased jailing in place), otherwise I can get started on bridge tax.

[byte-bandit]

There are now two open PRs to address this issue:

• feat: hold SLC until JIT valset update is processed palomachain/paloma#1015
• feat: valset ID crosscheck palomachain/pigeon#311

With those changes, I was not able to reproduce the Incorrect Checkpoint error on private test net any longer:
https://bscscan.com/address/0xc408282883Bcf98E7199Bc999B86BB4f67f5cF78

[taariq]

Closing as fixed.