New rewards accounting issue

[erikd]

On the shelley_qa network with commit 70ab190 (on master) I get:

[2021-10-12 00:08:35.75 UTC] validateEpochRewards: total rewards that become spendable in epoch 2917 is 
    101795.447987 ADA
1 errors, eg
  stake_test1uqt78w8gmktel2rd8u0yn9tuvyew22qy9ch3zk5u28r8cmcdkesdj rewards are 1496308.539254 ADA 
    and withdrawals are 1496313.48387 ADA

The sum of the rewards is 4.944616 ADA less than the sum of the withdrawals which obviously makes no sense.

[erikd]

Getting the stake address:

select hash_raw from stake_address
    where view = 'stake_test1uqt78w8gmktel2rd8u0yn9tuvyew22qy9ch3zk5u28r8cmcdkesdj' ;
                           hash_raw                           
--------------------------------------------------------------
 \xe017e3b8e8dd979fa86d3f1e49957c6132e528042e2f115a9c51c67c6f
(1 row)

I was hoping to use @JaredCorduan's stake history tool to debug this, but IntersectMBO/cardano-node#3086 has not been merged yet, that version does not understand Alonzo and it currently does not build.

[erikd]

For that address, the address registration:

select stake_registration.* from stake_registration inner join stake_address
    on stake_address.id = stake_registration.addr_id
    where view = 'stake_test1uqt78w8gmktel2rd8u0yn9tuvyew22qy9ch3zk5u28r8cmcdkesdj' ;
 id | addr_id | cert_index | epoch_no | tx_id 
----+---------+------------+----------+-------
 15 |      51 |          0 |      863 | 18835
(1 row)

but it has never been de-registered:

select stake_deregistration.* from stake_deregistration inner join stake_address
    on stake_address.id = stake_deregistration.addr_id
    where view = 'stake_test1uqt78w8gmktel2rd8u0yn9tuvyew22qy9ch3zk5u28r8cmcdkesdj' ;
 id | addr_id | cert_index | epoch_no | tx_id | redeemer_id 
----+---------+------------+----------+-------+-------------
(0 rows)

[erikd]

Currently getting a PastHorizon error with Jared's tool. Waiting for a fix.

[erikd]

stake-credential-history tool is now working.

According to the tool withdrawals with EpochNo < 2917 (with SlotNo 20984084 being in epoch 2916):

> grep WDRL stake-address.txt
WDRL -------------- SlotNo 9575825, Lovelace 507717303587
WDRL -------------- SlotNo 14683297, Lovelace 513381841129
WDRL -------------- SlotNo 19355359, Lovelace 370334576371
WDRL -------------- SlotNo 20984084, Lovelace 104879762783

which sums to 1496313483870 Lovelace which is 1496313.48387 ADA.

For rewards:

> grep REWARD-END stake-address.txt  | sed 's/.*Lovelace //' | awk '{s+=$1} END {print s}'
1496308539254

which is 1496308.539254 ADA.

These total rewards and the total withdrawals agree with db-sync.

[JaredCorduan]

I've found the problem. The problem stems from how different the shelley-qa network is from mainnet.

During epoch 2819 on shelley-qa, the reward calculation does not complete before the end of the epoch, and is forced to completion on the epoch boundary rollover. This has the unfortunate effect of making the rewards from epoch 2819 invisible to tools like db-sync and the stake credential history tool.

I think we should force the completion of the reward calculation 2k/f-many slots before the end of the epoch. This is way more than enough time for it to complete on mainnet.

[erikd]

On the shelley_qa network I hit this issue on the epochs [2917, 3190, 4967, 4992, 4993, 4994, 4995] and a different, but almost certainly related issue (missing MIR reward) in epoch 5035 .

[erikd]

There is a fix for this in ledger-specs. I have tried using that commit in db-sync but when I do so, the consensus build fails. Therefore, I will not be able to test this until consensus (and probably also node) have been updated.