[6.7.0] Balance Difference bug

[WhoSoup]

This report came from @ilzheev in discord. One of the nodes in the open node cluster reported a different factoid balance. The node in question is https://cl.factomd.net, which reports itself as running version 6.7.0 and was, at the time, synced to the latest block (266519).

The FCT address in question is FA3RqGvKruW9BPTPHqRGAop76HgJm4fHoit7wW4aqmPyHtrjCy1M, which is reported at 51.2 FCT by most nodes and 57.6 FCT by the cl node.

I verified it myself by comparing it to my local mainnet node:

who@spoon:~$ curl -d '{"jsonrpc": "2.0", "id": 0, "method": "factoid-balance", "params": {"address": "FA3RqGvKruW9BPTPHqRGAop76HgJm4fHoit7wW4aqmPyHtrjCy1M"}}' https://cl.factomd.net/v2
{"jsonrpc":"2.0","id":0,"result":{"balance":5760000000}}
who@spoon:~$ curl -d '{"jsonrpc": "2.0", "id": 0, "method": "factoid-balance", "params": {"address": "FA3RqGvKruW9BPTPHqRGAop76HgJm4fHoit7wW4aqmPyHtrjCy1M"}}' http://localhost:8088/v2
{"jsonrpc":"2.0","id":0,"result":{"balance":5120000000}}

I have asked Anton to provide the node's FastBoot file so we can see if it's an isolated issue or if there are more differing balances, and we're going to try deleting the FastBoot file and restarting the node to see if that fixes the incorrect state.

[WhoSoup]

I wrote a script that downloads all FBlocks between 167,000 and 266,519, goes through transactions, and tallies up the balance to see if the error was at the block level (which would be very bad) and I'm happy to report that the FBlocks of the node seem to be intact.

The raw data of both sums, on my local node, and the CL node are identical: https://gist.github.com/WhoSoup/5b813976774659750320c274848388fe

While gathering this, I talked to @afenrir (many thanks for the additional info). The address FA3RqGvKruW9BPTPHqRGAop76HgJm4fHoit7wW4aqmPyHtrjCy1M is the payout address of Factoshi's ANO servers. Factoshi ran two servers, one of which was retired around two weeks ago at height 264,291. The server that's still in the set is at an efficiency of 20 and receives 5.12 FCT, the server that retired ran at an efficiency of 80, which is 1.28 FCT.

Alex pointed out that the discrepancy of 6.4 FCT is equal to 5 payouts of 1.28 FCT. Due to this, investigating this bug should focus on the coinbase process, particularly with removed servers, first.

[ilzheev]

Also 6.4 FCT is equal to the single payout of 5.12 FCT + 1.28 FCT.

[WhoSoup]

Also 6.4 FCT is equal to the single payout of 5.12 FCT + 1.28 FCT.

Oh yeah, that's much more likely 😸

[zvrso]

FastBoot_MAIN_v13.zip
FastBoot file from the KP-L2-CL node