Performance Regression in Geth v1.10.21

[gabronick]

System information

Geth version: Geth v1.10.21
OS & Version: Linux Amazon EC2 5.10.130-118.517.amzn2.aarch64

Expected behaviour

blocks are processed very fast, several blocks usually take a few seconds, the server runs on NVME

example:

Imported new chain segment blocks=76 txs=12404 mgas=1108.491 elapsed=8.108s mgasps=136.710 number=15,197,808 hash=aab71b..b5f1e8 age=5d11h4m dirty=123.00MiB

Actual behaviour

Imported new chain segment blocks=5 txs=1250 mgas=115.102 elapsed=59.160s mgasps=1.946 number=15,232,749 hash=3fd81a..e5dd9d age=16m36s dirty=5.14GiB

the blocks start to process so slowly that the node lags, the only way is to restart the node

Steps to reproduce the behaviour

let the node run for a while,

UPDATE:

after going back to version .20 the stability has improved, we have never seen this behavior before and the workload is constant

[ligi]

Thanks for the report!

Can you let us now the command line parameters that you start geth with?

It may be related to this change:

We have reverted to an older version of goleveldb because recent versions have buggy compaction and manifest handling. (#25413)

but needs more investigation.

[fjl]

A theory why it could be slow:

In geth v1.10.20, we upgraded to a broken version of leveldb which may skip database compaction in some cases. When you upgrade to .21, it could be that leveldb needs to perform significant compaction work, which would reduce performance until that's done.

[MariusVanDerWijden]

Closing this for now, since it went pretty stale. If you still see these performance regressions with v1.10.25 please open another issue. thanks for reporting