Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rejected header marked as bad #11473

Closed
directsyndikat opened this issue Aug 3, 2024 · 5 comments
Closed

Rejected header marked as bad #11473

directsyndikat opened this issue Aug 3, 2024 · 5 comments
Assignees
Labels
erigon3 imp2 Medium importance

Comments

@directsyndikat
Copy link

Currently facing the following:

The node itself has been running for quite some days without issues up until today.
It keeps spamming the following logs, restarting the node does not seem to work.

From reading previous issues i attempt to
integration state_stages --unwind=10 --datadir="/root/erigon/.local/share/erigon/bor-mainnet" --chain="bor-mainnet"
However it does not recognize 'integration'




WARN[08-03|18:46:48.726] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.728] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.730] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.732] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.734] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.737] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.746] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.748] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.750] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.755] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.756] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.772] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.784] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.786] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.789] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.791] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.796] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.799] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
WARN[08-03|18:46:48.805] [downloader] Rejected header marked as bad hash=0xbfef129a747e5861405e2aceef1d76c7b8bf42d3aadde35b62a4cabf6ff2a5ad height=60137759
@bambuleDZ
Copy link

Are you running thorax/erigon:2.60.5-amd64 by any chance?

We had an issue as well and found out that the integration cmd is not built in anymore.
We rolled back to thorax/erigon:v2.60.4 and there it is while on thorax/erigon:2.60.4-amd64 integration is not included.

Maybe someone here can explain what's the background is on this change? @AskAlexSharov

@directsyndikat
Copy link
Author

I am running [erigon_3.0.0-alpha1_linux_arm64.tar.gz]
seems like files such as history/v1-accounts.2688-2752.v' = 'fbe6a3a701cda8ff6adfc4a8b33f9e1bb2b1781b' are not present in the manifest file

@yperbasis yperbasis added the imp2 Medium importance label Aug 9, 2024
@yperbasis
Copy link
Member

Does this still happen with v3.0.0-alpha2?

@AskAlexSharov
Copy link
Collaborator

AskAlexSharov commented Aug 24, 2024

yes. and it's duplicate of #10734

@taratorio taratorio self-assigned this Oct 21, 2024
AskAlexSharov pushed a commit that referenced this issue Oct 22, 2024
…12404)

relates to #11387,
#11473,
#10734

tried to simulate the OOM using
#11799

What I found was infinitely growing alloc of headers when receiving new
header messages in sentry's `blockHeaders66` handler (check screenshot
below).

It looks like this is happening because in the case of a bad child
header: we delete it from the `links` map, however its parent link still
holds a reference to it so the deleted link & header never get gc-ed.
Furthermore if new similar bad hashes arrive after deletion they get
appended to their parent header's link and the children of that link can
grow indefinitely
([here](https://github.com/erigontech/erigon/blob/main/turbo/stages/headerdownload/header_algos.go#L1085-L1086)).
Ie confirmed with debug logs (note link at 13450415 has 140124
children):
```
DBUG[10-21|18:18:05.003] [downloader] InsertHeader: Rejected header parent info hash=0xb488d67deaf4103880fa972fd72a7a9be552e3bc653f124f1ad9cb45f36bcd07 height=13450415 children=140124
```

<br/>

The solution for this is to remove the bad link from its parent child
list
[here](https://github.com/erigontech/erigon/blob/main/turbo/stages/headerdownload/header_algos.go#L544)
so that 1) it gets gc-ed and 2) the children list does not grow
indefinitely.

<br/>


![oom-heap-profile2](https://github.com/user-attachments/assets/518fa658-c199-48b6-aa2d-110673264144)
@taratorio
Copy link
Member

this should be fixed in v2.60.9 (by #12405) and v3.0.0-alpha5 (by #12404), please re-open if you see it again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
erigon3 imp2 Medium importance
Projects
None yet
Development

No branches or pull requests

5 participants