Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

diagnostics: add packet loss and cacheServerList for speedtest #10259

Merged
merged 4 commits into from
May 12, 2024
Merged

diagnostics: add packet loss and cacheServerList for speedtest #10259

merged 4 commits into from
May 12, 2024

Conversation

r3inbowari
Copy link
Contributor

No description provided.

@AskAlexSharov AskAlexSharov requested a review from dvovk May 11, 2024 02:42
@dvovk dvovk merged commit 3e54eaf into erigontech:main May 12, 2024
15 of 19 checks passed
@r3inbowari r3inbowari deleted the fix/speedtest branch May 12, 2024 13:20
@AskAlexSharov
Copy link
Collaborator

@r3inbowari why do we need cached client object?

@AskAlexSharov AskAlexSharov mentioned this pull request Jul 13, 2024
Merged
@r3inbowari
Copy link
Contributor Author

@r3inbowari why do we need cached client object?

Fallback to the last cache when the server fails to fetch.

@AskAlexSharov
Copy link
Collaborator

@r3inbowari ah, it's flipped go's pattern of error handling. my bran did know if err == nil { exists in this world:

	serverList, err := speedtestClient.FetchServers()
	// Ensure that the server list can rolled back to the previous cache.
	if err == nil {
		cacheServerList = serverList
	}
        targets, _ := cacheServerList.FindServer([]int{})

so, we ignore error. i see...
then it's nil in first run. and if first call will return error - we will get nil-ptr panic. probably need add nil-check.

@r3inbowari
Copy link
Contributor Author

if first call will return error - we will get nil-ptr panic. probably need add nil-check.

Hi, @AskAlexSharov, nil-checks are always good. In the original FetchServers in speedtest-go, an error and empty array will be returned. I haven't modified this part of the code.

AskAlexSharov added a commit that referenced this pull request Jul 14, 2024
@AskAlexSharov
diag: thread-safety step5 - race in speedtest (#11138)
ad150b8 
- attempt to upgrade speedtest - to fix race:
showwin/speedtest-go#109 (comment)

It didn't help. Created:
showwin/speedtest-go#223

I see: 
```
==================
WARNING: DATA RACE
Write at 0x00c2167c2088 by goroutine 70275:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1.1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:203 +0x84
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:74 +0xf0
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:65 +0x44
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:200 +0xb1

Previous read at 0x00c2167c2088 by goroutine 69927:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).Read()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:452 +0x64
  io.(*nopCloser).Read()
      <autogenerated>:1 +0x6c
  io.(*LimitedReader).Read()
      /usr/local/go/src/io/io.go:479 +0xc5
  io.copyBuffer()
      /usr/local/go/src/io/io.go:429 +0x29a
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x6f
  net.genericReadFrom()
      /usr/local/go/src/net/net.go:689 +0x12
  net.(*TCPConn).readFrom()
      /usr/local/go/src/net/tcpsock_posix.go:54 +0xc9
  net.(*TCPConn).ReadFrom()
      /usr/local/go/src/net/tcpsock.go:130 +0x64
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x95
  net/http.persistConnWriter.ReadFrom()
      /usr/local/go/src/net/http/transport.go:1824 +0x12
  bufio.(*Writer).ReadFrom()
      /usr/local/go/src/bufio/bufio.go:794 +0x2b0
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.CopyBuffer()
      /usr/local/go/src/io/io.go:402 +0x8f
  net/http.(*transferWriter).doBodyCopy()
      /usr/local/go/src/net/http/transfer.go:416 +0x144
  net/http.(*transferWriter).writeBody()
      /usr/local/go/src/net/http/transfer.go:371 +0x75c
  net/http.(*Request).write()
      /usr/local/go/src/net/http/request.go:755 +0x1413
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/transport.go:2447 +0x379
  net/http.(*Transport).dialConn.gowrap3()
      /usr/local/go/src/net/http/transport.go:1800 +0x33

Goroutine 70275 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:267 +0x3ef
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.gowrap1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:279 +0x41

Goroutine 69927 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1800 +0x27fe
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1485 +0x124
  net/http.(*Transport).queueForDial.gowrap1()
      /usr/local/go/src/net/http/transport.go:1449 +0x44
==================

==================
WARNING: DATA RACE
Write at 0x00c2167c2088 by goroutine 63832:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1.1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:203 +0x84
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:74 +0xf0
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:65 +0x44
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:200 +0xb1

Previous read at 0x00c2167c2088 by goroutine 57836:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).DownloadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:408 +0x3ee
  github.com/showwin/speedtest-go/speedtest.downloadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:183 +0x624
  github.com/showwin/speedtest-go/speedtest.(*Server).downloadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:119 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Goroutine 63832 (running) created at:
  time.goFunc()
      /usr/local/go/src/time/sleep.go:177 +0x44

Goroutine 57836 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).downloadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:122 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).DownloadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:109 +0x1ee
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:66 +0x1d2
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2
==================
==================
WARNING: DATA RACE
Write at 0x00c2167c2068 by goroutine 63836:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Previous read at 0x00c2167c2068 by goroutine 63840:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:438 +0x179
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Goroutine 63836 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2

Goroutine 63840 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2
==================
==================
WARNING: DATA RACE
Write at 0x00c2167c2068 by goroutine 63840:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Previous write at 0x00c2167c2068 by goroutine 63835:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Goroutine 63840 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2

Goroutine 63835 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2
==================
==================
WARNING: DATA RACE
Read at 0x00c2167c2100 by goroutine 63833:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:258 +0x1c6
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.gowrap1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:279 +0x41

Previous write at 0x00c2167c2100 by goroutine 63888:
  sync/atomic.AddInt64()
      /usr/local/go/src/runtime/race_amd64.s:289 +0xb
  sync/atomic.AddInt64()
      <autogenerated>:1 +0x15
  io.(*nopCloser).Read()
      <autogenerated>:1 +0x6c
  io.(*LimitedReader).Read()
      /usr/local/go/src/io/io.go:479 +0xc5
  io.copyBuffer()
      /usr/local/go/src/io/io.go:429 +0x29a
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x6f
  net.genericReadFrom()
      /usr/local/go/src/net/net.go:689 +0x12
  net.(*TCPConn).readFrom()
      /usr/local/go/src/net/tcpsock_posix.go:54 +0xc9
  net.(*TCPConn).ReadFrom()
      /usr/local/go/src/net/tcpsock.go:130 +0x64
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x95
  net/http.persistConnWriter.ReadFrom()
      /usr/local/go/src/net/http/transport.go:1824 +0x12
  bufio.(*Writer).ReadFrom()
      /usr/local/go/src/bufio/bufio.go:794 +0x2b0
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.CopyBuffer()
      /usr/local/go/src/io/io.go:402 +0x8f
  net/http.(*transferWriter).doBodyCopy()
      /usr/local/go/src/net/http/transfer.go:416 +0x144
  net/http.(*transferWriter).writeBody()
      /usr/local/go/src/net/http/transfer.go:371 +0x75c
  net/http.(*Request).write()
      /usr/local/go/src/net/http/request.go:755 +0x1413
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/transport.go:2447 +0x379
  net/http.(*Transport).dialConn.gowrap3()
      /usr/local/go/src/net/http/transport.go:1800 +0x33

Goroutine 63833 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:253 +0x464
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:195 +0x4a4
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2

Goroutine 63888 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1800 +0x27fe
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1485 +0x124
  net/http.(*Transport).queueForDial.gowrap1()
      /usr/local/go/src/net/http/transport.go:1449 +0x44
==================
```
- also i don't understand why do we need cached client object
#10259
blxdyx added a commit to node-real/bsc-erigon that referenced this pull request Jul 24, 2024
@blxdyx @AskAlexSharov
@dvovk @antonis19 @Giulio2002 @chuwt @sudeepdino008 @lystopad @mh0lt @stellrust @yperbasis @awskii @JkLondon @domiwei @lupin012 @somnathb1 @taratorio @Leeyqiongyuan @mriccobene @VBulikov @manav2401 @yingshanghuangqiao
Upstream erigon V3.0.0-alpha1 (#453)
00f899f 
* diag: thread-safety step1 - json marshal under mutex (erigontech#11134)

* diag: thread-safety step2 - unlock mutex in defer (erigontech#11135)

* diag: thread-safety step4 - remove dedicated shutdown listener goroutine (erigontech#11137)

reason: 
- we already have 1 goroutine for saving data:
```
func (d *DiagnosticClient) runSaveProcess(rootCtx context.Context) {
	ticker := time.NewTicker(5 * time.Minute)
	go func() {
		for {
			select {
			case <-ticker.C:
				d.SaveData()
			case <-rootCtx.Done():
				ticker.Stop()
				return
			}
		}
	}()
}
```
no reason to save it from one more goroutine. just save it right here -
in `case <-rootCtx.Done()` section. less concurrency - better.

rootContext already subscribed to sigterm

* diag: thread-safety step3 - `PeerStatistics.Clone()` and `PeerStats.mutex` (erigontech#11136)

Co-authored-by: dvovk <vovk.dimon@gmail.com>

* dl: manifest-verify green CI (erigontech#11142)

- skip `erigon2-v2` buckets - until
erigontech#10967

* bor: fix race in `LockedMilestoneIDs` access (erigontech#11139)

for erigontech#11129

* Diagnostics: refactor bulk execution thread safety (erigontech#11143)

* Call UnwindTo with tx instead of nil in sync_test.go (erigontech#11133)

Co-authored-by: antonis19 <antonis19@users.noreply.github.com>

* Caplin: Tweaked CGO_Flags (erigontech#11144)

* diag: thread-safety step5 - race in speedtest (erigontech#11138)

- attempt to upgrade speedtest - to fix race:
showwin/speedtest-go#109 (comment)

It didn't help. Created:
showwin/speedtest-go#223

I see: 
```
==================
WARNING: DATA RACE
Write at 0x00c2167c2088 by goroutine 70275:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1.1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:203 +0x84
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:74 +0xf0
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:65 +0x44
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:200 +0xb1

Previous read at 0x00c2167c2088 by goroutine 69927:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).Read()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:452 +0x64
  io.(*nopCloser).Read()
      <autogenerated>:1 +0x6c
  io.(*LimitedReader).Read()
      /usr/local/go/src/io/io.go:479 +0xc5
  io.copyBuffer()
      /usr/local/go/src/io/io.go:429 +0x29a
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x6f
  net.genericReadFrom()
      /usr/local/go/src/net/net.go:689 +0x12
  net.(*TCPConn).readFrom()
      /usr/local/go/src/net/tcpsock_posix.go:54 +0xc9
  net.(*TCPConn).ReadFrom()
      /usr/local/go/src/net/tcpsock.go:130 +0x64
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x95
  net/http.persistConnWriter.ReadFrom()
      /usr/local/go/src/net/http/transport.go:1824 +0x12
  bufio.(*Writer).ReadFrom()
      /usr/local/go/src/bufio/bufio.go:794 +0x2b0
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.CopyBuffer()
      /usr/local/go/src/io/io.go:402 +0x8f
  net/http.(*transferWriter).doBodyCopy()
      /usr/local/go/src/net/http/transfer.go:416 +0x144
  net/http.(*transferWriter).writeBody()
      /usr/local/go/src/net/http/transfer.go:371 +0x75c
  net/http.(*Request).write()
      /usr/local/go/src/net/http/request.go:755 +0x1413
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/transport.go:2447 +0x379
  net/http.(*Transport).dialConn.gowrap3()
      /usr/local/go/src/net/http/transport.go:1800 +0x33

Goroutine 70275 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:267 +0x3ef
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.gowrap1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:279 +0x41

Goroutine 69927 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1800 +0x27fe
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1485 +0x124
  net/http.(*Transport).queueForDial.gowrap1()
      /usr/local/go/src/net/http/transport.go:1449 +0x44
==================

==================
WARNING: DATA RACE
Write at 0x00c2167c2088 by goroutine 63832:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1.1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:203 +0x84
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:74 +0xf0
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:65 +0x44
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:200 +0xb1

Previous read at 0x00c2167c2088 by goroutine 57836:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).DownloadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:408 +0x3ee
  github.com/showwin/speedtest-go/speedtest.downloadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:183 +0x624
  github.com/showwin/speedtest-go/speedtest.(*Server).downloadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:119 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Goroutine 63832 (running) created at:
  time.goFunc()
      /usr/local/go/src/time/sleep.go:177 +0x44

Goroutine 57836 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).downloadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:122 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).DownloadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:109 +0x1ee
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:66 +0x1d2
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2
==================
==================
WARNING: DATA RACE
Write at 0x00c2167c2068 by goroutine 63836:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Previous read at 0x00c2167c2068 by goroutine 63840:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:438 +0x179
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Goroutine 63836 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2

Goroutine 63840 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2
==================
==================
WARNING: DATA RACE
Write at 0x00c2167c2068 by goroutine 63840:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Previous write at 0x00c2167c2068 by goroutine 63835:
  github.com/showwin/speedtest-go/speedtest.(*DataChunk).UploadHandler()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:440 +0x247
  github.com/showwin/speedtest-go/speedtest.uploadRequest()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:188 +0x113
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:150 +0x85
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start.func2()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:218 +0xca

Goroutine 63840 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2

Goroutine 63835 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:212 +0x70a
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2
==================
==================
WARNING: DATA RACE
Read at 0x00c2167c2100 by goroutine 63833:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.func1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:258 +0x1c6
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture.gowrap1()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:279 +0x41

Previous write at 0x00c2167c2100 by goroutine 63888:
  sync/atomic.AddInt64()
      /usr/local/go/src/runtime/race_amd64.s:289 +0xb
  sync/atomic.AddInt64()
      <autogenerated>:1 +0x15
  io.(*nopCloser).Read()
      <autogenerated>:1 +0x6c
  io.(*LimitedReader).Read()
      /usr/local/go/src/io/io.go:479 +0xc5
  io.copyBuffer()
      /usr/local/go/src/io/io.go:429 +0x29a
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x6f
  net.genericReadFrom()
      /usr/local/go/src/net/net.go:689 +0x12
  net.(*TCPConn).readFrom()
      /usr/local/go/src/net/tcpsock_posix.go:54 +0xc9
  net.(*TCPConn).ReadFrom()
      /usr/local/go/src/net/tcpsock.go:130 +0x64
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.Copy()
      /usr/local/go/src/io/io.go:388 +0x95
  net/http.persistConnWriter.ReadFrom()
      /usr/local/go/src/net/http/transport.go:1824 +0x12
  bufio.(*Writer).ReadFrom()
      /usr/local/go/src/bufio/bufio.go:794 +0x2b0
  io.copyBuffer()
      /usr/local/go/src/io/io.go:415 +0x22e
  io.CopyBuffer()
      /usr/local/go/src/io/io.go:402 +0x8f
  net/http.(*transferWriter).doBodyCopy()
      /usr/local/go/src/net/http/transfer.go:416 +0x144
  net/http.(*transferWriter).writeBody()
      /usr/local/go/src/net/http/transfer.go:371 +0x75c
  net/http.(*Request).write()
      /usr/local/go/src/net/http/request.go:755 +0x1413
  net/http.(*persistConn).writeLoop()
      /usr/local/go/src/net/http/transport.go:2447 +0x379
  net/http.(*Transport).dialConn.gowrap3()
      /usr/local/go/src/net/http/transport.go:1800 +0x33

Goroutine 63833 (running) created at:
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).rateCapture()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:253 +0x464
  github.com/showwin/speedtest-go/speedtest.(*TestDirection).Start()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/data_manager.go:195 +0x4a4
  github.com/showwin/speedtest-go/speedtest.(*Server).uploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:153 +0x2e8
  github.com/showwin/speedtest-go/speedtest.(*Server).UploadTestContext()
      /home/ubuntu/go/pkg/mod/github.com/showwin/speedtest-go@v1.7.5/speedtest/request.go:140 +0x25c
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).runSpeedTest()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:71 +0x240
  github.com/ledgerwatch/erigon-lib/diagnostics.(*DiagnosticClient).setupSpeedtestDiagnostics.func1()
      /home/ubuntu/erigon/erigon-lib/diagnostics/speedtest.go:35 +0xf2

Goroutine 63888 (running) created at:
  net/http.(*Transport).dialConn()
      /usr/local/go/src/net/http/transport.go:1800 +0x27fe
  net/http.(*Transport).dialConnFor()
      /usr/local/go/src/net/http/transport.go:1485 +0x124
  net/http.(*Transport).queueForDial.gowrap1()
      /usr/local/go/src/net/http/transport.go:1449 +0x44
==================
```
- also i don't understand why do we need cached client object
erigontech#10259

* rlpgen: step 1 (erigontech#11112)

- no behavior changes
- move encbuffer and encreader to `encbuffer.go`
- copy-paste rlpgen package

* bor: finality race - step 2 (erigontech#11151)

```
==================
WARNING: DATA RACE
Write at 0x00c015bf2208 by goroutine 110:
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).RemoveMilestoneID()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist/milestone.go:190 +0x164
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleNoAckMilestone()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:237 +0x12c
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:159 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:117 +0x57
  github.com/ledgerwatch/erigon/polygon/bor/finality.startNoAckMilestoneService()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:102 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap3()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:65 +0x17

Previous read at 0x00c015bf2208 by goroutine 109:
  github.com/ledgerwatch/erigon/polygon/bor/finality/whitelist.(*milestone).ProcessFutureMilestone()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist/milestone.go:277 +0x166
  github.com/ledgerwatch/erigon/polygon/bor/finality.handleMilestone()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:208 +0x1e5
  github.com/ledgerwatch/erigon/polygon/bor/finality.retryHeimdallHandler()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:159 +0x5c4
  github.com/ledgerwatch/erigon/polygon/bor/finality.RetryHeimdallHandler()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:117 +0x5c
  github.com/ledgerwatch/erigon/polygon/bor/finality.startMilestoneWhitelistService()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:93 +0x2a
  github.com/ledgerwatch/erigon/polygon/bor/finality.Whitelist.gowrap2()
      /home/ubuntu/erigon/polygon/bor/finality/whitelist.go:64 +0x17
```

* integration: wait for caplin snap open finish (erigontech#11124)

* rpc bottleneck: block files mutex (e3) (erigontech#11156)

for: erigontech#11090

thank you [tholcman](https://github.com/tholcman) for finding

* dl: use native `filepath.IsLocal` and `filepath.Clean` funcs (erigontech#11141)

they was vendored because of go1.19 compatibility which we dropped

* chore: confusing log (erigontech#11043)

* execution spec tests update for devnet-1 (erigontech#11127)

updating to version
[devnet-1@v1.3.0](https://github.com/ethereum/execution-spec-tests/releases/tag/devnet-1%40v1.3.0)

- update to 7702: some extra json fields displayed in fixtures (rather
than just rlps)
- some refactoring to fixture json ordering means that a huge number of
test files were changed
- [refactor
requested](erigontech#10812 (comment))
by Alex done in this pr
[here](https://github.com/ledgerwatch/erigon/pull/11127/files#diff-8c398ef0a79f97ba6d497a99247a815b3a51918de9e16954215b6073f907c92e).

* Delete as not needed cmd/release/. (erigontech#11161)

Discussed with @AskAlexSharov

* Fix downloader completion set and races (erigontech#11182)

fixes: erigontech#11060

Also fixes several races, which should include: 

erigontech#11123
erigontech#11102

* chore: fix some comments (erigontech#11170)

Signed-off-by: stellrust <gohunter@foxmail.com>

* tests: auto-close temporal db (erigontech#11160)

* discovery table: revalidate race fix (erigontech#11159)

for erigontech#11158

* Caplin: Added POST `Validators`  (erigontech#11152)

close erigontech#11150

* diagnostics: refactor network peers mutex (erigontech#11178)

Refactor to pattern which was suggested by @AskAlexSharov 

- move business-logic inside private methods
- move mutex locking inside public methods
- call private methods from public. don't call public methods from
private.

* move interfaces from ledgerwatch to erigontech (erigontech#11194)

* increase snapshot semaphore default limit (erigontech#11193)

* p2p receipts (erigontech#11010)

closes erigontech#10320 and closes erigontech#11014

---------

Co-authored-by: JkLondon <ilya@mikheev.fun>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* Caplin: Optimization and Parallelization of processes and reduction of Goroutines (erigontech#11058)

Optimizations:

1) Single goroutine for tracking expiry of gossip subscriptions
2) tweaking of parameters on operations retention in cache
3) moving the dumping of BeaconState after forkchoice, so that we do not
mess up block times with I/O

* Caplin: Look for peers if not avaiable within subnet (erigontech#11057)

Actually look for peers of needed subnet for better performance on
holesky and mainnet

---------

Co-authored-by: Kewei <kewei.train@gmail.com>

* Downloader: Bump up defaults (erigontech#11197)

* `erigon_getLatestLogs` add limit parameter (erigontech#11095)

If user provides logCount param the API should returns logCount records,
even if the block contains more logs that satisfy the filter.
The logCount param is already supported, this PR avoids to return the
log records in excess.
Moreover I have created 14 integration tests for this API (run with
success on erigon2).

* refine ProcessBlock and ProcessBlindedBlock (erigontech#10923)

- Apply generic interface `GenericBlock` for blinded block and normal
block.
- Refine function `ProcessBlock()` so as to remove duplicated code.

* less logs on CI (erigontech#11122)

* revert logging of binary data (erigontech#11199)

* pool: do fsync by non-empty update (e3) (erigontech#11198)

for erigontech#11163

* engineapi: Fix req list check (erigontech#11191)

* make `latest seen blocks` visible earlier (erigontech#11204)

* move secp256k1 to erigontech (erigontech#11209)

* move erigon-snapshot to erigontech (erigontech#11213)

* replaced speedtest lib with our fork (erigontech#11217)

Replaced speed test lib with our for which uses erigon cloudflare webbed
to test download speed. I decided to change speedtest servers to erigon
servers as users can complain why there is traffic servers which is not
related to erigon infrastructure.
Our for of speedtest https://github.com/erigontech/speedtest

* Caplin: Better old-state pruning (erigontech#11219)

* Caplin: Remove TmpDB from `BlockCollector` (erigontech#11215)

Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>

* upstrem

* Attempt to use go21  (erigontech#11207)

* post validator balances (erigontech#11218)

impl: erigontech#11055

* eth/stagedsync: polygon sync stage to use new heimdall service (erigontech#11196)

part 1 of erigontech#11186

* polygon/heimdall: remove old duplicate heimdall component (erigontech#11214)

part 2 of erigontech#11186

* enable `madv_normal` for .kv files of > 0 lvl (erigontech#11223)

enable `KV_MADV_NORMAL_NO_LAST_LVL=accounts,storage,code,commitment` by
default

* Move from ledgerwatch to erigontech (erigontech#11224)

* add log

* add log

* erigon-lib: tidy up slices,generics,cmp commons (erigontech#11216)

* Update module path of erigonwatch to erigontech (erigontech#11226)

* Fix previous download completion processing (erigontech#11227)

This fixes downloads stalling if a locally produced file appears not
downloaded
It also will re-complete hashes without re-downloading if the download
db is removed
It should also complete locally produced files when they are verified,
if not it will self mend on restart

---------

Co-authored-by: Giulio <giulio.rebuffo@gmail.com>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* Bring back dirs to  BaseAPI (erigontech#11228)

This PR brings back `datadir.Dirs` to `BaseAPI`, and removes a redundant
`Aggregator` argument to `EngineServer.Start` .

---------

Co-authored-by: antonis19 <antonis19@users.noreply.github.com>

* Logs checking (erigontech#11229)

changed erigon api logs to e3 closes erigontech#11117

---------

Co-authored-by: JkLondon <ilya@mikheev.fun>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* ots: nil-ptr in rpc (erigontech#11232)

* txpool changes for set_code_tx support (erigontech#11235)

* Erigon 3.0: Smarter full and minimal nodes (erigontech#11220)

* add log

* add log

* added var inside WriteMap for chain-like interface (erigontech#11241)

closes erigontech#11202

---------

Co-authored-by: JkLondon <ilya@mikheev.fun>

* tracer: add support bailout on evm.create() (erigontech#11237)

Add bailout mgt on evm.create() method (it is already mgt on evm.cal()).
When approved I will create PR on rel 2.60.x

* added rpc info feature (erigontech#11242)

closes erigontech#11157 but a bit of scared that we probably don't have some
MetadataFromContext method like in
ethereum/go-ethereum#24255

Co-authored-by: JkLondon <ilya@mikheev.fun>

* [test] remove unused variables (erigontech#10938)

remove unused variables

* force fsync after notifications sent (erigontech#11244)

* Fixed nil pointer exception (erigontech#11253)

* fix panic due to nil validator set (erigontech#11260)

fix erigontech#11027
root cause: specific slot state data not found leads to empty validator
set

* qa-tests: increase test time of sync-from-scratch for minimal node (erigontech#11256)

the last few runs of the test did not complete on time

* qa-tests: fix sync-from-scratch test result uploading (erigontech#11245)

The test results are uploaded to the github actions test run page at the
end of the test. As this test has 2 jobs, we need to give the uploaded
test results different names to avoid clashes.

* Caplin: Add support for beacon snapshots (also stops relying on Engine API) (erigontech#11250)

I had to:

* Add Caplin snapshot download
* Fix Snapshot Downloader on Holesky
* Fixed Holesky's chainspec

* add log

* fix chapel bt hash

* rm prints (erigontech#11261)

* fix chapel bt hash

* Enable `madv_normal` for level0 .kv (erigontech#11265)

* HexOrDecimal - to accept unquoted numbers - in json (e3) (erigontech#11262)

accept in `genesis.json`:` "nonce": 0,`
now see: 
```
Fatal: invalid genesis file: json: cannot unmarshal number into Go struct field Genesis.alloc of type *math.HexOrDecimal64
```

See also `https://github.com/ethereum/go-ethereum/pull/26758`

* e3: make getLatest layers visible in pprof  (erigontech#11266)

* hack to prevent early download finish (erigontech#11267)

* add log

* add log

* add log

* add log

* add log

* add log

* add log

* add log

* add log

* Wire OverridePragueTime into txpool (erigontech#11234)

also extract common logic from `(p *TxPool) isShanghai()`, `isCancun()`,
`isPrague()`.

* readme update (erigontech#11275)

* diag: race in updateIndexingStatus  (erigontech#11274)

for erigontech#11268

* Bump version to 3.0.0-alpha1 (erigontech#11276)

Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>

* fix body data

* fix goreleaser after update (erigontech#11281)

Fixes
https://github.com/erigontech/erigon/actions/runs/10041015251/job/27748204997#step:9:72
(caused by PR erigontech#10726 apparently). See also
https://github.com/goreleaser/goreleaser-cross-example/blob/master/Makefile

* fix body data

* fix body data

* bor: loopbreaker - to stop check `LoopBlockLimit` and rely on stage_headers progress (erigontech#11286)

* Special logs for near-chain-tip execution (erigontech#11288)

* fix upstream

* e3 use same goreleaser-cross version as in e2 (erigontech#11285)

- switched to `v1.21.5` 
- added to UI checkbox "Publish Artifacts" - which is disabled by
default. if not set: `make release-dry-run`

* add bool variable to auto-release  (erigontech#11290)

* on chain-tip: if batch is full - stop execution stage - to allow commit and reduce db size (erigontech#11287)

* add log

* PIP-35: enforce 25gwei gas config for all polygon chains (erigontech#11294)

Remove the checks for amoy added previously to prepare for mainnet
release.

Sets `txpool.pricelimit`, `miner.gasprice` and `gpo.ignoreprice` to
25gwei for all polygon based networks.

* refactor: rename fields in hex patricia trie (erigontech#11296)

Small refactor to improve the readability of some fields in the
`hex_patricia_hashed.go` .

Co-authored-by: antonis19 <antonis19@users.noreply.github.com>

* add log

* add log

* add log

* new dirWalk for test purposes (erigontech#11277)

closes erigontech#10086 but it uses old lib
(https://github.com/karrick/godirwalk). This branch could be used for
tests for someone who experiences troubles with RAM with RemoveContents
func. (For example for this guy
https://discord.com/channels/687972960811745322/1233600171821240380)


Maybe we should fork this lib :) thing for future milestone

---------

Co-authored-by: JkLondon <ilya@mikheev.fun>
Co-authored-by: alex.sharov <AskAlexSharov@gmail.com>

* chore: fix some comments (erigontech#11273)

Signed-off-by: yingshanghuangqiao <yingshanghuangqiao@foxmail.com>

* remove useless logs

---------

Signed-off-by: stellrust <gohunter@foxmail.com>
Signed-off-by: yingshanghuangqiao <yingshanghuangqiao@foxmail.com>
Co-authored-by: Alex Sharov <AskAlexSharov@gmail.com>
Co-authored-by: dvovk <vovk.dimon@gmail.com>
Co-authored-by: antonis19 <antonis19@mail.com>
Co-authored-by: antonis19 <antonis19@users.noreply.github.com>
Co-authored-by: Giulio rebuffo <giulio.rebuffo@gmail.com>
Co-authored-by: chuwt <weitaochu@gmail.com>
Co-authored-by: sudeep <sudeepdino008@gmail.com>
Co-authored-by: lystopad <oleksandr.lystopad@erigon.tech>
Co-authored-by: Mark Holt <135143369+mh0lt@users.noreply.github.com>
Co-authored-by: stellrust <gohunter@foxmail.com>
Co-authored-by: Andrew Ashikhmin <34320705+yperbasis@users.noreply.github.com>
Co-authored-by: awskii <awskii@users.noreply.github.com>
Co-authored-by: Ilya Mikheev <54912776+JkLondon@users.noreply.github.com>
Co-authored-by: JkLondon <ilya@mikheev.fun>
Co-authored-by: Kewei <kewei.train@gmail.com>
Co-authored-by: lupin012 <58134934+lupin012@users.noreply.github.com>
Co-authored-by: Somnath <snb895@outlook.com>
Co-authored-by: milen <94537774+taratorio@users.noreply.github.com>
Co-authored-by: LEE <Leeyqiongyuan@gmail.com>
Co-authored-by: Michelangelo Riccobene <michelangelo.riccobene@gmail.com>
Co-authored-by: VBulikov <vlad.bulikov@gmail.com>
Co-authored-by: yperbasis <andrey.ashikhmin@gmail.com>
Co-authored-by: Manav Darji <manavdarji.india@gmail.com>
Co-authored-by: yingshanghuangqiao <yingshanghuangqiao@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dvovk dvovk dvovk approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@r3inbowari @AskAlexSharov @dvovk