Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(vendor): update nim-libp2p repo path #6068

Closed
wants to merge 129 commits into from

Conversation

kaiserd
Copy link
Contributor

@kaiserd kaiserd commented Mar 12, 2024

The nim-libp2p repo has been moved from the status github org to the vacp2p org.

tersec and others added 30 commits February 20, 2024 05:14
- test refc in CI in Nim 2.0 and later
)

- update instructions for tracking upstream MIRACL Core
- bump `bls12-381-tests` to `v0.1.2`
- bump `miracl-core` to `0f67878bee7c4108405deb2b0b5e4e58d1ae30fc`
- test refc in CI in Nim 2.0 and later
- rename `milagro.nims` -> `miracl.nims`
- rename `milagro.nim` -> `miracl.nim`
- rename `milagro(Path|_func)` -> `miracl(Path|_func)`
- rename `milagro` references -> `miracl` in documentation
…5918)

- Upgrade to github actions v4
- Silence declared but unused warnings when there is no loggging code generated
- Reduce compiler warnings when using new chronos
- Upgrade github action to v3
- Fix autobahn tests
- Reenable tests
- Restrict autobahn tests run to merged PR only
- Upgrade github action setup-python to v3 and up/dn artifact to v3
- Reduce compiler warnings
- Fix github actions badge url
- Reduce compiler warnings
- Upgrade to github actions v4
- Revert "Fix github actions badge url"
- Downgrade upload artifact action to v3
- use non-EOL macOS version for GitHub Actions CI
- bump `csources` to `v0.4.1`
- test refc in CI in Nim 2.0 and later
- bump `csources` to `v0.4.2`
* track latest duration instead of total in new timing metrics

Change `db_checkpoint_seconds` and `state_replay_seconds` metrics to
record the latest duration instead of the total. `nim-metrics` already
synthesizes a `_total` metric from these implicitly.

* still have to use inc, metrics only synthesizes the name not the sum

* prefix with `beacon_dag`
- Add push raises to shims/macros module
- rename `RangeError` -> `RangeDefect` in test name
)

- Client also handle error message if id is null
- Client pass meaningful error to newFut when processMessage failed
- Refactoring: extract rpc handler from HTTP and WebSocket server
- Use pragma push/pop pair to disable warning
- http server better exception handling
- Fix CI badge url
- Upgrade github actions to v4
- Revert "Fix CI badge url"
- HttpAuthHook use async raises
- Move CancelledError handling to outer try/except of RpcWebsocketServer
- Implement RPC batch call both in servers and clients
- v0.4.0
- Should compile if chronicles log turned on
- Add framework to support more optional types
- v0.4.2
- test refc in CI in Nim 2.0 and later
- use non-EOL macOS version for GitHub Actions CI
- avoid failing uninitialized `Future`
- Improve batch call example and wrapper comments
- Fix ws and socket client error handling and add test to #212
- Add build test with chronicles to json enabled
- use non-EOL macOS version for GitHub Actions CI and rm AppVeyor and Travis configs
…5925)

- Dynlib fix for status-go integration
- Add ORC testing for the devel branch
- Upgrade github actions to v4
- Add copyright to source code
- Add push raises
- add missing `{.raises.}` to `addConfigFileContent`
- Refactor p2pProtocol internals
- Fix chronos related deprecated warnings in discv5 and uTP
- normalise nimble, ci
- Fix chronos related deprecated warnings in uTP code part II
- Remove usage of stew/shim/net
- Upgrade github actions to v4
- Fix two bugs in Receipts RLP encoding/decoding
- Move Ethereum specific RLP encoding tests under tests/common
- Reduce compiler warnings in rlp
- test refc in CI in Nim 2.0 and later
- test Nim 2.0 in CI and use non-EOL macOS version
- Use asyncraises in p2p
- Don't auto write p2pProtocol macro expansion to file
- Fix improper yield usage in rlpx and refine exception handling
…5927)

- test refc in CI in Nim 2.0 and later
- use supported version of macOS
- stop testing Nim C++ backend in GitHub Actions
)

- align maximum body size between Websock and HTTP transports
- Upgrade to github actions v4
- Fix 'value' template compilation error
- fix missing import
- Partial instantiation of 'value' template
- bump `csources` to `v1.3.1`
- test refc in CI in Nim 2.0 and later
- use non-EOL macOS version for GitHub Actions CI
…5928)

- test refc in CI in Nim 2.0 and later
- use non-EOL macOS version for GitHub Actions CI
…#5931)

- bump `whereami` to `c5f6b015c36964790493841b9b4623184c0f7c3f`
- require minimum of Nim 1.6
This PR allows sharing the pubkey data between validators by using a
thread-local cache for pubkey data, netting about a 400mb mem usage
reduction on holesky due to us keeping 3 permanent + several ephemeral
state copies in memory at all times and each state copy holding a full
validator.

The PR also introduces a hash cache for the key which gives ~14% speedup
for a full state `hash_tree_root` - the key makes up for a large part of
the `Validator` htr time.

Finally, the time it takes to copy a state goes down as well from ~80m
ms to ~60, for reasons similar to htr.

We use a `ptr` even if a `ref` could in theory have been used - there is
not much practical benefit to a `ref` (given it's mutable) while a `ptr`
is cheaper and easier to copy (when copying temporary states).

We could go further and cache a cooked pubkey but it turns out this is
quite intrusive - in all the relevant places, we're already using a
cooked key from the immutable validator data so there are no immediate
performance gains of doing so while managing the compressed -> cooked
key mapping would become more difficult - something for a future PR
perhaps.

Co-authored-by: Etan Kissling <etan@status.im>
- use non-EOL macOS version for GitHub Actions CI
Extend `{.push raises: [].}` to all files with spec references that are
periodically updated.

- #5942
…5937)

- unittest2.nim: ensure the testTeardownIMPL is performed at the end
- Add test for PR #35
- only run `teardown` if `setup` completed
- Fix allow break in test body regression
- Upgrade github actions to v4
- use non-EOL macOS version for GitHub Actions CI
…5936)

- Resolve ambiguity between chronicles and macros on nim v2
- Upgrade github action to v3
- use non-EOL macOS version for GitHub Actions CI
#5935)

- Upgrade to github actions v4
- Silence the infamous Conversion from itself warnings
#5934)

- bump `miniupnp` to `miniupnpc_2_2_5`
- require minimum of Nim 1.6
- Replace eth_getBlockReceipts return type with Opt[T] instead of Option[T]
- use non-EOL macOS version and fix spelling
- test refc in CI in Nim 2.0 and later
EIP-4844 requires BLST via `nim-kzg4844`; MIRACL Core is not supported.
Therefore, remove support for the MIRACL Core backend.

- status-im/nim-blscurve#170
etan-status and others added 25 commits March 7, 2024 18:42
EIP-4881 was never correctly implemented, the `DepositTreeSnapshot`
structure has nothing to do with its actual definition. Reflect that
by renaming the type to a Nimbus-specific `DepositContractSnapshot`,
so that an actual EIP-4881 implementation can use the correct names.

- https://eips.ethereum.org/EIPS/eip-4881#specification

Notably, `DepositTreeSnapshot` contains a compressed sequence in
`finalized`, only containing the minimally required intermediate roots.

That also explains the incorrect REST response reported in #5508.

The non-canonical representation was introduced in #4303 and is also
persisted in the database. We'll have to maintain it for a while.
This reverts commit 21601f5.

Reason: `/eth/v1/beacon/light_client/updates?start_period=0&count=128`
provides malformed data. Reverting fixes it locally.

Download JSON and SSZ, then use this script to validate (Holesky).
Need to export some symbols for parsing.

```nim
import
  stew/io2,
  ./beacon_chain/spec/[digest, forks, helpers, presets],
  ./beacon_chain/spec/eth2_apis/[eth2_rest_serialization, rest_light_client_calls]

let jsonBytes = io2.readAllBytes("/Users/etan/Downloads/json_updates.json").get()

let
  (cfg, unknowns) = readRuntimeConfig("/Users/etan/Documents/Repos/nimbus-eth2/vendor/holesky/custom_config_data/config.yaml")
  genesis_validators_root = Eth2Digest.fromHex("0x9143aa7c615a7f7115e2b6aac319c03529df8242ae705fba9df39b79c59fa8b1")
  forkDigests = newClone ForkDigests.init(cfg, genesis_validators_root)

let updates = seq[ForkedLightClientUpdate].decodeJsonLightClientObjects(jsonBytes, cfg, forkDigests)

var entries = newSeqOfCap[RestVersioned[ForkedLightClientUpdate]](updates.len)
for update in updates:
  let
    contextEpoch = withForkyUpdate(update):
      when lcDataFork > LightClientDataFork.None:
        forkyUpdate.contextEpoch
      else:
        continue
    contextFork = cfg.consensusForkAtEpoch(contextEpoch)
  entries.add RestVersioned[ForkedLightClientUpdate](
    data: update,
    jsonVersion: contextFork,
    sszContext: forkDigests[].atconsensusFork(contextFork))

let res =
  block:
    var default: seq[byte]
    try:
      var stream = memoryOutput()
      for e in entries:
        withForkyUpdate(e.data):
          when lcDataFork > LightClientDataFork.None:
            var cursor = stream.delayFixedSizeWrite(sizeof(uint64))
            let initPos = stream.pos
            stream.write e.sszContext.data
            var writer = SszWriter.init(stream)
            writer.writeValue forkyUpdate
            cursor.finalWrite (stream.pos - initPos).uint64.toBytesLE()
      stream.getOutput(seq[byte])
    except SerializationError:
      default
    except IOError:
      default

let sszBytes = io2.readAllBytes("/Users/etan/Downloads/ssz_updates.ssz").get()

let sszUpdates = seq[ForkedLightClientUpdate].decodeSszLightClientObjects(res, cfg, forkDigests)
doAssert updates.len == sszUpdates.len
for i in 0 ..< updates.len:
  doAssert updates[i].kind == sszUpdates[i].kind
  withForkyUpdate(updates[i]):
    when lcDataFork > LightClientDataFork.None:
      doAssert forkyUpdate == sszUpdates[i].forky(lcDataFork)

doAssert sszBytes == res
```
- Delete `genesis_deposit_contract_snapshot.ssz` file
- fix conversion error with `or` on futures with `{.async: (raises: []).}`
- Fix `or` deadlock issue.
- Ensure that `OwnCancelSchedule` flag will not be removed from `wait()` and `withTimeout()`.
- Add missing parts of defaults buffer size increase.
- Avoid `ValueError` effect in varargs `race`/`one`
- fix circular reference in timer
- Ensure `transp.reader` is reset to `nil` on error
)

- Introduce custom application error with optional data object
…#6044)

- bump `libbacktrace` to `f41cf1fecbc01966e0a3c32a1a54ff9cb0a19bb9`
- fix(hole-punching-interop): update nim to 1.6.16
- fix: compilation issue when `libp2p_expensive_metrics` is enabled.
- fix: more metrics issues when libp2p_expensive_metrics is enabled
- tests(flaky): Increase check timeouts
- docs: add comments and improve yamux readability
- test(flaky): Log checkExpiring failure
- fix: yamux metrics
- improvement: enhanced checkExpiring macro with custom timeout
- docs(general): Improve docs
Fix the `/eth/v1/beacon/deposit_snapshot` API to produce proper EIP-4881
compatible `DepositTreeSnapshot` responses. The endpoint used to expose
a Nimbus-specific database internal format.

Also fix trusted node sync to consume properly formatted EIP-4881 data
with `--with-deposit-snapshot`, and `--finalized-deposit-tree-snapshot`
beacon node launch option to use the EIP-4881 data. Further ensure that
`ncli_testnet` produces EIP-4881 formatted data for interoperability.
)

Corrects a regression from #5998 that led to crashes in #6046.
In `trustedNodeSync` mode, the config does not contain genesis keys,
so attempting to load from them is a `Defect`.
Provide additional context in the `syncEth1Chain tick` debug log to aid
with understanding of flow when debugging on a more precise basis than
just having the metrics.
* Reapply "ssz: bump (#5996)" (#6037)

This reverts commit a299d17.

Fixed in status-im/nim-faststreams#51

* bump
* allow specifying get_proposer_reward block root at state.slot

* Add consensus_block_value calculation.

* Address review comments.

* Post-rebase adjustments.

* Use proper state to calculate consensus block value.

* Revert "allow specifying get_proposer_reward block root at state.slot"

This reverts commit 9fef9a8.

* Fix post-revert problems.
Return back to Gwei.

* Adding test which is not working.

* Do not use test suite if it does not have post-state.

* Add debug logging.

* Increase logging to track sources of balance changes.

* Fix sync committee rewards/penalties calculation.

* Revert "Increase logging to track sources of balance changes."

This reverts commit 32feb20.

* Adopt new vision to block rewards.

* Add block produce logging to VC.

* Remove rewards.nim.

* Eliminate toWei changes.

* Improve UInt256 shortLog.

* Fix conversion procedure.

* Address review comments.

* Fix test.

* Revert "Fix test."

This reverts commit 4948b2c.

---------

Co-authored-by: tersec <tersec@users.noreply.github.com>
Co-authored-by: Etan Kissling <etan@status.im>
- fix: allFuturesThrowing compilation issue on daily
- feat: Yamux timeout
- Fix empty path crash issue for MultiAddresses `unix`, `ip6zone`, `dns***`.
- fix: move transport interop tests to nim-libp2p repo
- use chronos 4.0.0
- fix typo in `ProtoMessage.toString()`
- increase tolerance of `simple heartbeat` test
- remove unused `MultiBase.encode(..., Cid)` function
- move `allFutureThrowing` helper to tests
- use new Chronos `trackCounter` APIs for leaks checks in tests
- remove unused `LPStreamError` types
- change `SecioError` and `NoiseError` to descendants of `LPStreamError`
- do not log yamux buffers without sanitization (trace log level)
- don't forget closing the stream when final `{Fin}` fails in yamux
- use `race` instead of `or` to avoid lockup
- in yamux, do not write `{Rst}` packet to stream that's in use
- avoid `KeyError` in edge case of yamux handler
- avoid triple lookup in `m.flushed` yamux table
- catch `WebSocketError` in `wstransport`
- define proper parent error type for `YamuxError`
- document known `--mm:orc` crash
- add `{.async: (raises).}` to `libp2p/stream` modules
Signed-off-by: ksr <kaiserd@users.noreply.github.com>
@kaiserd kaiserd closed this Mar 12, 2024
@kaiserd
Copy link
Contributor Author

kaiserd commented Mar 12, 2024

accidentally opened onto stable. Will reopen onto unstable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants