chore: move `libp2p` crate to a dedicated directory #3536

thomaseizinger · 2023-03-02T06:07:23Z

Description

Currently, our top-level Cargo.toml manifest represents a crate AND a workspace. This causes surprising behaviour (e.g. #2949) where we need to explicitly pass --workpace to every command to run it on the entire workspace and not just the meta crate.

My moving the meta crate into its own directory, the root manifest file is a virtual manifest only and thus, every cargo command will automatically default to running on the entire workspace.

On top of this, I personally find it easier to understand if workspace and crate manifests are not mixed.

Notes

This is #3012 but started fresh because it had so many conflicts. Also, I am not renaming any crates in here.

Links to any relevant issues

Open Questions

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

mxinden

Change looks good to me. Feel free to merge whenever you see fit @thomaseizinger.

You could consider adding the final squash commit to https://github.com/libp2p/rust-libp2p/blob/master/.git-blame-ignore-revs.

mergify · 2023-03-07T14:06:51Z

This pull request has merge conflicts. Could you please resolve them @thomaseizinger? 🙏

thomaseizinger · 2023-03-07T18:23:09Z

You could consider adding the final squash commit to https://github.com/libp2p/rust-libp2p/blob/master/.git-blame-ignore-revs.

That is a good idea, I'll send a follow-up once this is merged.

thomaseizinger · 2023-03-08T10:52:00Z

Getting some very strange compile errors here. Somehow, cargo seems to treat these packages differently, now that the root manifest is a virtual workspace. I'll keep digging but that is a good validation to me that we should definitely move into this direction. Who knows how that might affect users.

thomaseizinger · 2023-03-08T11:48:57Z

Getting some very strange compile errors here. Somehow, cargo seems to treat these packages differently, now that the root manifest is a virtual workspace. I'll keep digging but that is a good validation to me that we should definitely move into this direction. Who knows how that might affect users.

No luck so far. Somehow, ring gets pulled in as a dependency as part of snow through the default-resolver feature but only if I compile for wasm32-unknown-emscripten and wasm32-wasi. At least that is what I could gather from the cargo tree output. Weirdly, looking at snow's source code, I cannot see how ring is meant to be a dependency. I am starting to think that this is a bug in cargo.

thomaseizinger · 2023-03-08T11:55:27Z

OMG, I think I found it. What a mess 😂

So previously, our root crate was an edition = 2021 crate which implies version 2 of the dependency resolver. That one is necessary to correctly resolve target specific dependencies (don't ask me why I know that, way too many hours wasted prior to this PR already). Now that it is a virtual manifest, cargo needs to be explicitly told that we want resolver = 2 in order to honor the cfg expressions within libp2p-noise's Cargo.toml.

thomaseizinger · 2023-03-08T12:03:29Z

I opened an issue with cargo: rust-lang/cargo#11811

Related: #3536. Pull-Request: #4984.

* ci: unset `RUSTFLAGS` value in semver job Don't fail semver-checking if a dependency version has warnings, such as deprecation notices. Related: libp2p#4932 (comment). Related: obi1kenobi/cargo-semver-checks#589. Pull-Request: libp2p#4942. * deps(webrtc): bump alpha versions Bumps versions of `libp2p-webrtc` and `libp2p-webrtc-websys` up one minor version. Fixes: libp2p#4953. Pull-Request: libp2p#4959. * feat(request-response): derive `PartialOrd`,`Ord` for `{Out,In}RequestId` Pull-Request: libp2p#4956. * refactor(connection-limits): make `check_limit` a free-function Pull-Request: libp2p#4958. * chore(webrtc-utils): bump version to allow for new release We didn't bump this crate's version despite it depending on `libp2p_noise`. As such, we can't release `libp2p-webrtc-websys` at the moment because it needs a new release of this crate. Pull-Request: libp2p#4968. * feat(webrtc-websys): hide `libp2p_noise` from the public API Currently, `libp2p-webrtc-websys` exposes the `libp2p_noise` dependency in its public API. It should really be a private dependency of the crate. By wrapping it in a new-type, we can achieve this. Pull-Request: libp2p#4969. * fix(kad): iterator progress to be decided by any of new peers Pull-Request: libp2p#4932. * chore(quic): set `max_idle_timeout` to quinn default timeout Resolves libp2p#4917. Pull-Request: libp2p#4965. * feat(core): impl Display on ListenerId Fixes: libp2p#4935. Pull-Request: libp2p#4936. * feat(server): support websocket Pull-Request: libp2p#4937. * feat(swarm): implement `Copy` and `Clone` for `FromSwarm` We can make `FromSwarm` implement `Copy` and `Close` which makes it much easier to a) generate code in `libp2p-swarm-derive` b) manually wrap a `NetworkBehaviour` Previously, we couldn't do this because `ConnectionClosed` would have a `handler` field that cannot be cloned / copied. Related: libp2p#4076. Related: libp2p#4581. Pull-Request: libp2p#4825. * deps: bump wasm-bindgen-futures from 0.4.38 to 0.4.39 Pull-Request: libp2p#4946. * feat(connection-limit): add function to mutate `ConnectionLimits` Resolves: libp2p#4826. Pull-Request: libp2p#4964. * deps: bump web-sys from 0.3.65 to 0.3.66 Pull-Request: libp2p#4976. * deps: bump wasm-bindgen-test from 0.3.38 to 0.3.39 Pull-Request: libp2p#4975. * fix(kad): don't assume `QuerId`s are unique We mistakenly assumed that `QueryId`s are unique in that, only a single request will be emitted per `QueryId`. This is wrong. A bootstrap for example will issue multiple requests as part of the same `QueryId`. Thus, we cannot use the `QueryId` as a key for the `FuturesMap`. Instead, we use a `FuturesTupleSet` to associate the `QueryId` with the in-flight request. Related: libp2p#4901. Resolves: libp2p#4948. Pull-Request: libp2p#4971. * fix(webrtc example): clarify idle connection timeout When I ran the `example/browser-webrtc` example I discovered it would break after a ping or two. The `Ping` idle timeout needed to be extended, on both the server and the wasm client, which is what this PR fixes. I also added a small note to the README about ensuring `wasm-pack` is install for the users who are new to the ecosystem. Fixes: libp2p#4950. Pull-Request: libp2p#4966. * docs(examples/readme): fix broken link Related: libp2p#3536. Pull-Request: libp2p#4984. * feat(yamux): auto-tune (dynamic) stream receive window libp2p/rust-yamux#176 enables auto-tuning for the Yamux stream receive window. While preserving small buffers on low-latency and/or low-bandwidth connections, this change allows for high-latency and/or high-bandwidth connections to exhaust the available bandwidth on a single stream. Using the [libp2p perf](https://github.com/libp2p/test-plans/blob/master/perf/README.md) benchmark tools (60ms, 10Gbit/s) shows an **improvement from 33 Mbit/s to 1.3 Gbit/s** in single stream throughput. See libp2p/rust-yamux#176 for details. To ship the above Rust Yamux change in a libp2p patch release (non-breaking), this pull request uses `yamux` `v0.13` (new version) by default and falls back to `yamux` `v0.12` (old version) when setting any configuration options. Thus default users benefit from the increased performance, while power users with custom configurations maintain the old behavior. Pull-Request: libp2p#4970. * deps: bump actions/deploy-pages from 2 to 3 Pull-Request: libp2p#4978. * deps: bump the axum group with 2 updates Pull-Request: libp2p#4943. * chore(webrtc-websys): remove unused dependencies Pull-Request: libp2p#4973. * chore(quic): fix link to PR in changelog Pull-Request: libp2p#4993. * deps: bump tokio from 1.34.0 to 1.35.0 Pull-Request: libp2p#4995. * deps: bump syn from 2.0.39 to 2.0.40 Pull-Request: libp2p#4996. * deps: bump once_cell from 1.18.0 to 1.19.0 Pull-Request: libp2p#4998. --------- Co-authored-by: Predrag Gruevski <2348618+obi1kenobi@users.noreply.github.com> Co-authored-by: Doug A <douganderson444@gmail.com> Co-authored-by: Darius Clark <dariusc93@users.noreply.github.com> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: maqi <qi.ma@maidsafe.net> Co-authored-by: stormshield-frb <144998884+stormshield-frb@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: NAHO <90870942+trueNAHO@users.noreply.github.com>

* ci: unset `RUSTFLAGS` value in semver job Don't fail semver-checking if a dependency version has warnings, such as deprecation notices. Related: libp2p#4932 (comment). Related: obi1kenobi/cargo-semver-checks#589. Pull-Request: libp2p#4942. * deps(webrtc): bump alpha versions Bumps versions of `libp2p-webrtc` and `libp2p-webrtc-websys` up one minor version. Fixes: libp2p#4953. Pull-Request: libp2p#4959. * feat(request-response): derive `PartialOrd`,`Ord` for `{Out,In}RequestId` Pull-Request: libp2p#4956. * refactor(connection-limits): make `check_limit` a free-function Pull-Request: libp2p#4958. * chore(webrtc-utils): bump version to allow for new release We didn't bump this crate's version despite it depending on `libp2p_noise`. As such, we can't release `libp2p-webrtc-websys` at the moment because it needs a new release of this crate. Pull-Request: libp2p#4968. * feat(webrtc-websys): hide `libp2p_noise` from the public API Currently, `libp2p-webrtc-websys` exposes the `libp2p_noise` dependency in its public API. It should really be a private dependency of the crate. By wrapping it in a new-type, we can achieve this. Pull-Request: libp2p#4969. * fix(kad): iterator progress to be decided by any of new peers Pull-Request: libp2p#4932. * chore(quic): set `max_idle_timeout` to quinn default timeout Resolves libp2p#4917. Pull-Request: libp2p#4965. * feat(core): impl Display on ListenerId Fixes: libp2p#4935. Pull-Request: libp2p#4936. * feat(server): support websocket Pull-Request: libp2p#4937. * feat(swarm): implement `Copy` and `Clone` for `FromSwarm` We can make `FromSwarm` implement `Copy` and `Close` which makes it much easier to a) generate code in `libp2p-swarm-derive` b) manually wrap a `NetworkBehaviour` Previously, we couldn't do this because `ConnectionClosed` would have a `handler` field that cannot be cloned / copied. Related: libp2p#4076. Related: libp2p#4581. Pull-Request: libp2p#4825. * deps: bump wasm-bindgen-futures from 0.4.38 to 0.4.39 Pull-Request: libp2p#4946. * feat(connection-limit): add function to mutate `ConnectionLimits` Resolves: libp2p#4826. Pull-Request: libp2p#4964. * deps: bump web-sys from 0.3.65 to 0.3.66 Pull-Request: libp2p#4976. * deps: bump wasm-bindgen-test from 0.3.38 to 0.3.39 Pull-Request: libp2p#4975. * fix(kad): don't assume `QuerId`s are unique We mistakenly assumed that `QueryId`s are unique in that, only a single request will be emitted per `QueryId`. This is wrong. A bootstrap for example will issue multiple requests as part of the same `QueryId`. Thus, we cannot use the `QueryId` as a key for the `FuturesMap`. Instead, we use a `FuturesTupleSet` to associate the `QueryId` with the in-flight request. Related: libp2p#4901. Resolves: libp2p#4948. Pull-Request: libp2p#4971. * fix(webrtc example): clarify idle connection timeout When I ran the `example/browser-webrtc` example I discovered it would break after a ping or two. The `Ping` idle timeout needed to be extended, on both the server and the wasm client, which is what this PR fixes. I also added a small note to the README about ensuring `wasm-pack` is install for the users who are new to the ecosystem. Fixes: libp2p#4950. Pull-Request: libp2p#4966. * docs(examples/readme): fix broken link Related: libp2p#3536. Pull-Request: libp2p#4984. * feat(yamux): auto-tune (dynamic) stream receive window libp2p/rust-yamux#176 enables auto-tuning for the Yamux stream receive window. While preserving small buffers on low-latency and/or low-bandwidth connections, this change allows for high-latency and/or high-bandwidth connections to exhaust the available bandwidth on a single stream. Using the [libp2p perf](https://github.com/libp2p/test-plans/blob/master/perf/README.md) benchmark tools (60ms, 10Gbit/s) shows an **improvement from 33 Mbit/s to 1.3 Gbit/s** in single stream throughput. See libp2p/rust-yamux#176 for details. To ship the above Rust Yamux change in a libp2p patch release (non-breaking), this pull request uses `yamux` `v0.13` (new version) by default and falls back to `yamux` `v0.12` (old version) when setting any configuration options. Thus default users benefit from the increased performance, while power users with custom configurations maintain the old behavior. Pull-Request: libp2p#4970. * deps: bump actions/deploy-pages from 2 to 3 Pull-Request: libp2p#4978. * deps: bump the axum group with 2 updates Pull-Request: libp2p#4943. * chore(webrtc-websys): remove unused dependencies Pull-Request: libp2p#4973. * chore(quic): fix link to PR in changelog Pull-Request: libp2p#4993. * deps: bump tokio from 1.34.0 to 1.35.0 Pull-Request: libp2p#4995. * deps: bump syn from 2.0.39 to 2.0.40 Pull-Request: libp2p#4996. * deps: bump once_cell from 1.18.0 to 1.19.0 Pull-Request: libp2p#4998. * deps: bump hkdf from 0.12.3 to 0.12.4 Pull-Request: libp2p#5009. * deps: bump clap from 4.4.10 to 4.4.11 Pull-Request: libp2p#4997. * deps: bump thiserror from 1.0.50 to 1.0.51 Pull-Request: libp2p#5010. * deps: bump syn from 2.0.40 to 2.0.41 Pull-Request: libp2p#5011. * deps: bump async-io from 2.2.1 to 2.2.2 Pull-Request: libp2p#5012. * deps: bump rust-embed from 8.0.0 to 8.1.0 Pull-Request: libp2p#5000. * chore(deps): bump golang.org/x/crypto from 0.7.0 to 0.17.0 Pull-Request: libp2p#5019. * deps: bump libc from 0.2.150 to 0.2.151 Pull-Request: libp2p#5002. * docs: remove security@libp2p.io I no longer have access to the mailing list. See libp2p#5007. Pull-Request: libp2p#5020. * chore: fix typos Pull-Request: libp2p#5021. * fix(derive): restore support for inline generic type constraints Fixes the `#[NetworkBehaviour]` macro to support generic constraints on behaviours without a where clause, which was the case before v0.51. Pull-Request: libp2p#5003. * deps: bump actions/deploy-pages from 3 to 4 Pull-Request: libp2p#5022. * chore: fix several typos in documentation Pull-Request: libp2p#5008. * deps: bump async-trait from 0.1.74 to 0.1.75 Pull-Request: libp2p#5029. * deps: bump anyhow from 1.0.75 to 1.0.76 Pull-Request: libp2p#5030. * deps: bump futures-util from 0.3.29 to 0.3.30 Pull-Request: libp2p#5031. * deps: bump syn from 2.0.41 to 2.0.43 Pull-Request: libp2p#5033. * deps: bump tokio from 1.35.0 to 1.35.1 Pull-Request: libp2p#5034. * deps: bump reqwest from 0.11.22 to 0.11.23 Pull-Request: libp2p#5035. * deps: bump futures from 0.3.29 to 0.3.30 Pull-Request: libp2p#5032. * deps: bump trybuild from 1.0.85 to 1.0.86 Pull-Request: libp2p#5036. * deps: bump proc-macro2 from 1.0.69 to 1.0.71 Pull-Request: libp2p#5041. * deps: bump actions/upload-pages-artifact from 2.0.0 to 3.0.0 Pull-Request: libp2p#5023. * deps: bump Rust to 1.75 and fix clippy lints Pull-Request: libp2p#5043. * deps: bump thiserror from 1.0.51 to 1.0.53 Pull-Request: libp2p#5044. * deps: bump clap from 4.4.11 to 4.4.12 Pull-Request: libp2p#5046. * deps: bump tempfile from 3.8.1 to 3.9.0 Pull-Request: libp2p#5047. * deps: bump rust-embed from 8.1.0 to 8.2.0 Pull-Request: libp2p#5049. * deps: bump serde_json from 1.0.108 to 1.0.109 Pull-Request: libp2p#5050. * deps: bump anyhow from 1.0.76 to 1.0.78 Pull-Request: libp2p#5051. * deps: bump proc-macro2 from 1.0.71 to 1.0.73 Pull-Request: libp2p#5054. * deps: bump quote from 1.0.33 to 1.0.34 Pull-Request: libp2p#5055. * deps: bump anyhow from 1.0.78 to 1.0.79 Pull-Request: libp2p#5062. * deps: bump serde_json from 1.0.109 to 1.0.111 Pull-Request: libp2p#5063. * deps: bump thiserror from 1.0.53 to 1.0.56 Pull-Request: libp2p#5064. * deps: bump libc from 0.2.151 to 0.2.152 Pull-Request: libp2p#5065. * deps: bump trybuild from 1.0.86 to 1.0.88 Pull-Request: libp2p#5068. * deps: bump proc-macro2 from 1.0.73 to 1.0.76 Pull-Request: libp2p#5069. * deps: bump clap from 4.4.12 to 4.4.13 Pull-Request: libp2p#5070. * deps: bump Swatinem/rust-cache from 2.7.1 to 2.7.2 Pull-Request: libp2p#5076. * deps: bump tj-actions/glob from 17 to 18 Pull-Request: libp2p#5058. * deps: bump the axum group with 1 update Pull-Request: libp2p#5045. * deps: bump quote from 1.0.34 to 1.0.35 Pull-Request: libp2p#5071. * deps: bump async-trait from 0.1.75 to 0.1.77 Pull-Request: libp2p#5081. * ci: add dependabot group for webrtc Pull-Request: libp2p#5082. * deps: bump base64 from 0.21.5 to 0.21.7 Pull-Request: libp2p#5086. * deps: bump trybuild from 1.0.88 to 1.0.89 Pull-Request: libp2p#5087. * deps: bump js-sys from 0.3.66 to 0.3.67 Pull-Request: libp2p#5091. * deps: bump wasm-bindgen from 0.2.89 to 0.2.90 Pull-Request: libp2p#5089. * add PeerId to ListenFailure --------- Co-authored-by: Predrag Gruevski <2348618+obi1kenobi@users.noreply.github.com> Co-authored-by: Doug A <douganderson444@gmail.com> Co-authored-by: Darius Clark <dariusc93@users.noreply.github.com> Co-authored-by: zhiqiangxu <652732310@qq.com> Co-authored-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: maqi <qi.ma@maidsafe.net> Co-authored-by: stormshield-frb <144998884+stormshield-frb@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: NAHO <90870942+trueNAHO@users.noreply.github.com> Co-authored-by: alex <152680487+bodhi-crypo@users.noreply.github.com> Co-authored-by: Akosh Farkash <aakoshh@gmail.com> Co-authored-by: Frieren <153332328+Frierened@users.noreply.github.com>

thomaseizinger added 2 commits March 2, 2023 17:03

Move libp2p crate to a dedicated directory

0e29efe

Move changelog

36ce4cb

thomaseizinger requested a review from mxinden March 2, 2023 06:07

This comment was marked as resolved.

Sign in to view

Merge branch 'master' into feat/root-manifest-virtual

6d07373

This comment was marked as resolved.

Sign in to view

Merge branch 'master' into feat/root-manifest-virtual

d80a809

mxinden approved these changes Mar 7, 2023

View reviewed changes

thomaseizinger added 5 commits March 8, 2023 11:21

Merge branch 'master' into feat/root-manifest-virtual

1f3dece

Add license to interop-tests

ec5b790

Move uds to wasm gated transports

ef9b720

Pin pre-release versions again

8c61dbc

Restore order of members keys

013b53b

License examples

641cda1

Manually specify resolver

68d254c

thomaseizinger mentioned this pull request Mar 8, 2023

Moving repository to a virtual manifest can cause surprising dependency resolution errors due to resolver fallback rust-lang/cargo#11811

Closed

thomaseizinger added the send-it label Mar 8, 2023

Merge branch 'master' into feat/root-manifest-virtual

a6eed5e

mergify bot merged commit d53cfed into master Mar 8, 2023

mergify bot deleted the feat/root-manifest-virtual branch March 8, 2023 13:20

trueNAHO mentioned this pull request Dec 5, 2023

docs(examples/readme): fix broken link #4984

Merged

4 tasks

mergify bot pushed a commit that referenced this pull request Dec 5, 2023

docs(examples/readme): fix broken link

06d80fb

Related: #3536. Pull-Request: #4984.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: move `libp2p` crate to a dedicated directory #3536

chore: move `libp2p` crate to a dedicated directory #3536

thomaseizinger commented Mar 2, 2023 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

mxinden left a comment

mergify bot commented Mar 7, 2023

thomaseizinger commented Mar 7, 2023

thomaseizinger commented Mar 8, 2023

thomaseizinger commented Mar 8, 2023

thomaseizinger commented Mar 8, 2023

thomaseizinger commented Mar 8, 2023

chore: move libp2p crate to a dedicated directory #3536

chore: move libp2p crate to a dedicated directory #3536

Conversation

thomaseizinger commented Mar 2, 2023 • edited Loading

Description

Notes

Links to any relevant issues

Open Questions

Change checklist

This comment was marked as resolved.

This comment was marked as resolved.

mxinden left a comment

Choose a reason for hiding this comment

mergify bot commented Mar 7, 2023

thomaseizinger commented Mar 7, 2023

thomaseizinger commented Mar 8, 2023

thomaseizinger commented Mar 8, 2023

thomaseizinger commented Mar 8, 2023

thomaseizinger commented Mar 8, 2023

chore: move `libp2p` crate to a dedicated directory #3536

chore: move `libp2p` crate to a dedicated directory #3536

thomaseizinger commented Mar 2, 2023 •

edited

Loading