feat: migrate to quick-protobuf #3312

kckeiks · 2023-01-07T01:23:15Z

Description

Instead of relying on protoc and buildscripts, we generate the bindings using pb-rs and version them within our codebase. This makes for a better IDE integration, a faster build and an easier use of rust-libp2p because we don't force the protoc dependency onto them.

Resolves #3024.

Change checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
A changelog entry has been made in the appropriate crates

thomaseizinger

Nice, this is what I was looking for, thanks!

Once we delete the other proto files, those should hopefully show up as moved rather than added.

.github/workflows/ci.yml

thomaseizinger · 2023-01-08T23:52:46Z

Dang, just saw CI doesn't actually like this. I am no bash expert but maybe try globbing into a variable first?

mxinden

Thanks to both of you for the work here!

core/src/generated/mod.rs

kckeiks · 2023-01-11T20:43:27Z

Dang, just saw CI doesn't actually like this. I am no bash expert but maybe try globbing into a variable first?

I'll try that. I'm no expert on that either so I will probably have to push quite a bit to test it so I apologize in advance. Also, will update this PR soon.

kckeiks · 2023-01-12T03:21:55Z

Looks like it works https://github.com/libp2p/rust-libp2p/actions/runs/3898831952/jobs/6657940569

kckeiks · 2023-01-12T03:23:50Z

Looks like the diff is not working though. This run should have failed. https://github.com/libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252

thomaseizinger · 2023-01-12T05:14:14Z

Looks like the diff is not working though. This run should have failed. libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252

Why did you expect it to fail? Did you modify any of the generated code in a commit?

kckeiks · 2023-01-12T15:43:21Z

Looks like the diff is not working though. This run should have failed. libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252

Why did you expect it to fail? Did you modify any of the generated code in a commit?

I modified the glob so every *.proto file should be getting compiled including the ones outside /generated.

thomaseizinger · 2023-01-12T23:04:11Z

Looks like the diff is not working though. This run should have failed. libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252

Why did you expect it to fail? Did you modify any of the generated code in a commit?

I modified the glob so every *.proto file should be getting compiled including the ones outside /generated.

The problem is that those are untracked files so they don't seem to "fail" the git diff command. I tried by modifying an existing generated file, committing it and git diff fails properly on those.

This is unnecessary as we can see the command being executed in the action log. See https://github.com/libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252#step:5:4.

This reverts commit 391a892.

kckeiks · 2023-01-12T23:33:31Z

Looks like the diff is not working though. This run should have failed. libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252

Why did you expect it to fail? Did you modify any of the generated code in a commit?

I modified the glob so every *.proto file should be getting compiled including the ones outside /generated.

The problem is that those are untracked files so they don't seem to "fail" the git diff command. I tried by modifying an existing generated file, committing it and git diff fails properly on those.

Thanks for the help!

Are we good with the CI then?
What is the plan with prost-codec? Will it be removed by another PR? Migrate from protoc to quick-protobuf #3066 (comment)

mergify · 2023-02-28T02:13:43Z

This pull request has merge conflicts. Could you please resolve them @kckeiks? 🙏

thomaseizinger · 2023-03-01T03:08:10Z

@mxinden This is ready to merge from my end. You will need to merge libp2p/github-mgmt#128 before you can merge here to update the required jobs.

mergify · 2023-03-01T04:36:54Z

This pull request has merge conflicts. Could you please resolve them @kckeiks? 🙏

Approvals have been dismissed because the PR was updated after the send-it label was applied.

thomaseizinger · 2023-03-02T11:19:19Z

I think some of the changelog diffs have been applied to the wrong version as of the last release now.

mxinden · 2023-03-02T11:23:52Z

Good catch. @thomaseizinger can you send a pull request in the next couple of days?

mxinden · 2023-03-02T11:24:17Z

Thanks @kckeiks and @thomaseizinger for all the work here!

thomaseizinger · 2023-03-02T21:15:30Z

Good catch. @thomaseizinger can you send a pull request in the next couple of days?

My mind is engineering a semverlog tool as we speak to avoid this.

I can send a PR although it might take a while, have a busy weekend ahead and flying to Europe on Tuesday morning.

thomaseizinger · 2023-03-03T02:43:43Z

Sent it here: #3541

Whilst #3312 was in development, we pushed a new release out and forgot to move the changelog entries to the new version. Unfortunately, this is all still very manual until we have a solution for #2902 so this stuff keeps happening. Pull-Request: #3541.

Instead of relying on `protoc` and buildscripts, we generate the bindings using `pb-rs` and version them within our codebase. This makes for a better IDE integration, a faster build and an easier use of `rust-libp2p` because we don't force the `protoc` dependency onto them. Resolves libp2p#3024. Pull-Request: libp2p#3312.

Whilst libp2p#3312 was in development, we pushed a new release out and forgot to move the changelog entries to the new version. Unfortunately, this is all still very manual until we have a solution for libp2p#2902 so this stuff keeps happening. Pull-Request: libp2p#3541.

Hasn't been needed since libp2p 0.51.1. libp2p/rust-libp2p#3312

kckeiks added 4 commits January 6, 2023 19:13

Add generated directory

c250f68

Add generated code

cac57a0

Add ci

44b3465

Try adding quotes

4dca4d0

kckeiks mentioned this pull request Jan 7, 2023

Migrate from protoc to quick-protobuf #3066

Closed

4 tasks

thomaseizinger approved these changes Jan 8, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

.github/workflows/ci.yml Outdated Show resolved Hide resolved

mxinden reviewed Jan 10, 2023

View reviewed changes

core/src/generated/mod.rs Show resolved Hide resolved

kckeiks added 9 commits January 11, 2023 21:30

Add input variable

6b19385

Move inputs

9d4b864

Move inputs 2

c9853f7

Glob into variable

9716fe5

Glob into variable 2

1d1dff0

Use glob

b2aabcd

Remove quotes

559a1d3

Remove refs

d62b50d

Test glob pattern

b894ac7

Bad commit

391a892

thomaseizinger added 5 commits January 13, 2023 10:04

Remove explicit echo

2991994

This is unnecessary as we can see the command being executed in the action log. See https://github.com/libp2p/rust-libp2p/actions/runs/3898847742/jobs/6657969252#step:5:4.

Fail CI on untracked files

8a25c84

Revert "Bad commit"

ccc4541

This reverts commit 391a892.

Cache installed pb-rs binary

f464e3b

Show bad proto files to the user

d2ed2c6

thomaseizinger mentioned this pull request Feb 27, 2023

Reduce dependency tree #3515

Open

kckeiks added 2 commits February 28, 2023 21:18

Merge branch 'master' into add-quickprotobuf-take2

08ce33f

Make an error type that does not expose anything

dfc55b8

kckeiks mentioned this pull request Mar 1, 2023

Wrap quick-protobuf::Error #3534

Closed

obi1kenobi mentioned this pull request Mar 1, 2023

Filling in state of art section tonowak/bachelors-thesis#9

Merged

Merge branch 'master' into add-quickprotobuf-take2

99cee6e

mxinden previously approved these changes Mar 2, 2023

View reviewed changes

Merge branch 'master' into add-quickprotobuf-take2

f939e06

mxinden added the send-it label Mar 2, 2023

Merge branch 'master' into add-quickprotobuf-take2

d2be0bf

mergify bot merged commit db82e02 into libp2p:master Mar 2, 2023

thomaseizinger mentioned this pull request Mar 3, 2023

fix: move changelog entries to correct version #3541

Merged

4 tasks

bajtos mentioned this pull request Mar 30, 2023

feat: remove protoc dependency filecoin-station/zinnia#165

Merged

kckeiks mentioned this pull request Apr 26, 2023

Refactor: Unix domain sockets for abci communication fleek-network/ursa#526

Closed

3 tasks

hrxi added a commit to nimiq/core-rs-albatross that referenced this pull request May 3, 2024

protoc isn't needed by libp2p anymore

ce8530c

Hasn't been needed since libp2p 0.51.1. libp2p/rust-libp2p#3312

hrxi mentioned this pull request May 3, 2024

protoc isn't needed by libp2p anymore nimiq/core-rs-albatross#2426

Merged

jsdanielh pushed a commit to nimiq/core-rs-albatross that referenced this pull request May 3, 2024

protoc isn't needed by libp2p anymore

bdbf5e9

Hasn't been needed since libp2p 0.51.1. libp2p/rust-libp2p#3312

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: migrate to quick-protobuf #3312

feat: migrate to quick-protobuf #3312

kckeiks commented Jan 7, 2023 •

edited by thomaseizinger

Loading

thomaseizinger left a comment

thomaseizinger commented Jan 8, 2023

mxinden left a comment

kckeiks commented Jan 11, 2023

kckeiks commented Jan 12, 2023

kckeiks commented Jan 12, 2023

thomaseizinger commented Jan 12, 2023

kckeiks commented Jan 12, 2023

thomaseizinger commented Jan 12, 2023

kckeiks commented Jan 12, 2023

mergify bot commented Feb 28, 2023

thomaseizinger commented Mar 1, 2023

mergify bot commented Mar 1, 2023

thomaseizinger commented Mar 2, 2023

mxinden commented Mar 2, 2023 •

edited

Loading

mxinden commented Mar 2, 2023

thomaseizinger commented Mar 2, 2023

thomaseizinger commented Mar 3, 2023

feat: migrate to quick-protobuf #3312

feat: migrate to quick-protobuf #3312

Conversation

kckeiks commented Jan 7, 2023 • edited by thomaseizinger Loading

Description

Change checklist

thomaseizinger left a comment

Choose a reason for hiding this comment

thomaseizinger commented Jan 8, 2023

mxinden left a comment

Choose a reason for hiding this comment

kckeiks commented Jan 11, 2023

kckeiks commented Jan 12, 2023

kckeiks commented Jan 12, 2023

thomaseizinger commented Jan 12, 2023

kckeiks commented Jan 12, 2023

thomaseizinger commented Jan 12, 2023

kckeiks commented Jan 12, 2023

mergify bot commented Feb 28, 2023

thomaseizinger commented Mar 1, 2023

mergify bot commented Mar 1, 2023

thomaseizinger commented Mar 2, 2023

mxinden commented Mar 2, 2023 • edited Loading

mxinden commented Mar 2, 2023

thomaseizinger commented Mar 2, 2023

thomaseizinger commented Mar 3, 2023

kckeiks commented Jan 7, 2023 •

edited by thomaseizinger

Loading

mxinden commented Mar 2, 2023 •

edited

Loading