Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add bottlerocket tools to twoliter binary #47

Merged
merged 7 commits into from
Sep 14, 2023
Merged

add bottlerocket tools to twoliter binary #47

merged 7 commits into from
Sep 14, 2023

Conversation

webern
Copy link
Contributor

@webern webern commented Sep 5, 2023 •
edited
Loading

Issue #, if available:

Closes #14
Closes #9
Closes #32
Partly Addresses #43 (We install it here, but should probably still eliminate it in favor of a lib)

Description of changes:

Embed tools such as buildsys, and scripts such as rpm2img into the Twoliter binary. Use them when building Bottlerocket

Testing

  • Build Bottlerocket on an x86_64 host
  • Build Bottlerocket on an Arm host

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@webern webern marked this pull request as draft September 5, 2023 23:15
webern
webern commented Sep 6, 2023
View reviewed changes
.cargo/config.toml Outdated Show resolved Hide resolved
tools/buildsys/src/builder.rs Show resolved Hide resolved
tools/buildsys/src/builder.rs Outdated Show resolved Hide resolved
tools/buildsys/src/gomod.rs Outdated Show resolved Hide resolved
tools/buildsys/src/gomod.rs Outdated Show resolved Hide resolved
twoliter/embedded/Dockerfile Outdated Show resolved Hide resolved
twoliter/embedded/Makefile.toml Outdated Show resolved Hide resolved
twoliter/src/cmd/make.rs Outdated Show resolved Hide resolved
twoliter/src/docker/image.rs Outdated Show resolved Hide resolved
twoliter/src/tools/mod.rs Outdated Show resolved Hide resolved
@webern webern changed the title move bottlerocket tools to twoliter add bottlerocket tools to twoliter binary Sep 6, 2023
@webern
Copy link
Contributor Author

webern commented Sep 6, 2023

https://github.com/bottlerocket-os/twoliter/compare/03e54a7a9a6f157f339e99d65a811bc50b1d09e1..b6c81c329538bc1259413081d604fd2646419f66

Clean some things up.

@webern webern force-pushed the tools-install branch 2 times, most recently from ae17df4 to 139880e Compare September 7, 2023 15:42
@webern webern force-pushed the tools-install branch 3 times, most recently from 49c31da to 9035614 Compare September 7, 2023 16:27
@webern webern force-pushed the tools-install branch 2 times, most recently from 82cb765 to 6ebb528 Compare September 7, 2023 18:44
bcressey
bcressey approved these changes Sep 7, 2023
View reviewed changes
tools/buildsys/src/builder.rs Show resolved Hide resolved
twoliter/embedded/Dockerfile Outdated Show resolved Hide resolved
twoliter/embedded/Makefile.toml Outdated Show resolved Hide resolved
twoliter/embedded/Makefile.toml Show resolved Hide resolved
twoliter/embedded/Makefile.toml Outdated Show resolved Hide resolved
twoliter/build.rs Show resolved Hide resolved
@webern webern mentioned this pull request Sep 8, 2023
Merged
18 tasks
@webern
Copy link
Contributor Author

webern commented Sep 8, 2023

https://github.com/bottlerocket-os/twoliter/compare/6ebb528c1b64159e6242e897d08784e82381b069..8a945cf94116f92a32ab08191bb0410c298794d6

Split commits

@webern
Copy link
Contributor Author

webern commented Sep 8, 2023

https://github.com/bottlerocket-os/twoliter/compare/8a945cf94116f92a32ab08191bb0410c298794d6..c6b7e0961a849d2fbd6a330457a9efa2794090f9

Fixup bcressey's suggestions

cbgbt
cbgbt approved these changes Sep 8, 2023
View reviewed changes
.github/workflows/rust.yml Outdated Show resolved Hide resolved
@webern webern force-pushed the tools-install branch 2 times, most recently from cda4f71 to 544cf34 Compare September 11, 2023 22:55
@webern webern force-pushed the tools-install branch 3 times, most recently from 5406dc0 to 14a9919 Compare September 13, 2023 03:10
@webern
Copy link
Contributor Author

webern commented Sep 13, 2023

https://github.com/bottlerocket-os/twoliter/compare/5406dc0f0187d491c24103f23a6c2f29d2cf3376..14a9919c7609da32e8551c03558104d7c09a608d

Substantial rework of how the tools are embedded. As it turns out, and this makes sense, build-dependencies are always built for the host machine. We need the bindeps to be dependencies so they will be built for the --target.

I cannot figure out a way to compress these bytes right now. The include-flate macro requires the path to be an actual string literal (not a macro like env! that evaluates to a string literal). This might be an oversight (not sure how it all works) that is fixable. But also note that the compilation of include-flate seemed to take forever. I'm not sure what is going on there.

cbgbt
cbgbt approved these changes Sep 13, 2023
View reviewed changes
rust-toolchain.toml Outdated Show resolved Hide resolved
bcressey
bcressey reviewed Sep 13, 2023
View reviewed changes
Copy link
Contributor

@bcressey bcressey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to reduce twoliter size a bit, I tested combinations of these settings in Cargo.toml:

[profile.release]
strip = "debuginfo"
codegen-units = 1
lto = true

At a bare minimum, I'd recommend stripping debuginfo.

Setting codegen units to 1 is the biggest one but costs the most in terms of added time. If we go there then we may as well go full LTO for the extra 18% size reduction. This seems worthwhile if our release builders can handle it, but definitely not for CI.

@webern
Copy link
Contributor Author

webern commented Sep 13, 2023

https://github.com/bottlerocket-os/twoliter/compare/552f9243c9d7282219fe53e4cd33aab5a821cc6e..9af3a998650d46758e1b34990e39fc23f5f7a7f7

Add release profile with one codegen unit.

@webern
Copy link
Contributor Author

webern commented Sep 13, 2023 •
edited
Loading

https://github.com/bottlerocket-os/twoliter/compare/9af3a998650d46758e1b34990e39fc23f5f7a7f7..84b795a80985ea12524b7d6a75d37e2ac3ef1eb6

Rebase over the latest tools changes from Bottlerocket.

Includes:

  • rpm2img xfs fix
  • testsys changes

@webern
Copy link
Contributor Author

webern commented Sep 13, 2023

At a bare minimum, I'd recommend stripping debuginfo.

Nice. This change took the binary from 380M to 80M!

@webern
Copy link
Contributor Author

webern commented Sep 13, 2023

https://github.com/bottlerocket-os/twoliter/compare/84b795a80985ea12524b7d6a75d37e2ac3ef1eb6..158e5179fe5e55e39c5e0b505ba598fc8e0aeee7

Fix a typo found by Sean.

@webern webern marked this pull request as ready for review September 13, 2023 22:55
@webern
Copy link
Contributor Author

webern commented Sep 13, 2023 •
edited
Loading

a527e07

Tacked on a commit to stop the unhelpful printing of the entire cargo make invocation which makes it harder for people to find the real error.

@webern
remove an unused item from .gitignore
5c777a4
@webern
use large github actions runners
8fe3131
@webern
add bottlerocket build tools to twoliter
f1ec39f 
Adds Rust tools and scripts, originally located in the Bottlerocket
repo, to Twoliter's embedded tools.
@webern
buildsys: integrate with embedded twoliter tools
20b1b3b 
Buildsys requires these changes to work with the tools embedded in the
Twoliter binary.
@webern
add debug command to twoliter
7a0315d 
Adds a command that can be used to dump the tools for further
inspection.
@webern
add version flag to twoliter
2e431c8 
Enable printing the current version with twoliter --version
@webern
stop printing the entire cargo make command
ace7b30 
It is not a good idea to print the entire cargo make command when an
error occurs. This is making it harder for people to find the real'
error and could reveal secrets. Best to be quiet about what command was
invoked under the hood.
@webern
Copy link
Contributor Author

webern commented Sep 14, 2023

https://github.com/bottlerocket-os/twoliter/compare/a527e076c885834efd25a22479c714f4cec21c70..ace7b30e6400df5a9af943dfe25b4ebd2cf10536

Rebase to pickup changes from Bottlerocket.

Base automatically changed from tools to develop September 14, 2023 00:10
@webern webern merged commit 2e0741a into develop Sep 14, 2023
1 check passed
@webern webern deleted the tools-install branch September 14, 2023 00:12
This was referenced Sep 18, 2023
Closed
Closed
@webern webern mentioned this pull request Oct 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bcressey bcressey bcressey approved these changes

@cbgbt cbgbt cbgbt approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

change build tools install dir twoliter make can build bottlerocket copy tools
3 participants
@webern @bcressey @cbgbt