Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pubsys: validate-repo without tokio runtime #1521

Merged
merged 1 commit into from
Apr 24, 2021
Merged

pubsys: validate-repo without tokio runtime #1521

merged 1 commit into from
Apr 24, 2021

Conversation

webern
Copy link
Contributor

@webern webern commented Apr 23, 2021

Issue number:

Description of changes:

We were seeing pubsys validate-repo hang and we believe it was because of multiple tokio runtimes. Although it's not clear why tokio::spawn was working before, re-implementing this to use std::thread::spawn instead has fixed the problem.

Testing done:

Created an Infra.toml like this:

[repo.2020-07-07]
metadata_base_url = "https://updates.bottlerocket.aws/2020-07-07/"
targets_url = "https://updates.bottlerocket.aws/targets/"
root_role_url = "https://cache.bottlerocket.aws/root.json"
root_role_sha512 = "90393204232a1ad6b0a45528b1f7df1a3e37493b1e05b1c149f081849a292c8dafb4ea5f7ee17bcc664e35f66e37e4cfa4aae9de7a2a28aa31ae6ac3d9bea4d5"

And this a few times:

cargo make validate-repo \
    -e PUBLISH_REPO=2020-07-07 \
    -e BUILDSYS_VARIANT=aws-k8s-1.15 \
    -e BUILDSYS_ARCH=x86_64 \
    -e PUBLISH_INFRA_CONFIG_PATH="${INFRA_TOML}"

I also edited a target in a repo and verified that the program errors out when the repo is bad:

Failed to validate repository: Failed to download and write target 'migrate_v1.0.6_metricdog-init.lz4': Hash mismatch for https://somewhere.amazonaws.com/somewhere/targets/17a0a6e4f95471734b77091212ccabf564d3312062fc98c995002df2fbf7c6db.migrate_v1.0.6_metricdog-init.lz4: calculated 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae, expected 17a0a6e4f95471734b77091212ccabf564d3312062fc98c995002df2fbf7c6db

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Copy link
Contributor

@tjkirch tjkirch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I'd like to see a test confirming that bad cases still fail as we expect, though, ideally. ✔️

tools/pubsys/src/repo/validate_repo/mod.rs Outdated Show resolved Hide resolved
tools/pubsys/src/repo/validate_repo/mod.rs Show resolved Hide resolved
Re-implement the parallelism of pubsys validate-repo so that it does not
need a tokio runtime. The runtime was interfering with the runtime
created by reqwest::blocking.
@webern webern merged commit 010745f into bottlerocket-os:develop Apr 24, 2021
@webern webern deleted the de-async-repo-validation branch May 6, 2021 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants