Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[vcpkg scripts] Simplify vcpkg_download_distfile output handling for Asset Caching #40945

Closed
wants to merge 4 commits into from
Closed

[vcpkg scripts] Simplify vcpkg_download_distfile output handling for Asset Caching #40945

wants to merge 4 commits into from

Conversation

JavierMatosD
Copy link
Contributor

This PR updates the vcpkg_download_distfile script to remove direct messaging and rely on output from the vcpkg executable. This change ensures that messages consider asset caching.

Before:
(Example log with redundant messaging)

~/dev/vcpkg % vcpkg install fmt
Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Downloading https://github.com/Kitware/CMake/releases/download/v3.29.2/cmake-3.29.2-macos-universal.tar.gz
Extracting cmake...
The following packages will be built and installed:
    fmt:arm64-osx@10.2.1#2
  * vcpkg-cmake:arm64-osx@2024-04-23
  * vcpkg-cmake-config:arm64-osx@2024-05-23
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet arm64-osx...
Compiler found: /Library/Developer/CommandLineTools/usr/bin/c++
Restored 0 package(s) from /Users/javiermatos/.cache/vcpkg/archives in 7.58 us. Use --debug to see more details.
Installing 1/3 vcpkg-cmake:arm64-osx@2024-04-23...
Building vcpkg-cmake:arm64-osx@2024-04-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/copyright
-- Performing post-build validation
Stored binaries in 1 destinations in 16 ms.
Elapsed time to handle vcpkg-cmake:arm64-osx: 57.2 ms
vcpkg-cmake:arm64-osx package ABI: ab01a101fdcc710bfb6d8a1b6cc2824f8a5d1ea5fcc7d0f54f1b52df68c7da50
Installing 2/3 vcpkg-cmake-config:arm64-osx@2024-05-23...
Building vcpkg-cmake-config:arm64-osx@2024-05-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/copyright
-- Skipping post-build validation due to VCPKG_POLICY_EMPTY_PACKAGE
Stored binaries in 1 destinations in 14.2 ms.
Elapsed time to handle vcpkg-cmake-config:arm64-osx: 196 ms
vcpkg-cmake-config:arm64-osx package ABI: 2e73acf69b779ce7be5136d726e4307ae589be20e90cf4a06baaac8633fd82a7
Installing 3/3 fmt:arm64-osx@10.2.1#2...
Building fmt:arm64-osx@10.2.1#2...
-- Downloading https://github.com/fmtlib/fmt/archive/10.2.1.tar.gz -> fmtlib-fmt-10.2.1.tar.gz...
[DEBUG] To include the environment variables in debug output, pass --debug-env
[DEBUG] Trying to load bundleconfig from /Users/javiermatos/dev/vcpkg/vcpkg-bundle.json
[DEBUG] Failed to open: /Users/javiermatos/dev/vcpkg/vcpkg-bundle.json
[DEBUG] Bundle config: readonly=false, usegitregistry=false, embeddedsha=nullopt, deployment=Git, vsversion=nullopt
[DEBUG] Metrics enabled.
[DEBUG] Feature flag 'binarycaching' unset
[DEBUG] Feature flag 'compilertracking' unset
[DEBUG] Feature flag 'registries' unset
[DEBUG] Feature flag 'versions' unset
[DEBUG] Feature flag 'dependencygraph' unset
[DEBUG] 1000: execute_process(curl --fail -L https://github.com/fmtlib/fmt/archive/10.2.1.tar.gz --create-dirs --output /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz.20477.part)
[DEBUG] 1000: cmd_execute_and_stream_data() returned 0 after   566531 us
[DEBUG] Trying to hash /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz.20477.part
[DEBUG] /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz.20477.part has hash 27df90c681ec37e55625062a79e3b83589b6d7e94eff37a3b412bb8c1473f757a8adb727603acc9185c3490628269216843b7d7bd5a3cb37f0029da5d1495ffa
Downloading https://github.com/fmtlib/fmt/archive/10.2.1.tar.gz
[DEBUG] /Users/runner/work/1/s/src/vcpkg/commands.download.cpp(148): 
[DEBUG] Time in subprocesses: 566531us
[DEBUG] Time in parsing JSON: 4us
[DEBUG] Time in JSON reader: 0us
[DEBUG] Time in filesystem: 1362us
[DEBUG] Time in loading ports: 0us
[DEBUG] Exiting after 578 ms (575746us)

-- Extracting source /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-10.2.1.tar.gz
-- Applying patch fix-visibility.patch
-- Applying patch fix-write-batch.patch
-- Using source at /Users/javiermatos/dev/vcpkg/buildtrees/fmt/src/10.2.1-a991065f88.clean
-- Found external ninja('1.11.1').
-- Configuring arm64-osx
-- Building arm64-osx-dbg
-- Building arm64-osx-rel
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/lib/pkgconfig/fmt.pc
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/debug/lib/pkgconfig/fmt.pc
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/usage
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/copyright
-- Performing post-build validation
Stored binaries in 1 destinations in 70.5 ms.
Elapsed time to handle fmt:arm64-osx: 3.8 s
fmt:arm64-osx package ABI: a6d6451d7953939a412dcafadbd5f80ab424b3c39a37dd8f6fa0d13530682c71
Total install time: 4.1 s
The package fmt provides CMake targets:

    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt)

    # Or use the header-only version
    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt-header-only)

~/dev/vcpkg %

After:
(Example log with streamlined output that reflects asset caching)

~/dev/vcpkg % vcpkg install fmt --no-binarycaching                 
Computing installation plan...
A suitable version of cmake was not found (required v3.29.2).
Asset cache hit for cmake-3.29.2-macos-universal.tar.gz; downloaded from: file:///Users/javiermatos/dev/asset-cache/26aab0163965f3d560dfd6b1f72c5e77192338237ebe286099fd62f243f1bbd4857b9193118386b801c00dc5cfbc5bc8af84814692fcfadcf56c7d2faab52533
Extracting cmake...
The following packages will be built and installed:
    fmt:arm64-osx@11.0.2
  * vcpkg-cmake:arm64-osx@2024-04-23
  * vcpkg-cmake-config:arm64-osx@2024-05-23
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet arm64-osx...
Compiler found: /Library/Developer/CommandLineTools/usr/bin/c++
Installing 1/3 vcpkg-cmake:arm64-osx@2024-04-23...
Building vcpkg-cmake:arm64-osx@2024-04-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake_arm64-osx/share/vcpkg-cmake/copyright
-- Performing post-build validation
Elapsed time to handle vcpkg-cmake:arm64-osx: 61.1 ms
vcpkg-cmake:arm64-osx package ABI: 44ec445be8ff2dfe41221664683af0e4d54a584531320d608d72d533f9403db8
Installing 2/3 vcpkg-cmake-config:arm64-osx@2024-05-23...
Building vcpkg-cmake-config:arm64-osx@2024-05-23...
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/vcpkg-port-config.cmake
-- Installing: /Users/javiermatos/dev/vcpkg/packages/vcpkg-cmake-config_arm64-osx/share/vcpkg-cmake-config/copyright
-- Skipping post-build validation due to VCPKG_POLICY_EMPTY_PACKAGE
Elapsed time to handle vcpkg-cmake-config:arm64-osx: 126 ms
vcpkg-cmake-config:arm64-osx package ABI: 66faabcff87a98b7f4eee82713f982c1d16b7890aa2e128d751dddeab46afd0e
Installing 3/3 fmt:arm64-osx@11.0.2...
Building fmt:arm64-osx@11.0.2...
Asset cache hit for fmtlib-fmt-11.0.2.tar.gz; downloaded from: file:///Users/javiermatos/dev/asset-cache/47ff6d289dcc22681eea6da465b0348172921e7cafff8fd57a1540d3232cc6b53250a4625c954ee0944c87963b17680ecbc3ea123e43c2c822efe0dc6fa6cef3
-- Extracting source /Users/javiermatos/dev/vcpkg/downloads/fmtlib-fmt-11.0.2.tar.gz
-- Applying patch fix-write-batch.patch
-- Using source at /Users/javiermatos/dev/vcpkg/buildtrees/fmt/src/11.0.2-463dc5ff21.clean
-- Found external ninja('1.11.1').
-- Configuring arm64-osx
-- Building arm64-osx-dbg
-- Building arm64-osx-rel
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/lib/pkgconfig/fmt.pc
-- Fixing pkgconfig file: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/debug/lib/pkgconfig/fmt.pc
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/usage
-- Installing: /Users/javiermatos/dev/vcpkg/packages/fmt_arm64-osx/share/fmt/copyright
-- Performing post-build validation
Elapsed time to handle fmt:arm64-osx: 3.1 s
fmt:arm64-osx package ABI: 3fcdbaa4539f8a1e2e1e4f103c34f523364ab26bc81f599dd06eb1268a870191
Total install time: 3.3 s
The package fmt provides CMake targets:

    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt)

    # Or use the header-only version
    find_package(fmt CONFIG REQUIRED)
    target_link_libraries(main PRIVATE fmt::fmt-header-only)

~/dev/vcpkg %

@JavierMatosD JavierMatosD mentioned this pull request Sep 12, 2024
Closed
@JonLiu1993 JonLiu1993 changed the title Simplify vcpkg_download_distfile output handling for Asset Caching [vcpkg scripts] Simplify vcpkg_download_distfile output handling for Asset Caching Sep 13, 2024
@JonLiu1993 JonLiu1993 added category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:internal This PR or Issue was filed by the vcpkg team. labels Sep 13, 2024
@JavierMatosD
Copy link
Contributor Author

CI failure is related to a known issue with sentry-native and not related to this PR.

@ras0219-msft ras0219-msft added requires:tool-release An issue that has been fixed in the microsoft/vcpkg-tool repo and is waiting for a release thereof depends:different-pr This PR or Issue depends on a PR which has been filed and removed requires:tool-release An issue that has been fixed in the microsoft/vcpkg-tool repo and is waiting for a release thereof labels Sep 23, 2024
@JavierMatosD JavierMatosD marked this pull request as draft October 1, 2024 14:14
@JavierMatosD JavierMatosD marked this pull request as ready for review October 15, 2024 22:19
@JonLiu1993 JonLiu1993 removed the depends:different-pr This PR or Issue depends on a PR which has been filed label Oct 17, 2024
JonLiu1993
JonLiu1993 previously approved these changes Oct 17, 2024
View reviewed changes
@JonLiu1993 JonLiu1993 added the info:reviewed Pull Request changes follow basic guidelines label Oct 17, 2024
@BillyONeal BillyONeal added requires:tool-release An issue that has been fixed in the microsoft/vcpkg-tool repo and is waiting for a release thereof and removed info:reviewed Pull Request changes follow basic guidelines labels Oct 18, 2024
BillyONeal
BillyONeal previously approved these changes Oct 18, 2024
View reviewed changes
Copy link
Member

@BillyONeal BillyONeal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approval is contingent on fixing the tool output e.g. microsoft/vcpkg-tool#1513

@BillyONeal
Copy link
Member

I cancelled the PR build run because this work won't merge as part of this PR anyway.

BillyONeal added a commit that referenced this pull request Nov 12, 2024
@BillyONeal
Update vcpkg-tool to 2024-11-12.
3a762c2 
https://github.com/microsoft/vcpkg-tool/releases/tag/2024-11-12

In particular, this includes the improvements to x-download's output that go with #40945
@BillyONeal BillyONeal mentioned this pull request Nov 12, 2024
Merged
@BillyONeal
Copy link
Member

Closing in favor of #42115

@BillyONeal BillyONeal closed this Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@konjac konjac konjac left review comments

@data-queue data-queue data-queue approved these changes

@JonLiu1993 JonLiu1993 JonLiu1993 left review comments

@BillyONeal BillyONeal BillyONeal left review comments

Assignees

@JonLiu1993 JonLiu1993

Labels
category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:internal This PR or Issue was filed by the vcpkg team. requires:tool-release An issue that has been fixed in the microsoft/vcpkg-tool repo and is waiting for a release thereof
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@JavierMatosD @BillyONeal @konjac @JonLiu1993 @data-queue @ras0219-msft