Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix all download failures claiming that the download was disabled by x-block-origin. #1513

Merged
merged 25 commits into from
Nov 11, 2024
Merged

Fix all download failures claiming that the download was disabled by x-block-origin. #1513

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
d269012
stop saying x-block-origin enabled when it is not
Oct 16, 2024
8f9ade9
remove unused message
Oct 16, 2024
1d44ff4
regen message map
Oct 16, 2024
fb140fa
fix test
Oct 16, 2024
1f2787f
format:
Oct 16, 2024
2202f1e
format
Oct 16, 2024
29c815c
show x-block-origin is set
Oct 22, 2024
220e737
undo debug change to test
Oct 22, 2024
02fb23b
format
Oct 22, 2024
0ab5003
add test cases for smoke tests
Nov 4, 2024
ba76a76
generate mesages
Nov 4, 2024
160463e
change to macos-13
Nov 4, 2024
505e8dd
fix unrecognized escape sequence
Nov 4, 2024
e8399d6
Merge remote-tracking branch 'upstream/main' into fix_asset_cache_bug
Nov 4, 2024
188d7b8
use .Contains instead of regex pattern matching
Nov 4, 2024
c9af32b
Accidentally removed a bit of test code that shouldn't have been
Nov 4, 2024
fec25ee
update test case to match new output
Nov 4, 2024
a92df59
respond to Billy Review
Nov 11, 2024
f91cbcc
add newlines
Nov 11, 2024
9be6a3a
add proxy message
Nov 11, 2024
d2fe966
remove newline replacements
Nov 11, 2024
a8741ff
modify proxy message
Nov 11, 2024
c513777
generate message map
Nov 11, 2024
d5f72ac
remove newline replacement
Nov 11, 2024
e93ce22
Format and regenerate messages
JavierMatosD Nov 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 163 additions & 2 deletions azure-pipelines/end-to-end-tests-dir/asset-caching.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,7 @@ $actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("install", "vcpkg
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"A suitable version of .* was not found \(required v[0-9\.]+\)."
"Asset cache miss; downloading from .*"
"Successfully stored .* to .*."
) -join "`n"

if (-not ($actual -match $expected)) {
Expand Down Expand Up @@ -105,3 +103,166 @@ $expectedPattern = "(?s)" +
if (-not ($actual -match $expectedPattern)) {
throw "Failure: couldn't find expected debug promises (asset caching disabled + x-block-origin disabled)"
}


# azurl (no), x-block-origin (no), asset-cache (n/a), download (fail)
# Expected: Download failure message, nothing about asset caching
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://localhost:1234/foobar.html"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"error: https://localhost:1234/foobar.html: curl failed to download with exit code 7"
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (no), x-block-origin (no), asset-cache (n/a), download (fail)"
}

#azurl (no), x-block-origin (no), asset-cache (n/a), download (sha-mismatch)
#Expected: Download message with "you might need to configure a proxy" and expected/actual sha
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: No "you might need to configure a proxy". I could add back in, but it feels like a potentially misleading message.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering it was in the CMake block we are removing I think it should be there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, we aren't removing the "you might need to configure a proxy" message from cmake world. https://github.com/microsoft/vcpkg/pull/40945/files#diff-95f7d8e8b25d1928ee2738036384adec841ddd80c51a526f45e9df167fea5b37L31

https://github.com/microsoft/vcpkg/pull/40945/files#diff-95f7d8e8b25d1928ee2738036384adec841ddd80c51a526f45e9df167fea5b37L270

Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73b", "--url", "https://example.com"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"error: File does not have the expected hash:"
"url: https://example.com"
"File: .*"
"Expected hash: d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73b"
"Actual hash: d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a"
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (no), x-block-origin (no), asset-cache (n/a), download (sha-mismatch)"
}

# azurl (no), x-block-origin (no), asset-cache (n/a), download (succeed)
# Expected: Download success message, nothing about asset caching
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com"))
$actual = $actual -replace "`r`n", "`n"
$expected = @(
"Downloading example3.html"
"Successfully downloaded example3.html."
JavierMatosD marked this conversation as resolved.
Show resolved Hide resolved
) -join "`n"
if (-not ($actual -match $expected)) {
throw "Failure: azurl (no), x-block-origin (no), asset-cache (n/a), download (succeed)"
}

# azurl (no), x-block-origin (yes), asset-cache (n/a), download (n/a)
# Expected: Download failure message, nothing about asset caching, x-block-origin complaint
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=clear;x-block-origin"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"error: Missing example3.html and downloads are blocked by x-block-origin."
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (no), x-block-origin (yes), asset-cache (n/a), download (n/a)"
}

# azurl (yes), x-block-origin (no), asset-cache (miss), download (fail)
# Expected: Download failure message, asset cache named, nothing about x-block-origin
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://localhost:1234/foobar.html", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"Asset cache miss; downloading from https://localhost:1234/foobar.html"
"Downloading example3.html"
"error: file://$AssetCache/.*: curl failed to download with exit code 37"
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (no), asset-cache (miss), download (fail)"
}

# azurl (yes), x-block-origin (no), asset-cache (hit), download (n/a)
# Expected: Download success message, asset cache named, nothing about x-block-origin
Refresh-TestRoot
Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite"))
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"Asset cache hit for example3.html; downloaded from: file://$AssetCache/.*"
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (no), asset-cache (hit), download (n/a)"
}

# azurl (yes), x-block-origin (no), asset-cache (miss), download (sha-mismatch)
# Expected: Download message with "you might need to configure a proxy" and expected/actual sha
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73b", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"Asset cache miss; downloading from https://example.com"
"Downloading example3.html"
"error: file://$AssetCache/.*: curl failed to download with exit code 37"
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (no), asset-cache (miss), download (sha-mismatch)"
}

$expected = @(
"error: File does not have the expected hash:"
"url: https://example.com"
"File: .*"
"Expected hash: d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73b"
"Actual hash: d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a"
) -join "`n"

if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (no), asset-cache (miss), download (sha-mismatch)"
}

# azurl (yes), x-block-origin (no), asset-cache (miss), download (succeed)
# Expected: Download success message, asset cache upload, nothing about x-block-origin
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"Asset cache miss; downloading from https://example.com"
"Downloading example3.html"
"Successfully downloaded example3.html."
"Successfully stored example3.html to file://$AssetCache/.*."
) -join "`n"
if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (no), asset-cache (miss), download (succeed)"
}

# azurl (yes), x-block-origin (yes), asset-cache (miss), download (n/a)
# Expected: Download failure message, which asset cache was tried, x-block-origin complaint
Refresh-TestRoot
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite;x-block-origin"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"Asset cache miss for example3.html and downloads are blocked by x-block-origin."
"error: Missing example3.html and downloads are blocked by x-block-origin."
) -join "`n"
if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (yes), asset-cache (miss), download (n/a)"
}

# azurl (yes), x-block-origin (yes), asset-cache (hit), download (n/a)
# Expected: Download success message, asset cache named, nothing about x-block-origin
Refresh-TestRoot
Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite"))
$actual = Run-VcpkgAndCaptureOutput -TestArgs ($commonArgs + @("x-download", "$downloadsRoot/example3.html", "--sha512", "d06b93c883f8126a04589937a884032df031b05518eed9d433efb6447834df2596aebd500d69b8283e5702d988ed49655ae654c1683c7a4ae58bfa6b92f2b73a", "--url", "https://example.com", "--x-asset-sources=x-azurl,file://$AssetCache,,readwrite;x-block-origin"))
$actual = $actual -replace "`r`n", "`n"

$expected = @(
"Asset cache hit for example3.html; downloaded from: file://$AssetCache/.*"
) -join "`n"
if (-not ($actual -match $expected)) {
throw "Failure: azurl (yes), x-block-origin (yes), asset-cache (hit), download (n/a)"
}
9 changes: 5 additions & 4 deletions include/vcpkg/base/message-data.inc.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,11 @@ DECLARE_MESSAGE(ArtifactsSwitchX86, (), "", "Forces host detection to x86 when a
DECLARE_MESSAGE(ArtifactsSwitchWindows, (), "", "Forces host detection to Windows when acquiring artifacts")
DECLARE_MESSAGE(AssetCacheHit, (msg::path, msg::url), "", "Asset cache hit for {path}; downloaded from: {url}")
DECLARE_MESSAGE(AssetCacheMiss, (msg::url), "", "Asset cache miss; downloading from {url}")
DECLARE_MESSAGE(AssetCacheMissBlockOrigin,
(msg::path),
"x-block-origin is a vcpkg term. Do not translate",
"Asset cache miss for {path} and downloads are blocked by x-block-origin.")
DECLARE_MESSAGE(DownloadSuccesful, (msg::path), "", "Successfully downloaded {path}.")
DECLARE_MESSAGE(DownloadingUrl, (msg::url), "", "Downloading {url}")
DECLARE_MESSAGE(AssetCacheProviderAcceptsNoArguments,
(msg::value),
Expand Down Expand Up @@ -1228,10 +1233,6 @@ DECLARE_MESSAGE(MissingShaVariable,
(),
"{{sha}} should not be translated",
"The {{sha}} variable must be used in the template if other variables are used.")
DECLARE_MESSAGE(AssetCacheMissBlockOrigin,
(msg::path),
"x-block-origin is a vcpkg term. Do not translate",
"Asset cache miss for {path} and downloads are blocked by x-block-origin.")
DECLARE_MESSAGE(FailedToExtract, (msg::path), "", "Failed to extract \"{path}\":")
DECLARE_MESSAGE(FailedToFetchRepo, (msg::url), "", "Failed to fetch {url}.")
DECLARE_MESSAGE(FailedToFindPortFeature,
Expand Down
21 changes: 13 additions & 8 deletions src/vcpkg/base/downloads.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -905,6 +905,8 @@ namespace vcpkg
MessageSink& progress_sink) const
{
std::vector<LocalizedString> errors;
bool block_origin_enabled = m_config.m_block_origin;

if (urls.size() == 0)
{
if (auto hash = sha512.get())
Expand Down Expand Up @@ -936,6 +938,14 @@ namespace vcpkg
msg::url = replace_secrets(read_url, m_config.m_secrets));
return read_url;
}
else if (block_origin_enabled)
{
msg::println(msgAssetCacheMissBlockOrigin, msg::path = download_path.filename());
JavierMatosD marked this conversation as resolved.
Show resolved Hide resolved
}
else
{
msg::println(msgAssetCacheMiss, msg::url = urls[0]);
}
}
else if (auto script = m_config.m_script.get())
{
Expand Down Expand Up @@ -985,16 +995,16 @@ namespace vcpkg
}
}

if (!m_config.m_block_origin)
if (!block_origin_enabled)
JavierMatosD marked this conversation as resolved.
Show resolved Hide resolved
{
if (urls.size() != 0)
{
msg::println(msgDownloadingUrl, msg::url = download_path.filename());
auto maybe_url = try_download_file(
fs, urls, headers, download_path, sha512, m_config.m_secrets, errors, progress_sink);
if (auto url = maybe_url.get())
{
m_config.m_read_url_template.has_value() ? msg::println(msgAssetCacheMiss, msg::url = urls[0])
: msg::println(msgDownloadingUrl, msg::url = urls[0]);
msg::println(msgDownloadSuccesful, msg::path = download_path.filename());

if (auto hash = sha512.get())
{
Expand All @@ -1012,11 +1022,6 @@ namespace vcpkg
}
}
}
// Asset cache is not configured and x-block-origin enabled
JavierMatosD marked this conversation as resolved.
Show resolved Hide resolved
if (m_config.m_read_url_template.has_value())
{
msg::println(msgAssetCacheMissBlockOrigin, msg::path = download_path.filename());
}
else
{
msg::println_error(msgMissingAssetBlockOrigin, msg::path = download_path.filename());
Expand Down
Loading