Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

corrupt package in staging? #42

Closed
jeremyd2019 opened this issue May 24, 2024 · 6 comments
Closed

corrupt package in staging? #42

jeremyd2019 opened this issue May 24, 2024 · 6 comments

Comments

@jeremyd2019
Copy link
Member

jeremyd2019 commented May 24, 2024 •
edited
Loading 

$ wget https://repo.msys2.org/staging/mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst
--2024-05-24 14:48:17--  https://repo.msys2.org/staging/mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst
Loaded CA certificate '/usr/ssl/certs/ca-bundle.crt'
Resolving repo.msys2.org (repo.msys2.org)... 178.63.98.68
Connecting to repo.msys2.org (repo.msys2.org)|178.63.98.68|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25311536 (24M) [application/octet-stream]
Saving to: ‘mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst’

mingw-w64-clang-aar 100%[===================>]  24.14M  8.24MB/s    in 2.9s

2024-05-24 14:48:21 (8.24 MB/s) - ‘mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst’ saved [25311536/25311536]


$ tar -tf mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst
.BUILDINFO
.MTREE
.PKGINFO
clangarm64/
clangarm64/bin/
clangarm64/bin/libLLVM-18.dll
/*stdin*\ : Decoding error (36) : Restored data doesn't match checksum
clangarm64/bin/libLTO.dll
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

I was getting these errors trying to update from staging from pacman, so decided to test in isolation. The original in msys2-arm/msys2-autobuild must be ok, because autobuild is not failing with these errors...
@jeremyd2019
Copy link
Member Author

jeremyd2019 commented May 24, 2024 •
edited
Loading

The original in msys2-arm/msys2-autobuild must be ok, because autobuild is not failing with these errors...

$ wget https://github.com/msys2-arm/msys2-autobuild/releases/download/staging-clangarm64/ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin
--2024-05-24 14:51:42--  https://github.com/msys2-arm/msys2-autobuild/releases/download/staging-clangarm64/ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin
Loaded CA certificate '/usr/ssl/certs/ca-bundle.crt'
Resolving github.com (github.com)... 140.82.116.3
Connecting to github.com (github.com)|140.82.116.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/382477994/72b8b251-d2f0-49f9-905c-297cfe38dab1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240524T215143Z&X-Amz-Expires=300&X-Amz-Signature=fffd88bb213566483730ccb1eb8f55bbd64e0b130e52fdc9812bf28a206610af&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=382477994&response-content-disposition=attachment%3B%20filename%3Dede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin&response-content-type=application%2Foctet-stream [following]
--2024-05-24 14:51:43--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/382477994/72b8b251-d2f0-49f9-905c-297cfe38dab1?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240524T215143Z&X-Amz-Expires=300&X-Amz-Signature=fffd88bb213566483730ccb1eb8f55bbd64e0b130e52fdc9812bf28a206610af&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=382477994&response-content-disposition=attachment%3B%20filename%3Dede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25311536 (24M) [application/octet-stream]
Saving to: ‘ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin’

ede1cc4b2234b6ec390 100%[===================>]  24.14M  56.9MB/s    in 0.4s

2024-05-24 14:51:44 (56.9 MB/s) - ‘ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin’ saved [25311536/25311536]


$ tar -tf ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin
.BUILDINFO
.MTREE
.PKGINFO
clangarm64/
clangarm64/bin/
clangarm64/bin/libLLVM-18.dll
clangarm64/bin/libLTO.dll
clangarm64/bin/libRemarks.dll
clangarm64/share/
clangarm64/share/licenses/
clangarm64/share/licenses/llvm-libs/
clangarm64/share/licenses/llvm-libs/LICENSE

@jeremyd2019
Copy link
Member Author

The files are the same size, but differ in contents:

$ cmp mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin
mingw-w64-clang-aarch64-llvm-libs-18.1.6-1-any.pkg.tar.zst ede1cc4b2234b6ec3905ffe3dc13dc2c650196cf8c9b23ceb1c7570fd4fff5ed.bin differ: byte 14458540, line 45675

@lazka
Copy link
Member

lazka commented May 25, 2024 •
edited
Loading

Yeah, we had that last week too (a file in the repo had a bitflip, but all other files were OK, and we restored it from backup). It seems the disks or the server might be dying.

@lazka
Copy link
Member

lazka commented May 25, 2024 •
edited
Loading

Should be fixed now after pushing the (freshly downloaded) package to the repo

@lazka
Copy link
Member

lazka commented May 25, 2024 •
edited
Loading

I'm gonna try adding a decompression check to the download command, so this fails early, for starters.

We already do that when adding to the repo, but not for staging: https://github.com/msys2/msys2-devtools/blob/44e62b0cee835aee99af85ec6df86cd2f2f0aab4/msys2-dbadd#L26-L27

lazka added a commit to lazka/msys2-autobuild that referenced this issue May 25, 2024
@lazka
fetch-assets: test all downloaded files with zstd
b240a28 
Test them before moving them to the final location.
This makes the download fial of there is some file corruption etc.

This adds a dependency on the zstd exectuable for the fetch-assets
command.

Motivated by msys2/msys2-main-server#42
@lazka lazka mentioned this issue May 25, 2024
Merged
lazka added a commit to msys2/msys2-autobuild that referenced this issue May 25, 2024
@lazka
fetch-assets: test all downloaded files with zstd
59bb7f6 
Test them before moving them to the final location.
This makes the download fial of there is some file corruption etc.

This adds a dependency on the zstd exectuable for the fetch-assets
command.

Motivated by msys2/msys2-main-server#42
@lazka
Copy link
Member

lazka commented Jun 13, 2024

disks were dying, and we are currently working on moving things over to a new machine, so this can be closed.

@lazka lazka closed this as completed Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lazka @jeremyd2019