[Bug]: OpenSearch Snapshot failing checksum validation

[ashwin-pc]

Describe the bug

This is similar to a bug we faced earlier #2114 where the snapshot for the latest main build fails checksum validation. Related error reported on Dashboards opensearch-project/OpenSearch-Dashboards#3787. This also breaks our CI workflows.

To reproduce

Run yarn opensearch snapshot

Expected behavior

No response

Screenshots

Host / Environment

OSD main

Additional context

No response

Relevant log output

info Installing from snapshot
   │ info version: 3.0.0
   │ info install path: /__w/OpenSearch-Dashboards/OpenSearch-Dashboards/.opensearch/3.0.0
   │ info license: oss
   │ info Verifying snapshot URL at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz
   │ info downloading artifact from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz
   │ info downloading artifact checksum from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz.sha512
   │ERROR artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz does not match expected checksum
   │        expected: 16e9c818034f17dcac24017a614300952de9c1d4c99c2708e85ef7d02d646a6c452b5e4e766a115a794ea28350cca5ea746e5bf033aa23d504eddb9b06464580
   │        received: f1008401e42b524768a86ecef288cd746638b205df00c2e4743f6f51475072581730aeaa96951d06c2b9c274501cc148cbe8cdae01a024479d512895f1c7b876
   │        headers: 
   │          accept-ranges: bytes
   │          age: 143
   │          connection: close
   │          content-length: 240730749
   │          content-type: application/x-gzip
   │          date: Wed, 05 Apr 2023 17:35:22 GMT
   │          etag: "eac155b9d05cbf94b122eaadd3cdf728-46"
   │          last-modified: Wed, 05 Apr 2023 01:41:18 GMT
   │          server: AmazonS3
   │          via: 1.1 38a1b6b88186539189dcd472aa234808.cloudfront.net (CloudFront)
   │          x-amz-cf-id: g6rMVBFUy_TUNyC6iN5kgH_7n9gIx-CUjN6pxl1P4qB0Uv9QVOlYdg==
   │          x-amz-cf-pop: LAX53-P3
   │          x-amz-server-side-encryption: AES256
   │          x-amz-version-id: 8wlhswK4CcPrNJ39UrYyW0spN8IcwY2g
   │          x-cache: Hit from cloudfront
   │        content[500 / 240730749 base64]: H4sIAAAAAAAA/+y6c5RlzZYvmuZO27Zt27Ztm5VmpZ2Vtm3bdlbaViUreavv6X73nq+7z+h+94z7z3tz7LV2RKw5Y0SsGRO/OZaDo5m9i5mRs4klHQs9Iz0jnYq8kKKKpIIqA9A/jRgZWRk52NgIGf9Gf/3/n20mVlYmFiYWNhYWdkJGJhZ2FhYgQrZ/3hL+c3JzcTVy/rPE/9N5/rq5f8LS/m+Qw3+qf1sr43/SGfjv65+VlZHp/9f//w36x/r/T5/SW/95af9VYmRiZGRnZf2P9M/OysLBxMjO/Pf6Z2ZiYuEAIvw/1sl/hf4/rn9FGVAwVCAgKKg/baH/OQLyr0+g/1xyYqpCdFLy4gygf2HM3+ajSQQGApr+w436vzPKCclLiYupqNLLibslyjksMSKG3ST8IiB2AiayMOExwJG/QgcuNkOChnJySVwfoqwn+aYaajBbZ9apJBbnf5KR4WHuS5PMG+B2iINJ01xBjE1h5HhBQTGOkS6oWUZswZtX5iMAvI7DJQkmcYKuY8ce7FrocCA/i9Md0EAoy5E4fJo71z6NX/xKOQwV1m+Pat2XiHditfGkmNo0SXXopdNklZDW95pcquWepzce5Y8DsTMuPxv9vvG7fmWdPi6Jk+ixNHr28c2vscTBFxJm5pB9JqFJmtY=

[jordarlu]

Hi, @ashwin-pc , it is possible the same issue as the PR you refered to and the comment at #2114 (comment)
Can you try it again in a few hours later and if you still see the same issue?
Or , if you can use https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-3.0.0-linux-x64.tar.gz instead ?

thanks,

[ashwin-pc]

@jordarlu can we get clarification about the difference between the 2 domains? We were hesitant to change the domain for a few reasons.

1. For our CI we need both the manifest and snapshot to run the CI correctly. Is the CI endpoint manifest url the same as artifact? based on what i see they seem to be different.
2. We also run quite a lot of CI workflows using these snapshots, a build failure in the snapshot process would bring our CI workflows to a halt. based on what i read, the bake time for artifact.opensearch.org gives us the buffer to make sure that the builds break less often. This is my assumption, correct me if im worng here.

Based on some of the older conversations, it appears that the reason this is happening is because we cache the checksum but not the snapshot. Cant we simply not cache the checksum and generate it for every build? seems a little weird to use a cached checksum for a build.

[jordarlu]

Hello, @ashwin-pc .. unfortunately the cache mechanism is on CloudFront where we don't have management there as I learned, and let me add @peterzhuamazon and @gaiksaya here to keep me honest..

thanks,

[peterzhuamazon]

Hi @ashwin-pc can you give us the exact step to reproduce this?
Can you show me the code of your repo that actually does the check?

I want to understand the root cause of this as this has been going on before in another issue. Thanks!

[ashwin-pc]

Sure here is the code: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/packages/osd-opensearch/src/artifact.js

[rishabh6788]

Hi @ashwin-pc Could you please help us understand what is the limitation in using ci.opensearch.org instead of artifacts.opensearch.org to download the artifact?
Meanwhile we will try to replicate this issue.
@peterzhuamazon

[jordarlu]

Hello, there ... trying to see how we can move forward on this issue :
@ashwin-pc , have you got a chance to look into using ci.opensearch.org instead of artifacts.opensearch.org to download the artifact?
@rishabh6788 , I bellieve we were unable to reproduce the issue, did we?
cc : @peterzhuamazon

[ashwin-pc]

Hey, sorry i have not had the time to look into that unfortunately. @kavilla has more context here, looping him in.

[jordarlu]

thanks, @ashwin-pc ,,
Hi, @kavilla , wondering if you ever got a time to take a look on this issue, and if any comment on using ci.opensearch.org instead of artifacts.opensearch.org to download the artifact? thanks !

[prudhvigodithi]

Closing this issue, @ashwin-pc @kavilla please re-open if required. Thanks

[ciesiolka]

I'm facing the same issue:

C:\opensearch\OpenSearch-Dashboards>yarn run opensearch snapshot --network-timeout 1000000
yarn run v1.22.19
$ scripts/use_node scripts/opensearch snapshot --network-timeout 1000000
 info Installing from snapshot
   │ info version: 3.0.0
   │ info install path: C:\opensearch\OpenSearch-Dashboards\.opensearch\3.0.0
   │ info license: oss
   │ info Verifying snapshot URL at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip
   │ info downloading artifact from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip
   │ info downloading artifact checksum from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip.sha512
   │ERROR artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip does not match expected checksum
   │        expected: 565d4792a00913f094e77ea8d0e0047dc628fcb27d7e2b870126213aa90dd66c12ab76f8ff61d57cede6294ee7018afb7ea2e86b6e1ff91abd526350dc3fc890
   │        received: 1f7dabac5c98117078b8289eb410dacec71513a765c94017b00329cab81388e1781b0afce4b759ae5e07b03c9ab6be1010080b9ebf0b9640c4316aa18542fd9f
   │        headers:
   │          accept-ranges: bytes
   │          connection: close
   │          content-length: 246071451
   │          content-type: application/zip
   │          date: Thu, 14 Dec 2023 09:49:04 GMT
   │          etag: "4134a72eb78c6832804653e566e2393d-47"
   │          last-modified: Thu, 14 Dec 2023 07:16:08 GMT
   │          server: AmazonS3
   │          via: 1.1 b42eb3d87622a28f769121f374a92fda.cloudfront.net (CloudFront)
   │          x-amz-cf-id: ZGjEmtcIpLselNIp-9g10JO2el1qrH_9hPrXf73cpLIfaFatfyKCfg==
   │          x-amz-cf-pop: WAW51-P4
   │          x-amz-replication-status: COMPLETED
   │          x-amz-server-side-encryption: AES256
   │          x-amz-version-id: CIuO5c5BuhitcP0vpMsLvBEuQppPP68j
   │          x-cache: Hit from cloudfront
   │        content[500 / 246071451 base64]: UEsDBBQAAAgIAAAAQQAAAAAAAgAAAAAAAAAaAAAAb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC8DAFBLAwQUAAAICAAAAEEAAAAAAAIAAAAAAAAAHgAAAG9wZW5zZWFyY2gtMy4wLjAtU05BUFNIT1QvbGliLwMAUEsDBBQAAAgIAAAAQQAa+HlGSsvKAJcY3AA7AAAAb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC9saWIvb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC5qYXKsugOUJVGyLlzmKdu2bdu2bZtdZpdtV5dtddl2ddm22X/PP/euN3gz7973bq6TefbOjIiz4uzQF7kVZUDBUIGAoKCAgICEgP5ygAD99YD+c8qJqQrRScmLM4ACKf4d4Zd2mUohMBDQ5h9q1L8llBOSlxIXU1GllxN3S9RxWGREDN3W+PUbMRIDscHkuEXK85c9TT4GeoIjkoM8r2kMMk9DDvHLo6wKXWTF3QnXdjaXybu85SNoNoVBL7eZK5bjeEtWZ/lzlG8Qfkgw8407CC0216NgrLobTF/ScxHNQJ3KFytbE5uIpfgsfIidgmnTEUUTP69xeaYMat/lkSNyh8hE6Zh712hUaXw8O/gMYZznwLqkuXIdqYNp5pDXd5OfCcFY4o8=
Done in 8.95s.

OpenSearch-Dashboard@3268e3722