git tree SHA mismatch

[alecloudenback]

Thank you for the package - one thing that I encountered and maybe it's the way its supposed to work? But felt like it might be an issue.

tl;dr: replaced original artifact with new one and got sha1 mismatch

What I did:

1. Created tarball and uploaded to Github: SOA_Tables.tar.gz
2. used add_artifact! to create the Artifacts.toml file
3. touched up my package to use the artifact and had it working
4. Decided that I wanted to timestamp the tarball, so created and uploaded SOA_Tables_20201022.tar.gz
5. Deleted the contents of Artifacts.toml
6. add_artifact! pointing to the updated URL
7. Then I ran tests on my local machine and it worked.
8. Pushed to Github branch, and tests failed with:

┌ Error: Tree Hash Mismatch!
│   Expected git-tree-sha1:   eda74e474e6619c1f4db573758712809f531a9c0
│   Calculated git-tree-sha1: 52b77526be6e7c36ebf8e03009e04ef52f802925
└ @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Artifacts.jl:788

9. I've updated the Artifacts.toml in a new commit with the "Expected" sha1 and now don't have that error (though I still have failing tests, I think because of something on Julia nightly)

[simeonschaub]

Hmm, so it did not update the tree hash at all? Or did it calculate a wrong tree hash? Might be related to JuliaPackaging/BinaryBuilder.jl#527 or JuliaMath/SpecialFunctions.jl#193. What OS and Julia version are you running locally? Could you maybe rerun CI with fail-fast: false, so we can check whether this fails on all platforms?

[alecloudenback]

Windows 10, Julia 1.5.2

The error did stop after step 9 (JuliaActuary/MortalityTables.jl@a2f48d7) above where I manually replaced the sha1 in the Artifacts.toml file.

I did also add fail-fast:false because CSV.jl is still failing on nightly but that's a separate issue I think.

[simeonschaub]

Could you try this: JuliaPackaging/BinaryBuilder.jl#527 (comment)

[alecloudenback]

julia> Pkg.PlatformEngines.probe_platform_engines!()

julia> Pkg.PlatformEngines.gen_unpack_cmd("file.tar.gz", "/tmp/file")
pipeline(pipeline(pipeline(`'C:\Users\alecl\AppData\Local\Programs\Julia 1.5.2\bin\..\libexec\7z.exe' x file.tar.gz -y -so`, stdout=`'C:\Users\alecl\AppData\Local\Programs\Julia 1.5.2\bin\..\libexec\7z.exe' x -si -y -ttar -o/tmp/file`), stdout>Base.DevNull()), stderr>Base.DevNull())

[simeonschaub]

Ok, I think it's not unlikely that 7zip extracts the tarball slightly differently than GNU tar, resulting in a different tree hash. Could you perhaps try this on 1.6? I think that should always use Tar.jl instead.

[StefanKarpinski]

I think we compute incorrect hashes on Windows. cc @staticfloat

[StefanKarpinski]

The right way to do this is to create tarballs first (using Tar.jl) and then compute the hash of the tarball with Tar.tree_hash. That way at least the hash will always match the tarball. Not sure what the status of Elliot's fixes to libuv file attributes is, but with those fixes, Tar.jl should create correct tarballs even on Windows.

[staticfloat]

This is a bug in old Julia versions, which we've fixed the underlying issue with, but Pkg hasn't quite caught up yet. This should fix it, but is of course only reliable on v1.6+: JuliaLang/Pkg.jl#2253

[simeonschaub]

@alecloudenback Would you be able to check whether this is fixed on Julia nightly? I unfortunately don't have a Windows machine to test this.

[alecloudenback]

@simeonschaub I've never built nightly. I still have this issue in my notification log and was going to try once a test release of 1.6 came out.

[giordano]

No need to build if you can't, just download the binary from https://julialang.org/downloads/nightlies/

[alecloudenback]

TIL that there's prebuilt nightlies!

Anyway, I just tested this on the same machine as original issue and I get the correct SHA: 52b77526be6e7c36ebf8e03009e04ef52f802925

Can close the issue?

[simeonschaub]

Awesome, thank you very much for checking! Yes, this can be closed then.