Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deterministic hash - refactor and fix deployment #804

Merged
merged 2 commits into from
Mar 11, 2019

Conversation

antho1404
Copy link
Member

Fix for #731

After a long time trying to go deep on why the hash was different between git and the tarball from github and our tarball I found out 2 things:

  • Git override the date of creation, what @krhubert already saw and fixed with this
filepath.Walk(path, func(p string, fi os.FileInfo, _ error) error {
	return os.Chtimes(p, time.Time{}, time.Time{})
})
  • The tarball had different file permissions than the git
GIT:
DEBU[0011] /package.json  IsDir=false ModTime="2019-03-08 16:24:12.514313 +0000 UTC" Mode=-rw-r--r-- Name=package.json Size=364
TARBALL:
DEBU[0028] /package.json  IsDir=false ModTime="2019-01-22 05:18:44 +0000 UTC"        Mode=-rw-rw-r-- Name=package.json Size=364

The docker untar function we used was the cause so I wrote a function to untar directly the content without the extra work that docker needs (here we don't care about docker).

I also did some refactoring on the FromArchive/FromURL function to have the url that uses the untar system and the tarball received is also untared to make sure the process is the same whatever the source of deployment and with all that now everything works fine

Here are some manual tests

./dev-cli service deploy https://github.com/mesg-foundation/service-webhook
> f4af4efd2c11e25655d41eb8ef10aa4d0abc2dda008249bc691f0cabc0b06cc4

./dev-cli service deploy https://api.github.com/repos/mesg-foundation/service-webhook/tarball/master
> f4af4efd2c11e25655d41eb8ef10aa4d0abc2dda008249bc691f0cabc0b06cc4

./dev-cli service deploy service-webhook
> f4af4efd2c11e25655d41eb8ef10aa4d0abc2dda008249bc691f0cabc0b06cc4


./dev-cli service deploy https://github.com/mesg-foundation/service-ethereum/archive/master.tar.gz
> 89961e225f1526f324d568e24f9f05849d0211451d4fb4581fddf25d4188c20c

./dev-cli service deploy https://github.com/mesg-foundation/service-ethereum
> 89961e225f1526f324d568e24f9f05849d0211451d4fb4581fddf25d4188c20c

./dev-cli service deploy service-ethereum/
> 89961e225f1526f324d568e24f9f05849d0211451d4fb4581fddf25d4188c20c

Feel free to test it yourself and verify that you get the same hashes

@antho1404 antho1404 force-pushed the feature/deterministic-hash-fix branch from 7452d66 to 59fd1d5 Compare March 9, 2019 17:33
@antho1404 antho1404 mentioned this pull request Mar 9, 2019
@antho1404 antho1404 requested review from krhubert, NicolasMahe and ilgooz and removed request for NicolasMahe March 9, 2019 18:10
@NicolasMahe
Copy link
Member

NicolasMahe commented Mar 11, 2019

@antho1404 could you explain a more clearly what the function docker.archive.untar is doing that should not be doing?

archive.untar accept a bunch of options. Is there any way to set the right option to get the expected result? https://godoc.org/github.com/docker/docker/pkg/archive#TarOptions
Otherwise, a lower-level function: https://godoc.org/github.com/docker/docker/pkg/archive#TarModifierFunc

I really think we should avoid to untar by "hand". I feel there are many things that is missing in this simple implementation of untar and will fire back on more complicated archive.
If we cannot use docker.archive.untar, maybe another lib like https://github.com/mholt/archiver.

@NicolasMahe NicolasMahe changed the title refactor and fix deployment Deterministic hash - refactor and fix deployment Mar 11, 2019
@krhubert krhubert merged commit 4e8d9c7 into feature/determistic-hash Mar 11, 2019
@ilgooz ilgooz deleted the feature/deterministic-hash-fix branch March 11, 2019 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants