R4R: Infrastructure for reproducible builds #4262

alessio · 2019-05-02T22:17:25Z

This change set introduces support for building gaia with gitian
on the following GOOS/GOARCH pairs:

darwin/386
darwin/amd64
linux/386
linux/amd64
linux/arm
linux/arm64
windows/386
windows/amd64

cmd/gaia/contrib/gitian-descriptors/ contains gitian descriptor files.

cmd/gaia/contrib/gitian-keys/ contains:

a keys.txt file that is meant to list core developers and gitian builders PGP keys.
README.me to provide instructions on how to import the keys into one's
personal GPG keyring.

The gosum utility is removed, so is the go.sum hashsum bit from gaiacli/gaiad
version string. It was meant to be a provisional mitigation to the lack of a
reproducible build process.

GOBIN is removed from all Makefiles. When GOBIN is set, go refuses to cross-compiles
binaries for foreign architectures. export GOBIN=$GOPATH/bin is unnecessary anyway
as by default go install places built binaries in $GOPATH/bin. Developers are required
to update their enviornment files and replace $GOBIN with $GOPATH/bin in PATH.

circleci configuration file is amended accordingly.

Instructions at https://github.com/cosmos/cosmos-sdk/wiki/Reproducible-builds

Closes: #4027
Closes: #4280

For Admin Use:

Added appropriate labels to PR (ex. wip, ready-for-review, docs)
Reviewers Assigned
Squashed all commits, uses message "Merge pull request #XYZ: [title]" (coding standards)

codecov · 2019-05-03T01:18:40Z

Codecov Report

Merging #4262 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4262   +/-   ##
======================================
  Coverage    59.1%   59.1%           
======================================
  Files         217     217           
  Lines       14595   14595           
======================================
  Hits         8627    8627           
  Misses       5330    5330           
  Partials      638     638

Instructions follow: Make sure ruby is installed in your system and lives in /usr/bin/ruby: [ `which ruby` = '/usr/bin/ruby' ] && echo yes || echo no Clone gitian: git clone https://github.com/devrandom/gitian-builder Prepare a docker image for the build: cd gitian-build bin/make-base-vm --docker bin/make-base-vm --docker --arch amd64 export USE_DOCKER=1 Download go: mkdir inputs gopkg=go1.12.4.linux-amd64.tar.gz ; \ curl -L https://dl.google.com/go/$gopkg > inputs/$gopkg Build: bin/gbuild $SDK-PATH/cmd/gaia/contrib/gitian-linux.yml \ --commit cosmos-sdk=v0.34.3

…pilation

…ck of reproducible builds

Update version package accordingly.

This reverts commit 4bce5c2.

This reverts commit e245095.

This reverts commit c60eb1b.

cmd/gaia/Makefile

alessio · 2019-05-09T23:42:50Z

Bnaries are now built with CGO enabled. I've successfully tested ledger support on Linux amd64.

cmd/gaia/contrib/gitian-keys/README.md

cmd/gaia/contrib/gitian-descriptors/gitian-windows.yml

liamsi

If I run ./cmd/gaia/contrib/gitian-build.sh on macosx, I get a ./cmd/gaia/contrib/gitian-build.sh: line 196: 1: unbound variable. Update: OK, I see this needs a param passed in, e.g. ./cmd/gaia/contrib/gitian-build.sh linux pwd.

Now:

Start the build
bin/gbuild:252:in ``': No such file or directory - sha256sum (Errno::ENOENT)
        from bin/gbuild:252:in `<main>'

OK, looks like coreutils installation failed. NVM.

liamsi · 2019-05-10T09:57:46Z

The instructions should probably contain a sentence on how to verify that the build I've created matches the official build / I successfully reproduced it.

alessio · 2019-05-10T12:09:30Z

@liamsi dixit:

The instructions should probably contain a sentence on how to verify that the build I've created matches the official build / I successfully reproduced it.

I will certainly do! And I will include instructions on how to retrieve and import builder keys into your GPG keyring, etc.

liamsi · 2019-05-10T13:18:32Z

Update: coreutils wasn't in PATH ... NVM

I ran git pull to get the latest changes (and deleted the gaia.sigs and the gitian-build-linux dirs, and restarted the build (via ./cmd/gaia/contrib/gitian-build.sh -s Ismail.Khoffi@gmail.com linux `pwd`) and got:

Updating apt-get repository (log in var/install.log)
Installing additional packages (log in var/install.log)
Upgrading system, may take a while (log in var/install.log)
Creating package manifest
Creating build script (var/build-script)
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... 
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
bin/gbuild:147:in `block (2 levels) in build_one_configuration': error looking up author date in cosmos-sdk (RuntimeError)
        from bin/gbuild:143:in `each'
        from bin/gbuild:143:in `block in build_one_configuration'
        from bin/gbuild:127:in `open'
        from bin/gbuild:127:in `build_one_configuration'
        from bin/gbuild:309:in `block (2 levels) in <main>'
        from bin/gbuild:304:in `each'
        from bin/gbuild:304:in `block in <main>'
        from bin/gbuild:302:in `each'
        from bin/gbuild:302:in `<main>'

liamsi · 2019-05-10T15:57:57Z

Build reports: https://github.com/Liamsi/gaia.sigs

jackzampolin

Builds the darwin for me!!

alessio added 13 commits May 3, 2019 18:18

Use Ubuntu bionic

50680c6

Rename gitian build descriptor

fe6175c

Build for multiple Linux archs, remove GOBIN as it prevents cross-com…

3ccf1f1

…pilation

Pass commit, remove invalid archs

7e3eb43

Make gosum

5421ad9

Make sure gosum is available

6172e34

We can ignore go.sum hash, it was a temporary hack to mitigate the la…

454e582

…ck of reproducible builds

Clean before build

688d429

Add gitian-keys directory

b5428ad

Kill gosum

639a9c9

Update version package accordingly.

Replace $GOBIN with $GOPATH/bin

bd46ee5

Remove unused anchor

9f28890

alessio force-pushed the alessio/reproducible-builds branch from 2022bb9 to 9f28890 Compare May 3, 2019 17:19

alessio added 7 commits May 3, 2019 18:20

Set a different gopath

e245095

Build tools, set custom gopath

4bce5c2

Revert "Build tools, set custom gopath"

907ee8c

This reverts commit 4bce5c2.

Revert "Set a different gopath"

9703514

This reverts commit e245095.

Add /tmp/workspace/bin to PATH

06aaba9

Customise DESTDIR for tools

bda49a7

Try to fix docker_image build

f48c9a9

alessio changed the title ~~Add Linux descriptor for Gitian builds~~ Start building the infrastructure for reproducible builds May 3, 2019

Correct tar file order

3b14f09

alessio marked this pull request as ready for review May 3, 2019 18:53

alessio requested review from alexanderbez, jackzampolin and rigelrozanski as code owners May 3, 2019 18:53

alessio requested review from mircea-c and zmanian May 3, 2019 18:53

alessio added 7 commits May 9, 2019 12:50

Update tarball's name

3269581

Cache Go src tarball, add distclean to Makefile

1e40f28

Test circleci

34b6903

Install ruby

aee3aab

Disable ci jobs for testing purpose

c60eb1b

Revert "Disable ci jobs for testing purpose"

3a70f4f

This reverts commit c60eb1b.

Revert

526d1b0

liamsi reviewed May 9, 2019

View reviewed changes

cmd/gaia/Makefile Show resolved Hide resolved

alessio added 5 commits May 9, 2019 16:23

Add extra envvars for testing with gitian forks

7764bde

Remove GOBIN from installation notes

c467324

Add pending entries

9481e01

Add docs

9e190c8

Merge branch 'master' into alessio/reproducible-builds

3d1539c

Build with go 1.12.5

c7e21c8

liamsi reviewed May 10, 2019

View reviewed changes

cmd/gaia/contrib/gitian-keys/README.md Show resolved Hide resolved

liamsi reviewed May 10, 2019

View reviewed changes

cmd/gaia/contrib/gitian-descriptors/gitian-windows.yml Show resolved Hide resolved

liamsi reviewed May 10, 2019

View reviewed changes

Fix windows binary extension

3b0216a

alessio added 2 commits May 11, 2019 19:00

Set git describe --abbrev flag to 9

6d4aa09

Merge branch 'master' into alessio/reproducible-builds

9210caf

alessio requested a review from liamsi May 13, 2019 16:44

jackzampolin approved these changes May 13, 2019

View reviewed changes

alessio requested a review from cwgoes May 13, 2019 19:22

alessio merged commit c0486aa into master May 13, 2019

alessio deleted the alessio/reproducible-builds branch May 13, 2019 22:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

R4R: Infrastructure for reproducible builds #4262

R4R: Infrastructure for reproducible builds #4262

alessio commented May 2, 2019 •

edited

Loading

codecov bot commented May 3, 2019 •

edited

Loading

alessio commented May 9, 2019

liamsi left a comment •

edited

Loading

liamsi commented May 10, 2019

alessio commented May 10, 2019

liamsi commented May 10, 2019 •

edited

Loading

liamsi commented May 10, 2019

jackzampolin left a comment

R4R: Infrastructure for reproducible builds #4262

R4R: Infrastructure for reproducible builds #4262

Conversation

alessio commented May 2, 2019 • edited Loading

codecov bot commented May 3, 2019 • edited Loading

Codecov Report

alessio commented May 9, 2019

liamsi left a comment • edited Loading

Choose a reason for hiding this comment

liamsi commented May 10, 2019

alessio commented May 10, 2019

liamsi commented May 10, 2019 • edited Loading

liamsi commented May 10, 2019

jackzampolin left a comment

Choose a reason for hiding this comment

alessio commented May 2, 2019 •

edited

Loading

codecov bot commented May 3, 2019 •

edited

Loading

liamsi left a comment •

edited

Loading

liamsi commented May 10, 2019 •

edited

Loading