Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

have Travis and Appveyor build official release packages #2861

Closed
derekbruening opened this issue Mar 3, 2018 · 5 comments · Fixed by #4089 or #4092
Closed

have Travis and Appveyor build official release packages #2861

derekbruening opened this issue Mar 3, 2018 · 5 comments · Fixed by #4089 or #4092

Comments

@derekbruening
Copy link
Contributor

derekbruening commented Mar 3, 2018

Previously official packages were built on developer machines. Now that we
have Travis and Appveyor we should leverage it for a centralized, standardized
official release process.

Complications include:

  • Having Dr. Memory as a sub-package
  • Having clean, simple path names

Xref package.cmake command lines at https://github.com/DynamoRIO/dynamorio/wiki/New-Release

For weekly builds (#1967) I ended up having each build job create its own
package, for efficiency of having all platforms have their own without
separate jobs to invoke package.cmake. But, this splits i386 and x86_64,
and makes packages that don't look like official packages. As part of
making official packages we may want to change the weekly to use
package.cmake too: since they are only weekly the extra jobs should be ok.

@derekbruening
Copy link
Contributor Author

We should install zlib on Travis so nightly and official builds have compressed drmemtrace support.

@derekbruening derekbruening self-assigned this Feb 9, 2020
derekbruening added a commit that referenced this issue Feb 9, 2020
Switches to a separate invocation of package.cmake for cronbuilds, to
make them look like release packages.

Checks out Dr. Memory to include it in the package.

Fixes #2861
derekbruening added a commit that referenced this issue Feb 9, 2020
Switches to a separate invocation of package.cmake for cronbuilds, to
make them look like release packages.  Cronbuilds on Travis and Appveyor
no longer run the same jobs as a regular build but instead run one
package.cmake job.

Checks out Dr. Memory to include it in the package.
Does a shallow clone, assuming 250 is enough for the embedded DR.
Excludes drmemory/ from vera checks.

Removes the "package" parameter from non-cron builds which should make
them a little faster (xref i#4059).

There are still further decisions and actions but now the cronbuilds have the same
content as past manual builds.

Issue: #2861, #4059
@derekbruening
Copy link
Contributor Author

There are a number of other issues here:

  • This needs to be fully tested with a cron trigger and ensure the artifacts are uploaded by both Travis and Appveyor and that the Travis tag is still triggering the Appveyor build.

  • We should install zlib so drcachesim in the package will compress.

  • add ARM and AArch64 cronbuilds to the weekly build releases #3002: We should add ARM, AArch64, and Android package jobs.

  • We should make a decision on whether to switch to truly automated official releases via committing a tag or whether a manual cronjob is used for a new release. For manual, we need to make sure we can specify the patchlevel version number from the outside and have it be the version and package file name. For automated:

    • We would need to get the patchlevel from the tag.
    • Add a Travis tag trigger
    • Change Downloads.wiki
    • Add automated publishing of cronbuild docs to Github Pages #3194 deploy the html docs in an automated fashion.
    • Have the changelist created in an automated fashion and placed both in the github release description and in the docs. This might be most easily done by taking the list of commits and using their titles or some tags inside them.

@derekbruening derekbruening reopened this Feb 9, 2020
@derekbruening
Copy link
Contributor Author

zlib was installed by 674b71d.

derekbruening added a commit that referenced this issue Feb 9, 2020
Adds jobs to the Travis cronbuild matrix to build ARM, AArch64, and
Android packages.  Adds toolchain setting based on the existing env
var and parameter scheme into package.cmake to match runsuite.cmake.
Adds logic to exclude the Dr. Memory build for AArch64.  Adds ignoring
of a "git tag" failure to handle races among the four jobs.

Issue: #2861
Fixes #3002
@derekbruening
Copy link
Contributor Author

Another question:

  • Should we remove the build number from the package dir and file name? I.e., instead of DynamoRIO-Linux-7.91.18301-1.tar.gz we would have DynamoRIO-Linux-7.91.18301.tar.gz (and for an official release DynamoRIO-Linux-8.0.0.tar.gz instead of DynamoRIO-Linux-8.0.0-1.tar.gz). The build number being part of that was used for manual releases when going through and testing release candidates so we could easily tell them apart, and then we'd release the final one that passed the extra pre-release testing: so sthg like DynamoRIO-Linux-6.1.0-4.tar.gz. But these days the team is too small to do extra pre-release testing and we're moving toward automated releases so I don't think the build number is useful anymore.

derekbruening added a commit that referenced this issue Feb 10, 2020

Partially verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
We cannot verify signatures from co-authors, and some of the co-authors attributed to this commit require their commits to be signed.
Adds jobs to the Travis cronbuild matrix to build ARM, AArch64, and
Android packages.  Adds toolchain setting based on the existing env
var and parameter scheme into package.cmake to match runsuite.cmake.
Adds logic to exclude the Dr. Memory build for AArch64.  Adds ignoring
of a "git tag" failure to handle races among the four jobs.

Adds the 32_only and 64_only args to package.cmake to match runsuite.cmake.
Fixes the Windows Dr. Memory checkout directory.
Makes a Dr. Memory embedded build location error fatal.

Adds proper naming of cross-compiled packages, including adding a new PACKAGE_SUBSYS CMake variable for setting -EABIHF, etc. suffixes.

Issue: #2861
Fixes #3002
@derekbruening
Copy link
Contributor Author

Here is what I'm doing:

  • If build# is 0 src/CMakeLists.txt omits a -NNN from the package name.
  • .travis.yml takes in VERSION_NUMBER and uses it; it has to include the
    -build# if that's desired, and that will be part of the tag;
    if VERSION_NUMBER is not set it does the default.
  • runsuite_wrapper.pl parses VERSION_NUMBER and if it has a -NNN it passes that
    as the build= arg to package.cmake.

Testing:

$ TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=0;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"

$ VERSION_NUMBER=8.1.2 TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=0;version=8.1.2;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"

$ VERSION_NUMBER=8.21.42-39 TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=39;version=8.21.42;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"

Then to make an official release, I would do a manual cron build and say:

env:
  global:
    - TRAVIS_EVENT_TYPE=cron
    - VERSION_NUMBER=8.0.0

We'll go with that for now, keeping the manual releases and manual changelists. One step at a time.

derekbruening added a commit that referenced this issue Feb 10, 2020
If the build number is 0, we omit it from the package name, rather than appending it.

.travis.yml now takes in VERSION_NUMBER and uses it directly.  There
is no more TAG_SUFFIX: VERSION_NUMBER has to include the build number
if that's desired, and that will be part of the tag as well.
If VERSION_NUMBER is not set it uses the default as before.

runsuite_wrapper.pl parses VERSION_NUMBER and if it has a -NNN it passes that
as the build= arg to package.cmake.  It also passes a version= argument.

Testing:
$ TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=0;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"
$ VERSION_NUMBER=8.1.2 TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=0;version=8.1.2;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"
$ VERSION_NUMBER=8.21.42-39 TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=39;version=8.21.42;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"

Fixes #2861
derekbruening added a commit that referenced this issue Feb 10, 2020
If the build number is 0, we omit it from the package name, rather than appending it.

.travis.yml now takes in VERSION_NUMBER and uses it directly.  There
is no more TAG_SUFFIX: VERSION_NUMBER has to include the build number
if that's desired, and that will be part of the tag as well.
If VERSION_NUMBER is not set it uses the default as before with a cronbuild- prefix for the tag.

runsuite_wrapper.pl parses VERSION_NUMBER and if it has a -NNN it passes that
as the build= arg to package.cmake.  It also passes a version= argument.

Testing:
$ TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=0;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"
$ VERSION_NUMBER=8.1.2 TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=0;version=8.1.2;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"
$ VERSION_NUMBER=8.21.42-39 TRAVIS_EVENT_TYPE=cron ~/dr/git/src/suite/runsuite_wrapper.pl travis 64_only
Running ctest -VV -S "/home/bruening/dr/git/src/suite/../make/package.cmake,travis;64_only;build=39;version=8.21.42;invoke=/home/bruening/dr/git/src/suite/../drmemory/package.cmake;drmem_only"

Fixes #2861
derekbruening added a commit that referenced this issue Feb 10, 2020
Fix an error where the Android package build is not listed as deploying.

Issue: #2861
derekbruening added a commit that referenced this issue Feb 10, 2020
We trigger a custom build by setting VERSION_NUMBER in Travis.
But Appveyor does not have access to that.
Here we extract that number from the git tag that the Travis build set.

Issue: #2861
derekbruening added a commit that referenced this issue Feb 15, 2020
Turns out the TRAVIS_EVENT_TYPE env var is not set during jobs matrix
creation.  Instead we have to use the "type = cron" condition.

Issue: #2861
derekbruening added a commit that referenced this issue Feb 15, 2020
Turns out the TRAVIS_EVENT_TYPE env var is not set during jobs matrix
creation.  Instead we have to use the "type = cron" condition.

Issue: #2861
derekbruening added a commit that referenced this issue Feb 15, 2020
We have to check both the Travis "type" and the env var to support
both manual and scheduled package builds.

Issue: #2861
derekbruening added a commit that referenced this issue Feb 15, 2020
We have to check both the Travis "type" and the env var to support
both manual and scheduled package builds.

Issue: #2861
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment