Skip to content
This repository has been archived by the owner on Aug 2, 2022. It is now read-only.

BATS bash tests for build scripts + various other improvements and fixes #7366

Merged
merged 16 commits into from
Jun 4, 2019

Conversation

NorseGaud
Copy link
Contributor

@NorseGaud NorseGaud commented May 16, 2019

Change Description

  • BATS-CORE is being used for bash unit testing: This will allow us to ensure bash scripts function as they should for users. It's perfect for regression testing and consistency. (see tests/bash-bats/README.md)
  • CICD Pipeline/steps run as the user cicd (instead of root)
    • Improvements to log output in buildkite showing a collapsed tab for each and every command the scripts run.
    • Auto retry failed step on -1 (Agent Lost)
  • Users are prompted if 'which' is missing (we will install it for them if they want): linux build scripts "which/c++: command not found" #7232
  • All prompts are now properly wrapped to be user friendly: scripts/eosio_build.sh is unfriendly when asking "Do you wish to update repositories"? #6931
  • New eosio path: ~/eosio/$VersionFromCmakeFile/ to store dependencies and eosio: scripts/eosio_build.sh is unfriendly when asking "Do you wish to update repositories"? #6931
  • Script filename extensions are now .bash, indicating that bash is required to properly execute.
  • Scripts now use #!/usr/bin/env to pull bash if it's installed into nonstandard /bin/ location
  • Better uninstaller script, removing old "full_uninstaller" (readme updated)
  • Build scripts support DRYRUN=true (doesn't execute anything, but shows you what it would do) and VERBOSE=true (shows the exact commands executed throughout script) for better troubleshooting.
  • Scripts use set -ieo pipefail so they fail properly when a command fails and also show colors for users executing the script (colors not supported on mac yet).
  • Using *_deps file for each distro
    • Package dependency files include flexibility by allowing us to specify what command is used to check if it exists or not (dpkg -s/rpm -qa)
    • no more extra space at end of deps file! (yay)
  • Using helpers/functions for modularity, avoiding having to repeat ourselves in each eosio_build_* script
  • Environment variables have been moved into a separate file for better flexibility
    • Test scripts can pull in the same variables
    • comments added for better comprehension

Compiler support

_You can see almost all of these tests passing in https://buildkite.com/EOSIO/eosio-base-images/builds/137

OS -P existing compiler existing c++/clang < required version no existing compiler
Darwin ✅ Will prompt for pin/install clang8 ✔️ (Hasn't been tested)
Centos 7 ✅ devtoolset-7 ✔️ (Hasn't been tested) ✅ Will prompt for install of build-essentials (fails: see below)
AmazonLinux 2 ✅ gcc-c++ ✔️ (Hasn't been tested) ✅ exits and notifies user
Ubuntu 16.04 ✔️ (Hasn't been tested) ✅ Will prompt for pin/install clang8 ✅ exits and notifies user
Ubuntu 18.04 ✔️ (Hasn't been tested) ✅ Will prompt for pin/install clang8 ✅ Will prompt for install of build-essentials

List of issues

  1. scripts/eosio_build.sh is unfriendly when asking "Do you wish to update repositories"? #6931 : Prompts are now humane + install into version specific folder
  2. centos build script has no sudo; implicitly encourages build as root #7120 : Centos 7 will now use sudo for yum commands if $(whoami) = root
  3. build script boost detection inconsistent and causes grief via not finding boost #7262 : New directory structure should fix this
  4. build script on clean install has compiler detection error #7260 : New logic should prevent this from being a problem
  5. linux build scripts "which/c++: command not found" #7232 : User prompts if missing which or c++ (and not using -P).

Consensus Changes

  • Consensus Changes

API Changes

  • API Changes

Documentation Additions

  • Documentation Additions

@NorseGaud NorseGaud requested review from spoonincode and larryk85 and removed request for spoonincode May 16, 2019 17:03
@spoonincode
Copy link
Contributor

I tried this on fresh 18.04 and it wouldn't use the system compiler is that intended?

@NorseGaud
Copy link
Contributor Author

I tried this on fresh 18.04 and it wouldn't use the system compiler is that intended?

Yep: The logic checks if c++/clang is version > 7. If it's < 7, it will install clang8

@NorseGaud NorseGaud requested a review from arhag May 20, 2019 14:31
@NorseGaud
Copy link
Contributor Author

Base image pipeline runs BATS and also runs the various builds with and without -P
https://buildkite.com/EOSIO/eosio-base-images/builds/61

All passing

New build/install/uninstall scripts

pipeline change

Cmake install path change (removing eosio) and .environment file

gitignore additions

[skip ci] removed 7 from function since devtoolset-8 is available and we might change it in the near future

[skip ci] bats readme updates

[skip ci] Architecture move

[skip ci] Removed set -i from ubuntu

eosio_install improvements

removing -P from pipeline config

Fix exception #

Update to appbase with max priority on main thread

Update to appbase with set_thread_priority_max method

Add call to set_thread_priority_max

(softfloat sync) clean up strict-aliasing rules warnings

this causes a lot of warnings on gcc7

Adds configuration for replay test pipeline

Fix `-b` flag for `cleos get table` subcommand

Add `boost/asio/io_context.hpp` header to `transaction_metadata.hpp`

[skip ci] Fixed build-esssential issue on ubuntu 18

[skip ci] Various fixes for bats

[skip ci] added back build-essentials line

[skip ci] bzip added back to ubuntu deps

[skip ci] More bats fixes to simplify c++ warning

[skip ci] Fix for prompting when all packages are already installed

[skip ci] bootstrap.sh for boost needs toolset=clang if we're pinning (fixes ubuntu16 build

[skip ci] quick removal of debug

[skip ci] Better placement for apt update prompt

[skip ci] Support for llvm/clang7 on ubuntu 16

[skip ci] regex match clang so we can support ehen readlink follows symlinks

[skip ci] Reverted support for clang/llvm7 on ubuntu 16 since it throws an error (Cannot find a C++ compiler that supports both C++11 and the specified C++ flags)

[skip ci] Show output for which install

[skip ci] Moved which to after CURRENT_USER and moved CURRENT_USER to environment file

[skip ci] Support for version inconsistencies

homebrew install fixes and move to helper

Various bats fixes and readme update

centos test fixes for clang/build-essential

amazonlinux2/centos7 gcc-c++ needed for -P

amazonlinux2/centos7 gcc-c++ needed for -P

[skip ci] amazonlinux2/centos7 gcc-c++ needed for -P

[skip ci] centos7 removal since it uses devtoolset-7

[skip ci] Putting centos gcc-c++ install back. Silly mistake

[skip ci] Adding devtoolset-7 install regardless as it's needed for cmake installation

[skip ci] output CXX and CC before build

v3 mac image

[skip ci] Various fixes for running as a user (instead of root)

[skip ci] quick bats fix for amazonlinux

revert of quiet mode for ubuntu

version bump for docker plugin and new mac tag name

version bump for docker plugin and new mac tag name

fixes for bats and ubuntu failing even on success due to teardown [skip ci]

added a check to uninstall-package to speed up tests [skip ci]

[skip ci] Removed working dir as it's no longer needed (Dockerfile defaults to /home/cicd

pipeline fixes for new cicd user home location

testing pipeline location

testing pipeline location

testing pipeline location

testing pipeline location

[skip ci] allow git clone

testing pipeline location

fixes

fixes

pipeline testing

pipeline testing

pipeline testing

pipeline testing

adding --- to Executing so it looks better in buildkite logs

test script fixes

submodule check simplification and fix for amazonlinux2

global VERBOSE=true

fixes for tests

test changes

testing tar of artifact

testing tar of artifact

testing tar of artifact

testing tar of artifact

testing tar of artifact

testing tar of artifact

testing tar of artifact

testing tar of artifact

python3 enable for centos7

package building fixes

centos package fix
@NorseGaud
Copy link
Contributor Author

NorseGaud commented May 23, 2019

Using BATS, I ran every situation I can think of (root user, cicd user, pinned, etc) here: https://buildkite.com/EOSIO/eosio-base-images/builds/137

remove anka-debug

port db_modes_test to python

increate timeout to 1 min

add free port check, current ulimit check

another try

another try

find free port automatically

throw error if trying to create non R1 key on SE or YubiHSM wallet

(fc sync) Fix certificate trust when trust settings is empty

When a certificate has no trust settings it is to be trusted

pipeline.jsonc in use [skip ci]

testing new pipeline [skip ci]

Removed pipeline files from eosio

add support for a simple header in the state database which defines versioning for the application level data.  The intent is to have a more direct informative message when the expectations for in-memory layout of application data has changed due to explicit changes to the structure (as opposed to the implicit change detected by our chainbase level checks)

allow in-place upgrade from unversioned to versioned state database

create the database header when loading from a snapshot

Remove undefined method

Add blk and trx options to connection for transactions only and block only connections

Add log for connection settting

Update handshake message to indicate type of connection, trx or blk or both

Remove unneeded reads_in_flight as there can only ever be one read in flight per connection. Add read_delay_count for giving up on a connection when too many consecutive read delays.

Improve logging of connection type

pipeline.jsonc changes [skip ci]

test scripts moved to pipelines repo

(chainbase) don’t keep file mapping active when in heap/locked mode

When in heap or locked mapping mode, pinnable_mapped_file initially maps the database so that it can copy its contents in to the heap/locked memory. Then on shutdown it copies the heap/locked memory back to that mapped file.

This means that during execution in heap/locked mode the process actually shows double memory usage as expected: once for the mapped file that is kept around, and once for the heap/locked memory.

This change closes the mapping after its contents have been copied to the heap/locked memory. It then reopens the mapping on shutdown so that the contents can be copied back. So now during execution the database is only consuming the expected memory

Use imported targets for boost

Update the main cmake file and submodules so that imported targets are used for boost. This allows the cmake config files in 1.70  to be used without activating legacy mode via Boost_NO_BOOST_CMAKE. Additionally, fc/appbase/chainbase are cleaned up so that they are usable outside of eosio without any quirks. To squelch a lot of repetition I’m relying on fc brining in a lot of deps like many boost components & threads — we link just about everything to fc

no need to start keosd for convert

no need to start keosd for cleos multisig review

Add option for cleos sign subcommand to ask keosd for signing

Fix missing signing when key is inputted from cin

rename option and explanation

dont echo private key when an error occurs

Fix description of cleos sign public key

txn json to file

add file open fail check
@EOSIO EOSIO deleted a comment from spoonincode Jun 3, 2019
@EOSIO EOSIO deleted a comment from spoonincode Jun 3, 2019
@NorseGaud
Copy link
Contributor Author

auto-buildkite-pipelines PR https://github.com/EOSIO/auto-buildkite-pipelines/pull/140

@NorseGaud NorseGaud merged commit 1e88237 into develop Jun 4, 2019
@brianjohnson5972 brianjohnson5972 deleted the bats-v3 branch November 21, 2020 15:08
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants