- Adding firewall entries for Jenkins workers
release-*
machines- RHEL7-S390X
- macOS
- AIX 7.1
- AIX 7.2 Install
- Windows (Azure/Rackspace)
- jenkins-workspace
- Docker hosts
- SmartOS
- Raspberry Pi
- IBM i
- z/OS
Workers must be added to the firewall config before they will be able to connect to the Jenkins master.
You must be part of the infra group and have setup the ssh keys and config file beforehand.
To add an entry do the following:
- ssh to the ci master:
ssh ci
- save the current config to a temporary file:
iptables-save >foo
- edit the temporary file with your favorite editor. Use one of
the existing lines as a template and add a new entry at the end
of the list of hosts just before the second
COMMIT
line near the end of the file. - restore the config from the temporary file:
iptables-restore foo
- remove the temporary file:
rm foo
- run
iptables-save > /etc/iptables/rules.v4
to ensure the changes persist across reboots
Once setup, they must have ~iojs/.ssh
cloned from another machine, so they
have the ssh setup and keys required to upload release artifacts to the
nodejs.org web server. The result will be 2 files, an id_rsa
containing
a private key, and a config
containing:
Host node-www
HostName direct.nodejs.org
User staging
IdentityFile ~/.ssh/id_rsa
Both the config
file and id_rsa
should be owned and only readable by the
user: chmod 700 .ssh && chmod 600 .ssh/*
.
Its necessary to accept the known_hosts
keys interactively on first ssh or
the release builds will fail. After setting up .ssh, do something like this:
ssh node-www date
// ... accept the host keys
In the case of Docker container release hosts, the SSH configuration above works
differently since the ~iojs
home directories are elsewhere on the host
machine. The Docker containers are started with /home/iojs
inside the
container mounted from /home/iojs/name-of-container/
on the host machine.
Therefore, the above SSH configuration should take place in
/home/iojs/name-of-container/.ssh/
, with permissions set appropriately.
known_hosts
can be primed and SSH tested from within the running containers:
- Find the running container ID using
docker ps
- Enter the container using
docker exec <containerid> -ti bash
- Run
ssh node-www date
(as above)
Previous notes: #1393
Xcode Command-line tools are not enough to perform a full notarization cycle, full Xcode must be installed manually.
As root:
- Download Xcode: https://developer.apple.com/download/more/ - find non-beta version, open Developer Tools in browser, Networking tab, start download (then cancel), in Networking tab "Copy as cURL" (available in Chrome & FF)
- Download onto release machine using the copied curl command (may need
-o xcode.xip
appended to curl command) to/tmp
- Extract:
xip --expand xcode.xip
- Move
Xcode.app
directory to/Applications
xcode-select --switch /Applications/Xcode.app
xcodebuild -license
- accept license
- Extract from secrets/build/release:
dotgpg cat Apple\ Developer\ ID\ Node.js\ Foundation.p12.base64 | base64 -D > /tmp/Apple\ Developer\ ID\ Node.js\ Foundation.p12
- Transfer to release machine (scp to /tmp)
sudo security import /tmp/Apple\ Developer\ ID\ Node.js\ Foundation.p12 -k /Library/Keychains/System.keychain -T /usr/bin/codesign -T /usr/bin/productsign -P 'XXXX'
(where XXXX is found in secrets/build/release/apple.md) (security unlock-keychain -u /Library/Keychains/System.keychain
may be required prior to running this command).
- security -i unlock-keychain (Enter the password for the machine located in secrets)
- security find-certificate -c "Developer ID Application" -p > /tmp/app.cert (outputs the PEM format of the cert so we can properly inspect it)
- security find-certificate -c "Developer ID Installer" -p > /tmp/installer.cert
- openssl x509 -inform PEM -text -in /tmp/app.cert | less
- openssl x509 -inform PEM -text -in /tmp/installer.cert | less
The last two steps will show the details of the certificates allowing to see expiry dates.
Example:
Not Before: Jan 22 03:40:05 2020 GMT
Not After : Jan 22 03:40:05 2025 GMT
First copy the rpms from a machine that already has them
scp -r test-ibm-rhel7-s390x-1:/data/devtoolset-6-s390x-rpms/ ~/devtoolset-6-s390x-rpms
Then copy them over to the target machine
scp -r ~/devtoolset-6-s390x-rpms {target host}:~/devtoolset-6-s390x-rpms/
Then install the rpms
yum install -y devtoolset-6-s390x-rpms/*
- Update Sudoers file:
this requires NOPASSWD
to be added to the sudoers file to enable elevation
sudo visudo
and change:
%admin ALL = (ALL) ALL
to
%admin ALL = (ALL) NOPASSWD:ALL
- Allow ssh access
sudo systemsetup -setremotelogin on
Some libuv/Node.js tests currently fail on AIX with a network interface
containing a link local address. This is being tracked in
nodejs/node#39143. In the meantime the en1
interface containing the link local address is removed.
sudo ifconfig en1 down detach
Use
ifconfig -a
to list the available interfaces. To add back the en1
interface, run
sudo autoconf6 -i en1
Most packages should be installed via ansible.
If there are any missing they should be installed via yum
What you do need to install manually is ccache
mkdir -p /opt/gcc-6.3 && cd /opt/gcc-6.3
curl -L https://ci.nodejs.org/downloads/aix/gcc-6.3-aix7.2.ppc.tar.gz | /opt/freeware/bin/tar -xzf -
mkdir -p /opt/ccache-3.7.4 && cd /opt/ccache-3.7.4
curl -L https://ci.nodejs.org/downloads/aix/ccache-3.7.4.aix7.2.ppc.tar.gz | /opt/freeware/bin/tar -xzf -
For AIX 7 and 6.1, needed for the file watcher unit tests.
Add the following to /etc/filesystems:
/aha:
dev = /aha
vfs = ahafs
mount = true
vol = /aha
and then:
mkdir /aha
mount /aha
- Download 16.1.0 packages from: https://testcase.boulder.ibm.com (username: xlcomp4, password: ask @mhdawson)
- scp them to target:/opt/ibm-xlc
- on target:
cd /opt/ibm-xlc
uncompress 16.1.0.3-IBM-xlCcmp-AIX-FP003.tar.Z
tar -xvf 16.1.0.3-IBM-xlCcmp-AIX-FP003.tar
uncompress IBM_XL_C_CPP_V16.1.0.0_AIX.tar.Z
tar -xvf IBM_XL_C_CPP_V16.1.0.0_AIX.tar
installp -aXYgd ./usr/sys/inst.images -e /tmp/install.log all
inutoc
installp -aXgd ./ -e /tmp/install.log all
- Find compilers in
/opt/IBM/xl[cC]/16.1.0/bin/
-
download gcc-c++ (with dependencies) from bullfreeware.com
-
scp 15412gcc-c++-6.3.0-1.aix7.2.ppc.rpm-with-deps.zip TARGET:/ramdisk0
- Note: / is too small
-
unzip 15412gcc-c++-6.3.0-1.aix7.2.ppc.rpm-with-deps.zip
-
contained wrong libstdc++-9.1, so downloaded bundle for libstdc++ 6.3.0-1
-
unpack the RPMs:
$ for f in *gcc* *stdc*; do rpm2cpio $f | /opt/freeware/bin/cpio_64 -idmv; done
-
Find absolute symlinks, and make them relative, example:
$ find . -type l | xargs file ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/ppc64/libatomic.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/libatomic.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/ppc64/libgcc_s.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/libgcc_s.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/ppc64/libstdc++.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/libstdc++.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/ppc64/libsupc++.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/libsupc++.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/ppc64/libatomic.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libatomic.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/ppc64/libgcc_s.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libgcc_s.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/ppc64/libstdc++.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libstdc++.a. ./opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/ppc64/libsupc++.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libsupc++.a.
bash-5.0# pwd /ramdisk0/aixtoolbox/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/ppc64
bash-5.0# ln -fs ../libatomic.a ../libgcc_s.a ../libstdc++.a ../libsupc++.a ./
bash-5.0# find . -type l | xargs file ./ppc64/libatomic.a: archive (big format) ./ppc64/libgcc_s.a: archive (big format) ./ppc64/libstdc++.a: archive (big format) ./ppc64/libsupc++.a: archive (big format) ./pthread/ppc64/libatomic.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libatomic.a. ./pthread/ppc64/libgcc_s.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libgcc_s.a. ./pthread/ppc64/libstdc++.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libstdc++.a. ./pthread/ppc64/libsupc++.a: symbolic link to /opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/6.3.0/pthread/libsupc++.a.
bash-5.0# cd pthread/ppc64/
bash-5.0# ln -fs ../libatomic.a ../libgcc_s.a ../libstdc++.a ../libsupc++.a ./
bash-5.0# file *.a libatomic.a: archive (big format) libgcc.a: archive (big format) libgcc_eh.a: archive (big format) libgcc_s.a: archive (big format) libgcov.a: archive (big format) libstdc++.a: archive (big format) libsupc++.a: archive (big format)
-
Move to target location and create a tarball with no assumptions on leading path prefix:
$ mkdir /opt/gcc-6.3 $ cd /opt/gcc-6.3 $ mv .../opt/freeware/* ./ $ tar -cvf ../gcc-6.3-aix7.2.ppc.tar *
Example above was for 6.3.0, but process for 4.8.5 is identical, other than the version numbers.
Example search for 4.8.5 gcc on bullfreeware:
Notes:
-
AIX tar doesn't know about the "z" switch, so use GNU tar.
-
Build tools create 32-bit binaries by default, so explicitly create 64-bit ones.
$ curl -L -O https://github.com/ccache/ccache/releases/download/v3.7.4/ccache-3.7.4.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 607 0 607 0 0 3281 0 --:--:-- --:--:-- --:--:-- 3281 100 490k 100 490k 0 0 586k 0 --:--:-- --:--:-- --:--:-- 60.4M $ /opt/freeware/bin/tar -xzf ccache-3.7.4.tar.gz $ cd ccache-3.7.4 $ ./configure CC="gcc -maix64" && gmake $ mkdir -p /opt/ccache-3.7.4/libexec /opt/ccache-3.7.4/bin $ cp ccache /opt/ccache-3.7.4/bin $ cd /opt/ccache-3.7.4/libexec $ ln -s ../bin/ccache c++ $ ln -s ../bin/ccache cpp $ ln -s ../bin/ccache g++ $ ln -s ../bin/ccache gcc $ ln -s ../bin/ccache gcov $ cd cd /opt/ccache-3.7.4 $ tar -cf /opt/ccache-3.7.4.aix7.2.ppc.tar.gz *
In order to get Windows machines to a state where Ansible can be run against them, some manual steps need to be taken so that Ansible can connect.
Machines should have:
- Remote Desktop (RDP) enabled, the port should be listed with the access credentials if it is not the default (3389).
- PowerShell access enabled, the port should be listed with the access credentials if it is not the default (5986).
Install the pywinrm
pip module: pip install pywinrm
The preparation script needs to be run in PowerShell (run as Administrator):
iwr -useb https://raw.githubusercontent.com/ansible/ansible/devel/examples/scripts/ConfigureRemotingForAnsible.ps1 | iex
Test the connection to the target machine with ansible HOST -m win_ping -vvvv
. If there is any issue, please refer to the official Ansible documentation in Setting up a Windows Host.
The hosts labelled jenkins-workspace are used to "execute" the coordination of Jenkins jobs. Jenkins uses them to do the initial Git work to figure out what needs to be done before farming off to the actual test machines. These machines are lower powered but have large disks so they can waste space with the numerous Git repositories Jenkins will create in this process. The use of these hosts takes a load off the Jenkins master and prevents the Jenkins master from filling up its disk with Git repositories.
Note that not all jobs can use jenkins-workspace servers for execution, some are tied to other hosts.
The jenkins-workspace hosts are setup as standard Node.js nodes but are only given the jenkins-workspace
label. After setup, they require the following manual steps:
- Download the Coverity Build Tool for Linux x64 at https://scan.coverity.com/download (requires a Coverity login)
- Extract to
/var
, e.g. so the resulting directory looks like/var/cov-analysis-linux64-2017.07/
or similar - Ensure that the node-coverity-daily job matches the path used in its explicit
PATH
setting
The hosts that run Docker images for "sharedlibs", Alpine Linux and a few other dedicated systems (hosts identified by grep _docker-x64- inventory.yml
) don't have Docker image reload logic built in to Ansible. Changes to Docker images (adding, deleting, modifying) involve some manual preparation.
The general steps are:
- Stop the concerned Jenkins systemd service(s) (
sudo systemctl stop jenkins-test-$INSTANCE
) - Disable the concerned Jenkins systemd service(s) (
sudo systemctl disable jenkins-test-$INSTANCE
) - Remove the Jenkins systemd service configuration (
rm /lib/systemd/system/jenkins-test-$INSTANCE.service
) systemctl daemon-reload
to reload systemd configuration from disksystemctl reset-failed
to remove the disabled and removed systemd service(s)- Clean up unnecessary Docker images (
docker system prune -fa
to clean everything up, or justdocker rm
for the images that are no longer needed and a lighterdocker system prune
after that to clean non-tagged images).
Steps 3-5 may not be strictly necessary in the case of a simple modification as the existing configurations will be reused or rewritten by Ansible anyway.
To completely clean the Jenkins and Docker setup on a Docker host to start from scratch, either re-image the server or run the follwing commands:
systemctl list-units -t service --plain --all jenkins* | grep jenkins-test | awk '{print $1}' | xargs -l sudo systemctl stop
systemctl list-units -t service --plain --all jenkins* | grep jenkins-test | awk '{print $1}' | xargs -l sudo systemctl disable
sudo rm /lib/systemd/system/jenkins-test-*
sudo systemctl daemon-reload
sudo systemctl reset-failed
sudo docker system prune -fa
To do this across multiple hosts, it can be executed with parallel-ssh
like so:
parallel-ssh -i -h /tmp/docker-hosts 'systemctl list-units -t service --plain --all jenkins* | grep jenkins-test | awk '\''{print $1}'\'' | xargs -l sudo systemctl stop'
parallel-ssh -i -h /tmp/docker-hosts 'systemctl list-units -t service --plain --all jenkins* | grep jenkins-test | awk '\''{print $1}'\'' | xargs -l sudo systemctl disable'
parallel-ssh -i -h /tmp/docker-hosts 'sudo rm /lib/systemd/system/jenkins-test-*'
parallel-ssh -i -h /tmp/docker-hosts 'sudo systemctl daemon-reload'
parallel-ssh -i -h /tmp/docker-hosts 'sudo systemctl reset-failed'
parallel-ssh -i -h /tmp/docker-hosts 'sudo docker system prune -fa'
Note that while this is being done across all Docker hosts, you should disable node-test-commit-linux-containered to avoid a queue and delays of jobs. The Alpine Linux hosts under node-test-commit-linux will also be impacted and may need to be manually cancelled if there is considerable delay. Leaving one or more Docker hosts active while reloading others will alleviate the need to do this.
Joyent SmartOS machines use libsmartsshd.so
for PAM SSH authentication in order to look up SSH keys allowed to access machines. Part of our Ansible setup removes this so we can only rely on traditional SSH authentication. Therefore, it is critcal to put nodejs_test_*
public keys into $USER/.ssh/authorized_keys
as appropriate or access will be lost and not recoverable after reboot or sshd restart (part of Ansible setup).
Raspberry Pi configuration is integrated into the standard Ansible playbooks and will run properly when the right hosts are executed.
The current configuration relies upon an NFS boot and NFS root architecture, although it should still be possible to connect a non-NFS Raspberry Pi to the Node.js CI and the Ansible playbooks are intended to be friendly to non-NFS hosts. It's possible that current Ansible scripts don't properly account for non-NFS hosts since these are not regularly included so some adjustments may be necessary.
This document covers much of the process we use: https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md
- The SD card in the Raspberry Pi should have an up to date bootcode.bin (found in the FAT partition of the Raspbian images and comes via regular updates) and ideally an updated firmware. It is possible to boot newer Pi's without SD card but the SD card is necessary for older models and our Ansible setup uses it for local swap space.
- Upon boot, the bootcode will cause the Pi to reach out for an initial DHCP discovery. A DHCP server should be configured to respond appropriately for that device's address along with some NFS boot signals, such as this configuration for dnsmasq:
enable-tftp
tftp-root=/var/tftpd
pxe-service=0,"Raspberry Pi Boot"
- The tftp server on the same DHCP server should be able to respond to boot requests.
- After looking in the root of the tftp server, the Pi will attempt to load files from a subdirectory named by its serial number. Obtain the serial number from /proc/cpuinfo from a running Pi, take the last 8 characters from the
Serial
field and use that to store individual boot files. - Copy the entirety of the boot partition of the Raspbian disk image into a subdirectory for each Pi. Symlinks are acceptable to map serial numbers to the names of the Pi's to keep them organised properly.
- Edit cmdline.txt inside the tftp subdirectory for each file and replace the contents with:
modprobe.blacklist=bcm2835_v4l2 root=/dev/nfs nfsroot=NFS_ROOT_SERVER_IP:/NFS_ROOT_FOR_THIS_PI,vers=3 rw ip=dhcp rootwait elevator=deadline
- Replacing
NFS_ROOT_SERVER_IP
andNFS_ROOT_FOR_THIS_PI
as appropraite. It is assumed that NFSv3 is used for the root server. Themodprobe.blacklist
is for a sound driver that has been causing problems in most 2019 versions of the Raspbian kernels (Stretch and Buster), this may be fixed at a later day and be unnecessary. Booting would freeze late in the process without this driver removed. - The DHCP / tftp server should export each of the boot directories via NFS so they can be mounted by the Pi's as /boot/ which will allow the files to be updated during system updates.
An NFS root server can be separate from NFS boot server, and could be different for each Pi.
- The NFS root server should export a shared .ccache directory to be mounted by all Pi's, so it should be exported in such a way as to be permissive with IP addresses. The export should have roughly these options:
(async,rw,all_squash,anonuid=1001,anongid=1002,no_subtree_check)
. - The NFS root server should export a root directory for each Pi. The IP of the server along with the path to the directory should be put in cmdline.txt on the NFS boot server. The exports should have roughly these options:
(rw,sync,no_subtree_check,no_root_squash)
/ - The ext4 partition of the Raspbian image file (second partition, not the boot FAT partition) should be extracted into this root directory.
etc/fstab
in the root directory should be edited to make it NFS compatible. Remove the existing/boot
and/
entries and replace them with:
/dev/mmcblk0p1 /mnt/mmcblk0p1 vfat defaults 0 0
NFS_ROOT_SERVER_IP:PATH_TO_SHARED_CCACHE_DIRECTORY /home/iojs/.ccache nfs4 rw,exec,async,noauto 0 0
NFS_BOOT_SERVER_IP:PATH_TO_TFTP_BOOT_EXPORT /boot nfs4 nfsvers=3,rw,noexec,async,noauto 0 0
- Ansible should also perform checks on
/etc/fstab
so these modifications may not be strictly necessary but it is helpful to have first-boot be into an appropriate state. - Mounting
/
is done during the NFS boot process so is omitted from/etc/fstab
. - The SD card is mounted at
/mnt/mmcblk0p1
and is assumed to be in this location by the Ansible scripts for swap file creation. /boot
is this Pi's tftp boot directory from the NFS boot server.- When powered on, the Pi should perform all mount steps and present with the standard initial Raspbian boot & login. Note that SSH is not enabled by default and this needs to be done manually before you can remotely access it. To streamline setup, these additional steps can be performed on an initial Pi and then its root directory copied to all other root directories with only minor modifications to
/etc/fstab
required:- Enable SSH with
raspi-config
- Add the
nodejs_build_test
public SSH key to~pi/.ssh/authorized_keys
(with appropriate permissions). - Change the default password for user
pi
to remove insecurity. This could even be disabled entirely since the SSH key is in place.
- Enable SSH with
After these steps are performed and the Pi's are running, Ansible can be run to finish setup. A reboot is recommended after initial setup to ensure the environment is configured correctly (locale and other settings that are changed).
There isn't a system start service on IBMi -- the machine should not be
rebooted, and after ansible is run, jenkins needs to be started with
jenkins-start.sh
. This will submit the job under the iojs user. If the
job is already running, the jenkins-start.sh
script will not start
another job.
See http://ibm.biz/ibmi-rpms (see "Installation" section)
/QOpenSys/usr/bin/system -kpib 'CRTUSRPRF USRPRF(NODEJS) PASSWORD() USRCLS(*SECOFR) SPCAUT(*USRCLS) PWDEXPITV(*NOMAX)'
mkdir -p /home/NODEJS
chown -R nodejs /home/NODEJS
Edit /QOpenSys/etc/profile
to contain:
PATH=/QOpenSys/pkgs/bin:$PATH
export PATH
This can be done by running the following commands from a shell:
echo 'PATH=/QOpenSys/pkgs/bin:$PATH' >> /QOpenSys/etc/profile
echo 'export PATH' >> /QOpenSys/etc/profile
After that is completed, copy to the .bashrc
file for the nodejs user
cp /QOpenSys/etc/profile /home/NODEJS/.bashrc
/QOpenSys/pkgs/bin/yum install chsh
/QOpenSys/pkgs/bin/chsh -s /QOpenSys/pkgs/bin/bash
/QOpenSys/pkgs/bin/chsh -s /QOpenSys/pkgs/bin/bash nodejs
The system Java installed is too old to be able to verify the SSL certificate
for our Jenkins servers and a more recent version has to be installed manually.
The script used to start the Jenkins agent expects to find the Java SDK in
/u/unix1/java/J8.0_64/
.
To install the Java SDK, obtain the latest Java 8 service refresh for z/OS from: https://developer.ibm.com/javasdk/support/zos/
Transfer the pax.Z file to the z/OS system (via sftp, do not use scp as that
will perform an unwanted character conversion). Log into the z/OS system and
extract the SDK via the pax
command:
e.g. if the pax.Z file is located in /u/unix1/SDK8_64bit_SR6_FP10.PAX.Z
mkdir -p /u/unix1/java
cd /u/unix1/java
pax -rf /u/unix1/SDK8_64bit_SR6_FP10.PAX.Z -ppx