pkgdev:bulk

Contents

1. Introduction
2. Host setup
3. Generate SSH key
4. Build pbulk tools (macOS/Linux)
5. Install Manta tools (optional)
6. Configure pbulk
7. pbulk.conf.local
8. pkgbuild.conf.local
9. mk.conf.local
10. Perform bulk build

Introduction

Bulk builds are a way to automatically build a large number of packages, generating a nice report at the end summarising the results. We use bulk builds to build our official package sets.

Results for various operating systems are posted regularly to the https://mail-index.netbsd.org/pkgsrc-bulk/ mailing list.

Note that the steps below should be NOT be performed inside a run-sandbox session, but on the primary OS. The bulk build software and build scripts set up sandbox chroots automatically as part of the build process.

Host Setup

Ensure you have followed the necessary setup instructions for your target OS first.

Generate SSH key

Bulk builds support distributed builds via SSH, but even for single host builds we use SSH to launch local builds. Generate a private key for the host in question to allow unprompted logins.

$ ssh-keygen -f ~/.ssh/pkgsrc_pbulk_rsa -N '' -b 4096 -t rsa
$ cat ~/.ssh/pkgsrc_pbulk_rsa.pub >>~/.ssh/authorized_keys

$ vi ~/.ssh/config
Host 127.0.0.1
	IdentityFile ~/.ssh/pkgsrc_pbulk_rsa

: Test that unprompted logins work and cache the host key.
$ ssh 127.0.0.1 echo

Build pbulk tools (macOS/Linux)

This step is not required on SmartOS, skip to the next step.

The first set of packages we need to build on macOS and Linux are the pbulk tools, and any software required to perform the main bulk builds.

First checkout the joyent/feature/pbulk/trunk branch. This contains some modifications to the pbulk code, for example to enable publishing to Manta.

$ cd /data/pkgsrc
$ git checkout -b joyent/feature/pbulk/trunk origin/joyent/feature/pbulk/trunk

Then execute the build script, passing an appropriate PKGBUILD configuration in the environment.

: Choose one of the options below based on your target OS.
$ export PKGBUILD=linux-trunk-pbulk32   # Generic Linux 32-bit
$ export PKGBUILD=linux-trunk-pbulk64   # Generic Linux 64-bit
$ export PKGBUILD=osx-trunk-pbulk32     # macOS 32-bit
$ export PKGBUILD=osx-trunk-pbulk64     # macOS 64-bit

: Execute the tools build script
$ /data/pkgbuild/scripts/run-jenkins-tools

This may take a while, during which time it will build a pkgsrc bootstrap kit and a bunch of packages.

Install Manta tools (optional)

If you wish to publish your results to Manta, install the Manta CLI into a separate directory, using a sandbox to avoid host pollution.

$ /data/pkgbuild/scripts/run-sandbox ${PKGBUILD}

: Within the sandbox
$ mkdir -p /data/manta
$ cd /data/manta
$ pkg_add nodejs
$ npm install manta
$ exit

Configure pbulk

Now we can switch to configuring our main bulk build. Start by setting PKGBUILD to an appropriate configuration.

: Choose one of the options below based on your target OS.  For SmartOS
: you may choose any of the supported branches and architectures.
$ export PKGBUILD=2017Q4-x86_64         # SmartOS 2017Q4 64-bit
$ export PKGBUILD=linux-trunk-i386      # Generic Linux 32-bit
$ export PKGBUILD=linux-trunk-x86_64    # Generic Linux 64-bit
$ export PKGBUILD=osx-trunk-i386        # macOS 32-bit
$ export PKGBUILD=osx-trunk-x86_64      # macOS 64-bit

Then edit the following files under that pkgbuild directory.

$ cd /data/pkgbuild/conf/${PKGBUILD}

pbulk.conf.local

This file contains overrides from the primary pbulk.conf file, so look at that to get an idea of what variables can be set. Here's similar to what is used for the Ubuntu Linux 16.10 builds.

$ vi pbulk.conf.local

pkgsrc=/data/pkgsrc
report_from_addr="pkgsrc@example.com"
report_from_name="pkgsrc bulk builds"
report_recipients="pkgsrc-reports@example.com"
build_chroots=8
scan_chroots=8
base_url=https://us-east.manta.joyent.com/pkgsrc/public/reports/Linux/trunk/x86_64
MANTA_USER="pkgsrc"
MANTA_SUBUSER="linux"
MANTA_KEY_ID="aa:bb:cc:dd:ee:ff:gg:00:11:22:33:44:55:66:77:88"
MANTA_URL="https://us-east.manta.joyent.com"
MANTA_PATH="/usr/pbulk/bin:/data/manta/node_modules/.bin"
report_manta_target="/pkgsrc/public/reports/Linux/trunk/x86_64"
report_platform="Ubuntu Linux 16.10/x86_64"

If you wish to perform a simpler build to start with, just to check that everything is working ok, or if you only need to build certain packages, then here are some recommended settings for a limited bulk build:

limited_list=/data/build-packages-list
build_chroots=1
scan_chroots=1

Then populate /data/build-packages-list with a list of package directories to build, for example:

$ echo "pkgtools/cwrappers" >/data/build-packages-list

If you'd prefer to simply rsync the report to an existing HTTP server rather uploading than Manta, configure the following variables and remove the manta_* variables:

report_rsync_args="-avz --delete-excluded"
report_rsync_target="192.168.1.10:/data/reports"
base_url="http://192.168.1.10/reports"

You will need to pre-create the ${report_rsync_target} directory to avoid rsync failures.

pkgbuild.conf.local

This file allows you to override any variables from the main pkgbuild.conf file. You will probably want to set FAILURE_RECIPIENTS so that you get notification when the pbulk scan fails, and you may also want to set PKGBUILD_GPG_SIGN_AS if signing your own packages.

$ vi pkgbuild.conf.local

FAILURE_RECIPIENTS=pkgsrc@example.com
PKGBUILD_GPG_SIGN_AS=DEADBEEF

mk.conf.local

You can set your own pkgsrc variables in mk.conf.local, for example if you wish to change the default package options.

$ vi mk.conf.local

# Enable compressed mailboxes with mutt
PKG_OPTIONS.mutt+=	mutt-compressed-mbox

Perform bulk build

Now that everything is set up we can perform the bulk build.

First, switch over to the target branch. This will likely be either one of the joyent/release/* branches if you wish to track a quarterly release branch, or will be the latest pkgsrc trunk from upstream.

: If you are using a pkgbuild image then it will already be on the appropriate
: branch and you may not need to do anything here.
$ cd /data/pkgsrc

: Build against a Joyent quarterly release..
$ git checkout -b joyent/release/2016Q4 origin/joyent/release/2016Q4

: ..or fetch the latest pkgsrc trunk and build that
$ git remote add upstream https://github.com/jsonn/pkgsrc.git
$ git fetch upstream
$ git checkout -b upstream/trunk upstream/trunk

Then, assuming you have correctly set PKGBUILD to an appropriate value, run the bulk build script.

$ /data/pkgbuild/scripts/run-jenkins-build