Home

Introduction

The IBM Advance Toolchain for Linux on Power is a set of open source development tools and runtime libraries which allows users to take leading edge advantage of IBM's latest POWER hardware features on Linux:

POWER8 enablement
POWER8 optimized scheduler
POWER8 Transactional Memory enablement
POWER8 Crypto Operations enablement
POWER8 Fusion enablement

Advance Toolchain 11.0 features:

Support for big endian (ppc64) and little endian (ppc64le)
POWER8 optimized runtime libraries

Advance Toolchain 10.0 features:

Support for big endian (ppc64) and little endian (ppc64le)
POWER8 optimized runtime libraries

Advance Toolchain 9.0 features:

Support for big endian (ppc64) and little endian (ppc64le)
POWER7 and POWER8 optimized runtime libraries

Advance Toolchain 8.0 features:

Support for big endian (ppc64) and little endian (ppc64le)
POWER7 and POWER8 optimized runtime libraries

Advance Toolchain 7.1 features:

Support for little endian (ppc64le)
POWER7 and POWER8 optimized runtime libraries

Advance Toolchain 7.0 features:

POWER7 and POWER8 optimized runtime libraries

Advance Toolchain 6.0 features:

POWER7 optimized runtime libraries

Advance Toolchain releases up to version 5.0 feature:

PPC970, POWER4, POWER5, POWER5+, POWER6, POWER6x and POWER7 optimized runtime libraries

This self-contained toolchain does not rely on the system toolchain and requires minimum dependencies. Nor does it override the default Linux distribution toolchain (it is installed in /opt).

The Advance Toolchain source code can be found at: https://github.com/advancetoolchain/advance-toolchain.

The latest release includes current stable versions of the following packages:

GNU Compiler Collection (gcc, g++ and gfortran), plus individually optimized gcc runtime libraries for supported POWER processors
GNU C library (glibc), individually optimized for supported POWER processors
GNU Binary Utilities (binutils)
Decimal Floating Point Library (libdfp), optimized with hardware DFP support for POWER6, POWER6x and POWER7 but also available in emulation for PPC970, POWER4, POWER5, and POWER5+
AUXV Library (libauxv)
GNU Debugger (gdb)
Performance analysis tools (oprofile, valgrind, itrace)
Muti-core exploitation libraries (TBB, Userspace RCU, SPHDE), starting in version 5.0-3
Plus several support libraries (libhugetlbfs, Boost, zlib, etc)

For more information about AUXV and Optimized Libraries, see this technical report.

Supported Linux Distributions

Supported Linux distributions according to the Advance Toolchain version and their respective links for manual download.

	Advance Toolchain 11.0 ¹	Advance Toolchain 10.0 ¹	Advance Toolchain 9.0 ¹
Fedora 19			Native & Cross-compiler
Fedora 22²		Native & Cross-compiler	Native & Cross-compiler
RHEL 6			Cross-compiler
RHEL 7	Native & Cross-compiler	Native & Cross-compiler	Native & Cross-compiler
SLES 12	Native & Cross-compiler	Native & Cross-compiler	Native & Cross-compiler
Ubuntu 14.04		Native & Cross-compiler	Native & Cross-compiler
Ubuntu 14.10			Native & Cross-compiler
Ubuntu 16.04	Native & Cross-compiler	Native & Cross-compiler
Debian 8		Native & Cross-compiler	Native & Cross-compiler
Debian 9	Native & Cross-compiler

¹ All versions of Advance Toolchain since 8.0 supports both big and little endian.
² Advance Toolchain for Fedora 22 supports little endian only.

This includes all the service packs and update releases for the mentioned Linux distributions.

In addition, we provide a runtime compatibility package for older Linux distributions, so binaries built by the Advance Toolchain can be run on those.

For more information about each individual release, please refer to the release notes file available at the Advance Toolchain file repositories below.

Installation

The gpg public keys gpg-pubkey-6976a827-5164221b (for RHEL6/7, SLES 12 and Ubuntu) and gpg-pubkey-3052930d-5175955a (for RHEL6) are provided in the repositories. These public keys can be used to verify the authenticity of both the Advance Toolchain packages and the repository contents. Download the gpg public key for your Linux distribution and import it using the following commands:

RPM based distributions:

wget ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7/gpg-pubkey-6976a827-5164221b
rpm --import gpg-pubkey-6976a827-5164221b

DEB based distributions:

wget ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu/dists/trusty/6976a827.gpg.key
sudo apt-key add 6976a827.gpg.key

Warnings:

RHEL6 requires both gpg public keys.
On SLES10 and RHEL5, please install the ncurses package (64-bit) before installing the Advance Toolchain.
atX.X in the instructions below should be replaced by the version you are using (for example, at6.0, at7.0, etc)

Using YaST or Zypper (SLES 12)

YaST

To install using YaST, execute yast2 as root. Then follow these steps:

Select Add-on Product.
Select the FTP Protocol:
(x) FTP...
Under Server Name:
ftp.unicamp.br
Under Directory on Server:
/pub/linuxpatch/toolchain/at/suse/SLES_12
You will get a warning about there being no product information available at the given location. This is because the repomd based repository does not contain the YaST product information. This is not a bug. Select [Continue].
Under the Software Management interface, search for "advance toolchain" and mark the advance-toolchain-atX.X-runtime, advance-toolchain-atX.X-devel, advance-toolchain-atX.X-perf and advance-toolchain-atX.X-mcore-libs packages for installation as necessary and click [Accept]. If you are installing a cross compiler, select advance-toolchain-atX.X-cross-ppc64 or advance-toolchain-atX.X-cross-ppc64le.

Zypper

To install using Zypper, you first need (as root) to add the Unicamp FTP to the repository list:

zypper addrepo ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/suse/SLES_12 "Advance Toolchain"

This will create a new repository entry called "Advance Toolchain" pointing to the Unicamp FTP site. Then, execute the following to install:

zypper install advance-toolchain-atX.X-runtime \
               advance-toolchain-atX.X-devel \
               advance-toolchain-atX.X-perf \
               advance-toolchain-atX.X-mcore-libs

Or to install the cross compiler, run:

zypper install advance-toolchain-atX.X-cross-ppc64

You will be prompted to trust the repository key (AdvanceToolchain05). Hit "a" for always trusting this key, then "y" to confirm packages installation. Optionally, you may also install the advance-toolchain-atX.X-mcore-libs package, which contains the multi-core exploitation libraries (versions 5.0-3 and above), and the advance-toolchain-atX.X-selinux package (versions 5.0-7 and above), which enables compatibility with SELinux.

Warning: Do not try to install the advance-toolchain-atX.X-selinux package if you do not have SELinux installed and enabled.

YaST and Zypper support package upgrades for new revision releases (i.e. 6.0-0 to 6.0-1). For new major releases (i.e. 5.0-8 to 6.0-1), please proceed as in a normal installation.

Warning: When upgrading using Zypper on SLES 10, use the following command:

zypper update -t package advance-toolchain-at5.0-runtime

The -t package option is required on SLES 10.

Using YUM and DNF (RHEL6/7 and Fedora 19/22)

First, if you have never created an Advance Toolchain repository configuration file, you need to do so. Create the following file using the text editor of your choice as root:

/etc/yum.repos.d/atX.X.repo

and add the following content:

# Beginning of configuration file
[atX.X]
name=Advance Toolchain Unicamp FTP
baseurl=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL7/gpg-pubkey-6976a827-5164221b
# End of configuration file

Notes

When installing on RHEL6 point to the RHEL6 baseurl, and append the additional gpgkey:

baseurl=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL6
gpgkey=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL6/gpg-pubkey-6976a827-5164221b
       ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/RHEL6/gpg-pubkey-3052930d-5175955a

Fedora 19 uses the same repository of RHEL7.

When installing on Fedora 22, update baseurl to the Fedora 22 repository:

baseurl=ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/redhat/Fedora22

When installing the advanced toolchain cross-compiler on an x86 or amd64 target, the entry gpgcheck=1 should be changed to gpgcheck=0 as these packages are not currently signed.

You may check if the repository was added correctly by running yum repolist. You will see your new atX.X repository listed if the configuration file is correct.

To install, execute yum install as root:

yum install advance-toolchain-atX.X-runtime \
            advance-toolchain-atX.X-devel \
            advance-toolchain-atX.X-perf \
            advance-toolchain-atX.X-mcore-libs

Or to install a cross compiler:

yum install advance-toolchain-atX.X-cross-ppc64

Answer "y" when YUM prompts for confirmation. Optionally, you may also install the advance-toolchain-atX.X-mcore-libs package, which contains the multi-core exploitation libraries (versions 5.0-3 and above), and the advance-toolchain-atX.X-selinux package (versions 5.0-7 and above), which enables compatibility with SELinux.

Warning: Do not try to install the advance-toolchain-atX.X-selinux package if you do not have SELinux installed and enabled.

YUM supports package upgrades for new revision releases (i.e. 5.0-0 to 5.0-1). For new major releases (i.e. 5.0-8 to 6.0-1), please proceed as in a normal installation.

Using IBM Linux on Power tools repository

Advance Toolchain is part of the IBM Linux on Power tools repository, you can get AT's repository configured by installing the ibm-power-repo package:

Download the RPM package from https://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/yum.html.
Install the ibm-power-repo RPM package, by running the following command with root user privileges:

rpm -ivh ibm-power-repo-latest.noarch.rpm

After the installation, run the following command with root user privileges:

/opt/ibm/lop/configure

Then you can install AT by using yum or zypper commands.

AT Downloader

There is a tool available to download AT called AT Downloader.

The AT Downloader is a script to download the latest version of all of the AT packages for a supported distribution and create an ISO image if desired.

You can then manually install the AT packages on your systems.

Note: The preferred method to download AT is configuring the repository (see Yast, Yum, Aptitude and IBM Linux on Power tools repository), the intention of this script is to help users that have limited internet access on their power servers.

Manual RPM installation

If you are installing the rpms manually, you will need to download them from our repositories and install in the following order (due to prerequisites):

advance-toolchain-atX.X-runtime-X.X-X
advance-toolchain-atX.X-devel-X.X-X
advance-toolchain-atX.X-perf-X.X-X
advance-toolchain-atX.X-mcore-libs-X.X-X

where X.X-X is the actual version of the package (i.e. 6.0-1). Optionally, you may also install the advance-toolchain-atX.X-mcore-libs package, which contains the multi-core exploitation libraries (versions 5.0-3 and above), and the advance-toolchain-atX.X-selinux package (versions 5.0-7 and above), which enables compatibility with SELinux. These packages depend on advance-toolchain-atX.X-runtime.

Warning: Do not try to install the advance-toolchain-atX.X-selinux package if you do not have SELinux installed and enabled.

If you are updating your current installation to a new revision release (i.e. 7.0-0 to 7.0-1), you need to update as a batch, so rpm can solve the dependencies:

rpm -Uvh advance-toolchain-atX.X-runtime-X.X-X.ppc64.rpm \
         advance-toolchain-atX.X-devel-X.X-X.ppc64.rpm \
         advance-toolchain-atX.X-perf-X.X-X.ppc64.rpm \
         advance-toolchain-atX.X-mcore-libs-X.X-X.ppc64.rpm

The rpm command should always be run as root. Failure to do so will result in an error similar to this:

error: can't create transaction lock on /var/lib/rpm/__db.000

In case you installed or updated the Advance Toolchain incorrectly, you might end up with an inconsistent rpm database (i.e. showing that the rpms are installed, but no files present in /opt/atX.X). In that case, you will have to force-remove the rpms and install again. To force-remove all the Advance Toolchain rpms:

rpm -e --force advance-toolchain-*

Then reinstall using any of the procedures described above.

Using Aptitude or apt (Ubuntu 14.04/14.10/16.04 and Debian 8)

Configure the repositories

If you have never configured Advance Toolchain repositories in this system, you need to do so. Add the following line to /etc/apt/sources.list using your editor of choice as root.

On i386 or amd64:

deb [arch=i386] ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu trusty atX.X

AT 10.0 and above don't support i386, use:

deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu trusty atX.X

On ppc64el:

deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu trusty atX.X

Notes

When installing on Ubuntu 16.04 (xenial) or Ubuntu 14.10 (utopic), point to its respective repository:

      deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu xenial atX.X
      deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/ubuntu utopic atX.X

When installing on Debian 8 (jessie) or Debian 9 (stretch), point to its respective repository:

      deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/debian jessie atX.X
      deb ftp://ftp.unicamp.br/pub/linuxpatch/toolchain/at/debian stretch atX.X

Aptitude

After configuring the repository, refresh the aptitude cache by running:

sudo aptitude update

Finally, install the Advance Toolchain package running the following command:

sudo aptitude install advance-toolchain-atX.X-runtime \
                      advance-toolchain-atX.X-devel \
                      advance-toolchain-atX.X-perf \
                      advance-toolchain-atX.X-mcore-libs

Or to install a cross compiler:

sudo aptitude install advance-toolchain-atX.X-cross-ppc64

Aptitude supports package upgrades for new revision releases (i.e. 7.1-0 to 7.1-1). For new major releases (i.e. 7.1-1 to 8.0-0), please proceed as in a normal installation.

apt

After configuring the repository, refresh the apt cache by running:

sudo apt-get update

Finally, install the Advance Toolchain package running the following command:

sudo apt-get install advance-toolchain-atX.X-runtime \
                     advance-toolchain-atX.X-devel \
                     advance-toolchain-atX.X-perf \
                     advance-toolchain-atX.X-mcore-libs

Or to install a cross compiler:

sudo apt-get install advance-toolchain-atX.X-cross-ppc64

APT supports package upgrades for new revision releases (i.e. 7.1-0 to 7.1-1). For new major releases (i.e. 7.1-1 to 8.0-0), please proceed as in a normal installation.

Packages Descriptions

In most cases, you do not need to install all of the packages that are provided with the Advance Toolchain. The following list describes when they are needed:

advance-toolchain-atX.X-runtime
Provides base functionality to run Advance Toolchain applications. This package is always required.
advance-toolchain-atX.X-runtime-compat
Substitutes the above package on previous distributions versions. More information about SLES9 and RHEL4 or SLES10 and RHEL5
advance-toolchain-atX.X-devel
Provides development tools. This package is only needed when developing applications.
advance-toolchain-atX.X-perf
Provides tools for measuring performance. It's only useful on some development environments.
advance-toolchain-atX.X-mcore-libs
Provides libraries for multi-thread development, like Boost, SPHDE and Threading Building Blocks. This package is also required on servers running the applications developed with those libraries.
advance-toolchain-atX.X-runtime-atZZ-compat
Install this package only if you need to run an application built with the previous version of the Advance Toolchain on top of the current version. More information on Runtime Compatibility Between Advance Toolchain Versions
advance-toolchain-atX.X-selinux
Provides SELinux settings. Only required when developing or running Advance Toolchain applications on a SELinux enabled environment.
advance-toolchain-atX.X-golang
Provides Golang for ppc64le, it doesn't have any dependencies with the other packages. The files will be installed into /usr/local/go/.
advance-toolchain-atX.X-cross-ppc64 or advance-toolchain-atX.X-cross-ppc64le
Provide a cross compiler for big endian (ppc64) or little endian (ppc64le). These packages are available for x86 (i386) or x86-64 (amd64) in order to generate binaries for POWER.
advance-toolchain-atX.X-cross-common
Provides files common to both cross compiler packages (ppc64 and ppc64le). This package is mandatory for cross compiler installation starting from version 8.0.
advance-toolchain-atX.X-cross-ppc64(le)-runtime-extras
Provides additional libraries to the cross compiler packages (ppc64 or ppc64le).
advance-toolchain-atX.X-cross-ppc64(le)-mcore-libs
Provides the libraries for multi-thread development to the cross compiler (ppc64 or ppc64le).
advance-toolchain-atX.X-<package_name>-debuginfo or advance-toolchain-atX.X-<package_name>-dbg
Provides the .debug files that contain the DWARF debuginfo for the files in <package_name>, this is useful to debug and profile the applications built with AT.

Important Notes

advance-toolchain-atX.X-runtime-compat-X.X-X
- Advance Toolchain 5.x: DO NOT INSTALL on SLES10 and beyond or RHEL5 and beyond. These are intended for runtime compatibility on SLES9 and RHEL4 only (see below). By doing so, you will corrupt your entire Advance Toolcahin installation and you will have to force-remove all the rpms and perform a new installation (see above for instructions on how to do it). If you are getting some strange "Illegal instruction" or "Segmentation fault" errors in runtime when you are absolutely sure those shouldn't be happening, you may want to verify if the compatibility rpm is installed:
  rpm -qa | grep advance-toolchain
  
  And check if it returns advance-toolchain-atX.X-runtime-compat-X.X-X.
- Advance Toolchain 6.x and 7.x: DO NOT INSTALL on SLES11 and beyond or RHEL6 and beyond, for the same reasons cited above.
- Advance Toolchain 8.x, 9.x and 10.x: DO NOT INSTALL on RHEL7 and beyond, for the same reasons cited above.
The Advance Toolchain rpms were designed so that you can install multiple versions of it in your machine if necessary. Different versions will install in different paths. (for example, /opt/at7.0 and /opt/at8.0 for Advance Toolchain 7.0-x and Advance Toolchain 8.0-y).

Installation on SLES9 and RHEL4 — Runtime Compatibility (version 5.x)

Users running applications on SLES9 or RHEL4 may install the compatibility rpm advance-toolchain-atX.X-runtime-compat-X.X-X in order to get the Advance Toolchain optimized runtime libraries. For SLES9 and RHEL4, optimized libraries are available for:

PPC970
POWER4
POWER5
POWER5+

Installation on SLES10, RHEL5/6 — Runtime Compatibility (version 6.x and beyond)

Users running applications on SLES10, RHEL5 or RHEL6 may install the compatibility rpm advance-toolchain-atX.X-runtime-compat-X.X-X in order to get the Advance Toolchain runtime libraries. There are no optimized libraries for the runtime compatibility rpm starting on version 6.0-0.

Runtime compatibility between Advance Toolchain versions

Users running applications built with an older version of the Advance Toolchain may install the compatibility rpm advance-toolchain-atX.X-runtime-atZ.Z-compat-X.X-X in order to run these applications on top of a newer version of the Advance Toolchain runtime, for example, using AT 7.0 to run applications built with AT 6.0.

After installing this package, please run:

/etc/rc.d/init.d/atX.X-runtime-atZ.Z-compat start

Using the Cross Compiler (x86)

The cross compiler provides a way to compile programs for POWER using a x86 (i386) or x86-64 (amd64) machine, helping teams to start developing for POWER without having access to a server in the early development phase. However it does not completely remove the requirement of a POWER server to run a complete test in a proper environment.

For more information about using the cross compiler, visit Building with Advance Toolchain cross compiler documentation.

Usage

After completing the installation steps, the Advance Toolchain is ready for usage. Just call the program directly, for example /opt/atX.X/bin/gcc.

However, some applications have complex build systems (for example: autotool, make, cmake) in which is necessary to correctly set the environment PATH, for example:

PATH=/opt/atX.X/bin:/opt/atX.X/sbin:$PATH make

On cmake build systems, it is also necessary to set the CMAKE_PREFIX_PATH with AT's path, for example:

CMAKE_PREFIX_PATH=/opt/atX.X/

Manual Pages

In order for the system man application to pick up Advance Toolchain installed manual pages, you must export the location of the AT manual pages in the MANPATH variable prior to invoking man. This is best accomplished with the following commands:

unset MANPATH
export MANPATH="/opt/atX.X/share/man:`manpath`"
man <topic>

Or you may override the current environment MANPATH as demonstrated in the following example:

MANPATH="/opt/atX.X/share/man:`manpath`" man lsauxv

Optimization Selection

Directing gcc to build an application for a particular CPU can take advantage of processor-specific instruction selection. In some cases, it can significantly improve performance. Building without selecting a particular CPU simply causes gcc to select the default (lowest common denominator) instruction set.

-mcpu=power4
-mcpu=970
-mcpu=power5
-mcpu=power5+
-mcpu=power6
-mcpu=power6x
-mcpu=power7
-mcpu=power8

Warnings

On Advance Toolchain 10.0 and 11.0 the compiler defaults to -mcpu=power8 -mtune=power8
On Advance Toolchain 7.x, 8.0, and 9.0, the compiler defaults to -mcpu=power7 -mtune=power8
On Advance Toolchain 6.0, the compiler defaults to -mcpu=power6 -mtune=power7
When using -mcpu=power7, DO NOT disable Altivec (for example, -mno-altivec) without also disabling VSX (for example, -mno-vsx). The following combination is illegal:
-mcpu=power7 -mno-altivec

Common GCC options

-fpeel-loops
Peels the loops so that there is enough information that they do not roll much (from profile feedback). This value also turns on complete loop peeling; complete removal of loops with small constant number of iterations.
-funroll-loops
Unroll loops whose number of iterations can be determined at compile time or upon entry to the loop. The -funroll-loops value implies the use of the -frerun-cse-after-loop value. This option makes the code larger, and may or may not make it run faster.
-ftree-vectorize
Perform loop vectorization on trees. This flag is enabled by default at -O3, starting at the GCC 4.3 time frame.
-ffast-math
Sets the -fno-math-errno, -funsafe-math-optimizations, -ffinite-math-only, -fno-rounding-math, -fno-signaling-nans and -fcx-limited-range values. This option causes the preprocessor macro _ FAST_MATH _ to be defined. This option is not turned on by any -O option because it can result in incorrect output for programs which depend on an exact implementation of IEEE or ISO rules or specifications for math functions. It may, however, yield faster code for programs that do not require the guarantees of these specifications.
-mcmodel=medium
Generate PowerPC64 code for the medium model: The TOC and other static data may be up to a total of 4G in size.

GCC options for older Linux distributions

The Advance Toolchain shows significant gains over the older (SLES10 SP3) Enterprise Linux Distributions, because these releases use an older GCC-4.1 compiler that is not enabled for POWER7. In addition, these releases are restricted to POWER6 compatibility mode and cannot leverage the new instructions enabled for POWER7 for applications or runtime libraries. There are, however, some best practices that you can use to maximize your performance.

Best practice for the -mcpu and -mtune values

If your program will be running on the same POWER systems for the foreseeable future, built with the matching -mcpu value.
If your program needs to run on multiple systems, the best strategy is to build with the -mcpu set for the oldest supported system. If your program mostly runs on a newer system with the older system for accommodation or backup, then use the -mcpu set for the oldest system combined with the -mtune value for system that you want to optimize for performance.
If your program in running in POWER6 compatibility mode on a POWER7 System, use the -mcpu=power6 and -mtune=power7 values.
If your code is organized in dynamic libraries, you can compile and build your libraries multiple times using the -mcpu value for the specific Power platform and then install those libraries into the matching /lib64/power directory. This allows the dynamic linker to automatically select the dynamic library optimized for the specific system.

Best practice for large programs

Avoid compiling with the -mminimal-toc value as this adds extra levels of indirection for static data accesses
Compile with the -mcmodel=medium value to optimize static data access and allow the linker to perform additional optimizations on the final program or library image.
For programs with static data exceeding 2 GB, you may need to use the -mcmodel=large value.

Platform and Hardware Capabilities Determination

Since around AT 9.0, glibc includes the function getauxval which can be used at runtime to query the capabilities of the hardware. Users of Advanced Toolchain 9.0 or newer are encouraged to use this instead of libauxv. A manual page exists which documents this function. libauxv may be removed in a future major release of the Advanced Toolchain.

The following example tests whether the runtime CPU has POWER7 Vector extensions:

#include <sys/auxv.h>
#include <stdbool.h>
#include <stdio.h>
bool
has_vector(void)
{
unsigned long int hwcap_mask = (unsigned long int) getauxval (AT_HWCAP);
bool has_vec = (hwcap_mask & PPC_FEATURE_HAS_VSX) != 0;
if (has_vec)
printf("CPU has POWER7 vector extensions\n");
return has_vec;
}

Advance Toolchain versions 10.0 and below include libauxv and lsauxv, a system library and application, respectively, which provide a mechanism for querying the system platform information from the kernel's auxiliary vector. The system hardware capabilities (hwcap) may be queried through the auxiliary vector as well. For example, the platform can be queried dynamically with the following function:

char * platform = (char *) query_auxv (AT_PLATFORM);

Information from the hwcap can be queried in the following manner:

unsigned long int hwcap_mask = (unsigned long int) query_auxv (AT_HWCAP);
if (hwcap_mask & PPC_FEATURE_HAS_FPU)
    printf(" HAS_FPU\n");

For more information about using libauxv and lsauxv can be found in the auxv and lsauxv manual pages provided by the Advance Toolchain (see Manual Pages above).

Relinking a Pre-built Application with the Advance Toolchain

Locate all of the application's .o files. You can also link .a files to pick them all up at once. These will be needed for the relink.

Locate the paths to all of the necessary linked shared-object files, for example:

/usr/X11R6/lib for libXrender
/opt/gnome/lib for libgtk-x11-2.0

Edit /opt/atX.X/etc/ld.so.conf and add the directories to all of the shared object files to the end of this file. Don't forget lib64 for the 64-bit equivalent libraries if applicable, for example:

/opt/gnome/lib/
/opt/gnome/lib64/
/usr/X11R6/lib
/usr/X11R6/lib64/

Run the Advance Toolchain's ldconfig application to regenerate /opt/atX.X/etc/ld.so.cache, for example:

sudo /opt/atX.X/sbin/ldconfig

The loader uses /opt/atX.X/etc/ld.so.cache to find the libraries the application was linked against.

Relink using the Advance Toolchain's compiler:

/opt/atX.X/bin/gcc -g -O2 -o <application_name> <list_of_dot_o_files> \
                   <list_of_dot_a_files> -L<path_to_libraries> \
                   -l<one_for_each_library_needed_for_the_link>

For example:

/opt/at5.0/bin/gcc -g -O2 -o mandelbrot callbacks.o interface.o \
    main.o quadmand.o support.o mandel_internals.a \
    -L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/opt/gnome/lib -lgtk-x11-2.0
    -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 \
    -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl \
    -lglib-2.0 -lfreetype -lfontconfig \
    -lXrender -lX11 -lXext -lpng12 -lz -lglitz -lm -lstdc++ -lpthread \
    -lgthread-2.0

If ld gives an error like the following then you're missing the path to that library in the link stage. Add it with -L<path to library>, e.g.

/opt/at5.0/bin/ld: cannot find -lgtk-x11-2.0

Add -L/opt/gnome/lib/ to the gnome compilation line. You need to tell the linker where to find all of the libraries.

When running the relinked application, if you get an error like the following:

./mandelbrot: error while loading shared libraries: libglib-2.0.so.0: cannot open shared object file: No such file or directory.

You need to add the path to the library in question to /opt/atX.X/etc/ld.so.conf and re-run /opt/atX.X/sbin/ldconfig. The Advance Toolchain's loader needs to know where to find the libraries and uses the generated /opt/atX.X/etc/ld.so.cache to find them.

You can verify that the Advance Toolchain libraries were picked up by running the application prefaced with LD_DEBUG=libs, e.g.

LD_DEBUG=all ./mandelbrot

Warning: do NOT use LD_LIBRARY_PATH to point to the Advance Toolchain libraries if your applications are not relinked with the Advance Toolchain. Doing so can result in ld.so and libc.so version mismatch and cause runtime failures.

Library Search Paths

/opt/atX.X/etc/ld.so.conf already includes /etc/ld.so.conf in the search order, but you may need to re-run /opt/atX.X/sbin/ldconfig in order populate /opt/atX.X/etc/ld.so.cache with the specialized search paths you have added to /etc/ld.so.conf after an Advance Toolchain installation.

If you are running ldd against your binary and it is showing that some libraries are not found, you may need to re-run /opt/atX.X/sbin/ldconfig to fix that.Although, Advance Toolchain has a daemon (atxx-x-cachemanager.service) that monitors the system ld.so.cache and updates /opt/atX.X/etc/ld.so.cache when needed.

The environment variable LD_LIBRARY_PATH is a colon-separated set of directories where libraries should be searched first before the standard set of directories. This list is used in preference to any runtime or default system linker path.

We do not recomend the use of LD_LIBRARY_PATH, you can usually rely on the list compiled in path, but if you have to use it (e.g. your build system uses LD_LIBRARY_PATH or has it as part of a wrapper) be sure that a system directory never appears before the Advance Toolchain directory.

Example:

LD_LIBRARY_PATH=/opt/atX.X/lib64:/lib64:$LD_LIBRARY_PATH

Lock Elision support in glibc

Transactional Lock Elision (TLE) is a technique, implemented on top of the Hardware Transactional Memory, which allows critical sections of code to be speculatively executed by multiple threads, potentially without serializing. In the ideal case, this allows multiple threads to execute such a section of code in parallel. When a data race does occur, each thread will make several attempts to elide the lock before falling back to traditional locking for a period of time. No source modifications are needed to enable this feature, however with profiling, source code can be altered to change things such as structure padding and member placement to minimize false sharing conflicts. Such changes are likely beneficial even without TLE.

It should be noted that TLE may not benefit all applications, and the benefits depend largely on how supported mutexes are used. In essence, TLE lets the hardware track any data races which occur while executing a critical section. POWER8 transactional resources are shared among hardware threads per core, thus enabling more threads per core may result in lower performance depending on how much memory is touched within a critical section. TLE is not recommended for applications which make system calls inside the critical section as the kernel does not support system calls from inside a memory transaction.

Advance Toolchain 9.0 and above has support for Transactional Lock Elision (TLE) on glibc (since Advance Toolchain 9.0-3). TLE is disabled by default, a script to enable it is provided as /opt/atX.X/scripts/tle_on.sh. Note, that the internals of how the script work are subject to change at any time.

Below is a sample program which demonstrates how to use the tle_on.sh script, and how to use it:

#include <stdio.h>
#include <pthread.h>
#include <htmintrin.h>
int main()
{
pthread_mutex_t t = PTHREAD_MUTEX_INITIALIZER;
int elided = 0;
pthread_mutex_lock (&t);
if (_HTM_STATE (__builtin_tcheck ()) == _HTM_TRANSACTIONAL)
elided = 1;
pthread_mutex_unlock (&t);
if (elided)
puts ("Hurray! We are elided!");
else
puts ("Shucks! We are not elided!");
return 0;
}

Compiling this with the following command:

/opt/at9.0/bin/gcc tle.c -mhtm -o tle -O2 -pthread

It can be run as follows:

/opt/at9.0/scripts/tle_on.sh -e yes ./tle

or

/opt/at9.0/scripts/tle_on.sh ./tle

Likewise

/opt/at9.0/scripts/tle_on.sh -h

Will print the help information for the script.

Using the Advance Toolchain with libhugetlbfs-2.0 (and later)

The Advance Toolchain provides its own 32-bit and 64-bit versions of libhugetlbfs 2.x. Please refer to the Advance Toolchain's libhugetlbfs man page for more information on using libhugetlbfs-2.0:

unset MANPATH
export MANPATH="/opt/atX.X/share/man:`manpath`"
man libhugetlbfs

Using the Advance Toolchain with libhugetlbfs-1.0

Warning: Libhugetlbfs 1.0 is deprecated for new users. Please use libhugetlbfs 2.0 (or later) that's provided directly by the Advance Toolchain. If you must use libhugetlbfs 1.0 follow these instructions.

The /opt/atX.X/scripts/createldhuge.sh script is provided (until Advance Toolchain 8.0), which copies /opt/atX.X/bin/ld to /opt/atX.X/bin/ld.orig and creates a wrapper script in /opt/atX.X/bin/ld. You only need to run this if you want the Advance Toolchain to work with libhugetlbfs.

The new /opt/atX.X/bin/ld is a wrapper script which detects whether the --hugetlbfs-link or --hugetlbsf-align switches have been passed to the linker. If so then it sets a script-local LD environment variable to /opt/atX.X/bin/ld.orig and invokes the system's ld.hugetlbfs, e.g.

LD="/opt/atX.X/bin/ld.orig" /usr/share/libhugetlbfs/ld.hugetlbfs *switches*

If it doesn't detect the hugetlbfs-link/hugetlbfs-align switch, then it simply forwards the linker invocation to /opt/atX.X/bin/ld.orig directly.

If libhugetlbfs support is desired, the first thing to do is backup the original Advance Toolchain linker just in case there are problems and you need to restore it manually.

cp -p /opt/atX.X/bin/ld /opt/atX.X/bin/ld.backup

The scripts in /opt/atX.X/scripts/ will do the rest of the work for you:

createldhuge.sh restoreld.sh

Invoke createldhuge.sh to create the wrapper ld:

sudo sh createldhuge.sh /<prefix-to-libhugetlbfs>/share/libhugetlbfs/ld.hugetlbfs /opt/atX.X

This MUST be executed as sudo (or root) for the ld wrapper script to be created properly.

When/If you want to restore the original Advance Toolchain linker, simply run:

sudo sh restoreld.sh.

The Advance Toolchain gcc always ignores the -B/<prefix-to-libhugetlbfs>/share/libhugetlbfs directive because it has been built to always invoke /opt/atX.X/bin/ld directly. You can use the gcc invocation you have always used, e.g.

/opt/atX.X/bin/gcc temp.c -v -o temp -B/<prefix-to-libhugetlbfs>/share/libhugetlbfs/ -Wl,--hugetlbfs-link=BDT

Note: If you invoke /opt/atX.X/bin/ld --hugetlbfs-link=BDT directly, you'll need to supply a -m* flag which is normally provided by gcc directly (see man ld for supported emulations).

Packaging an application built with the Advance Toolchain using RPM

Due to a limitation in the way that RPM generates dependency lists, the symbols from the Advance Toolchain may collide with the ones provided by the Linux distribution. There are two ways to prevent this:

Manually set your dependencies, using the steps below:
- Change your spec file, adding the entry Autoreq: 0 to disable the auto requirements check;
- Perform the build of your application as usual and then check every component manually for its dependencies (usually using ldd to find out shared library dependencies);
- Use the shared library list obtained in the previous step and replace the Advance Toolchain entries found by a single advance-toolchain-atX.X-runtime, where X.X is the Advance Toolchain version being used for the build;
  Note: If any libraries from the Advance Toolchain mcore-libs package have being used, their entries must be replaced by a single advance-toolchain-atX.X-mcore-libs entry instead;
- Add an entry of Requires: <list-found> on your spec file, replacing the "<list-found>" with the list assembled in the previous step;
Note: These steps are only required for the first build to adjust your spec file for packaging, unless changes in the build system, or other packaging spec defined changes occur (in which case, it must be revised to guarantee its functionality).
Use a script to get personalized require dependency filters, using the steps below:
- The /opt/atX.X/scripts/find_dependencies.sh script is provided (since Advance Toolchain 9.0-2) to help set the dependencies. Since RPM version 4.8, you can replace the macro that checks for dependencies. This script accept a list of files as input and provide a list of found dependencies as output, it looks for standard dependencies found inside the Advance Toolchain install path and replace them by a single output of advance-toolchain-atX.X-runtime, or advance-toolchain-atX.X-mcore-libs if the dependency found was a multi core library provided by Advance Toolchain.
- RPM by default points the %__find_requires macro to its standard script located on /usr/lib/rpm/find_requires, to use the find_dependencies.sh script you need to change your spec file, redefining the macro required using the following entries:
  %define __find_requires /opt/atx.x/scripts/find_dependencies.sh </full/path/to/files>
  %define _use_internal_dependency_generator 0
  Take a look at this link for more information about RPM dependencies.

Note: The second method is best used with RPM version 4.8 or later. With earlier RPM versions, the first method is recommended.

Using the Advance Toolchain with IBM XLC and XLF

When compiling binaries using XLC or XLF, the user must add the -F <path_to_cfg_file> option to the compiler command line. The Advance Toolchain provides a script that creates those files in /opt/atX.X/scripts. This script is automatically run during installation. If you need to re-run it later (i.e. you installed XLC/XLF after the Advance Toolchain installation), execute the following command:

/opt/atX.X/scripts/at-create-ibmcmp-cfg.sh

Please notice the absolute path when calling the script. DO NOT call it using a relative path. The script creates the config files in /opt/atX.X/scripts.

This procedure does not affect the default XLC/XLF configuration.

Using the Advance Toolchain with MASS libraries

The IBM's Mathematical Acceleration Subsystem (MASS) libraries consist of a set of mathematical functions for C, C++, and Fortran-language applications that are tuned for specific POWER architectures. The libraries are available for ppc64 and ppc64le.

To use it with AT you need to pass the option -mveclibabi=mass to GCC.

Debugging

gdb can be asked to output _Decimal[32|64|128] formatted floating point registers by default, using its printf command.

When using objdump to inspect POWER6 code, make sure to use the -Mpower6 flag:

/opt/atX.X/bin/objdump -d -Mpower6 <your_file>

The same applies to POWER7 code.

Time zones

Since AT 5.0-7 and AT 6.0-1, the Advance Toolchain automatically sets its time zone to the same time zone used by the system. This is done with the symbolic link /opt/atX.X/etc/localtime to /etc/localtime. To replicate this behavior in the Advance Toolchain 5.0-6, 6.0-0 and earlier versions, it is necessary to replace the old symbolic link:

rm -f /opt/atX.X/etc/localtime && ln -s /etc/localtime /opt/atX.X/etc/localtime

By doing so, further changes of system's time zone will automatically reflects in Advance Toolchain's time zone.

If necessary, the AT can work with a different time zone. All the available time zones are listed in the directory /opt/atX.X/share/zoneinfo.

Firstly, remove the symbolic link /opt/atX.X/etc/localtime:

rm -f /opt/atX.X/etc/localtime

Then, create a symbolic link to the new time zone, i.e.:

ln -s /opt/atX.X/share/zoneinfo/Europe/Athens /opt/atX.X/etc/localtime

SELinux issue

There is a bug on RHEL6 that prevents restorecon to restore the context of symbolic links. This bug affects AT's time zone settings. While Red Hat works to solve this bug, a workaround is available.

Replace the symbolic link by a hard link:

rm -f /opt/atX.X/etc/localtime
ln /etc/localtime /opt/atX.X/etc/localtime

However, if /etc and /opt are stored in different file systems the only solution available is to copy the system's time zone:

rm -f /opt/atX.X/etc/localtime cp /etc/localtime /opt/atX.X/etc/localtime

Using Go compilers provided by the Advance Toolchain

There are two compilers provided by the Advance Toolchain that can be used to compile Go source code on ppc64le starting in AT 9.0: the Golang compiler and gccgo.

The Advance Toolchain provides an IBM-optimized version of the Go compiler from the Golang community for ppc64le starting on AT 9.0.

It is installed by the package advance-toolchain-golang-at-X.Y-Z.ppc64le.rpm. This package is standalone and does not depend on any of the other packages offered by the Advance Toolchain.

This compiler and runtime is based on the final release versions provided by the Go community, plus the newest bug fixes and performance optimizations relevant to Power. The major version of the compiler (i.e. 1.6, 1.7, etc) is upgraded with every Advance Toolchain major release. Please refer to the Release Notes file for the compiler version and included bug/performance fixes included.

The Go compiler for Golang is installed here:

/usr/local/go

When /usr/local/go/bin is added to $PATH, then this go tool is the default. Otherwise, the go tool can be invoked directly by doing:

/usr/local/go/bin/go

The Advance Toolchain also includes gccgo. When the full install of GCC is done from Advance Toolchain, gccgo is included and the go tool which uses gccgo is in the default path. If both GCC (including gccgo) and the Golang compiler from Advance Toolchain are installed, and the Go compiler from Golang is to be used instead of gccgo, then the PATH must be set as described above.

Advance Toolchain Limitations and Known Issues

The cross package upgrade from 9.0-4, 10.0-5, 11.0-1 or older versions to a newer release need to be forced.
The upgrade from 9.0-0 to a newer release need to be forced. The problem was fixed in 9.0-1 and does not affect that, nor newer releases.
oprofile's operf tool running on SLES 11: if running operf as the root user, only --system-wide or --pid modes are supported. This is a permanent restriction.
gcc no longer has support for Graphite loop optimizations starting on Advance Toolchain 6.0-0.
libdfp has some known limitations as indicated in the libdfp README.user documentation.
oprofile currently doesn't support profiling using JVMPI.
oprofile Java profiling restriction: To use oprofile for profiling Java VMs 1.5 or greater, users generally have two options available when invoking the JVM: -agentlib or -agentpath. However, when using the Advance Toolchain's oprofile, you must use
-agentpath:/opt/at5.0/lib64/oprofile/libjvmti_oprofile.so
for 64-bit JVMs, or
agentpath:/opt/at5.0/lib/oprofile/libjvmti_oprofile.so
for 32-bit JVMs.
Upgrading from 5.0-0, 5.0-1 or 5.0-2 to a newer release is not working. The problem was fixed in 5.0-3 and does not affect that, nor newer releases. If you need to upgrade from 5.0-0, 5.0-1 or 5.0-2, please uninstall the Advance Toolchain and then install the new version.

Frequently Asked Questions

How do I find which package provides a specific feature?
What does ";undefined reference to sem_*" mean?
Why doesn't the Advance Toolchain find my library?
Why does my program fail with "No such file or directory"?
Can I set RPATH/RUNPATH with the Advance Toolchain?
Can I mix parts of different glibc versions?
How to verify if the Advance Toolchain is corrupted?
When should I use Advance Toolchain?

How do I find which package provides a specific feature?

On RHEL based systems you can run:

yum provides <feature>

On SLES you need to run the following command:

zypper what-provides <feature>

What does "undefined reference to sem_*" mean?

These kind of errors usually appear when trying to link a multithreaded program without enabling multithreading support (i.e. specify -pthread when calling the linker), e.g.:

/opt/at7.0/bin/gcc main.c -pthread

Why doesn't the Advance Toolchain find my library?

The loader keeps a cache of all libraries available in the library path (/opt/atX.X/etc/ld.so.conf). Every time a new library is installed in one of the directories of the library path or when the library path is changed, it is necessary to update this cache by running /opt/atX.X/sbin/ldconfig as root.

Why does my program fail with "No such file or directory"

The symptom of this issue is: a program built with the Advance Toolchain fails to execute with the error message bash: ./program: No such file or directory.

This message is indicating the system was not able to run this program. Usually this happens because the system wasn't able to find the correct major version of the Advance Toolchain. In other words, Bash didn't find the loader specified in the .interp section of the executable.

You can identify which Advance Toolchain version built this program by running ldd <program>. Then, look for /opt/atX.X/lib64/ld64.so.2. Ensure to install the latest update of the major version you identified.

Can I set RPATH/RUNPATH with the Advance Toolchain?

Yes. The Advance Toolchain uses the RPATH and RUNPATH tags to identify where its libraries are located, but it is possible to add new directories using the linker paramenter -rpath=dir.

By default, the Advance Toolchain will prepend its own lib directory to the RPATH/RUNPATH. However, in case the user changes this behavior, he or she must ensure the Advance Toolchain lib directory will always appear before a directory with another libc, i.e. /usr/lib, /usr/lib64, /lib or /lib64. This will guarantee you won't mix parts of different glibc versions.

Can I mix parts of different glibc versions?

No. The GNU C Library has not been designed to support mixing parts of different glibc versions, e.g.: use the loader provided by the Linux distribution and glibc libraries provided by the Advance Toolchain.

When these libraries are mixed up, the program usually aborts complaining about missing symbols in one of the libraries provided by glibc.

To avoid these issues, the user has to:

Ensure the Advance Toolchain lib directories appear before a directory with another libc in the library search path, i.e. /usr/lib, /usr/lib64, /lib or /lib64. Which includes LD_LIBRARY_PATH, RPATH/RUNPATH and /opt/atX.X/etc/ld.so.conf.
Force to use the Advance Toolchain loader when forcing the preload of an Advance Toolchain library with a program compiled using the system's toolchain, e.g.: LD_PRELOAD=/opt/atX.X/lib64/libtcmalloc.so /opt/atX.X/lib64/ld-X.XX.so /usr/bin/printenv PATH

How to verify if the Advance Toolchain is corrupted?

The package manager provides .

On RPM-based systems (RHEL, SLES, Fedora):

Run rpm --verify <list of packages>.

Replace <list of packages> by the list of Advance Toolchain packages installed in the system, for example: rpm --verify advance-toolchain-atX.X-runtime advance-toolchain-atX.X-devel advance-toolchain-atX.X-perf advance-toolchain-atX.X-mcore-libs

If this command creates any output, some of the packages are corrupted. If it returns 0, the installation is fine.

On DEB-based systems (Ubuntu):

Run debsums <list of packages>.

Replace <list of packages> by the list of Advance Toolchain packages installed in the system, for example: debsums advance-toolchain-atX.X-runtime advance-toolchain-atX.X-devel advance-toolchain-atX.X-perf advance-toolchain-atX.X-mcore-libs

This command is going to print the status for each file. If it returns 2, there is at least one corrupted file. If it returns 0, the installation is fine.

In case the previous commands report any issue, uninstall and install the corrupted package again.

When should I use Advance Toolchain?

The Advance Toolchain provides earlier toolchain functionality and a group of optimized libraries. AT is highly recommended when you want to build an optimized CPU-bound application on POWER or want some of the new toolchain functionalities on POWER before they made into a distribution. Also, you must use AT when you link with a library built with AT.

In some scenarios you will not get many advantages using the Advance Toolchain, and it will be easier to use the distro's toolchain that usually performs quite well. Such cases include when you are building applications that are not CPU-bound and on I/O constrained environments.

Support

Customer support for the Advance Toolchain is provided in one of the following ways:

If you are using the Advance Toolchain as directed by an IBM product team (i.e. IBM XL Compiler or PowerVM Lx86), please report the suspected issues to the IBM Support using that product name and entitlement.
IBM's Support Line for Linux Offerings now provides support for the Advance Toolchain as well. If you have a contract with a Support Line for Linux, place a call to IBM Support, e.g. for United States:
1. Dial 1-800-426-IBM-SERV.
2. Option #2 (Other business products or solutions).
3. Option #2 (Software).
4. Option #7 (Other OS/Linux).
For other countries click here.
All other users can use an electronic forum that is monitored Monday through Friday. For questions regarding the use of the Advance Toolchain or to report a suspected defect, click here.
1. Open the Advance Toolchain topic.
2. Select "Post a New Reply".
3. Enter and submit your question or problem.
4. An initial response will be attempted within 2 business days.
Also, if you have a GitHub account you can open an issue in the Advance Toolchain repository.

The Advance Toolchain is supported on many Linux distributions.

Home

YaST

Configure the repositories

Aptitude

apt

Best practice for the -mcpu and -mtune values

Best practice for large programs

Clone this wiki locally