Cross Compile for ARMV8

[Derrekito]

I'm trying to cross compile for ARMV8. My build parameters and errors are as followed:

make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=CORTEXA73 USE_OPENMP=1

../../libopenblas_cortexa73p-r0.3.23.dev.a
make[2]: ../../libopenblas_cortexa73p-r0.3.23.dev.a: Permission denied
make[2]: *** [Makefile:626: ../../libopenblas_cortexa73p-r0.3.23.dev.a] Error 127
rm la_xisnan.mod
make[2]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib/SRC'
make[1]: *** [Makefile:27: lapacklib] Error 2
make[1]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib'
make: *** [Makefile:260: netlib] Error 2

OR

make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=ARMV8 USE_OPENMP=1

../../../libopenblas_armv8p-r0.3.23.dev.a
make[2]: ../../../libopenblas_armv8p-r0.3.23.dev.a: Permission denied
make[2]: *** [Makefile:84: ../../../libopenblas_armv8p-r0.3.23.dev.a] Error 127
make[2]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib/TESTING/MATGEN'
make[1]: *** [Makefile:49: tmglib] Error 2
make[1]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib'
make: *** [Makefile:261: netlib] Error 2

[martin-frbg]

the "permission denied" suggests you do not have write access in your OpenBLAS source directory. Did you perhaps unpack the source archive as root ?

[Derrekito]

I simply cloned the repo and tried compiling it using the parameters I mentioned. Also, I am able to natively compile OpenMP using just make. To be clear, I recreated the error (as a normal user) using the following steps:

rm -rf OpenBLAS
git clone https://github.com/xianyi/OpenBLAS.git
make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=ARMV8 USE_OPENMP=

../../libopenblas_armv8p-r0.3.23.dev.a
make[2]: ../../libopenblas_armv8p-r0.3.23.dev.a: Permission denied
make[2]: *** [Makefile:626: ../../libopenblas_armv8p-r0.3.23.dev.a] Error 127
rm la_xisnan.mod
make[2]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLAS/lapack-netlib/SRC'
make[1]: *** [Makefile:27: lapacklib] Error 2
make[1]: Leaving directory '/home/derrekito/Projects/docker-cuda-agx/OpenBLASls -la | grep libopenblas_armv8p
lrwxrwxrwx 1 derrekito derrekito 32 Apr 17 15:33 libopenblas.a -> libopenblas_armv8p-r0.3.23.dev.a
-rw-rw-r-- 1 derrekito derrekito 18438972 Apr 17 15:33 libopenblas_armv8p-r0.3.23.dev.a/lapack-netlib'
make: *** [Makefile:260: netlib] Error 2

In case this helps:

ls -la | grep libopenblas_armv8p
lrwxrwxrwx  1 derrekito derrekito       32 Apr 17 15:33 libopenblas.a -> libopenblas_armv8p-r0.3.23.dev.a
-rw-rw-r--  1 derrekito derrekito 18438972 Apr 17 15:33 libopenblas_armv8p-r0.3.23.dev.a

[martin-frbg]

Strange, I cannot reproduce this (without doing something evil, as follows...) . The error message looks a bit like it is trying to invoke libopenblas.a at this point, instead of running a command (which would be either ar or ranlib) on it. What do you have in the generated OpenBLAS/lapack-netlib/make.inc file ? (An empty entry on the RANLIB= line would probably cause this, but I think this could only happen if you are actively setting RANLIB empty somewhere in your environment)

[Derrekito]

I'm also able to reproduce the error in an Ubuntu 20.04 Docker environment using my exact make line as shown earlier.

Here is my make.inc file

FC          = aarch64-linux-gnu-gfortran-11
override FFLAGS      = -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -fno-tree-vectorize
FFLAGS_DRV  = -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a
POPTS       = -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -pg -fno-inline
FFLAGS_NOOPT       = -O0 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a
PNOOPT      = -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -pg -fno-inline -O0
LDFLAGS     = -O2 -Wall -frecursive -fno-optimize-sibling-calls -fopenmp -fPIC -march=armv8-a -lm -lpthread -lgfortran
CC          = aarch64-linux-gnu-gcc-11
override CFLAGS      = -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=20 -DMAX_PARALLEL_NUMBER=1 -DBUILD_SINGLE=1 -DBUILD_DOUBLE=1 -DBUILD_COMPLEX=1 -DBUILD_COMPLEX16=1 -DVERSION="0.3.23.dev" -march=armv8-a -UASMNAME -UASMFNAME -UNAME -UCNAME -UCHAR_NAME -UCHAR_CNAME -DASMNAME= -DASMFNAME=_ -DNAME=_ -DCNAME= -DCHAR_NAME="_" -DCHAR_CNAME="" -DNO_AFFINITY -I. -DHAVE_LAPACK_CONFIG_H
AR          = ar
ARFLAGS     =  -ru
LAPACKLIB   = ../../libopenblas_armv8p-r0.3.23.dev.a
TMGLIB      = ../../../libopenblas_armv8p-r0.3.23.dev.a
BLASLIB     = ../../../libopenblas_armv8p-r0.3.23.dev.a
LAPACKELIB  = ../../../libopenblas_armv8p-r0.3.23.dev.a
LAPACKLIB_P = ../libopenblas_armv8p-r0.3.23.dev_p.a
SUFFIX      = o
PSUFFIX     = po
CEXTRALIB   = -lm -lpthread -lgfortran
TIMER       = INT_ETIME
LOADER      = aarch64-linux-gnu-gfortran-11 -pthread
BUILD_DEPRECATED      = 1
BUILD_SINGLE      = 1
BUILD_DOUBLE      = 1
BUILD_COMPLEX      = 1
BUILD_COMPLEX16      = 1
LAPACKE_WITH_TMG      = 1
SHELL = /bin/sh
PLAT = _LINUX
DRVOPTS  = $(NOOPT)
#ARCHFLAGS= $(ARFLAGS) -ru
#RANLIB   = ranlib

Changing my build arguments to this fixed the issue!

make CC=aarch64-linux-gnu-gcc-11 FC=aarch64-linux-gnu-gfortran-11 HOSTCC=gcc-11 TARGET=ARMV8 USE_OPENMP=1 RANLIB=ranlib

I'll leave this ticket open for a short while to allow any additional remarks.

[martin-frbg]

Thanks, that appears to confirm my idea - but it is still unclear to me why (and where) you would have an empty RANLIB= entry in your environment. Would that be a default in Ubuntu 20 - but I do not think that would make sense ?

[Derrekito]

Well, the main system I tried on is Pop-OS 22.04. The Docker image is Ubuntu 20.04. The behavior was the same either way. I haven't tested on any other environment (e.g., Arch or Slackware). I believe I can share my Dockerfile if there is any additional interest in tracking down the issue. Otherwise, I'm happy to close this ticket. Please advise. I appreciate all your help!

[martin-frbg]

Can you check what running set to dump the environment tells about the presence of a RANLIB variable ? (Btw it is strange that the make.inc you posted does not have an (uncommented) RANLIB line at all - is this a hand-edited version already ? make.inc gets generated by the toplevel Makefile, and that uses unconditional echo commands to write individual lines. There is no mechanism for querying what RANLIB should be, just a simple "set it to "ranlib" (with the cross-compiler prefix if necessary) unless it is already set" in Makefile.system

[Derrekito]

I have not hand-edited any part of the OpenMP project. My only input is manually setting environment vars at the command line, appended to make, as originally shown. Anyway, running the following does not print any output:

set | grep RANLIB

As an aside, I can compile natively just fine without these issues by simply executing make without setting any additional environment variables.

[martin-frbg]

Thanks, I guess it will have to remain a mystery then - anyway glad that we could solve the problem.

[Dirreke]

I have the same Issue when cross compile for armv8 on Ubuntu22.04

[martin-frbg]

Hmm, does your cross-compiler toolchain have its own ranlib (named with the same prefix as your cross-gcc, so something like aarch64-linux-gnu-ranlib ?) Maybe the flaw in the current OpenBLAS Makefile is assuming that such a program will always exist (and be required) if there are toolchain-gcc and toolchain-ld present.

[Dirreke]

I use the toolchain from linaro. And it does have aarch64-linux-gnu-ranlib

[martin-frbg]

Thank you - however I cannot reproduce this on Ubuntu 22 with the gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu tarball. The library is compiled without any issues, and lapack-netlib/make.inc got created with the line RANLIB=/opt/gcc-linaro-6.3.1-2017.02-x86_64-aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc-ranlib matching the actual program in the toolchain directory.

[Dirreke]

I do some check about it. I found that, actually, Makefile get the RANLIB="aarch64-linux-gnu-ranlib, it has an extra quotes. It result the echo get an error /bin/sh: 1: Syntax error: Unterminated quoted string, which cause the ranlib is not printed to make.inc

[Dirreke]

And the CROSS_SUFFIX in Makefile.conf is wrong

(I use the armv7 to test now)

[Dirreke]

the c-check get the basename bn which is like "aarch64_linux-gnu-gcc". cross-suffix cut by "-" from basename, and get the value with a single quote

[martin-frbg]

Which version are you trying to build ? I still cannot reproduce this with a fresh checkout of the current develop branch. I did commit a broken "fix" for a cross-build problem on April 7 however (#3996) which I only reworked on April 17 (#4010), the same day this issue was created. So any checkouts between these two dates may well have been broken (with respect to crossbuilds).

[Dirreke]

I'm at the newest commit. I can see the #4010 in my local repo

[isuruf]

I can reproduce with a simple,

CC=x86_64-w64-mingw32-gcc HOSTCC=gcc NO_CBLAS=1 NO_LAPACK=1 TARGET=CORE2 make

and can confirm that #3996 is the culprit and that #4010 did not fix it.

[martin-frbg]

Seems basename is handling the quotes differently when there is no leading directory path to remove

[Dirreke]

bn=`basename "$compiler_name"`
is ok.No need to use \"

[martin-frbg]

assuming that works for all the funky combinations of CC="my compiler -with-options" that started this mess...

[Dirreke]

if \" is used , " will be a part of content.

When the compiler_name has the leading directory, the bn get the value like aarch64-linux-gnu-gcc", which has a single quote at the last of the string. Therefore, the cross_suffix will be right.

[Dirreke]

" is ok, and \ is needless

[Dirreke]

the " in ` does not require escape character