binaries to output as libblas.dll and liblapack.dll

[fommil]

Can you please output separate dynamic libraries for your project:

• libblas3.dll containing BLAS and CBLAS
• liblapack3.dll containing LAPACK (and no LAPACKE, in keeping with industry norms)

instead of bundling it all into a single DLL. This allows upstream developers to swap the BLAS/LAPACK implementation at runtime to OpenBLAS (if compiled against the reference implementation, for example) or to change to another implementation for performance reasons (e.g. to cuBLAS for the BLAS portion).

This is directly impacting me in fommil/netlib-java#27 on Windows, but it would appear that Linux distribution managers already take care of this.

[xianyi]

Hi @fommil ,

Thank you for the suggestion.

For BLAS, it's no problem. However, OpenBLAS optimized some functions in LAPACK, which called OpenBLAS internal functions. When this separate liblapack3.dll links other BLAS library (e.g. cuBLAS or ATLAS), it will cause the error.

Xianyi

[fommil]

so create libblas and liblapack which dynamically load libopenblas (or inline your functions). the important thing is that you stick to the industry normed api :-)

[fommil]

@xianyi it also seems that you require a few more libraries to be available at runtime that most people wouldn't have:

86_64-w64-mingw32-objdump -p /opt/win-x86_64/openblas/lib/libopenblas.dll | grep DLL
    DLL
 vma:            Hint    Time      Forward  DLL       First
    DLL Name: ADVAPI32.dll
    DLL Name: libgfortran-3.dll
    DLL Name: KERNEL32.dll
    DLL Name: msvcrt.dll

libgfortran-3.dll can be included statically... have a look at how I do this with MinGW and I'm not 100% sure if modern machines actually have ADVAPI32.dll. I'm certainly getting link problems on Windows 8 64 bit.

[fommil]

AND, libgfortran-1-3.dll depends on libquadmath-0.dll and libgcc_s_seh-1.dll :-P

I use these MinGW command linker parameters on 64 bit

https://github.com/fommil/netlib-java/blob/master/native_system/win-x86_64/pom.xml#L90

and these on 32 bit

https://github.com/fommil/netlib-java/blob/master/native_system/win-i686/pom.xml#L90

in my project.

NOTE the bizarre -Wl,--kill-at on 32bit Windows! Essential to ensure your methods are named as you think they are.

[kkofler]

FYI, unfortunately, it looks like ATLAS is also using different library names these days. See this discussion:
https://lists.fedoraproject.org/pipermail/devel/2013-September/189467.html

For GNU/Linux distributions, different library names mean that the BLAS/LAPACK implementation has to be picked at compile time, and that build systems of dependent packages may need patching to support the new names.

[susilehtola]

fommil: libgfortran-1-3.dll, libquadmath-0.dll and libgcc_s_seh-1.dll are gcc runtime libraries. There's no sense in linking them statically, since otherwise the binary size will explode. You'll still have them on any MinGW system.

[fommil]

@kkofler that isn't true for debian and I doubt it ever will be.

@susilehtola the benefit is to the end user. these are small libs.

[susilehtola]

That'd be 1.5MB extra for each binary / library. I don't think it's that small. There's no sense in duplicating system libraries, because you'll have to recompile whenever bug fixes are made and the system libraries are always available.

I'm all for maintaining the current system of having a single library, since only then can you get consistent behavior. As a package maintainer I want to be able to select the library to be used, and not some frankensystem where the output is different for different users.

[susilehtola]

Case in point: parallel software. This would be totally broken if the undelying library can change, e.g. serial <-> pthreads <-> openmp.

[kkofler]

Well, I'm also a Fedora packager, and I think that, as long as we want to support multiple implementations of BLAS/LAPACK, the system which has been used for ATLAS in Fedora until now (i.e. the implementations all have the same sonames, and ld.so.conf.d is used to tell the linker to pick a specific implementation at runtime) is much better. For us packagers, it means that we only have to BuildRequire lapack-devel and not have to worry about what the most optimized implementation available for the given (possibly secondary) architecture is. For the end users, it means that they can install the most optimized implementation we provide for their CPU, or even build their own (tuned for the exact properties of their machine), without having to recompile all the packages using BLAS and/or LAPACK.

In addition, as I have already written on the Fedora devel list, shipping multiple implementations in a way where different packages can (easily) link a different implementation (technically, this was also possible with the old solution by abusing rpath, but then the rpath is your problem, rpaths are evil) is a surefire recipe for symbol conflicts. The situation I'm thinking of is an application linking one BLAS implementation, but the libraries it uses linking one or more other one(s). In the best case, you'd end up with an unexpected implementation (and so you're no better off than before if you were relying on getting a specific one). In the worst case, you end up with symbols from multiple implementations which are incompatible enough to crash the program or cause it to fail with an unresolved symbol. (What happens if the linker loads the reference libblas.so.3 first and resolves the BLAS symbols against that, then loads libopenblas.so.* and tries to resolve the LAPACK symbols against that?) So the only realistic solution I see for Fedora if libopenblas.so is what we're going to be stuck with is to build everything against OpenBLAS and support only OpenBLAS. Of course, you as OpenBLAS maintainers probably like this solution. ;-) But I'm not convinced that this is realistically doable in Fedora due to the politics involved. We haven't even managed to prevent MySQL-Community and Apache OpenOffice from being (re)introduced next to the default MariaDB and LibreOffice. So the symbol conflicts would likely become a serious practical problem.

But yes, incompatibilities with parallelization are a problem. The default implementation (shipped as libblas.so.3 and liblapack.so.3) can only realistically be serial. But shipping the libraries with different names doesn't help there, unless you're also renaming all the symbols. The previous paragraph on symbol conflicts applies here too! To properly support parallel LAPACK/BLAS, you have to build separate serial and parallel versions of ALL the libraries that use LAPACK and/or BLAS, even indirectly! That would probably be many more packages than you expect.

[fommil]

@susilehtola for the avoidance of doubt, I'm only referring to the convenience Windows builds when talking about static linking. For all other platforms, which tend to have sane package managers, of course the dynamic linking is the preference.

[fommil]

+1 for what @kkofler is saying. If you ship a single file then it is next to impossible for users to use your library at runtime using standard conventions. You may as well change all your symbol names and demand that users code against your API instead of BLAS/LAPACK.

[susilehtola]

But as I stated above, none of this really is relevant because as nowadays all cpus are multicore, shortly every program will link to a parallel library which are not interchangeable.

[fommil]

@susilehtola if your LAPACK impl is dependent on your BLAS impl, then document it. If anybody does mix and match implementations, it is perfectly acceptable to have undefined runtime behaviour. Can you give an example of how this works: I struggle to see how you can introduce incompatibilties when implementing the standard BLAS API. If you have your own routines that are BLAS-like, then it is acceptable to put them into a separate library and your BLAS and LAPACK can both access them.

You're missing the fact that there are decades of binaries that link to the libblas and liblapack libraries and you will break binary link compatibility for no good reason, and also force vendor dependence. For linear algebra developers, it is ideal to be able to build against the reference API and then users swap the implementation at runtime. If it is not possible to do that with your library, then you are not implementing BLAS/LAPACK, period: you are implementing something BLAS-like and something LAPACK-like. Just like cuBLAS and clBLAS.

[susilehtola]

fommil: if you have an OpenMP program that uses LAPACK/BLAS functions, and underlying LAPACK/BLAS library is switched from OpenMP to a pthreads version, then things will go crazy and you will break your performance.

This situation is totally analogous to MPI, which is a standard which has multiple implementations (e.g. OpenMPI, MPICH, Intel MPI just to name a few). The only sane way to handle having multiple mutually incompatible runtimes is to separate them totally, e.g. using environment modules.

[fommil]

@susilehtola with separate dynamic libraries for BLAS/LAPACK, users can still code against a specific implementation if they want to: those vendor specific libs are still there. However, what you're doing is cutting off decades of binary compatibility and simply removing yourself from being an option for existing binaries... who may have no specific memory or thread model requirements.

I still don't see why different threading models can cause such a problem. So long as the array pointers are pointers into the heap (and not into some other memory model, like in GPUs), there should be no stability problems, and threading models are simply a machine/user decision. If that runs suboptimally, then boo hoo for the sysadmin. If memory location is really a big issue, then standard BLAS / LAPACK are probably not the best options anyway.

In any case, I have no personal plans to use OpenBLAS. But please know that by doing this you are essentially forcing a lot of end-users away from your library, simply because you're now positioning yourself as a vendor-specific implementation of BLAS/LAPACK. I recommend that you change all the symbol names to reflect this (which is farcical, I know... but there really is no point in implementing BLAS/LAPACK if users can't switch to you at runtime).

[xianyi]

Hi all,

Sorry for the delay response. I juggling projects for weeks.

For binary compatibility issue, I think you can try build with NO_LAPACK=1 flag. Then, OpenBLAS will only contain BLAS/CBLAS functions. The user can switch it at runtime, e.g. renaming libopenblas.dll to libblas3.dll.

Xianyi

[kkofler]

Unfortunately, I don't think that is really a good solution. Sure, it allows to replace BLAS and use the reference LAPACK, but then the users miss out on your optimized LAPACK routines. :-( FYI, what Debian does is to symlink both libblas.so.3 and liblapack.so.3 to the libopenblas.so.*, but that's also a bit of a hack, because then the programs are linking the same .so twice.

You may be interested in the discussion of this issue in downstream Fedora:
https://fedorahosted.org/fpc/ticket/352

(I don't know yet what the outcome of that discussion will be in the end.)

[kkofler]

To solve the problem with the internal routines, I guess it should be possible to build 3 shared objects: libopenblasinternal.so.0, libblas.so.3 and liblapack.so.3.

[xianyi]

Hi @kkofler ,
Thank you for the suggestion about building 3 shared library. I think I can implement this feature.I will add a flag in Makefile.rule. If the user enables this flag, OpenBLAS will generate 3 shared library (liblapack.so.3, libblas.so.3, libopenblasinternal.so.0).

[fommil]

sounds perfect @xianyi

BTW, how did you get the auto-bounty link on the ticket to appear?

[xianyi]

@fommil , I just added this project on https://www.bountysource.com/ . Then, I created some bounties on ARM porting issue. I think bountysource.com add the auto-bounty link.

[pforai]

👍 for the multiple shared libraries options. When will this be implemented? Thanks.

[xianyi]

In 1 or 2 weeks.

2014/1/10 Petar Forai notifications@github.com

[image: 👍] for the multiple shared libraries options. When will this
be implemented? Thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/296#issuecomment-32018500
.

[TimothyGu]

@xianyi what's the status of this? Have you implemented this?

[xianyi]

@TimothyGu ,
Sorry, I didn't implement it yet. I wrote my dissertation in the last few months.

Now, I am debugging issue #340 . After #340, I think I can start implementing this feature.

[fommil]

Hi @xianyi I'm giving a talk at scala exchange in a few weeks about high performance linear algebra (on the JVM) and I was going to include a (complementary! 😄) slide about GotoBLAS and its successor OpenBLAS.

I was wondering if you had yet been able to figure out how to distribute OpenBLAS as a standard BLAS/LAPACK fortran binary API? i.e. provide libblas.so.3 and liblapack.so.3 as separate binaries that can be accessed from either Fortran or CBLAS/LAPACKE C wrappers?

From my perspective, I'm actually 100% happy with libblas.so.3 being able to be provided this way and liblapack.so.3 requiring that it be linked to the openblas-specific implementation of libblas. The important thing is that a user program compiled against the reference implementation of BLAS/LAPACK can be pointed at OpenBLAS at runtime and get the performance benefits.

[xianyi]

@fommil , is it possible to create two symbol links to libopenblas.so?

[fommil]

not really, one ends up with a messy system and symbol clashes. You'd still want to allow people to use their own LAPACK on top of your BLAS.

I think ubuntu and debian basically take the approach of only allowing OpenBLAS to be used for BLAS... you might want to find the maintainer/packager and have a discussion with them about this. That means if someone wants to use OpenBLAS for LAPACK they need to manually override the alternatives system and set up their own links... which is messy and won't even be allowed in enterprise situations. I doubt redhat is much better (which basically means OpenBLAS cannot be used in corporate environments). Usually a sysadmin would set all this up, and - in my experience - that means developers don't get to ask for special symbolic links.

The bottom line is that NETLIB have defined four very clear APIs here:

• BLAS (fortran) libblas.so.3
• CBLAS (C) libcblas.so.3
• LAPACK (fortran) liblapack.so.3
• LAPACKE (C) liblapacke.so.3

and any deviation from that means that a library is no longer compliant.

I think it's fair enough that OpenBLAS' liblapack.so.3 must be linked against the OpenBLAS libblas.so.3 (or libcblas.so.3) at runtime.

Note that some distros screw up the above, the most common mistake being the inclusion of CBLAS in libblas.so.3

[tkelman]

Note that MKL puts everything in the same library and yet is widely used in corporate environments and otherwise.

This seems like a fairly simple matter of packaging to me, and I'd personally rather see Xianyi spend his time on the core computational pieces of OpenBLAS rather than packaging details that can be handled by third parties. You already can build OpenBLAS without the LAPACK pieces, call that libblas, and build a Netlib LAPACK on top of it. Or you can build a full OpenBLAS and mark it as providing both libblas and liblapack, and maybe conflicting with the blas-only version of OpenBLAS.

Virtually every application that uses Blas or Lapack can be configured to use an alternate implementation, and due to MKL's example most of those also do a check for whether the Lapack library provides Blas (or the other way around), so only one library is needed.

[jakirkham]

So, I create symbolic links for libblas.so and liblapack.so on Linux (or dylib on Mac) and libblas.a and liblapack.a and this works fine. It would be nice if OpenBLAS did this by default for these platforms. However, I have not explored this on Windows.

[opoplawski]

Has any more work been done on this?

[sjjpo2002]

There is only libgfortran-3.dll in mingw. Unless by libgfortran-1-3.dll you mean libgfortran-1.dll or libgfortran-2.dllor libgfortran-3.dll